#  Missci: Reconstructing Fallacies in Misrepresented Science

Max Glockner♣, Yufang Hou♢♣, Preslav Nakov♠ Iryna Gurevych♣
♣Ubiquitous Knowledge Processing Lab (UKP Lab),
TU Darmstadt and Hessian Center for AI (hessian.AI)
♢IBM Research, Ireland, ♠MBZUAI
www.ukp.tu-darmstadt.de

###### Abstract

Health-related misinformation on social networks can lead to poor decision-
making and real-world dangers. Such misinformation often misrepresents
scientific publications and cites them as “proof” to gain perceived
credibility. To effectively counter such claims automatically, a system must
explain how the claim was falsely derived from the cited publication. Current
methods for automated fact-checking or fallacy detection neglect to assess the
(mis)used evidence in relation to misinformation claims, which is required to
detect the mismatch between them. To address this gap, we introduce Missci, a
novel argumentation theoretical model for fallacious reasoning together with a
new dataset for real-world misinformation detection that misrepresents
biomedical publications. Unlike previous fallacy detection datasets, Missci
(_i_) focuses on implicit fallacies between the relevant content of the cited
publication and the inaccurate claim, and (_ii_) requires models to verbalize
the fallacious reasoning in addition to classifying it. We present Missci as a
dataset to test the critical reasoning abilities of large language models
(LLMs), which are required to reconstruct real-world fallacious arguments, in
a zero-shot setting. We evaluate two representative LLMs and the impact of
providing different levels of detail about the fallacy classes to the LLMs via
prompts. Our experiments and human evaluation show promising results for GPT
4, while also demonstrating the difficulty of this task.111Code and data are
available at: https://github.com/UKPLab/acl2024-missci.

##  1 Introduction

Figure 1:  Fallacious Argument Reconstruction: The claim is falsely derived
from the cited study (_green_) by relying on the content of $p_{0}$. The model
generates and classifies the fallacious reasoning (_orange_) that needs to be
applied when concluding the claim based on all relevant study content
(including $s_{1}$ and $s_{2}$).

False or misleading narratives spread rapidly on social media (Vosoughi et
al., 2018; Wardle, 2018), posing challenges for non-experts in discerning
credible information, and exceeding the capabilities of human fact-checkers
(HFC). The need to support HFC has accelerated the research in automated fact-
checking (AFC) and related tasks (Guo et al., 2022; Schlichtkrull et al.,
2023a). Yet, the real-world applicability of AFC systems is limited due to the
lack of trustworthiness (Nakov et al., 2021), or their reliance on counter-
evidence, which may not exist (Glockner et al., 2022). Often, misinformation
distorts genuine information rather than creating new content (Brennen et al.,
2020). For example, the claim in Figure 1 that “ _hydroxychloroquine is a cure
for COVID-19_ ” contains a kernel of truth and relies on some content
(referred to as _accurate premise)_ of the cited study that found “
_chloroquine reduced infection of the coronavirus_.” However, further content
from the study shows that it did not conduct human experiments ($s_{1}$) and
focused on SARS-CoV-1 ($s_{2}$), different from the virus causing COVID-19.
Only when knowing this additional information ($s_{1}$ and $s_{2}$) about the
cited study, one can detect the applied fallacies (_Fallacy of Composition_
and _False Equivalence_).

In this work, we focus on inaccurate claims that misrepresent scientific
publications. We assume that the misrepresented publication is presented
alongside the claim as a “proof” amplifying the claim’s impact through
increased perceived credibility. With this assumption, we can access the cited
sources, which is essential for detecting fallacious reasoning and implements
human verification strategies (Silverman, 2014). Our goal is to automatically
outline the fallacious reasoning and explain why the claim is incorrect, a
crucial aspect of debunking misinformation (Schmid and Betsch, 2019;
Lewandowsky et al., 2020).

Earlier work on fallacy detection focused on surface-level fallacies like _Ad
Hominem_ or _Loaded Language_. More recent work (Jin et al., 2022; Alhindi et
al., 2022) also included logical fallacies like _False Cause_ or _Hasty
Generalization_. Yet, they did not adapt the task definition to account for
the fact that these fallacies may need information beyond what is explicitly
stated in the text. This hinders their applicability to real-world fallacies
that rely on external information. To bridge this gap, we introduce Missci, a
new argumentation theoretical model for fallacious reasoning, accompanied by a
new fallacy detection dataset derived from real-world misinformation. Unlike
prior work, we (_i_) treat inaccurate claims as the conclusion of a logical
argument, encompassing all necessary information to detect the applied
fallacies (_green_ in Figure 1). We (_ii_) formulate a distinct fallacy
inventory drawn from literature to express fallacies when misrepresenting
scientific publications. Finally, inspired by Cook et al. (2018), we (_iii_)
explicitly verbalize the fallacious reasoning via premises (_orange_) that
only implicitly exist based on the claim to reconstruct the fallacious
argument.

Our focus lies on the reasoning abilities to reconstruct fallacious arguments,
and we manually paraphrase the relevant publication content (_green_) based on
HFC articles, rather than using the original misrepresented document.
Motivated by recent progress in zero-shot performance of large language models
(LLMs) (Kojima et al., 2022), we evaluate the reasoning abilities of two
state-of-the-art LLMs in reconstructing the fallacious reasoning on Missci and
define the task in a zero-shot setup, exemplified in Figure 1. Given the
claim, an accurate premise, and the publication contexts, the model must
verbalize the fallacious reasoning and assign fallacy classes. Our
contributions are:

  * •

A new argumentation theoretical model with a new task formulation to
reconstruct the fallacious arguments.

  * •

A new dataset comprising complex fallacious arguments of real-world
misinformation.

  * •

Evaluation of two state-of-the-art LLMs and their critical reasoning abilities
to reconstruct fallacious arguments.

##  2 Related Work

Previous work has focused on surface-level fallacies for propaganda detection
(Da San Martino et al., 2019; Piskorski et al., 2023; Salman et al., 2023),
for online discussions (Habernal et al., 2018a; Sahai et al., 2021), or for
gamified settings (Habernal et al., 2017). The addressed fallacies typically
do not require information beyond what is stated explicitly in the text. Other
work targeted specific fallacies such as _Non Sequitur_ in law (Nakpih and
Santini, 2020) or _Ad Hominem_ in social media (Habernal et al., 2018a). More
similar to our work, some research (Jin et al., 2022; Musi et al., 2022;
Alhindi et al., 2022) focused on logical fallacies from the real world. Yet,
they neither verbalized the implicitly applied fallacies, nor considered the
underlying sources beyond what is explicitly stated in the text. Moreover, our
task design to generate fallacious premises differs from implicit premise
detection work (Habernal et al., 2018b; Chakrabarty et al., 2021) in that the
premises in Missci are inherently invalid and linked to applied fallacy
classes (see $\overline{p}_{1}$ and $\overline{p}_{2}$ in Figure 1; _orange_).
Existing work on fallacy generation focused on data generation (Huang et al.,
2023; Alhindi et al., 2023) rather than on articulating applied fallacious
reasoning within real-world fallacious arguments.

Scientific AFC work (Wadden et al., 2020; Sarrouti et al., 2021; Wadden et
al., 2022; Lu et al., 2023; Vladika and Matthes, 2023) considered external
sources, like us, but did not identify and articulate the nuanced fallacies
when concluding a claim from the cited study. Detecting distortions in
scientific communication is part of science communication research
(Augenstein, 2021), where studies have examined exaggerations in news articles
(Sumner et al., 2014; Bratton et al., 2019; Yu et al., 2020; Wright and
Augenstein, 2021), analyzed reporting certainty in scientific publications
(Pei and Jurgens, 2021), or quantified information mismatches between reported
and actual scientific findings (Wright et al., 2022). In parallel work, Wührl
et al. (2024) quantify and compare the original paper with other reporting of
their findings across fine-grained dimensions such as certainty or
generalization. In addition to our distinct task setup, our problem space
differs as we focus on harmful misinformation that comprises more severe
distortions, which are not necessarily bound to a study’s main findings.

##  3 Formalism of Missci

###  3.1 Misrepresented Science Arguments

Inspired by Cook et al. (2018), we reconstruct the fallacious reasoning in the
form of a logical argument. For an accurate claim $c$, a logical argument
would be structured as follows:

$\\{p_{0}\,,\,p_{1}\,,\,\ldots\,,\,p_{N}\\}\Rightarrow c$ (1)

where $P=\\{p_{0},p_{1},\ldots,p_{N}\\}$ is a set of _accurate_ premises that
jointly entail ($\Rightarrow$) the true claim $c$. For inaccurate claims
$\overline{c}$, the entailment relation does not hold based on accurate
premises, formulated as $P\not\Rightarrow\overline{c}$, where
$\not\Rightarrow$ denotes a corrupted entailment relation. To _reconstruct_ a
fallacious argument, a set of inaccurate (fallacious) premises
$\overline{p}_{i}\in\overline{P}$ must be applied such that
$P\cup\overline{P}\Rightarrow\overline{c}$. For example, consider the
following argument (simplified from Figure 2):

  * •

Accurate premise $p_{0}$: The viral COVID-19 spike protein inhibits repair of
DNA damage.

  * •

Fallacious premise $\overline{p}_{1,2}$: COVID-19 vaccine spike proteins are
as dangerous as viral COVID-19 spike proteins.

  * •

Conclusion ($\overline{c}$): Therefore, COVID-19 vaccines inhibit repair of
DNA damage.

Here, the accurate premise alone lacks sufficient support for the conclusion
(or claim). Establishing a support relationship between the accurate premise
and the conclusion requires the fallacious premise, which employs the _False
Equivalence_ fallacy. To debunk a claim $\overline{c}$, the argument can be
_deconstructed_ by highlighting the fallacies applied in each
$\overline{p}_{i}$. This invalidates the premises that are necessary to
establish the claim as a logical conclusion and renders the argument invalid.

Figure 2:  Interchangeable Fallacies: On the left, no distinction between the
different “spike proteins” (from the vaccine or the virus) is made; on the
right, both are assumed to behave alike. Only one of these premises is needed
to bridge the reasoning gap.

Misinformation that is falsely derived from a single credible publication $S$
can be formulated as $S\not\Rightarrow\overline{c}$. Each argument in Missci
has exactly one accurate premise, $P=\\{p_{0}\\}$, which is entailed by (parts
of) $S$ and represents the “kernel of truth” behind the inaccurate claim
$\overline{c}$. Since all arguments in Missci constitute misinformation, each
argument is illogical and contains at least one reasoning gap, which must be
bridged via a fallacious premise $\overline{p}_{i}$ that applies a fallacy
class $f_{i}$ to support the claim. This reasoning gap only becomes imminent
after observing relevant context ($s_{i}$) from the misrepresented publication
$S$. For example, consider Figure 2, where a study observed harm from SARS-
CoV-2 spike proteins (_accurate premise $p_{0}$_). The argument wrongly
applied this finding to mRNA vaccines, assuming that viral and vaccine spike
proteins behave identically (in both (alternative) fallacious premises).
Without detailed content about the misrepresented publication (that it was
observed on _viral_ spike proteins in $s_{1}$), this fallacy is undetectable.
In this work, we manually paraphrase this necessary context $s_{i}$ that
exhibits the reasoning gap from the publication $S$ where each $s_{i}$ is
entailed by $S$. Because the accurate premise $p_{0}$ is always entailed by
the publication, its publication context ($s_{0}$) is identical to the
accurate premise. We represent each fallacious reasoning $R_{i}\in R$ that
bridges one reasoning gap as a triplet $R_{i}=$ ($s_{i}$, $\overline{p}_{i}$,
$f_{i}$). To bridge all reasoning gaps and fully support the claim
$\overline{c}$ each fallacious reasoning $R_{i}\in R$ must be applied. The
overall formulation of a fallacious argument $\mathcal{A}$ is shown below:

$\biggl{\\{}\,\,\overset{\begin{subarray}{c}s_{0}\\\
=\end{subarray}}{p_{0}}\,,\,\underset{\begin{subarray}{c}\downarrow\\\
f_{1}\end{subarray}}{\overset{\begin{subarray}{c}s_{1}\\\
\downarrow\end{subarray}}{\overline{p}_{1}}}\,,\,\underset{\begin{subarray}{c}\downarrow\\\
f_{2}\end{subarray}}{\overset{\begin{subarray}{c}s_{2}\\\
\downarrow\end{subarray}}{\overline{p}_{2}}}\,,\,\ldots\,,\,\underset{\begin{subarray}{c}\downarrow\\\
f_{N}\end{subarray}}{\overset{\begin{subarray}{c}s_{N}\\\
\downarrow\end{subarray}}{\overline{p}_{N}}}\,\,\biggr{\\}}\,\,\Rightarrow\overline{c}$
(2)

Each argument $\mathcal{A}=(\overline{c},p_{0},R)$ comprises an inaccurate
claim $\overline{c}$, that builds on the accurate premise $p_{0}$ and applies
at least one fallacious reasoning $R_{i}\in R$. As shown in Figure 2, in some
cases, multiple fallacious premises with distinct fallacy classes can be used
interchangeably (i.e.,
$\overline{p}_{i}=\bigl{[}\overline{p}_{i,1},\>\overline{p}_{i,2}\bigr{]}$).
Interchangeable fallacies always share the identical publication context
($s_{i}$), but only one of them is necessary to bridge the reasoning gap.

###  3.2 Non-exhaustive, Inductive Arguments

Unlike Cook et al. (2018), we do not require arguments to deduce the claim,
which is unrealistic based on empirical evidence from scientific publications.
Instead, we consider _strong inductive support_ sufficient for
($\Rightarrow$). In inductive arguments, invalid premises, by definition,
weaken the conclusion without necessarily rendering it false. To avoid
labeling minor mismatches as fallacies, we ensure the relevance of each
fallacious reasoning $R_{i}\in R$ in the strong inductive argument by deriving
$R$ exclusively from the HFC article. By relying on the HFC, the extracted
fallacious reasoning lines are non-exhaustive and limited to the most pivotal
ones. Importantly, fallacies within the logical arguments do not necessarily
match the fallacies made by the claimant. For example, one can make the claim
in Figure 1 after only skimming parts of the study without ever knowing that
experiments were performed in cell cultures ($s_{1}$). In this case, the
_Fallacy of Exclusion_ rather than the _Fallacy of Composition_ was applied.
To account for these cases, we state our objective as detecting the necessary
fallacies needed to conclude the claim $\overline{c}$ from all relevant
content of the misrepresented publication $S$. This follows the _principle of
total evidence_ , which dictates that an inductive argument must consider all
available relevant evidence (Chakraborti, 2007).

###  3.3 Task Definition

We assess the ability to reconstruct the fallacious reasoning for each
fallacious argument $\mathcal{A}$ on the fallacious reasoning level: For each
fallacious reasoning $R_{i}\in R$, given the claim $\overline{c}$, the
accurate premise $p_{0}$ and the publication context $s_{i}$ from $R_{i}$, the
model must verbalise the fallacious premise $\hat{\overline{p}}_{i}$ and
predict the applied fallacy class $\hat{f_{i}}$ to bridge the reasoning gap,
so that ($\hat{\overline{p}}_{i}$, $\hat{f_{i}}$) constitute valid fallacies
as approximated via the annotated interchangeable fallacies
($\overline{p}_{i}$, $f_{i}$) of $R_{i}$.

##  4 Dataset

| Collect | Select | Reconstruct
---|---|---|---
HFC articles | 527 | 150 | 147
Links | 8,695 | 208 | 184
Arguments | – | – | 184
Fall. Reasoning $\mathbf{R_{i}}$ | – | – | 435
Table 1: Dataset Construction: Number of elements for all three steps during
dataset construction.

Our main annotator was a M.Sc. student in biology, covering early pilot
studies and post-annotation consolidation. Additionally, two more M.Sc.
students, one in biology and one in linguistics, were employed during
annotation. The annotators received a pay of 12.26 EUR per hour. We used Surge
AI222https://www.surgehq.ai/ as the annotation tool. Weekly meetings involving
all annotators and one of the authors were held throughout the project to
provide feedback and to refine the guidelines as needed, in line with the
recommendations of Klie et al. (2024). To create Missci, we (1) _collected_
HFC articles and pre-selected links that may point to a misrepresented
publication, (2) manually _selected_ all links that pointed to a
misrepresented publication, and (3) _reconstructed_ the fallacious arguments
from the HFC articles. A summary of these three steps is given in Table 1. We
collected a total of 527 fact-checking articles from
HealthFeedback333https://healthfeedback.org/ until January 2023, excluding
those that address accurate claims. HealthFeedback collaborates with
scientists in reviewing health and medical claims. From these HFC articles, we
annotated 8,695 links from reputable sources (cf. §A.1) to determine whether a
link pointed to a misrepresented scientific publication. Our annotators found
208 links pointing to misrepresented scientific publications across 150 HFC
articles (cf. §A.2; Krippendorff’s $\alpha$ was $0.728$).

###  4.1 Fallacious Argument Reconstruction

The annotators were instructed to generate all elements of the fallacious
argument $\mathcal{A}$ that falsely concludes the claim $\overline{c}$. This
included the accurate premise $p_{0}$ as well as the fallacious premise
$\overline{p}_{i}$, fallacy class $f_{i}$, and publication context $s_{i}$ for
each fallacious reasoning $R_{i}$ (cf. §B.1 for a list of all fallacy
classes). Each element had to be justified with an extracted statement from
the HFC article. Often, selecting a single definitive fallacious reasoning was
ambiguous and oxymoronic, akin to identifying the “correct invalid reasoning”
(cf. §3; Figure 2). This aligns with Bonial et al. (2022), who observed that
due to overlapping definitions, fallacies could often be reformulated to fit
the definition of a different fallacy. Hence, we allowed separate listing of
interchangeable fallacies, which were merged during consolidation. As we aimed
to detect the fallacious reasoning _between_ a scientific publication and an
inaccurate claim, our work rests on the assumption that the publication itself
is trustworthy. To verify the trustworthiness, the annotators rated the
credibility of the scientific document based on the HFC article, which we
analyzed in §C.1. Detailed instructions and the annotation process are
outlined in §B.2.

###  4.2 Inter-Annotator Agreement

We collected 520 annotated HITs for 208 potential arguments. After
consolidation (cf. §B.3), Missci contained 435 distinct fallacious reasoning
lines ($R_{i}$) bridging different reasoning gaps (with a total of 550
interchangeable fallacies) for 184 fallacious arguments. Each argument
involved 1-5 fallacious reasoning lines ($R_{i}$), averaging at 2.4 per
argument. Most of the arguments within Missci were related to the COVID-19
infodemic. We show the distribution of arguments over years and their relation
to COVID-19 in §C.2. Calculating the inter-annotator agreement for the fallacy
class annotations faced two challenges: interchangeable fallacies with
different but valid labels, and annotators identifying different (non-
interchangeable) fallacious reasoning lines that bridge different reasoning
gaps.

To address this, we used two complementary measures: We calculated the inter-
annotator agreement for the fallacy class $f_{i}$ among all 253 fallacious
reasoning $R_{i}$ identified by at least two annotators within the
consolidated arguments. When simulating a single-label classification setup,
the inter-annotator agreement, measured using Krippendorff’s $\alpha$, was
0.520. This is comparable to Cohen’s $\kappa$ of 0.47 (Alhindi et al., 2022)
and 0.52 (Musi et al., 2022) in similar work.444Our agreement measure differs
as we employed up to three rather than two annotators. We additionally
compared each fallacious reasoning $R_{i}$ identified by each individual
annotator to the consolidated argument and measured how many fallacious
reasoning $R_{i}$ of the consolidated argument a single annotator found on
average. Here, we considered all fallacies that were merged during
consolidation as identical, and did not differentiate whether the annotators
selected distinct interchangeable fallacies that apply different fallacy
classes. Here, we only considered 70 arguments, which were fully annotated by
all three annotators, for the computation to not artificially inflate the
coverage by a single annotator. On average, each annotator identified 72.5% of
the fallacious reasoning lines $R_{i}$ in the consolidated argument. We
examined how this affected the overall recall of the detected fallacies in
Missci in §C.3.

###  4.3 Fallacy Class Analysis

Figure 3:  Interchangeable Fallacy Classes: Heatmap of co-occurring
interchangeable fallacy classes of the consolidated arguments ordered by
frequency.

To understand which fallacy classes have been annotated together as
interchangeable fallacies, we show their co-occurrence matrix in Figure 3. The
definitions and the examples for all fallacies are given in §B.1. We observe
that most co-occurrences are between _False Equivalence_ and _Ambiguity_ (as
discussed in Figure 2). In contrast, _Hasty Generalization_ was the most
clear-cut fallacy in our annotations, likely because HFC typically explicitly
specify when a study lacks sufficient observations for the claim, and because
it had little overlap with other fallacy definitions. The majority class of
all fallacy class annotations is the _Fallacy of Exclusion_. This fallacy
omits critical information when drawing a conclusion and could theoretically
apply to every reasoning gap that depends on the information in the
publication context $s_{i}$, because each $s_{i}$ contains content that
undermines the claim. To address this, during annotation the annotators were
tasked to prioritize more specific fallacies before the _Fallacy of Exclusion_
(cf. §B.2), yet the fallacy class remains the most common.

We found two main reasons for the prevalence of the _Fallacy of Exclusion_ in
Missci: The first and primary reason for including this fallacy in our
inventory, is that parts of the misrepresented publication contradict the
claim $\overline{c}$. For example, the claim that “ _spike proteins induced by
RNA vaccines can damage blood vessels_ ” is based on a study which concludes
that “ _vaccination could protect against blood vessel damage_ ”. In this
case, the authors’ comment must be ignored and no other fallacy in our
inventory can be used to conclude a claim opposite to their conclusion.
Second, the _Fallacy of Exclusion_ often serves as a fallback class in Missci
due to its broad applicability. Given the inevitability of an incomplete
fallacy inventory, instances where the detected fallacies do not clearly align
with the predefined fallacy classes are frequently labeled as _Fallacy of
Exclusion_. This leads to co-occurrences with other fallacies in borderline
cases. For example, the claim that “ _Pfizer’s COVID-19 vaccine effectiveness
dropped from 100% to 20%_ ” relies on infection numbers, and ignores the
reported high effectiveness against severe disease. This flawed reasoning
could be interpreted as _False Equivalence_ , assuming mild and severe
COVID-19 cases are equivalent, or _Fallacy of Exclusion_ by omitting the
protection against severe diseases. A clear fallacy class can only be assigned
for a specific _verbalized_ fallacious premise.

##  5 Experiments

| _Fallacy_ | _Fallacious premise (@1)_ | _Consistency_
---|---|---|---
Model | P@1 | Arg@1 |  F1 (_micro_) | METEOR | BERTScore | NLI-A | NLI-S | Matches@1 (%)
_random_ \+ _claim_ | 0.131 | 0.264 | 0.117 | 0.181 | 0.611 | 0.120 | 0.130 | –
_random_ \+ $p_{0}$ | 0.188 | 0.599 | 0.062 | 0.067 | –
LLaMA 2 (D) | 0.223 | 0.416 | 0.233 | 0.222 | 0.617 | 0.123 | 0.148 | 40.5
LLaMA 2 (DE) | 0.209 | 0.422 | 0.232 | 0.229 | 0.621 | 0.124 | 0.148 | 34.7
LLaMA 2 (DL) | 0.196 | 0.409 | 0.211 | 0.203 | 0.616 | 0.130 | 0.143 | 41.0
LLaMA 2 (DLE) | 0.209 | 0.416 | 0.233 | 0.207 | 0.616 | 0.129 | 0.145 | 19.3
LLaMA 2 (L) | 0.193 | 0.377 | 0.208 | 0.253 | 0.627 | 0.140 | 0.165 | 54.5
LLaMA 2 (LE) | 0.212 | 0.409 | 0.222 | 0.180 | 0.609 | 0.121 | 0.134 | 43.0
GPT 4 (D) | 0.317 | 0.571 | 0.297 | 0.239 | 0.619 | 0.069 | 0.126 | 61.2
GPT 4 (L) | 0.292 | 0.526 | 0.290 | 0.238 | 0.613 | 0.064 | 0.140 | 61.4
Table 2: Argument Reconstruction Results: Evaluation of LLaMA 2 (70B) and GPT
4 over the predicted fallacy class and the generated fallacious reasoning. We
report the performance when using prompts with fallacy _(D)_ efinitions, _(L)_
ogical forms and/or _(E)_ xamples, and provide a consistency estimate of the
LLM by asking each LLM to separately classify the fallacy present in the
generated premise.

For each input ($\overline{c}$, $p_{0}$, $s_{i}$), comprising the incorrect
claim, the accurate premise, and the publication context linked to a fallacy,
the model must generate at least one fallacious premise
$\hat{\overline{p}}_{i}$ together with the applied fallacy class
$\hat{f_{i}}$. We only experiment in a zero-shot setting, since the dataset
construction depends on high-quality HFC articles, which limits size and
scalability. However, we separate 30 arguments as a validation split555No
misrepresented publication occurs in _test_ and _dev_. to allow for a prompt
selection without compromising the evaluation on the unseen test split, which
comprises the remaining 154 arguments with 363 fallacious reasoning lines
$R_{i}$ bridging different reasoning gaps.

###  5.1 Metrics

Even though multiple interchangeable fallacies may be applicable, only one of
them is required to reconstruct the fallacious argument. Hence, to evaluate
the fallacy classes, we report P@1 as our primary metric, where the top-ranked
predicted fallacy class $\hat{f}_{i,1}$ is considered correct if it matches
any gold fallacy class $f_{i,j}$ of the interchangeable fallacy classes in
$R_{i}$. Further, we model the fallacy classification as a multi-label, multi-
class classification problem, in which we ask the model to identify _all_
interchangeable fallacy classes within each $R_{i}$. While the single-label
classification measures sufficiency, the multi-label multi-class
classification relates to the comprehensiveness of the detected fallacy
classes. Akin to previous fallacy detection work (Dimitrov et al., 2021; Jin
et al., 2022) with high class-imbalances, we report the micro F1-score.
Additionally, we assume that correctly detecting at least one fallacy is
sufficient to reject the claim, and report argument-level accuracy, denoted as
_Arg@1_ , by considering an argument as rejected if the top-ranked fallacy
class prediction of any fallacious reasoning $R_{i}$ is correct.

To evaluate the generated fallacious premises, we first match the top-ranked
generated premises with the gold premises as reference texts via the predicted
fallacy class and cosine similarity (cf. §D.1). We then report METEOR score
(Banerjee and Lavie, 2005), which was used for rationales in the real-world
AFC dataset AVeriTeC (Schlichtkrull et al., 2023b), and BERTScore (Zhang et
al., 2020) to account for semantic similarity. Further, we follow Honovich et
al. (2022) who use a T5 (Raffel et al., 2020) model trained on NLI data and
consider the predicted probability for the entailment label as measure. Rather
than using the entailment probability given the reference premise
$\overline{p}_{i}$ as _premise_ and the generated premise
$\hat{\overline{p}}_{i}$ as _hypothesis_
$e(\overline{p}_{i},\hat{\overline{p}}_{i})$ (denoted as _NLI-A_), we
additionally compute a symmetric variant (_NLI-S_) via
$\mathrm{max}\bigl{[}e(\overline{p}_{i},\hat{\overline{p}}_{i});e(\hat{\overline{p}}_{i},\overline{p}_{i})\bigr{]}$
to not penalize a model if the generated premise is more specific than the
reference premise. More details about the metrics and matching with reference
text are provided in §D.1. Finally, we measure the LLM’s internal consistency
of the generated premise and fallacy class, by prompting the same LLM again to
classify the fallacy present in the generated fallacious premise
$\hat{\overline{p}}_{i}$ given ($\overline{c}$, $p_{0}$,
$\hat{\overline{p}}_{i}$). We report the percentage in which the same fallacy
class is predicted.

###  5.2 Models

We evaluated two baselines that predict a randomly selected fallacy class. For
the fallacious premise, these baselines either always predict the claim
$\overline{c}$ or the accurate premise $p_{0}$, both of which are topically
related to the gold fallacious premise but meaningless in their verbalized
reasoning.

We conducted experiments with two state-of-the-art LLMs: LLaMA 2 (70B)
(Touvron et al., 2023) as an open-source LLM which can be run on a local
machine, and GPT 4 (OpenAI, 2023) as a proprietary LLM. In line with our
annotation process, we prompted the LLM to generate a ranked list of multiple
pairs consisting of the fallacious premise and fallacy class
($\hat{\overline{p}}_{i,j}$, $\hat{f}_{i,j}$), which may express
interchangeable fallacy classes. We evaluated different prompts, varying in
the amount of information provided to the LLMs about the fallacy classes.
Specifically, we examined the impact of fallacy definitions (_D_), the logical
form (_L_), and the examples (_E_) from our fallacy inventory (cf. §B.1),
sourced from Bennett (2012) and Cook et al. (2018). The definitions offer
descriptive information about the fallacies. The logical forms abstract from
the content, but explicitly indicate the applied fallacious reasoning. For
instance, the logical form for the _Fallacy of Composition_ is _“A is part of
B. A has property X. Therefore, B has property X.”_. This resembles surface
patterns that were found to be beneficial in logical fallacies (Jin et al.,
2022). We hypothesize that different types of information have varying effects
on fallacy classification and fallacious premise generation. We selected the
best prompt based on the P@1 performance on the validation split (cf. §D.2).
For GPT 4, we only report the results based on the respective best LLaMA 2
prompts for fallacy premise generation and fallacy class P@1 on the test set
for comparison. Prompts and hyper-parameters are in §F.

###  5.3 Argument Reconstruction Results

Table 2 shows the results for reconstructing the fallacious argument. LLaMA 2
achieves its best fallacy detection (P@1) and fallacious premise generation
performance using (_D_) or (_L_) in the prompt, respectively, which is
consequently reported for GPT 4. For both LLMs, using only the fallacy
definition leads to the best fallacy classification performance. Here, GPT 4
outperforms LLaMA 2 by a large margin, correctly identifying at least one
fallacy in 57% of the arguments. For fallacious premise generation, each LLM
exhibits the best performance based on different prompts. In the fallacious
premise generation, even GPT 4 achieves low scores, particularly compared to
the random baselines. The generated premises perform primarily poorly when the
predicted fallacy class does not match the gold fallacy class of the reference
premise. When separately evaluating the generated fallacious premises over
correctly classified fallacies with matching classes only (cf. §D.3; Table
10), GPT 4 surpasses the random baseline and outperforms LLaMA 2 in METEOR
($0.264$ vs. $0.243$) and BERTScore ($0.637$ vs. $0.622$), reaching comparable
performance in NLI-S ($0.267$ vs. $0.266$). Finally, both models seem to show
(slightly) improved consistency (last column) when given the logical form,
suggesting that it helps the fallacy generation to match the expected form.
Overall, the consistency is much higher for GPT 4.

###  5.4 Fallacy Classification Results

LLM | Prompt | Acc. | F1
---|---|---|---
| – | 0.493 | 0.406
| Def. | 0.577 | 0.464
| Def. + Logical | 0.630 | 0.476
LLaMA 2 | Def. + Example | 0.637 | 0.476
| Def. + Logical + Example | 0.568 | 0.459
| Logical | 0.601 | 0.472
| Logical + Example | 0.645 | 0.499
| Def. | 0.738 | 0.649
GPT 4 | Logical | 0.744 | 0.624
| Logical + Example | 0.771 | 0.682
Table 3: Fallacy Classification: Performance when predicting the gold fallacy
class $f_{i,j}$ given the claim $\overline{c}$, the fallacy context $s_{i}$
and the verbalized fallacious premise $\overline{p}_{i,j}$. We report accuracy
and F1-score (macro). Figure 4:  Performance per Fallacy: F1-score per
predicted fallacy class from a multi-label multi-class perspective considering
all model predictions.

We instructed LLMs to classify the applied fallacy class $f_{i,j}$ based on
the _provided_ gold fallacious premise $\overline{p}_{i,j}$, along with the
claim $\overline{c}$, the accurate premise $p_{0}$, and the publication
context linked to a fallacy $s_{i}$, and report the results in Table 3. Since
each fallacious premise $p_{i,j}$ verbalizes a single fallacy class $f_{i,j}$
this becomes a single-label classification problem. These experiments help to
(_i_) compare the difficulty of detecting fallacies with _explicit_ fallacious
reasoning provided or not (as in §5.3), and (_ii_) re-evaluate the LLMs and
the prompts used to assess the consistency over the gold fallacious premises.
In addition to exploring the impact of (_D_ , _L_ , _E_), we also evaluated
the performance when only provided with the fallacy names (first row), to
assess whether sufficient fallacy knowledge was acquired during pretraining.
For GPT 4, we evaluated the best prompt based on LLaMA 2 performance, as well
as the prompts used to measure consistency in Table 2. Both LLMs performed
strong across all prompts, especially considering that this is a 9-way
classification problem. In §E.3, we further provide empirical evidence that
GPT 4 benefits from the premise generation task, when no gold fallacious
premise is available. The accuracy over the gold premises always exceeded the
consistency scores, suggesting that model-generated premises were not as
clear-cut to a single fallacy class compared to gold fallacious premises, even
by the LLMs’ own judgement. The primary misclassification for both LLMs
occured between _Ambiguity_ and _False Equivalence_ (cf. §D.4), two very
related fallacies (cf. Figures 2 & 3). However, LLaMA 2 overpredicted _False
Equivalence_ in general. The best performance was reached with access to the
logical form and the examples. We hypothesize that both were most influential
to our annotators, and are hence helpful for detecting their generated
fallacies.

##  6 Analysis

###  6.1 Fallacy-level Performance

We assess the performances per fallacy class for the best prompts (_D_) in
§5.3 for both LLMs in Figure 4. Specifically, we report the fallacy-level
F1-score in a multi-label multi-class setting. GPT 4 outperforms LLaMA 2 in
almost all classes. The strongest F1-score by LLaMA 2 is achieved for the
_False Equivalence_ class. This aligns with the (_LE_) prompted LLMs from
Table 3, analyzed in §D.4, and primarily stems from a high recall for
detecting this fallacy. For all other fallacy classes, GPT 4 achieves a
substantially higher recall, leading to an overall higher performance in terms
of F1-score, given the mostly similar precision across fallacy classes (cf.
§E.2; Figure 16). We observe the biggest difference for _Ambiguity_ and
_Impossible Expectations_ , which are frequently detected by GPT 4, but never
by LLaMA 2. The same was observed when prompting LLMs to predict fallacy
classes applied by the gold fallacious premises (cf. §D.4; Figures 13 & 14),
suggesting that the differences were inherent to the LLMs. Interestingly, both
LLMs perform best on the most frequent fallacy classes despite no fine-tuning
involved.

###  6.2 Allowing Multiple Predictions

Generally, we assume that our annotators identified the most fitting fallacies
that should be among the top model predictions. However, different applicable
fallacy classes may exist and our annotations cannot guarantee full recall. To
address this, in Figure 5, we evaluate all LLMs in a more lenient setting. We
borrow the HasPositive@$k$ metric from Shaar et al. (2020) and consider a
detected fallacy class correct, if any of the top $k$ predicted fallacy
classes is accurate. This approach avoids penalizing models for predicting
different fallacies, as long as they also predict a gold fallacy class. The
results demonstrate a consistent improvement in the performance of GPT 4 as
more predictions are considered. In contrast, LLaMA 2, in most cases, fails to
predict the gold fallacy, even within the top 6 predictions. This confirms
that GPT 4’s superiority on this task is not a result of subtle selection bias
for the top-ranked fallacy class, but arises from its better ability to
identify the required fallacy classes. GPT 4 especially benefits from
improving the detection of _Fallacy of Exclusion_ and _False Equivalence_ when
increasing $k$ (cf. §E.1), which account for 45.3% of all 550 interchangeable
fallacies in Missci, and nearly doubles the accuracy when considering the top
3 predictions. The argument-level performance (Arg@$k$) peaks at a maximum of
89.0% for GPT 4 and 59.7% for LLaMA 2 for $k=6$.

Figure 5: Relaxed Fallacy Detection: Performance when a fallacy is considered
as correct, if the model predicts a fallacy class within the top _k_ results
that matches any of the gold interchangeable fallacy classes.

###  6.3 Human Evaluation

We manually evaluated 240 predictions from the main experiments in Table 2 (60
per LLM with (_D_) and (_L_) prompts; 50% for correctly and incorrectly
classified fallacies based on P@1). Table 4 shows the estimated overall
results. Additional results and details are provided in §E.4. Our human
judgment found the fallacies produced by GPT 4, in particular (_L_), the most
plausible. Yet, the predicted fallacy class often did not match the premise.
Overall, we observed a major quality difference of generated premises across
both LLMs, with LLaMA 2 often repeating parts of the input. NLI-S showed the
strongest correlation with human judgements (Pearson $r$=0.209;
p-value=$0.001$; cf. §E.4). Due to the complexity of this task and its
automatic evaluation, we echo Schlichtkrull et al. (2023b) who argue that
human evaluation is necessary for robustness.

Model | Applicable premise | Correct class
---|---|---
LLaMA 2 (_L_) | 0.167 | 0.040
LLaMA 2 (_D_) | 0.233 | 0.107
GPT 4 (_L_) | 0.867 | 0.503
GPT 4 (D) | 0.674 | 0.481
Table 4: Human Evaluation: Assessment if the generated fallacious premises are
_applicable_ to bridge the reasoning gap, and if the predicted fallacy _class_
is applied by the generated and applicable premise.

##  7 Discussion

Following suggestions in Schlichtkrull et al. (2023a), we outline how our
research contributes to combating misinformation. The analyzed _data subjects_
and _data actors_ are social media users. For responsible applications, we
emphasize that _data owners_ should ideally have domain expertise, recognizing
that any system will inevitably be imperfect. We strictly did not ask the
models to assign an overall rating of the claim’s veracity. Instead, we kept
the user in the loop for decision-making and only assisted by outlining the
fallacious discrepancies between the cited publication and the claim. Clearly
communicating the inaccuracies behind a claim is important for effective
debunking (Schmid and Betsch, 2019; Lewandowsky et al., 2020) and can help to
increase digital literacy, which is important for building resilience against
misinformation (Lewandowsky and Van Der Linden, 2021; Musi et al., 2022).
While previous approaches taught digital literacy using serious games
(Roozenbeek and van der Linden, 2020; Musi et al., 2023) that require active
participation, we envision a system that supports passive consumers of social
media.

##  8 Conclusion and Future Work

We introduced Missci, a novel dataset to combat real-world misinformation that
misrepresents scientific publications. We proposed a novel task formulation to
automatically reconstruct the fallacious reasoning through logical arguments
based on the cited publication’s content. We showcased Missci as a testbed for
evaluating the reasoning abilities of LLMs. Our experiments on two LLMs
demonstrated the potential for reconstructing fallacious arguments. In future
work, we plan to use Missci with different LLMs, domains, and languages.

## Limitations

To reconstruct fallacious arguments, we solely relied on the expertise of a
single fact-checking organization. Missci is limited to this organization’s
selected claims, topics, and biases. While fallacies are derived from
reasoning flaws detected by the HFC, separating fallacious reasoning from
valid reasoning is not always clear-cut. Generalizing or abstracting from
specific observations is an essential part of reasoning (Bennett, 2012) and
some argue that fallacy theory in general has limited applicability for real-
world claims (Boudry et al., 2015). When selecting claims for fact-checking,
the virality of claims is a major factor (Arnold, 2020). It is, therefore,
likely that information about the claim, and why it is inaccurate may have
been acquired by the LLMs during pretraining (Magar and Schwartz, 2022),
similarly to leaked evidence effects observed in fact-checking (Glockner et
al., 2022). While Missci addresses real-world misinformation, it is just one
step toward detecting such fallacies: Our design choices exclude joint use of
multiple publications to derive a claim. Assessing a claim by its cited source
as done in this work is necessary but insufficient; verifying a claim in the
real world requires consultation with complementary sources and domain experts
(Silverman, 2014). Our approach requires knowledge of the misrepresented
publication, which may not always be provided together with the
claim.666Although, by not providing evidence for the claim, the _Evading the
Burden of Proof_ fallacy applies. Moreover, Missci does not consider the
original content of the misrepresented publication, but relies on paraphrasing
from HFC articles, which is not available in real-world applications. Finally,
Missci comprises pure misinformation, and our results offer no insight into
model performance over accurate claims. For practical utility, fallacy
detection systems must discern whether a fallacy is present before selecting
the specific type of fallacy. We note that including unbiased accurate claims
is challenging, as they likely differ in topic, specificity, and may cite
multiple scientific publications. Due to these limitations, neither the tested
models nor any derived from Missci in this form should be directly applied in
the real world.

## Ethics Statement

The objective of this work, to combat misinformation and to increase the
public resilience to it, is ethically uncritical and beneficial to society.
Nevertheless, our work bears the danger of undesired side effects. Although
our task definition is clearly bound to the content used when deriving a
claim, our evaluation may favor models that align with the best knowledge
available during COVID-19, which makes up the majority of our dataset. Yet,
scientific knowledge may change over time, which will not be reflected in
Missci. Moreover, we task the models to produce fallacious reasoning. This is
important to explain the fallacious reasoning behind a claim for debunking
(Lewandowsky et al., 2020), yet it may also be misused by malicious actors.
Nevertheless, we argue that our work is rather a further demonstration of how
generating fallacies in a controlled setup can be used for good, and aligns
with previous work that generated misinformation to improve NLP-based
approaches (Zellers et al., 2019; Huang et al., 2023; Alhindi et al., 2023).
We did not take any steps to anonymise the collected data. All claims in
Missci are taken from HFC articles which often focus on claims by public
figures. We neither contacted the individuals making the claim, nor the HFC.
Following Schlichtkrull et al. (2023b) we will remove claims from Missci upon
request by any individual that stated the claim, is subject of the claim or
created the HFC article.

## Acknowledgments

This research work has been funded by the German Federal Ministry of Education
and Research and the Hessian Ministry of Higher Education, Research, Science
and the Arts within their joint support of the National Research Center for
Applied Cybersecurity ATHENE, and by the the German Research Foundation (DFG)
as part of the UKP-SQuARE project (grant GU 798/29-1). Yufang Hou is supported
by the Visiting Female Professor Programme from TU Darmstadt. We gratefully
acknowledge the support of Microsoft with a grant for access to OpenAI GPT
models via the Azure cloud (Accelerate Foundation Model Academic Research). We
are grateful to our dedicated annotators who helped to create Missci. Finally,
we wish to thank Jan Buchmann, Nils Dycke, Aniket Pramanick, Luke Bates and
Jing Yang for their valuable feedback on an early draft of this work.

## References

  * Alex et al. (2010) Bea Alex, Claire Grover, Rongzhou Shen, and Mijail Kabadjov. 2010.  Agile Corpus Annotation in Practice: An Overview of Manual and Automatic Annotation of CVs.  In _Proceedings of the Fourth Linguistic Annotation Workshop_ , pages 29–37, Uppsala, Sweden. Association for Computational Linguistics.
  * Alhindi et al. (2022) Tariq Alhindi, Tuhin Chakrabarty, Elena Musi, and Smaranda Muresan. 2022.  Multitask Instruction-based Prompting for Fallacy Recognition.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8172–8187, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Alhindi et al. (2023) Tariq Alhindi, Smaranda Muresan, and Preslav Nakov. 2023.  Large Language Models are Few-Shot Training Example Generators: A Case Study in Fallacy Recognition.  _ArXiv preprint_ , abs/2311.09552.
  * Arnold (2020) Phoebe Arnold. 2020.  The challenges of online fact checking: how technology can (and can’t) help.  Technical report, FullFact.
  * Augenstein (2021) Isabelle Augenstein. 2021.  Determining the Credibility of Science Communication.  In _Proceedings of the Second Workshop on Scholarly Document Processing_ , pages 1–6, Online. Association for Computational Linguistics.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Bennett (2012) Bo Bennett. 2012.  _Logically Fallacious: The Ultimate Collection of Over 300 Logical Fallacies (Academic Edition)_.  eBookIt.com.
  * Bonial et al. (2022) Claire Bonial, Austin Blodgett, Taylor Hudson, Stephanie M. Lukin, Jeffrey Micher, Douglas Summers-Stay, Peter Sutor, and Clare Voss. 2022.  The Search for Agreement on Logical Fallacy Annotation of an Infodemic.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 4430–4438, Marseille, France. European Language Resources Association.
  * Boudry et al. (2015) Maarten Boudry, Fabio Paglieri, and Massimo Pigliucci. 2015.  The fake, the Flimsy, and the Fallacious: Demarcating Arguments in Real Life.  _Argumentation_ , 29:431–456.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Bratton et al. (2019) Luke Bratton, Rachel C Adams, Aimée Challenger, Jacky Boivin, Lewis Bott, Christopher D Chambers, and Petroc Sumner. 2019.  The association between exaggeration in health-related science news and academic press releases: a replication study.  _Wellcome open research_ , 4(148).
  * Brennen et al. (2020) J. Scott Brennen, Felix M. Simon, Philip N. Howard, and Rasmus Kleis Nielsen. 2020.  Types, Sources, and Claims of COVID-19 Misinformation.  Technical report, Reuters Institute for the Study of Journalism.
  * Chakrabarty et al. (2021) Tuhin Chakrabarty, Aadit Trivedi, and Smaranda Muresan. 2021.  Implicit Premise Generation with Discourse-aware Commonsense Knowledge Models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6247–6252, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Chakraborti (2007) Chhanda Chakraborti. 2007.  _LOGIC: Informal, Symbolic and Inductive_.  PHI Learning Pvt. Ltd.  Page 48.
  * Cook et al. (2018) John Cook, Peter Ellerton, and David Kinkead. 2018.  Deconstructing climate misinformation to identify reasoning errors.  _Environmental Research Letters_ , 13(2).
  * Da San Martino et al. (2019) Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeño, Rostislav Petrov, and Preslav Nakov. 2019.  Fine-Grained Analysis of Propaganda in News Article.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5636–5646, Hong Kong, China. Association for Computational Linguistics.
  * Dettmers et al. (2022) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022.  GPT3.int8(): 8-bit Matrix Multiplication for Transformers at Scale.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 30318–30332. Curran Associates, Inc.
  * Dimitrov et al. (2021) Dimitar Dimitrov, Bishr Bin Ali, Shaden Shaar, Firoj Alam, Fabrizio Silvestri, Hamed Firooz, Preslav Nakov, and Giovanni Da San Martino. 2021.  Detecting Propaganda Techniques in Memes.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6603–6617, Online. Association for Computational Linguistics.
  * Glockner et al. (2022) Max Glockner, Yufang Hou, and Iryna Gurevych. 2022.  Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5916–5936, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Guo et al. (2022) Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022.  A Survey on Automated Fact-Checking.  _Transactions of the Association for Computational Linguistics_ , 10:178–206.
  * Habernal et al. (2017) Ivan Habernal, Raffael Hannemann, Christian Pollak, Christopher Klamm, Patrick Pauli, and Iryna Gurevych. 2017.  Argotario: Computational Argumentation Meets Serious Games.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 7–12, Copenhagen, Denmark. Association for Computational Linguistics.
  * Habernal et al. (2018a) Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018a.  Before Name-Calling: Dynamics and Triggers of Ad Hominem Fallacies in Web Argumentation.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 386–396, New Orleans, Louisiana. Association for Computational Linguistics.
  * Habernal et al. (2018b) Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018b.  The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1930–1940, New Orleans, Louisiana. Association for Computational Linguistics.
  * He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021.  DeBERTa: Decoding-enhanced BERT with Disentangled Attention.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Honovich et al. (2022) Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022.  TRUE: Re-evaluating Factual Consistency Evaluation.  In _Proceedings of the Second DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering_ , pages 161–175, Dublin, Ireland. Association for Computational Linguistics.
  * Huang et al. (2023) Kung-Hsiang Huang, Kathleen McKeown, Preslav Nakov, Yejin Choi, and Heng Ji. 2023.  Faking Fake News for Real Fake News Detection: Propaganda-Loaded Training Data Generation.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 14571–14589, Toronto, Canada. Association for Computational Linguistics.
  * Jin et al. (2022) Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schoelkopf. 2022.  Logical Fallacy Detection.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 7180–7198, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Khot et al. (2018) Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018.  SciTaiL: A Textual Entailment Dataset from Science Question Answering.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1).
  * Klie et al. (2024) Jan-Christoph Klie, Richard Eckart de Castilho, and Iryna Gurevych. 2024.  Analyzing Dataset Annotation Quality Management in the Wild.  _Computational Linguistics_ , pages 1–48.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large Language Models are Zero-Shot Reasoners.  _Advances in neural information processing systems_ , 35:22199–22213.
  * Lewandowsky et al. (2020) Stephan Lewandowsky, John Cook, Ullrich Ecker, Dolores Albarracin, Michelle Amazeen, Panayiota Kendou, Doug Lombardi, E Newman, Gordon Pennycook, Ethan Porter, et al. 2020.  _The Debunking Handbook 2020_.
  * Lewandowsky and Van Der Linden (2021) Stephan Lewandowsky and Sander Van Der Linden. 2021.  Countering Misinformation and Fake News Through Inoculation and Prebunking.  _European Review of Social Psychology_ , 32(2):348–384.
  * Lu et al. (2023) Xinyuan Lu, Liangming Pan, Qian Liu, Preslav Nakov, and Min-Yen Kan. 2023.  SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 7787–7813, Singapore. Association for Computational Linguistics.
  * Magar and Schwartz (2022) Inbal Magar and Roy Schwartz. 2022.  Data Contamination: From Memorization to Exploitation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 157–165, Dublin, Ireland. Association for Computational Linguistics.
  * Musi et al. (2022) Elena Musi, Myrto Aloumpi, Elinor Carmi, Simeon Yates, and Kay O’Halloran. 2022.  Developing Fake News Immunity: Fallacies as Misinformation Triggers During the Pandemic.  _Online Journal of Communication and Media Technologies_ , 12(3).
  * Musi et al. (2023) Elena Musi, Elinor Carmi, Chris Reed, Simeon Yates, and Kay O’Halloran. 2023.  Developing Misinformation Immunity: How to Reason-Check Fallacious News in a Human–Computer Interaction Environment.  _Social Media + Society_ , 9(1).
  * Nakov et al. (2021) Preslav Nakov, David Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barrón-Cedeño, Paolo Papotti, Shaden Shaar, and Giovanni Da San Martino. 2021.  Automated Fact-Checking for Assisting Human Fact-Checkers.  In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21_ , pages 4551–4558. International Joint Conferences on Artificial Intelligence Organization.  Survey Track.
  * Nakpih and Santini (2020) Callistus Ireneous Nakpih and Simone Santini. 2020.  Automated Discovery of Logical Fallacies in Legal Argumentation.  _International Journal of Artificial Intelligence and Applications (IJAIA)_ , 11.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  _ArXiv preprint_ , abs/2303.08774.
  * Pei and Jurgens (2021) Jiaxin Pei and David Jurgens. 2021.  Measuring Sentence-Level and Aspect-Level (Un)certainty in Science Communications.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9959–10011, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Piskorski et al. (2023) Jakub Piskorski, Nicolas Stefanovitch, Nikolaos Nikolaidis, Giovanni Da San Martino, and Preslav Nakov. 2023.  Multilingual Multifaceted Understanding of Online News in Terms of Genre, Framing, and Persuasion Techniques.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3001–3022, Toronto, Canada. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Roozenbeek and van der Linden (2020) Jon Roozenbeek and Sander van der Linden. 2020.  Breaking Harmony Square: A game that “inoculates” against political misinformation.  _The Harvard Kennedy School Misinformation Review_ , 1(8).
  * Sahai et al. (2021) Saumya Sahai, Oana Balalau, and Roxana Horincar. 2021.  Breaking Down the Invisible Wall of Informal Fallacies in Online Discussions.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 644–657, Online. Association for Computational Linguistics.
  * Salman et al. (2023) Muhammad Salman, Asif Hanif, Shady Shehata, and Preslav Nakov. 2023.  Detecting Propaganda Techniques in Code-Switched Social Media Text.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 16794–16812, Singapore. Association for Computational Linguistics.
  * Sarrouti et al. (2021) Mourad Sarrouti, Asma Ben Abacha, Yassine Mrabet, and Dina Demner-Fushman. 2021.  Evidence-based Fact-Checking of Health-related Claims.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 3499–3512, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Schlichtkrull et al. (2023a) Michael Schlichtkrull, Nedjma Ousidhoum, and Andreas Vlachos. 2023a.  The Intended Uses of Automated Fact-Checking Artefacts: Why, How and Who.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 8618–8642, Singapore. Association for Computational Linguistics.
  * Schlichtkrull et al. (2023b) Michael Sejr Schlichtkrull, Zhijiang Guo, and Andreas Vlachos. 2023b.  AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Schmid and Betsch (2019) Philipp Schmid and Cornelia Betsch. 2019.  Effective strategies for rebutting science denialism in public discussions.  _Nature Human Behaviour_ , 3(9):931–939.
  * Schuster et al. (2021) Tal Schuster, Adam Fisch, and Regina Barzilay. 2021.  Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 624–643, Online. Association for Computational Linguistics.
  * Shaar et al. (2020) Shaden Shaar, Nikolay Babulkov, Giovanni Da San Martino, and Preslav Nakov. 2020.  That is a Known Lie: Detecting Previously Fact-Checked Claims.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3607–3618, Online. Association for Computational Linguistics.
  * Silverman (2014) Craig Silverman. 2014.  _Verification Handbook: An Ultimate Guideline on Digital Age Sourcing for Emergency Coverage_.  European Journalism Centre.
  * Sumner et al. (2014) Petroc Sumner, Solveiga Vivian-Griffiths, Jacky Boivin, Andy Williams, Christos A Venetis, Aimée Davies, Jack Ogden, Leanne Whelan, Bethan Hughes, Bethan Dalton, Fred Boy, and Christopher D Chambers. 2014.  The association between exaggeration in health related science news and academic press releases: retrospective observational study.  _BMJ_ , 349.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018.  FEVER: a Large-scale Dataset for Fact Extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  _ArXiv preprint_ , abs/2307.09288.
  * Vladika and Matthes (2023) Juraj Vladika and Florian Matthes. 2023.  Scientific Fact-Checking: A Survey of Resources and Approaches.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 6215–6230, Toronto, Canada. Association for Computational Linguistics.
  * Vosoughi et al. (2018) Soroush Vosoughi, Deb Roy, and Sinan Aral. 2018.  The spread of true and false news online.  _Science_ , 359:1146–1151.
  * Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020.  Fact or Fiction: Verifying Scientific Claims.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7534–7550, Online. Association for Computational Linguistics.
  * Wadden et al. (2022) David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Iz Beltagy, Lucy Lu Wang, and Hannaneh Hajishirzi. 2022.  SciFact-Open: Towards open-domain scientific claim verification.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 4719–4734, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wardle (2018) Claire Wardle. 2018.  5 Lessons for Reporting in an Age of Disinformation.  Technical report, First Draft.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-Art Natural Language Processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wright and Augenstein (2021) Dustin Wright and Isabelle Augenstein. 2021.  Semi-Supervised Exaggeration Detection of Health Science Press Releases.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10824–10836, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Wright et al. (2022) Dustin Wright, Jiaxin Pei, David Jurgens, and Isabelle Augenstein. 2022.  Modeling Information Change in Science Communication with Semantically Matched Paraphrases.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 1783–1807, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wührl et al. (2024) Amelie Wührl, Dustin Wright, Roman Klinger, and Isabelle Augenstein. 2024.  Understanding Fine-grained Distortions in Reports of Scientific Findings.  _ArXiv preprint_ , abs/2402.12431.
  * Yu et al. (2020) Bei Yu, Jun Wang, Lu Guo, and Yingya Li. 2020.  Measuring Correlation-to-Causation Exaggeration in Press Releases.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4860–4872, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. 2019.  Defending Against Neural Fake News.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , pages 9051–9062.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020.  BERTScore: Evaluating Text Generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_.
  * Zhang et al. (2019) Yuan Zhang, Jason Baldridge, and Luheng He. 2019.  PAWS: Paraphrase Adversaries from Word Scrambling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1298–1308, Minneapolis, Minnesota. Association for Computational Linguistics.

##  Appendix A Dataset Construction I: Selecting Misrepresented Scientific
Publications From HFC Articles

###  A.1 URL Filtering

For reproducibility we collect all HTML webpages via the Wayback
Machine777https://archive.org/web/. We apply the following filtering on URLs
that may be relevant to our instances. We initially only include URLs with the
following top level domains:

  * •

_“.gov”, “.org”, “.int”, “.edu”, “.gov.uk”, “.org.uk”, “.gov.au”, “.org.nz”,
“.edu.au”, “.gov.in”, “.org.au”, “.ac.uk”_.

We remove commonly occurring fact-checking organizations that are within the
applied filtering: _“fullfact.org”, “www.poynter.org”, “factcheck.org”,
“npr.org”_. We finally add known publishers of scientific content that would
otherwise be removed via our filtering step:

  * •

_“nature.com”, “jamanetwork.com”, “thelancet.com”, “researchgate.net”,
“academic.oup.com”, “bmj.com”, “onlinelibrary.wiley.com”,“www.mdpi.com”,
“www.ijidonline.com”, “link.springer.com”, “sciencedirect.com”,
“tandfonline.com”, “journals.lww.com”, “cell.com”, “papers.ssrn.com”,
“cebm.net”, “thejournal.ie”, “cebm.ox.ac.uk”, “elsevier.com”,
“biomedcentral.com”, “journalofinfection.com”, “journals.sagepub.com”,
“scientificamerican.com”, “pfizer.com”, “www.the-scientist.com”,
“www.cancer.net”, “www.ema.europa.eu”_

Finally, we keep archived URLs

  * •

_“archive.is”, “archive.ph”, “archive.md”, “archive.vn”, “perma.cc”,
“archive.fo”_

as it is unknown from the surface form if it refers to a scientific
publication or not.

###  A.2 Annotation Process

Figure 6:  Annotation (Step 1): A preprocessed HFC article. Links for
annotation are highlighted in color.

We selected 8,695 links for annotation, using a curated list of reputable
scientific publishers and top-level domains (cf. §A.1). Few, if any, links in
a fact-checking article point to a misrepresent scientific publication. HFC
articles must be assessed in their entirety as critical statements may be
scattered. Further, articles may cover multiple related claims, or various
subclaims, each possibly misrepresenting different publications. For efficient
annotation, we grouped up to 8 distinct links per fact-checking article into
one HIT, highlighting each link in a different color (Figure 6). This resulted
in 1,385 HITs for annotation.

Each highlighted link was given to the annotators within the original context
of the HFC article, to decide if the link points to a scientific publication
that is misrepresented by a non-true claim as discussed in the article at
hand. For each misrepresented publication, annotators had to provide triplets
consisting of (1) a non-true claim that misrepresents (2) a scientific
publication, and (3) a justification of the flawed reasoning. Each triplet
requires an extracted statement from the article as justification, explaining
how the document could be misused to support the claim
($S\Rightarrow\overline{c}$) and why it does not
($S\not\Rightarrow\overline{c}$). Consecutive HITs present the same HFC
article to prevent context switching. Annotators label a link as
“misrepresented” only if the claim explicitly relied on the linked
publication’s content. Indirect misrepresentation (e.g., through a press
release) is also classified as “misrepresented” as the claim relied on the
same content. Remaining links are grouped into three sub-categories during
annotation (cf. §A.3), but these are collapsed to “not applicable” during
dataset compilation and discarded from further annotation steps.

In a pilot study with all 927 selected links from 50 fact-checking articles we
found that single annotation instead of double annotation (Krippendorff’s
$\alpha$ of 0.728; cf. §A.4) results in a minor loss in recall only. Hence,
each instance is annotated by a single annotator. Note that redundant
annotations mainly impact the recall of misrepresented scientific documents.
Data quality remains unaffected, as errors in link selection are rectified in
later annotation tasks. In total, we identified 208 scientific publications
labeled as “misrepresented” across 150 HFC articles. 107 articles contained
only one misrepresented publication, while 43 articles contained 2-4
misrepresented publications.

##### Definitions.

When selecting the triplets consisting of (1) a non-true claim that
misrepresents (2) a scientific publication, and (3) a justification of the
flawed reasoning we consider the following definitions: We consider a claim as
_non-true_ and _misrepresenting_ if it is not fully accurate (e.g., including
_partly true_ or _misleading_ claims) and lack a valid entailment relationship
($\not\Rightarrow$) with the cited document. We exclude documents from the
misrepresented category if the claim can be validly inferred from an incorrect
source or if refutation requires additional external evidence. We consider a
publication as _scientific_ if it is published in a scientific venue, may be
submitted to such a venue (preprints), or constitutes a scientific report from
a credible institution, (e.g., annual CDC reports) and include non-peer-
reviewed documents because they can be misrepresented before being accepted.

###  A.3 Link Annotation: Fine-grained labels

Annotators categorize each link into one of four categories:

  * •

Misrepresented: This category is designated for scientific publications that
are explicitly misrepresented by a non-true claim. Such publications may be
referenced either directly or indirectly, for example, through a press
release.

  * •

Misrepresentable: This category is assigned to publications that were not
misrepresented but have the potential to be misrepresented. This occurs when
the HFC discuss related scientific documents for a comprehensive overview.
While these documents could be susceptible to misrepresentation by similar
claims, they haven’t been misrepresented by the claimant.

  * •

Maybe Misrepresentable: Annotators can choose this category when uncertain.
Uncertainty may stem from ambiguity regarding a document’s scientific status
or doubts about misrepresentation.

  * •

Not Applicable: This category applies to all other links not covered by the
previous categories.

Annotators must provide an explanation for labels other than “not applicable”.
In Missci, we focus exclusively on real-world misinformation involving
genuinely misrepresented scientific documents. Instances not labeled as
“misrepresented” are collapsed into the “not applicable” class and excluded.
While we exclude links labeled “misrepresentable” or “maybe misrepresentable”
from Missci, we provide all annotations using the fine-grained taxonomy.

###  A.4 Link Annotation: Pilot Study and Final Results

##### Agreement.

The annotators, alongside one author, annotated all 221 links from 16 randomly
chosen fact-checking articles. The inter-annotator agreement, assessed with
Krippendorff’s $\alpha$, was 0.360. Disagreement primarily arose from cases
initially marked as “misrepresentable” or “maybe misrepresentable” later
grouped into the “not applicable” category (as per §A.3). Using the grouped
labels, we calculated binary inter-annotator agreement between
“misrepresented” labels and “not applicable” labels. This resulted in an
inter-annotator agreement of 0.751. The annotators then double-annotated all
706 links from additional 34 randomly selected fact-checking articles,
achieving comparable inter-annotator agreement of 0.728.

##### Single annotations are sufficient.

We assess the value of having two annotations versus one using the double-
annotated data. Specifically, if at least one annotator labels an instance as
“misrepresented” we classify it as such. A single annotator identifies 78.3%
of the same instances as “misrepresented”. When we also consider instances
labeled as “misrepresentable” by the single annotator, 95.5% of the presumed
“misprepresented” double-annotated instances are detected. These additional
cases were labeled as “misrepresentable” only by the second annotator,
indicating more uncertainty. To reduce the workload while maintaining
sufficient coverage, all remaining instances were annotated by a single
annotator.

##### Results.

In total, we found 208 (2.4%) scientific publications labelled as
“misrepresented”, 425 (4.9%) labelled as “misrepresentable”, and 596 (6.9%)
labelled as “maybe misrepresentable”. The remaining 7,466 (85.9%) links were
unrelated to our problem.

##  Appendix B Dataset Construction II: Fallacious Argument Reconstruction

###  B.1 Fallacious Reasoning for Misrepresented Science

Definition |  Logical Form
---|---
Ambiguity
When an unclear phrase with multiple definitions is used within the argument; therefore, does not support the conclusion. |  Claim X is made. Y is concluded based on an ambiguous understanding of X.
Equivocation (merged with Ambiguity)
When the same word (here used also for phrase) is used with two different meanings. Equivocation is a subset of the ambiguity fallacy. |  Term X is used to mean Y in the premise. Term X is used to mean Z in the conclusion.
Impossible Expectations / Nirvana Fallacy
Comparing a realistic solution with an idealized one, and discounting or even dismissing the realistic solution as a result of comparing to a “perfect world” or impossible standard, ignoring the fact that improvements are often good enough reason. |  X is what we have. Y is the perfect situation. Therefore, X is not good enough.
False Equivalence
Assumes that two subjects that share a single trait are equivalent. |  X and Y both share characteristic A. Therefore, X and Y are [behave] equal.
False Dilemma
Presents only two alternatives, while there may be another alternative, another way of framing the situation, or both options may be simultaneously viable. |  Either X or Y is true.
Biased Sample Fallacy
Drawing a conclusion about a population based on a sample that is biased, or chosen in order to make it appear the population on average is different than it actually is. |  Sample S, which is biased, is taken from population P. Conclusion C is drawn about population P based on S.
Hasty Generalization
Drawing a conclusion based on a small sample size, rather than looking at statistics that are much more in line with the typical or average situation. |  Sample S is taken from population P. Sample S is a very small part of population P. Conclusion C is drawn from sample S and applied to population P.
False Cause Fallacy (use as Causal Simplification)
Post hoc ergo propter hoc — after this therefore because of this. Automatically attributes causality to a sequence or conjunction of events. |  A is regularly associated with B; therefore, A causes B.
Single Cause Fallacy (use as Causal Simplification)
Assumes there is a single, simple cause of an outcome. |  X is a contributing factor to Y. X and Y are present. Therefore, to remove Y, remove X.
Fallacy of Composition
Inferring that something is true of the whole from the fact that it is true of some part of the whole. |  A is part of B. A has property X. Therefore, B has property X.
Fallacy of Division (merged with Fallacy of Composition)
Inferring that something is true of one or more of the parts from the fact that it is true of the whole. |  A is part of B. B has property X. Therefore, A has property X.
Fallacy of Exclusion / Cherry Picking / Slothful Induction
When only select evidence is presented in order to persuade the audience to accept a position, and evidence that would go against the position is withheld (Cherry Picking). Ignores relevant and significant evidence when inferring to a conclusion (Slothful Induction – focus on neglect). |  Evidence A and evidence B is available. Evidence A supports the claim of person 1. Evidence B supports the counterclaim of person 2. Therefore, person 1 presents only evidence A.
Table 5: Fallacy Overview. Definition and logical form taken from Bennett
(2012) and Cook et al. (2018). Ambiguity
---
It is said that we have a good understanding of our universe. Therefore, we
know exactly how it began and exactly when.
Equivocation
A feather is light. What is light cannot be dark. Therefore, a feather cannot
be dark.
Impossible Expectations / Nirvana Fallacy
Seat belts are a bad idea. People are still going to die in car crashes.
False Equivalence
They are both Felidae, mammals in the order Carnivora, therefore there’s
little difference between having a pet cat and a pet jaguar.
False Dilemma
I thought you were a good person, but you weren’t at church today.
Biased Sample Fallacy
Based on a survey of 1000 American homeowners, 99% of those surveyed have two
or more automobiles worth on average $100,000 each. Therefore, Americans are
very wealthy.
Hasty Generalization
My father smoked four packs of cigarettes a day since age fourteen and lived
until age sixty-nine. Therefore, smoking really can’t be that bad for you.
False Cause Fallacy
Every time I go to sleep, the sun goes down. Therefore, my going to sleep
causes the sun to set.
Single Cause Fallacy
Smoking has been empirically proven to cause lung cancer. Therefore, if we
eradicate smoking, we will eradicate lung cancer.
Fallacy of Composition
Hydrogen is not wet. Oxygen is not wet. Therefore, water (H2O) is not wet.
Fallacy of Division
His house is about half the size of most houses in the neighborhood.
Therefore, his doors must all be about 3 1/2 feet high.
Fallacy of Exclusion / Cherry Picking / Slothful Induction
Employer: “It says here on your resume that you are a hard worker, you pay
attention to detail, and you don’t mind working long hours.”
Andy: “Yes sir.”
Employer: “I spoke to your previous employer. He says that you constantly
change things that should not be changed, you could care less about other
people’s privacy, and you had the lowest score in customer relations.”
Andy: “Yes, that is all true, as well.”
Table 6: Fallacy Examples (taken from Bennett (2012)).

##### Fallacy Inventory Selection.

To select a suitable fallacy inventory, we begin by examining the fallacies
employed by Cook et al. (2018) as they pertain to misinformation within the
scientific domain. A distinction lies in the relation to science, as they
focus on climate-change denial whereas our focus lies on the misrepresentation
of scientific documents that seemingly _support_ the claims. Consequently, we
exclude fallacies like _Red Herring_ which divert attention from opposing
arguments, or _Fake experts_ , which contradicts our requirement for credible
evidence. We select the remaining fallacies by examining instances of
misrepresentation of scientific publications, guided by the collection of
logical fallacies from Bennett (2012). An overview of all selected fallacies
can be found in Table 5, along with examples in Table 6.

##### Merged Fallacies.

After annotation, we merge several fallacy classes due to difficulties in
differentiation based solely on the information from the fact-checking
article, or because they share similar traits and one is very infrequent:

  * •

Fallacy of Division/Composition: Combines _Fallacy of Division_ and _Fallacy
of Composition_ as both involve generalizations through the _part-of_
relationship.

  * •

Causal Oversimplification: Merges _Single Cause_ and _False Cause_ as they are
often indistinguishable based solely on the fact-checking article.

  * •

Ambiguity: Combines _Ambiguity_ with its subtype _Equivocation_ , which relies
on the same vocabulary in the claim and premises, a detail not accessible
during annotation.

Fallacies annotated as _Other_ were resolved into one of the existing fallacy
classes. This was always possible, as _Fallacy of Exclusion_ can be applied in
almost all cases (as it ignores compromising content (or _context_) of $S$.

##### Fallacy Inventory Discussion.

Several fallacies, such as _False Causality_ , _Hasty Generalization_ , _False
Dilemma_ or _Ambiguity_ are present in most existing NLP fallacy inventories
in some form. A distinction of our selected fallacies lies in our exclusive
focus on corrupted support ($\not\Rightarrow$) relationships. For this reason,
we exclude several fallacies that are commonly used in fallacy detection
datasets and important for propaganda techniques (Da San Martino et al., 2019;
Piskorski et al., 2023) or misinformation in general (Musi et al., 2022;
Alhindi et al., 2022):

  1. 1.

Fallacies that attack (e.g. _Ad Hominem_).

  2. 2.

Fallacies that divert (e.g. _Strawman Fallacy_).

  3. 3.

Fallacies that use manipulation techniques like slogans or emotional language
(e.g. _Appeal to Emotion_).

  4. 4.

Fallacies that utilize non-credible evidence (e.g. _False Authority_), or omit
evidence altogether (e.g. _Evading the Burden of Proof_).

##### Fallacy Inventory Comparison.

In contrast to other works (Musi et al., 2022; Alhindi et al., 2022), a strong
focus lies on a detailed analysis of generalization fallacies. We employ
various specific generalization fallacies, such as _Hasty Generalization_ ,
_Biased Sample Fallacy_ and _False Equivalence_. The fallacy of _Cherry
Picking_ has been addressed in prior research. However, our interpretation of
this fallacy aligns more closely with _Slothful Induction_. This distinction
arises because our focus is not on presenting selectively chosen information
but rather on the omission of crucial aspects of the study that weaken the
claim’s validity. We find _False Analogy_ as employed by e.g. Alhindi et al.
(2022), to be comparable to _False Equivalence_. Both fallacies can be applied
to the same problems. Without access to the claimant’s specific reasoning, we
find it impossible to prioritize one over the other when generating fallacious
premises. For similar reasons, we adopt a broad definition of the _False
Dilemma_ fallacy, which assumes that only two options (or outcomes) exist
when, in reality, more options are available. We consolidate it with the
fallacy of _Affirming the Disjunct_ , which assumes an “either/or” possibility
among different options, even when these options are not mutually exclusive.
Despite their differences (in _False Dilemma_ the options are indeed mutually
exclusive but not exhaustive), they share similar characteristics in
exhibiting black-and-white thinking and may only differ in their specificity
of the explicit reasoning that is unavailable to us.

###  B.2 Argument Reconstruction Guidelines

Annotators should base their reconstruction of fallacious arguments on content
in the fact-checking article. The final argument should be:

  * •

Comprehensive: All fallacies identified by the fact-checker should be
incorporated.

  * •

Self-contained: Subsequent steps should not necessitate the use of the fact-
checking articles.

For all text generation tasks our annotators utilize
Grammarly888https://www.grammarly.com, integrated into the Surge AI tool, to
ensure high-quality text. Each of the previously detected 208 misrepresented
links was provided to the annotators together with the the preprocessed fact-
checking article with highlighted links, and the justification for labeling
the link as “misrepresented” (from §A.2) and annotators were tasked to
reconstruct all parts of the fallacious argument. Annotations were conducted
in batches of 30-40 arguments per annotator per week with weekly meetings,
adhering to agile annotation principles (Alex et al., 2010). Each argument was
independently assessed by at least two annotators. At the end of annotation,
annotators reviewed their own annotations, excluding the last batch, to
rectify errors resulting from initial guideline misunderstandings and enhance
consistency.

##### Claim Rewriting.

Figure 7: Claim annotation interface from Surge AI.

Fact-checking articles might discuss multiple related claims or complex
arguments with subclaims. Annotators should first understand the main claim of
the fact-checking article and the misrepresented scientific document. The
misrepresenting claim $\overline{c}$ should be formulated as such that
$S\cup\overline{P}\Rightarrow\overline{c}$. Annotators should use the main
claim of the fact-checking article if possible, and make minimal changes if
necessary. While the formulation of $\overline{c}$ is a prerequisite for
detecting fallacious reasoning lines $R$, the validity of
$S\cup\overline{P}\Rightarrow\overline{c}$ can only be checked after
constructing the argument. Therefore, annotators should re-evaluate
$\overline{c}$ after identifying all fallacies. The annotation interface,
including a link to the pre-processed fact-checking article and relevant
information is shown in Figure 7.

##### Accurate Premise Writing.

The accurate premise $p_{0}$ provides a correct description of the
misrepresented scientific document $S$. Its purpose is to offer logical
support for the claim ($p_{0}\Rightarrow\overline{c}$) but it falls short due
to the presence of fallacious reasoning. Fact-checkers always include an
accurate description of the misrepresented scientific document. Annotators
must locate all relevant information and formulate $p_{0}$ s.t.

  1. 1.

The wording is as precise as possible and uses the HFC vocabulary.

  2. 2.

All accurate content that strengthens $p_{0}\Rightarrow\overline{c}$ is
included.

  3. 3.

Any accurate content that weakens $p_{0}\Rightarrow\overline{c}$ is excluded.

We guide annotators to include information in the accurate premise $p_{0}$
only if it _increases_ the plausibility of $\overline{c}$. Any information
(from $S$) that _decreases_ plausibility is paraphrased as $s_{i}$ and is part
of $R_{i}$. The publication context $s_{i}$ is optional and required only if
additional information beyond the given $p_{0}$ is necessary.

##### Hidden Premise Writing.

Fact-checkers explain how the claim $\overline{c}$ relates to the scientific
publication $S$. This may include additional knowledge not found in
$\overline{c}$ or $S$.999This relies on subjective judgment since annotators
aren’t required to read the scientific document. For example, to understand
why the claim “Cucumber kills lung cancer cells.” was made based on the
scientific finding that “cucubitacin B promoted lung tumor cell death.” one
must know that cucumbers contain cucubitacin B. This information is likely not
provided by the misrepresented publication itself. Annotators can provide any
number of hidden premises which are concise, accurate statements that
complement $S$ and are essential in understanding the connection between
$\overline{c}$ and $S$. Each hidden premise should be a single sentence
derived from the fact-checking article.

##### Fallacy Class Selection Preference

Annotators are directed to prioritize fallacies that engage with the content
(all fallacies except _Fallacy of Exclusion_) rather than ignoring crucial
aspects. We allow interchangeable fallacies with distinct classes, but we
instruct annotators to prioritize more specific fallacies over broader ones.
When multiple fallacies share the same flawed reasoning, annotators should
select the most specific fallacy class. For example, when a conclusion is
drawn from a biased sample, it can be labeled as the _Biased Sample Fallacy_.
Alternatively, it might be seen as the _Single Cause Fallacy_ assuming that
the properties for which the sample is biased do not impact the conclusion. In
this example, both fallacy classes do not differ in their applied reasoning,
but only in their level of specificity. Therefore, the more specific _Biased
Sample Fallacy_ should be preferred.101010This is different to the example in
Figure 2, in which different fallacies apply different reasoning.. We provided
a taxonomy to the annotators to specify how to choose the more specific
fallacy class if multiple apply.

##### Fallacious Premise Writing.

Figure 8: Fallacy annotation interface from Surge AI.

Annotators should thoroughly review the fact-checking article to identify all
sections discussing the claim $\overline{c}$ misrepresenting the scientific
publication $S$. These discussions may be distributed throughout the article.
Annotators must focus solely on fallacies discernible from the content of $S$.
This excludes other scientific documents used by the HFC to refute
$\overline{c}$ or assessing the credibility of $S$ (e.g., if $S$ is retracted
or inaccurate). Each fallacy should be separately formulated by the annotator
and must be accompanied by a justification extracted from the fact-checking
article. The context for each fallacy must be constructed akin to $p_{0}$,
emphasizing the weakening of the claim rather than its strengthening. The
fallacious premise must align with the selected fallacy class and make the
fallacious reasoning explicit. Given that selecting the fallacious reasoning
is subjective since different fallacious premises $p_{i}$ can re-instantiate
$S\cup p_{i}\Rightarrow\overline{c}$, annotators are permitted to formulate
multiple alternative variants that they consider plausible. The annotation
interface for a fallacy is visualized in Figure 8. After constructing all
fallacies, annotators are tasked with reviewing their own constructed argument
to ensure the coherence of the fallacies, claim, and accurate premise, in
their HIT.

###  B.3 Argument Consolidation

Our primary annotator, with the most project experience, handled the
consolidation process. The consolidator aligned all annotated fallacious
reasoning lines, and selected the best verbalized candidate for each
$\overline{c}$, $\overline{p}_{i}$ and $s_{i}$, or paraphrased multiple such
candidates into an improved version. Interchangeable fallacies with different
classes were preserved. Only clearly unjustified fallacy annotations were
discarded or corrected if possible. Each consolidated argument underwent
double-checking by an author. The consolidator and one of the authors
collaborated on a final round of fallacious premise curation. Due to different
URLs used to link to the same scientific document, some arguments with
duplicate annotations were merged using normalized URLs (cf. §B.4), resulting
in 193 distinct fallacious arguments $\mathcal{A}$. Individual annotators
might discard arguments during annotation if they cannot reconstruct a
fallacious argument, for example because of insufficient information provided
by the HFC or violations of the credibility assumption of the cited
publication (we analyze the impact of such cases where fewer annotations are
available in §C.3). In seven cases no annotator could reconstruct
$\mathcal{A}$, and two more fallacious arguments were discarded during
consolidation.

###  B.4 Scientific Document Annotation.

Figure 9: Credibility annotation via Surge AI.

Fact-checkers may use varying links of the same publication within their fact-
checking articles. To detect duplicates these links must be normalized. To
assess the content of publicly available studies, we require them to be
available via PMC. Annotators are tasked to find a URL pointing to the
misrepresented scientific publication $S$ and select the first applicable URL
of the following list:

  1. 1.

PMC (available as _fulltext_)

  2. 2.

sciencedirect (available as _fulltext_)

  3. 3.

Original Publisher (available as _fulltext_)

  4. 4.

PubMed (available as _abstract only_)

  5. 5.

PMC (available as _abstract only_)

  6. 6.

sciencedirect (available as _abstract only_)

  7. 7.

Original Publisher (available as _abstract only_)

  8. 8.

as-is

We also asked whether the HFC highlighted flaws in the study itself (see
Figure 9).

##  Appendix C Dataset Analysis

###  C.1 Publication Credibility

Annotation | Documents | Krippendorff’s $\alpha$
---|---|---
Flawed/Criticised | 39 | 0.504
Preprint | 28 | 0.582
Retracted | 2 | 0.665
Outdated | 1 | 0.0
Any from Above | 61 | 0.577
Credible | 123
Table 7: Publication Credibility: Results of the credibility annotations of
the misrepresented publications per fallacious argument alongside with the
inter-annotator agreement per label.

Augenstein (2021) identifies two key challenges in science communication: (1)
assessing the credibility of scientific publications, and (2) preventing the
misrepresentation of a study’s findings. This study addresses the second
challenge, while assuming the credibility. To examine to which degree our
assumption holds, we report the credibility annotations of the used
publications in Table 7. Note that a publication may exhibit more than one
credibility issue (e.g., a preprint, that was criticised) We consider a
publication $S$ as “credible” only if no annotator detected any aspect
compromising its credibility in the HFC article, providing a conservative
estimate. For 123 arguments, no credibility violations were identified. Even
identified violations do not necessarily mean that the publication is not
credible or not misrepresented by $\overline{c}$. For instance, preprints lack
formal community approval but may still be credible. At the time of
annotation, 16 out of 28 publications marked as preprints by the HFC were
accepted. Further, criticism of a study does not make it scientifically
unsound. In fact, some critics highlight omitted limitations that could lead
to misunderstandings similar to the misrepresentations studied in this work.
The overall agreement in distinguishing credible documents from those with any
credibility issue is 0.577 (Krippendorff’s $\alpha$).

Figure 10: Heatmap of co-occurring credibility annotations per misrepresented
document a fallacious argument.

Figure 10 displays the co-occurring credibility annotations assigned to
fallacious arguments. We count an occurrence of a credibility label if at
least one annotator assigned it to the misrepresented document, and, hence,
ignore duplicate annotations of the same label.

###  C.2 Claims about the COVID-19 infodemic

Figure 11: COVID-19 Arguments: Collected arguments per year and whether they
constitute COVID-19 related misinformation.

Missci comprises fallacious arguments from 2014 to 2022. We manually evaluate
each argument for its association with COVID-19 misinformation, considering
context provided by the HFC, rather than explicit COVID-19 mentions. For
instance, we consider the claim “Ivermectin sterilizes most men who take it”
as COVID-19-related because it only exists due to the misinformation about
Ivermectin as a COVID-19 cure. Figure 11 illustrates argument distribution
over time and their COVID-19 relevance.

###  C.3 On the Coverage of Fallacies

In some cases, annotators may decide that an argument cannot be reconstructed
and discard the HIT. This can for example happen, if the annotator considers
the information provided by the HFC insufficient to reconstruct the argument,
or if the cited publication itself is non-credible. Table 8 compares the
annotation assignments and successful argument reconstructions. A total of 153
arguments were reconstructed by all assigned annotators. In 25 cases, one
annotator, and in 6 cases, two annotators could not reconstruct the argument.
Given the limited number of annotations and ambiguity when assigning fallacy
classes, we aim to shed light into the recall of our detected fallacies. To
this end, we compare the number of successfully reconstructed fallacious
arguments with the number of detected fallacies (excluding interchangeable
fallacies) in Figure 12. More annotators generally lead to more identified
fallacies. The majority of arguments with successful reconstructions of two
and three annotators comprise two or three distinct fallacies respectively.

The analysis indicates that Missci may lack comprehensiveness of fallacies, in
arguments with fewer annotations. However, this does not affect our main
objective since annotators are likely to identify the most severe fallacies,
highlighted by the HFC, that violate _strong inductive support_ (cf. §3.2). A
single study, under specific conditions, rarely gives unconditional support
for any claim $c$ without potential generalizations. While not necessarily or
severely fallacious, these generalizations may be less emphasized by HFC,
which makes them harder to detect as fallacious reasoning for the annotators.
In real-world texts, identifying different fallacies among annotators is not
unusual (Da San Martino et al., 2019), and distinguishing between fallacious
and accurate reasoning can be subtle (Boudry et al., 2015).

_Annotators_ | _Result_
---|---
Assigned | Successful | Num. Arguments
2 | 1 | 9
3 | 1 | 6
3 | 2 | 16
Incl. failed reconstructions | 31
2 | 2 | 83
3 | 3 | 70
No failed reconstruction | 153
Table 8: Annotation Assignments: Number of successfully reconstructed argument
versions by the number of initially assigned annotators. Figure 12: Argument
Annotations: Number of successfully constructed arguments (_y-axis_) compared
to the number of assigned annotators who manually reconstruct each argument
(_color_) per distinct fallacious reasoning lines $R_{i}$ in the argument post
consolidation (_x-axis_). The number of arguments shows if one (_blue_), two
(_orange_) or three (_green_) distinct arguments versions of the same argument
were constructed by the annotators.

##  Appendix D Experiments

###  D.1 Fallacious Premise Evaluation

We use BERTScore (Zhang et al., 2020) F1 with version 0.3.13, based on DeBERTa
(He et al., 2021), fine-tuned on MNLI (Williams et al., 2018), as recommended
at the time of this writing.111111https://github.com/Tiiiger/bert_score For
the NLI-based metric, we use the predicted probability of the entailment
label, following (Honovich et al., 2022). We use their provided T5-XXL model
(Raffel et al., 2020), fine-tuned on SNLI (Bowman et al., 2015), MNLI
(Williams et al., 2018), FEVER (Thorne et al., 2018), Scitail (Khot et al.,
2018), PAWS (Zhang et al., 2019), and VitaminC (Schuster et al., 2021). The
predicted probability for the token “1” (representing _entailment_) is used
for evaluation.

Multiple interchangeable reference fallacious premises $\overline{p}_{i,j}$
may exist. Therefore, before evaluating the generated fallacious premise
$\hat{\overline{p}}_{i,1}$, it needs to be matched with a reference text. For
the first-ranked predicted fallacy by the model, which consists of a
fallacious premise and fallacy class ($\hat{\overline{p}}_{i,1}$,
$\hat{f}_{i,1}$), we use the gold fallacious premise $\overline{p}_{i,j}$ as
the reference text if the accompanied gold fallacy class $f_{i,j}$ matches the
predicted fallacy class $\hat{f}_{i,1}$. In the absence of matches based on
the fallacy class, we select the $\overline{p}_{i,j}$ with the highest cosine
similarity to $\hat{\overline{p}}_{i,1}$ from all interchangeable fallacious
premises $\overline{p}_{i,j}$. Cosine similarity is measured using the
sentence embeddings produced by SBERT (Reimers and Gurevych, 2019) (all-mpnet-
base-v2).

###  D.2 Prompt Engineering for Fallacy Generation and Classification

Template | P@1 | Arg-1
---|---|---
p1-basic (D) | 0.208 | 0.467
p2-support (D) | 0.222 | 0.467
p3-undermine (D) | 0.208 | 0.467
p4-connect (D) | 0.278 | 0.533
p5-auto (D) | 0.194 | 0.433
p5-auto-connect (D) | 0.264 | 0.467
p1-basic (DE) | 0.208 | 0.467
p2-support (DE) | 0.194 | 0.433
p3-undermine (DE) | 0.208 | 0.467
p4-connect (DE) | 0.292 | 0.600
p5-auto (DE) | 0.194 | 0.433
p5-auto-connect (DE) | 0.278 | 0.500
p1-basic (DL) | 0.208 | 0.467
p2-support (DL) | 0.222 | 0.500
p3-undermine (DL) | 0.236 | 0.533
p4-connect (DL) | 0.361 | 0.700
p5-auto (DL) | 0.250 | 0.567
p5-auto-connect (DL) | 0.306 | 0.633
p1-basic (DLE) | 0.236 | 0.500
p2-support (DLE) | 0.236 | 0.533
p3-undermine (DLE) | 0.250 | 0.533
p4-connect (DLE) | 0.306 | 0.600
p5-auto (DLE) | 0.222 | 0.500
p5-auto-connect (DLE) | 0.236 | 0.500
p1-basic (L) | 0.181 | 0.367
p2-support (L) | 0.208 | 0.433
p3-undermine (L) | 0.236 | 0.500
p4-connect (L) | 0.278 | 0.567
p5-auto (L) | 0.194 | 0.400
p5-auto-connect (L) | 0.236 | 0.433
p1-basic (LE) | 0.208 | 0.467
p2-support (LE) | 0.250 | 0.567
p3-undermine (LE) | 0.222 | 0.500
p4-connect (LE) | 0.236 | 0.500
p5-auto (LE) | 0.222 | 0.500
p5-auto-connect (LE) | 0.264 | 0.600
Table 9: Argument Reconstruction: Prompt tuning using LLaMA 2 (70B)

The only existing prompts related to fallacy detection stem from Alhindi et
al. (2022). Yet, they were (a) not used in zero-shot experiments, instead
relying on fine-tuning, and (b) were used to only classify the fallacy within
a single text. Since our task differs and demands a zero-shot setup, we
manually select novel prompts for the task. We initially experiment on few
instances to derive promising prompt templates. We assess the performance of
each prompt with different combinations of (_D_ , _L_ , and _E_) using LLaMA 2
on the 30 arguments of the validation split in Table 9. All prompts and hyper-
parameters are listed in §F.

###  D.3 Premise Evaluation on Correct/Incorrect Fallacy Classes

| _Correct Fallacy Class_ | _Incorrect Fallacy Class_
---|---|---
Model | METEOR | BERTScore | NLI-A | NLI-S | METEOR | BERTScore | NLI-A | NLI-S
LLaMA 2 (_D_) | 0.214 | 0.616 | 0.181 | 0.231 | 0.224 | 0.617 | 0.106 | 0.124
LLaMA 2 (_DE_) | 0.238 | 0.615 | 0.144 | 0.174 | 0.227 | 0.622 | 0.119 | 0.142
LLaMA 2 (_DL_) | 0.218 | 0.608 | 0.105 | 0.120 | 0.199 | 0.618 | 0.136 | 0.148
LLaMA 2 (_DLE_) | 0.236 | 0.624 | 0.130 | 0.154 | 0.199 | 0.614 | 0.128 | 0.143
LLaMA 2 (_L_) | 0.243 | 0.622 | 0.239 | 0.266 | 0.256 | 0.628 | 0.116 | 0.141
LLaMA 2 (_LE_) | 0.194 | 0.605 | 0.149 | 0.168 | 0.177 | 0.610 | 0.114 | 0.125
GPT 4 (_D_) | 0.259 | 0.636 | 0.094 | 0.200 | 0.229 | 0.611 | 0.057 | 0.092
GPT 4 (_L_) | 0.264 | 0.637 | 0.109 | 0.267 | 0.227 | 0.604 | 0.046 | 0.088
Table 10: Fallacious Premise Evaluation based on Fallacy Class Correctness:
Automatic metrics separately evaluated over fallacious premises when the
predicted fallacy class was correct or not.

We closely examine the predictions in §5.3 and analyze the generated
fallacious premises under two conditions: (a) when accompanied by a correct
fallacy class, and (b) when not accompanied by an accurate fallacy class. The
results are presented in Table 10. Specifically, for the metrics METEOR,
BERTScore, and NLI-S, the fallacious premises produced by GPT 4 outperform
those generated by LLaMA 2 when the fallacy class was correctly predicted.
This differs strongly from the results when not separating correct from
incorrect fallacy predictions. LLaMA 2 (_D_), the previously superior model in
fallacious premise generation (from the main results in Table 2), maintains a
good performance in this evaluation. In almost all cases, the performance of
both LLMs is higher for correctly classified fallacies compared to incorrectly
classified premises. This is likely attributed to the fact that we only have
matching fallacious premises as reference text when the fallacy class is
accurately predicted, but also indicates a certain consistency between the
fallacy class and fallacious premise generation. While we believe it is
important to report the performance as done here, we chose to report the
overall performance in the main results in §5.3. This decision is made because
(a) models may (and do, see §E.4) produce correct premises but assign the
inaccurate fallacy class, and (b) to ensure comparability among models, as all
scores are based on identical instances and not influenced by the model’s
specific fallacy classification performance.

###  D.4 Class-wise analysis of the fallacy classification

Figure 13: Fallacy Classification for LLaMA 2 Confusion matrix based on LLaMA
2 (_Logical + Example_) of predicted and gold fallacy classes when provided
with $p_{0}$, $s_{i}$, $\overline{p_{i}}$ and $\overline{c}$. Figure 14:
Fallacy Classification for GPT 4 Confusion matrix based on GPT 4 (_Logical +
Example_) of predicted and gold fallacy classes when provided with $p_{0}$,
$s_{i}$, $\overline{p_{i}}$ and $\overline{c}$.

We examine the class-wise predictions of the best-performing models concerning
the fallacies applied in the gold fallacious premises $\overline{p}_{i}$. The
confusion matrices of LLaMA 2 (_LE_) and GPT 4 (_LE_) are presented in Figures
13 & 14. Both LLMs exhibit strong performances across the majority of fallacy
classes. For GPT 4, the most confusion arises between _Ambiguity_ and _False
Equivalence_ , as well as between _Biased Sample Fallacy_ and _Hasty
Generalization_. The commonalities between the former two fallacies align with
the frequently co-occurring fallacies identified by our annotators in §4.3.
Conversely, _Hasty Generalization_ , although clear-cut in our annotations, is
often used interchangeably with its superclass _Faulty Generalization_ in the
wild. We hypothesize that GPT 4 may have acquired knowledge during pretraining
from discussions where claims generalizing from biased subsets are labeled as
_Hasty Generalization_. Similarly, for LLaMA 2, most confusion primarily
arises between _False Equivalence_ and _Ambiguity_. However, we observe that
LLaMA 2 tends to overpredict _False Equivalence_ in general.

##  Appendix E Experiment Analysis

###  E.1 GPT 4 (D) Fallacy Detection Performance over k

Figure 15:  Number of correctly predicted fallacy classes of the GPT 4 (_D_)
model when considering all correctly predicted fallacies among the top $k$
predictions.

We report the number of correctly detected fallacies per distinct fallacy
class in Figure 15 for the GPT-4 (_D_) model with the best overall performance
from §5.3. By allowing the model to provide $k$ additional fallacy class
predictions, the number of correctly identified fallacies increases,
especially for the _Fallacy of Exclusion_ and _False Equivalence_.

###  E.2 Fallacy-wise classification performance

Figure 16:  Per-Fallacy Performance: F1 score per predicted fallacy class from
a multi-label multi-class classification perspective considering all
predictions by the respective model.

We visualize the precision (top) and recall (bottom) from a multi-class multi-
label perspective for each distinct fallacy class of LLaMA-2 and GPT-4 (both
(_D_)) in Figure 16.

###  E.3 The impact of the fallacious premise

To better understand the impact of the generated fallacious premise
$\overline{p}_{i}$, we additionally compare and evaluate the LLMs to see how
well they can predict a correct fallacy class given only the inaccurate claim
$\overline{c}$, the accurate premise $p_{0}$ and the context $s_{i}$, without
generating the fallacious premise $\overline{p}_{i}$. We compare the results
with the accuracy achieved in the fallacy reconstruction (Table 2), and when
the fallacious premise is provided (Table 3). The results are depicted in
Table 11. We report P@1 when the gold $\overline{p}_{i}$ is not provided to
the model (i.e. for _Reconstruct_ and _n/a_), since each of the
interchangeable fallacies is equally valid. When $\overline{p}_{i}$ is
provided (_Given_), we report the stricter accuracy metric in which only the
one applied fallacy is correct. Interestingly, LLaMA 2 consistently performs
better when it is not required to generate the fallacious reasoning for the
fallacy classification task.

| _Role of $\overline{p}_{i}$_
---|---
Model | Reconstruct | Given | n/a
LLaMA 2 (_D_) | 0.223 | 0.577 | 0.248
LLaMA 2 (_DE_) | 0.209 | 0.637 | 0.264
LLaMA 2 (_DL_) | 0.196 | 0.630 | 0.237
LLaMA 2 (_DLE_) | 0.209 | 0.568 | 0.259
LLaMA 2 (_LE_) | 0.212 | 0.645 | 0.267
LLaMA 2 (_L_) | 0.193 | 0.601 | 0.262
GPT 4 (_D_) | 0.317 | 0.738 | 0.267
GPT 4 (_L_) | 0.292 | 0.744 | 0.245
Table 11: Fallacy classification without fallacious premises: Performance when
predicting the applied fallacy class only, when required to _reconstruct_ the
fallacious premise, when the gold fallacious premise is _given_ to the model,
or when the fallacious premise is _n/a_ and does not need to be generated.

We hypothesize that this is due to the poor utility of the fallacious premises
generated by LLaMA 2 (cf. manual analysis in §E.4). For GPT 4, which produced
substantially better fallacious premises according to our human evaluation,
the model exhibits improved performance, similar to chain-of-thought prompting
(Wei et al., 2022), when tasked with generating the fallacious premise for
both evaluated prompts.

To further shed light on this, In Table 12 we compare the performance of GPT 4
when (a) it was able to generate a fallacious premise that matched the
reference premise according to human judgment, and (b) when it did not. We
report the P@1 (or accuracy) for the GPT 4 models121212We only report GPT 4
results here because we found too few instances in which LLaMA 2 generated a
matching fallacious premise in §E.4. when tasked to additionally generate the
fallacious premise, only classify the fallacy, or classify the fallacy given
the gold $\overline{p}_{i}$ in Table 11. The results show a substantial
performance difference among both evaluated splits of instances, regardless of
whether the fallacious premises needed to be generated or not, suggesting that
these instances are challenging for both objectives. When provided with the
gold reference $\overline{p}_{i}$, however, the difference in performance
decreases, showing that the verbalized fallacious reasoning overcomes these
challenges. For both prompts, the best fallacy classification performance was
achieved when the model was tasked with additionally generating the fallacious
premise and did so correctly. Note that we report P@1 if the fallacious
premise $\overline{p}_{i}$ is not provided, and accuracy otherwise.

|  | _Matching $\hat{\overline{p}}_{i}$_
---|---|---
Model | Setup | Yes | No
| classify $f_{i}$ and gen. $\overline{p}_{i}$ | 0.880 | 0.229
GPT 4 (_D_) | classify $f_{i}$ w/o $\overline{p}_{i}$ | 0.640 | 0.114
| classify $f_{i}$ given $\overline{p}_{i}$ | 0.788 | 0.689
| classify $f_{i}$ and gen. $\overline{p}_{i}$ | 0.867 | 0.133
GPT 4 (_L_) | classify $f_{i}$ w/o $\overline{p}_{i}$ | 0.533 | 0.133
| classify $f_{i}$ given $\overline{p}_{i}$ | 0.732 | 0.722
Table 12: Comparison to fallacy generation difficulty: We separate the 60
instances from our manual analysis per GPT 4 by whether the generated
fallacious premises apply the same fallacious reasoning as the reference gold
fallacious premises (_Yes_) or not (_No_) and report the performance in three
setups for both evaluated prompts.

###  E.4 Manual Evaluation of Generated Fallacies

For the first ranked generated fallacious premise $\hat{\overline{p}}_{i}$ and
predicted fallacy class $\hat{f}_{i}$ of 240 model predictions we assess if

  1. 1.

the LLM outputs a fallacious premise (Q1),

  2. 2.

the generated premise $\hat{\overline{p}}_{i}$ represents an applicable
premise within the argument bridges the reasoning gap based on the context
provided via $s_{i}$ (if exists) (Q2),

  3. 3.

Q2 and the generated premise $\hat{\overline{p}}_{i}$ applies the predicted
fallacy class $\hat{f}_{i}$ (Q3),

  4. 4.

the generated premise $\hat{\overline{p}}_{i}$ expresses the same fallacious
reasoning as the reference $\overline{p}_{i}$ (Q4).

We answer each question with yes/no, without awareness of the model or prompt
for each prediction. Only in a single instance, the LLM failed to generate any
fallacious premise (Q1). We provide the results of our manual analysis (Q2 and
Q3) in Table 14. Most differences can be seen among both LLMs. Premises
generated when a gold fallacy was assigned were considered valid slightly more
often compared to when no gold fallacy was predicted. When the gold fallacy
was predicted, the predicted fallacy class almost always matched the fallacy
of the generated premise. However, when no gold fallacy class was predicted,
often the predicted fallacy class did also not match the generated premise.
Since we used stratified sampling when selecting the predictions, we
approximate the overall performance $s$:

$s=\mathrm{P@1}\times h^{correct}+(1-\mathrm{P@1})\times h^{incorrect}$

where $h^{correct}$ (or $h^{incorrect}$) represent the manual evaluation among
all instances considered correct (or incorrect) by the automatic evaluation
via P@1. Examples are provided in Table 15. The first example contains
relatively similar premises generated by the LLM and the annotators. In the
second example, the LLM provides the same fallacious reasoning that applies
results from mice to humans, yet the premise is much more specific and
tailored to the provided content of the misrepresented publication. However,
the human premise does not entail the LLM-generated premises. The third
example exhibits semantically similar premises. However, the LLM-generated
premise makes a causal assumption, differing from the _Biased Sample Fallacy_
employed in the human-written premise. In the last example, the LLM-generated
premise looks similar but does not logically support the claim, which reasons
that in order to stop COVID-19, we need higher temperatures (and hence climate
change) since higher temperatures stop the spread of COVID-19.

We use Q4 to assess the quality of each applied metric in Table 13. We
separately report each metric among premise pairs when the generated premises
involves the same fallacious reasoning as the reference premise (score should
be high), and when not (score should be low). As commonly done (Banerjee and
Lavie, 2005; Zhang et al., 2020), we report Pearson correlation to human
judgment. When computing the Pearson correlation, we convert “yes” and “no”
into numerical values of 1 and 0. Unlike semantic similarity, we observed that
deciding whether the same reasoning flaw is verbalized is more often a binary
decision than a decision on a continuous scale.

We found NLI-S to be most correlated with human judgment, while the asymmetric
counterpart NLI-A exhibited the least correlation. This confirms our initial
hypothesis that LLMs may produce more specific premises than our annotators
(see examples in §E.4, Table 15), and vice versa, which should be accounted
for by the metric.

| _Matching premises_ | _Pearson r_
---|---|---
Metric | Yes | No | _r_ | P-value
METEOR | 0.280 | 0.227 | 0.153 | 0.017*
BERTScore | 0.638 | 0.622 | 0.092 | 0.158
NLI-A | 0.155 | 0.140 | 0.022 | 0.736
NLI-S | 0.317 | 0.150 | 0.209 | 0.001*
Table 13: Human evaluation (Q4): Comparison of the used metrics reported over 68 generated premises that match the reference premise and 172 premises that do not match the reference premise. We measure the correlation with the human judgment via Person _r_ and mark significant results with an asterisk.  | _Gold Fallacy Prediction_ | _No Gold Fallacy Prediction_ |  _Overall_ (approximated)
---|---|---|---
Model | Plausible | Matching | Plausible | Matching | Plausible | Matching
LLaMA 2 (_L_) | 5 (17%) | 5 (17%) | 2 (7%) | 1 (3%) | _0.167_ | _0.040_
LLaMA 2 (_D_) | 7 (23%) | 7 (23%) | 4 (13%) | 3 (10%) | _0.233_ | _0.107_
GPT 4 (_L_) | 26 (87%) | 26 (87%) | 25 (83%) | 11 (37%) | _0.867_ | _0.503_
GPT 4 (_D_) | 25 (83%) | 24 (80%) | 18 (60%) | 10 (33%) | _0.674_ | _0.481_
Table 14: Results of our manual analysis based on 240 ($30\times 4\times 2$) manually assessed model predictions. We report the number of generated premises deemed plausible, and the number of plausible fallacious premises in which the applied fallacy class matches the predicted fallacy class. Matches |  LLM |  Annotator
---|---|---
yes |  Since THC is a compound in marijuana and HU210 is more potent than THC, it follows that marijuana has the same brain cell growth properties as HU210. |  HU210 and THC have similar properties. Since HU210 increases the number of brain cells, THC also causes brain cells to grow.
yes |  Because HU210, a synthetic compound found in cannabis, increases the number of cells in the hippocampus of mice, the use of any cannabis-related substance, like marijuana, will yield the same results in humans. |  Mice and humans are both mammals. Therefore, marijuana grows brain cells in humans.
no |  The fluoride in the pineal gland comes from the fluoride in toothpaste. |  Elderly people are a subset of all people. Therefore, fluoride in toothpaste harms the pineal gland in all humans.
no |  Efforts to combat climate change will result in warmer average temperatures, therefore increasing the prevalence of the SARS-CoV-2 virus. |  Lower SARS-CoV-2 transmission was associated with lower temperature. Therefore, lower temperature facilitates SARS-CoV-2 transmission.
Table 15: Matching of Fallacious Premises: Examples of the manual evaluation
with fallacious premises produced by an LLM and by the annotators, together
with our rating whether they express the same reasoning flaw or not.

##  Appendix F Reproducibility

###  F.1 Hyperparameters

For reproducibility, experiments on LLaMA 2 use a batch size of 1, zero
temperature, and a beam size of 1, with a fixed random seed set to 1. LLaMA 2
experiments were implemented using the Huggingface transformers library (Wolf
et al., 2020). To address computational constraints, we employ 4-bit
quantization (Dettmers et al., 2022) for LLaMA 2 (70B), which we found
comparable to 8-bit quantization in preliminary experiments. All prompting
experiments wit LLaMA 2 models are executed on A100 GPUs. For GPT 4 we use the
GPT version=gpt-4 and the API version=2023-10-01-preview with a maximum new
token length of $1000$ and content filtering turned off, using the OpenAI
API131313https://platform.openai.com/docs/api-reference.

###  F.2 Prompts

####  F.2.1 Prompts for Fallacious Argument Reconstruction

We include _definitions_ , _logical forms_ or _examples_ from literature
(Bennett, 2012; Cook et al., 2018) (cf. §B.1). Notably, examples do not
constitute real-world fallacies, but constitute educational examples that
clearly outline the fallacious reasoning behind them, such as “A feather is
light. What is light cannot be dark. Therefore, a feather cannot be dark” (cf.
Table 6). We assess different combinations thereof in various prompt templates
outlined in Figure 17. Using this template, we evaluate various task
descriptions (Figures 18-23; key differences are highlighted in bold) for the
different combinations of definitions, logical forms and examples. We do not
argue that these are the respective best prompts to solve the introduced task.
Rather, they serve as useful baselines that for future research that and are
identical for both evaluated LLMs to allow for direct comparison as well as
the analysis of the impact of _definitions_ , _logical forms_ or _examples_
within these prompts.

Fallacy Inventory: Ambiguity: Definition 1: _When an unclear phrase with
multiple definitions is used within the argument; therefore, does not support
the conclusion._ Logical Form 1: _Claim X is made. Y is concluded based on an
ambiguous understanding of X._ Example 1: _It is said that we have a good
understanding of our universe. Therefore, we know exactly how it began and
exactly when._ Definition 2: _When the same word (here used also for phrase)
is used with two different meanings._ Logical Form 2: _Term X is used to mean
Y in the premise. Term X is used to mean Z in the conclusion._ Example 2: _A
feather is light. What is light cannot be dark. Therefore, a feather cannot be
dark._ Impossible Expectations: Definition 1: _Comparing a realistic solution
with an idealized one, and discounting or even dismissing the realistic
solution as a result of comparing to a “perfect world” or impossible standard,
ignoring the fact that improvements are often good enough reason._ Logical
Form 1: _X is what we have. Y is the perfect situation. Therefore, X is not
good enough._ Example 1: _Seat belts are a bad idea. People are still going to
die in car crashes._ <more fallacies> Task: <task-instruction with instance>
Figure 17: Argument Reconstruction Template: Overall template used to
reconstruct fallacious arguments in which _definition_ , _logical form_ and
_example_ are used. We evaluate different combinations thereof, as well as
different task descriptions. Examine the following fallacious argument:
Premise 1: “<$p_{0}$>” Premise 2: “<$s_{i}$>” Premise 3: “” Therefore:
“<claim>” Premises 1 and 2 are sourced from the same credible scientific
document. The claim is based on the information in Premise 1. However, Premise
2 suggests that the claim is an invalid conclusion from the scientific
document. Your task is to identify and verbalize the fallacious reasoning in
Premise 3 (the fallacious premise) that is necessary to support the claim,
despite the conflicting information in Premise 2. Only consider fallacies from
the provided fallacy inventory. Present each fallacious premise along with the
applied fallacy class in this format: Fallacious Premise: <fallacious
premise>; Applied Fallacy Class: <applied fallacy class>. If there are
multiple applicable fallacies, list them in order of relevance. Figure 18: P1
Basic: Our most basic task instruction to reconstruct the fallacious argument.
Examine the following fallacious argument: Premise 1: “<$p_{0}$>” Premise 2:
“<$s_{i}$>” Premise 3: “” Therefore: “<claim>” Premises 1 and 2 are sourced
from the same credible scientific document. The claim is based on the
information in Premise 1. However, Premise 2 suggests that the claim is an
invalid conclusion from the scientific document. Your task is to identify and
verbalize the fallacious reasoning in Premise 3 (the fallacious premise) that
is necessary to support the claim, despite the conflicting information in
Premise 2. This reasoning should be strong enough to support the claim and
counter any uncertainties raised by Premise 2. Only consider fallacies from
the provided fallacy inventory. Present each fallacious premise along with the
applied fallacy class in this format: Fallacious Premise: <fallacious
premise>; Applied Fallacy Class: <applied fallacy class>. If there are
multiple applicable fallacies, list them in order of relevance. Figure 19: P2
Support: The model is tasked to produce a fallacious premise that increases
the support behind the claim. Examine the following fallacious argument:
Premise 1: “<$p_{0}$>” Premise 2: “<$s_{i}$>” Premise 3: “” Therefore:
“<claim>” Premises 1 and 2 are sourced from the same credible scientific
document. The claim is based on the information in Premise 1. However, Premise
2 suggests that the claim is an invalid conclusion from the scientific
document. Your task is to identify and verbalize the fallacious reasoning in
Premise 3 (the fallacious premise) that is necessary to support the claim,
despite the conflicting information in Premise 2. This reasoning should
effectively support the claim, ensuring that Premise 2 does not undermine the
claim as a valid conclusion. Only consider fallacies from the provided fallacy
inventory. Present each fallacious premise along with the applied fallacy
class in this format: Fallacious Premise: <fallacious premise>; Applied
Fallacy Class: <applied fallacy class>. If there are multiple applicable
fallacies, list them in order of relevance. Figure 20: P3 Undermine: The task
is rather phrased negatively. The model must generate the fallacious premise
to avoid that $s_{i}$ undermines the claim. Examine the following fallacious
argument: Premise 1: “<$p_{0}$>” Premise 2: “<$s_{i}$>” Premise 3: “”
Therefore: “<claim>” Premises 1 and 2 are sourced from the same credible
scientific document. The claim is based on the information in Premise 1.
However, Premise 2 suggests that the claim is an invalid conclusion from the
scientific document. Your task is to identify and verbalize the fallacious
reasoning in Premise 3 (the fallacious premise) that is necessary to support
the claim, despite the conflicting information in Premise 2. Do not repeat the
claim itself, Premise 1, or Premise 2 when generating the fallacious Premise
3. Make sure the generated Premise 3 connects Premise 1 and Premise 2 to