* Sinha  _et al._ [2022] A. Sinha, U. Azad, and H. Singh, Proceedings of the AAAI Conference on Artificial Intelligence 36, 9935 (2022).
  * Zhou  _et al._ [2020c] X. Zhou, Y. Feng, and S. Li, in  _Proceedings of the 39th International Conference on Computer-Aided Design_, ICCAD ’20 (Association for Computing Machinery, New York, NY, USA, 2020).
  * Paler  _et al._ [2022] A. Paler, L. M. Sasu, A.-C. Florea, and R. Andonie, ACM Transactions on Quantum Computing (2022), 10.1145/3565271.
  * Saeedi  _et al._ [2011] M. Saeedi, R. Wille, and R. Drechsler, Quantum Information Processing 10, 355 (2011).
  * [383] R. Wille, O. Keszocze, M. Walter, P. Rohrs, A. Chattopadhyay, and R. Drechsler, in  _2016 21st Asia and South Pacific Design Automation Conference (ASP-DAC)_, pp. 292–297.
  * Hart  _et al._ [1968] P. E. Hart, N. J. Nilsson, and B. Raphael, IEEE Transactions on Systems Science and Cybernetics 4, 100 (1968).
  * Emerson  _et al._ [2005] J. Emerson, R. Alicki, and K. Życzkowski, Journal of Optics B: Quantum and Semiclassical Optics 7, S347 (2005).
  * Magesan  _et al._ [2011] E. Magesan, J. M. Gambetta, and J. Emerson, Physical Review Letters 106, 180504 (2011).
  * Dankert  _et al._ [2009] C. Dankert, R. Cleve, J. Emerson, and E. Livine, Physical Review A 80, 012304 (2009).
  * Lévi  _et al._ [2007] B. Lévi, C. C. López, J. Emerson, and D. G. Cory, Physical Review A 75, 022314 (2007).
  * Magesan  _et al._ [2012] E. Magesan, J. M. Gambetta, B. R. Johnson, C. A. Ryan, J. M. Chow, S. T. Merkel, M. P. Da Silva, G. A. Keefe, M. B. Rothwell, T. A. Ohki, _et al._ , Physical Review Letters 109, 080505 (2012).
  * Emerson  _et al._ [2007] J. Emerson, M. Silva, O. Moussa, C. Ryan, M. Laforest, J. Baugh, D. G. Cory, and R. Laflamme, Science 317, 1893 (2007).
  * Barends  _et al._ [2014] R. Barends, J. Kelly, A. Veitia, A. Megrant, A. Fowler, B. Campbell, Y. Chen, Z. Chen, B. Chiaro, A. Dunsworth, _et al._ , Physical Review A 90, 030303 (2014).
  * Onorati  _et al._ [2019] E. Onorati, A. Werner, and J. Eisert, Physical Review Letters 123, 060501 (2019).
  * Carignan-Dugas  _et al._ [2015] A. Carignan-Dugas, J. J. Wallman, and J. Emerson, Physical Review A 92, 060302 (2015).
  * Cross  _et al._ [2016] A. W. Cross, E. Magesan, L. S. Bishop, J. A. Smolin, and J. M. Gambetta, npj Quantum Information 2, 16012 (2016).
  * Helsen  _et al._ [2019] J. Helsen, X. Xue, L. M. Vandersypen, and S. Wehner, npj Quantum Information 5, 71 (2019).
  * França and Hashagen [2018] D. S. França and A. Hashagen, Journal of Physics A: Mathematical and Theoretical 51, 395302 (2018).
  * Proctor  _et al._ [2019] T. J. Proctor, A. Carignan-Dugas, K. Rudinger, E. Nielsen, R. Blume-Kohout, and K. Young, Physical Review Letters 123, 030503 (2019).
  * Zhang  _et al._ [2022d] Y. Zhang, W. Yu, P. Zeng, G. Liu, and X. Ma, arXiv:2203.10320 (2022d).
  * Gaebler  _et al._ [2012] J. P. Gaebler, A. M. Meier, T. R. Tan, R. Bowler, Y. Lin, D. Hanneke, J. D. Jost, J. Home, E. Knill, D. Leibfried, _et al._ , Physical Review Letters 108, 260503 (2012).
  * Gambetta  _et al._ [2012] J. M. Gambetta, A. D. Córcoles, S. T. Merkel, B. R. Johnson, J. A. Smolin, J. M. Chow, C. A. Ryan, C. Rigetti, S. Poletto, T. A. Ohki, _et al._ , Physical Review Letters 109, 240504 (2012).
  * McKay  _et al._ [2019] D. C. McKay, S. Sheldon, J. A. Smolin, J. M. Chow, and J. M. Gambetta, Physical Review Letters 122, 200502 (2019).
  * Garion  _et al._ [2021] S. Garion, N. Kanazawa, H. Landa, D. C. McKay, S. Sheldon, A. W. Cross, and C. J. Wood, Physical Review Research 3, 013204 (2021).
  * Helsen  _et al._ [2022] J. Helsen, I. Roth, E. Onorati, A. H. Werner, and J. Eisert, PRX Quantum 3, 020357 (2022).
  * Boixo  _et al._ [2018] S. Boixo, S. V. Isakov, V. N. Smelyanskiy, R. Babbush, N. Ding, Z. Jiang, M. J. Bremner, J. M. Martinis, and H. Neven, Nature Physics 14, 595 (2018).
  * Neill  _et al._ [2018] C. Neill, P. Roushan, K. Kechedzhi, S. Boixo, S. V. Isakov, V. Smelyanskiy, A. Megrant, B. Chiaro, A. Dunsworth, K. Arya, _et al._ , Science 360, 195 (2018).
  * Barends  _et al._ [2019] R. Barends, C. Quintana, A. Petukhov, Y. Chen, D. Kafri, K. Kechedzhi, R. Collins, O. Naaman, S. Boixo, F. Arute, _et al._ , Physical Review Letters 123, 210501 (2019).
  * Aaronson and Gunn [2019] S. Aaronson and S. Gunn, arXiv:1910.12085 (2019).
  * Barak  _et al._ [2020] B. Barak, C.-N. Chou, and X. Gao, arXiv:2005.02421 (2020).
  * Chen  _et al._ [2022a] J. Chen, D. Ding, C. Huang, and L. Kong, arXiv:2206.08293 (2022a).
  * Cross  _et al._ [2019] A. W. Cross, L. S. Bishop, S. Sheldon, P. D. Nation, and J. M. Gambetta, Physical Review A 100, 032328 (2019).
  * Horodecki  _et al._ [1999] M. Horodecki, P. Horodecki, and R. Horodecki, Physical Review A 60, 1888 (1999).
  * [412] “Quantinuum sets new record with highest ever quantum volume,” https://www.quantinuum.com/pressrelease/quantinuum-sets-new-record-with-highest-ever-quantum-volume.
  * [413] “Pushing quantum performance forward with our highest quantum volume yet,” https://research.ibm.com/blog/quantum-volume-256.
  * Wack  _et al._ [2021] A. Wack, H. Paik, A. Javadi-Abhari, P. Jurcevic, I. Faro, J. Gambetta, and B. Johnson, arXiv:2110.14108 (2021).
  * Proctor  _et al._ [2022] T. Proctor, K. Rudinger, K. Young, E. Nielsen, and R. Blume-Kohout, Nature Physics 18, 75 (2022).
  * Dong and Lin [2021] Y. Dong and L. Lin, Physical Review A 103, 062412 (2021).
  * Tomesh  _et al._ [2022] T. Tomesh, P. Gokhale, V. Omole, G. S. Ravi, K. N. Smith, J. Viszlai, X.-C. Wu, N. Hardavellas, M. R. Martonosi, and F. T. Chong, in  _2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_ (IEEE, 2022) pp. 587–603.
  * van der Schoot  _et al._ [2022] W. van der Schoot, D. Leermakers, R. Wezeman, N. Neumann, and F. Phillipson, in  _2022 IEEE International Conference on Quantum Software (QSW)_ (IEEE, 2022) pp. 9–16.
  * Lubinski  _et al._ [2021] T. Lubinski, S. Johri, P. Varosy, J. Coleman, L. Zhao, J. Necaise, C. H. Baldwin, K. Mayer, and T. Proctor, arXiv:2110.03137 (2021).
  * [420] “Algorithmic qubits: A better single-number metric,” https://ionq.com/posts/february-23-2022-algorithmic-qubits.
  * Markov  _et al._ [2018] I. L. Markov, A. Fatima, S. V. Isakov, and S. Boixo, arXiv:1807.10749 (2018).
  * De Raedt  _et al._ [2007] K. De Raedt, K. Michielsen, H. De Raedt, B. Trieu, G. Arnold, M. Richter, T. Lippert, H. Watanabe, and N. Ito, Computer Physics Communications 176, 121 (2007).
  * Smelyanskiy  _et al._ [2016] M. Smelyanskiy, N. P. Sawaya, and A. Aspuru-Guzik, arXiv:1601.07195 (2016).
  * Häner and Steiger [2017] T. Häner and D. S. Steiger, in  _Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis_, SC ’17 (Association for Computing Machinery, New York, NY, USA, 2017).
  * Pednault  _et al._ [2017] E. Pednault, J. A. Gunnels, G. Nannicini, L. Horesh, T. Magerlein, E. Solomonik, E. W. Draeger, E. T. Holland, and R. Wisnieff, arXiv:1710.05867 (2017).
  * Steiger  _et al._ [2018] D. S. Steiger, T. Häner, and M. Troyer, Quantum 2, 49 (2018).
  * Luo  _et al._ [2020] X.-Z. Luo, J.-G. Liu, P. Zhang, and L. Wang, Quantum 4, 341 (2020).
  * Jones  _et al._ [2019b] T. Jones, A. Brown, I. Bush, and S. C. Benjamin, Scientific Reports 9, 10736 (2019b).
  * Suzuki  _et al._ [2021] Y. Suzuki, Y. Kawase, Y. Masumura, Y. Hiraga, M. Nakadai, J. Chen, K. M. Nakanishi, K. Mitarai, R. Imai, S. Tamiya, T. Yamamoto, T. Yan, T. Kawakubo, Y. O. Nakagawa, Y. Ibe, Y. Zhang, H. Yamashita, H. Yoshimura, A. Hayashi, and K. Fujii, Quantum 5, 559 (2021).
  * Orús [2014] R. Orús, Annals of physics 349, 117 (2014).
  * Vidal [2003] G. Vidal, Physical Review Letters 91, 147902 (2003).
  * Vidal [2004] G. Vidal, Physical Review Letters 93, 040502 (2004).
  * Daley  _et al._ [2004] A. J. Daley, C. Kollath, U. Schollwöck, and G. Vidal, Journal of Statistical Mechanics: Theory and Experiment 2004, P04005 (2004).
  * Verstraete and Cirac [2004] F. Verstraete and J. I. Cirac, cond-mat/0407066 (2004).
  * Shi  _et al._ [2006] Y.-Y. Shi, L.-M. Duan, and G. Vidal, Physical Review A 74, 022320 (2006).
  * Vidal [2007] G. Vidal, Physical Review Letters 99, 220405 (2007).
  * Schollwöck [2011] U. Schollwöck, Annals of Physics 326, 96 (2011).
  * White [1992] S. R. White, Physical Review Letters 69, 2863 (1992).
  * White [1993] S. R. White, Physical Review B 48, 10345 (1993).
  * McCaskey  _et al._ [2018] A. McCaskey, E. Dumitrescu, M. Chen, D. Lyakh, and T. Humble, PLOS ONE 13, 1 (2018).
  * Wang  _et al._ [2017] D. S. Wang, C. D. Hill, and L. C. Hollenberg, Quantum Information Processing 16, 176 (2017).
  * Dang  _et al._ [2019] A. Dang, C. D. Hill, and L. C. Hollenberg, Quantum 3, 116 (2019).
  * Huang  _et al._ [2019b] H.-L. Huang, W.-S. Bao, and C. Guo, Physical Review A 100, 032305 (2019b).
  * Zhou  _et al._ [2020d] Y. Zhou, E. M. Stoudenmire, and X. Waintal, Physical Review X 10, 041038 (2020d).
  * Shang  _et al._ [2022] H. Shang, L. Shen, Y. Fan, Z. Xu, C. Guo, J. Liu, W. Zhou, H. Ma, R. Lin, Y. Yang, _et al._ , arXiv:2207.03711 (2022).
  * Bylaska  _et al._ [2021] E. J. Bylaska, D. Song, N. P. Bauman, K. Kowalski, D. Claudino, and T. S. Humble, Frontiers In Chemistry 9, 26 (2021).
  * Yalouz  _et al._ [2021] S. Yalouz, B. Senjean, J. Günther, F. Buda, T. E. O’Brien, and L. Visscher, Quantum Science and Technology 6, 024004 (2021).
  * Manrique  _et al._ [2021] D. Z. Manrique, I. T. Khan, K. Yamamoto, V. Wichitwechkarn, and D. M. Ramo, arXiv:2008.08694 (2021).
  * Xia and Kais [2020] R. Xia and S. Kais, Quantum Science and Technology 6, 015001 (2020).
  * Li  _et al._ [2021b] W. Li, Z. Huang, C. Cao, Y. Huang, Z. Shuai, _et al._ , Chemical Science 13, 8953 (2021b).
  * Liu  _et al._ [2020b] J. Liu, L. Wan, Z. Li, and J. Yang, Journal of Chemical Theory and Computation 16, 6904 (2020b).
  * Fan  _et al._ [2021b] Y. Fan, J. Liu, Z. Li, and J. Yang, The Journal of Physical Chemistry Letters 12, 8833 (2021b).
  * Kottmann  _et al._ [2021] J. S. Kottmann, P. Schleich, T. Tamayo-Mendoza, and A. Aspuru-Guzik, The Journal of Physical Chemistry Letters 12, 663 (2021).
  * Cao  _et al._ [2022] C. Cao, J. Hu, W. Zhang, X. Xu, D. Chen, F. Yu, J. Li, H.-S. Hu, D. Lv, and M.-H. Yung, Physical Review A 105, 062452 (2022).
  * Ryabinkin  _et al._ [2021] I. G. Ryabinkin, A. F. Izmaylov, and S. N. Genin, Quantum Science and Technology 6, 024012 (2021).
  * Xu  _et al._ [2022] Z. Xu, Y. Fan, H. Shang, and C. Guo, arXiv:2211.07983 (2022).
  * Hastings [2009] M. B. Hastings, Journal of mathematical physics 50, 095207 (2009).
  * Han  _et al._ [2018] Z.-Y. Han, J. Wang, H. Fan, L. Wang, and P. Zhang, Physical Review X 8, 031012 (2018).
  * Jiang  _et al._ [2008] H. C. Jiang, Z. Y. Weng, and T. Xiang, Physical Review Letters 101, 090603 (2008).
  * Corboz  _et al._ [2010] P. Corboz, R. Orús, B. Bauer, and G. Vidal, Physical Review B 81, 165104 (2010).
  * Jordan  _et al._ [2008] J. Jordan, R. Orús, G. Vidal, F. Verstraete, and J. I. Cirac, Physical Review Letters 101, 250602 (2008).
  * Lubasch  _et al._ [2014] M. Lubasch, J. I. Cirac, and M.-C. Bañuls, Physical Review B 90, 064425 (2014).
  * Levin and Nave [2007] M. Levin and C. P. Nave, Physical Review Letters 99, 120601 (2007).
  * Verstraete  _et al._ [2008] F. Verstraete, V. Murg, and J. I. Cirac, Advances in physics 57, 143 (2008).
  * Orús and Vidal [2009] R. Orús and G. Vidal, Physical Review B 80, 094403 (2009).
  * Xie  _et al._ [2009] Z. Y. Xie, H. C. Jiang, Q. N. Chen, Z. Y. Weng, and T. Xiang, Physical Review Letters 103, 160601 (2009).
  * Zhao  _et al._ [2010] H. H. Zhao, Z. Y. Xie, Q. N. Chen, Z. C. Wei, J. W. Cai, and T. Xiang, Physical Review B 81, 174411 (2010).
  * Xie  _et al._ [2012] Z. Y. Xie, J. Chen, M. P. Qin, J. W. Zhu, L. P. Yang, and T. Xiang, Physical Review B 86, 045139 (2012).
  * Villalonga  _et al._ [2019] B. Villalonga, S. Boixo, B. Nelson, C. Henze, E. Rieffel, R. Biswas, and S. Mandrà, npj Quantum Information 5, 86 (2019).
  * Villalonga  _et al._ [2020] B. Villalonga, D. Lyakh, S. Boixo, H. Neven, T. S. Humble, R. Biswas, E. G. Rieffel, A. Ho, and S. Mandrà, Quantum Science and Technology 5, 034003 (2020).
  * Markov and Shi [2008] I. L. Markov and Y. Shi, SIAM Journal on Computing 38, 963 (2008).
  * Gray and Kourtis [2021] J. Gray and S. Kourtis, Quantum 5, 410 (2021).
  * Huang  _et al._ [2021d] C. Huang, F. Zhang, M. Newman, X. Ni, D. Ding, J. Cai, X. Gao, T. Wang, F. Wu, G. Zhang, _et al._ , Nature Computational Science , 1 (2021d).
  * Chen  _et al._ [2022b] Y. Chen, Y. Liu, X. Shi, J. Song, X. Liu, L. Gan, C. Guo, H. Fu, D. Chen, and G. Yang, arXiv:2205.00393 (2022b).
  * Kalachev  _et al._ [2021] G. Kalachev, P. Panteleev, and M.-H. Yung, arXiv:2108.05665 (2021).
  * Chen  _et al._ [2018] J. Chen, F. Zhang, C. Huang, M. Newman, and Y. Shi, arXiv:1805.01450 (2018).
  * Chen  _et al._ [2020b] M.-C. Chen, R. Li, L. Gan, X. Zhu, G. Yang, C.-Y. Lu, and J.-W. Pan, Physical Review Letters 124, 080502 (2020b).
  * Li  _et al._ [2019b] R. Li, B. Wu, M. Ying, X. Sun, and G. Yang, IEEE Transactions on Parallel and Distributed Systems 31, 805 (2019b).
  * Sudarshan  _et al._ [1961] E. C. G. Sudarshan, P. M. Mathews, and J. Rau, Physical Review Journals Archive 121, 920 (1961).
  * Jordan and Sudarshan [1961] T. F. Jordan and E. C. G. Sudarshan, Journal of Mathematical Physics 2, 772 (1961).
  * Choi [1975] M.-D. Choi, Linear Algebra and its Applications 10, 285 (1975).
  * Griewank [1992] A. Griewank, Optimization Methods and Software 1, 35 (1992).
  * Guo and Poletti [2021] C. Guo and D. Poletti, Physical Review E 103, 013309 (2021).
  * Schuld and Killoran [2022] M. Schuld and N. Killoran, arXiv:2203.01340 (2022).
  * Tang [2019] E. Tang, in  _Proceedings of the 51st Annual ACM SIGACT Symposium on Theory of Computing_ (2019) pp. 217–228.
  * Ding  _et al._ [2021b] C. Ding, T.-Y. Bao, and H.-L. Huang, IEEE Transactions on Neural Networks and Learning Systems (2021b), 10.1109/TNNLS.2021.3084467.
  * Arrazola  _et al._ [2019] J. M. Arrazola, A. Delgado, B. R. Bardhan, and S. Lloyd, arXiv:1905.10415 (2019).
  * Peng  _et al._ [2020] T. Peng, A. W. Harrow, M. Ozols, and X. Wu, Physical Review Letters 125, 150504 (2020).
  * Ying  _et al._ [2022] C. Ying, B. Cheng, Y. Zhao, H.-L. Huang, Y.-N. Zhang, M. Gong, Y. Wu, S. Wang, F. Liang, J. Lin, _et al._ , arXiv:2207.14142 (2022).
  * Huang  _et al._ [2018b] H.-L. Huang, A. K. Goswami, W.-S. Bao, and P. K. Panigrahi, SCIENCE CHINA Physics, Mechanics & Astronomy 61, 060311 (2018b).


# Flee the Flaw: Annotating the Underlying Logic of
Fallacious Arguments Through Templates and Slot-filling

Irfan Robbani1 Paul Reisert2 Naoya Inoue1,3 Surawat Pothong1 Camélia
Guerraoui4,3,5
Wenzhi Wang4,3 Shoichi Naito6,4,3 Jungmin Choi3 Kentaro Inui7,4,3
1JAIST 2Beyond Reason 3RIKEN 4Tohoku University
5INSA Lyon 6Ricoh Company, Ltd. 7MBZUAI
<EMAIL_ADDRESS><EMAIL_ADDRESS>
{guerraoui.camelia.kenza.q4, wang.wenzhi.r7<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Prior research in computational argumentation has mainly focused on scoring
the quality of arguments, with less attention on explicating logical errors.
In this work, we introduce four sets of explainable templates for common
informal logical fallacies designed to explicate a fallacy’s implicit logic.
Using our templates, we conduct an annotation study on top of 400 fallacious
arguments taken from LOGIC dataset and achieve a high agreement score
(Krippendorf’s $\alpha$ of 0.54) and reasonable coverage (0.83). Finally, we
conduct an experiment for detecting the structure of fallacies and discover
that state-of-the-art language models struggle with detecting fallacy
templates (0.47 accuracy). To facilitate research on fallacies, we make our
dataset and guidelines publicly available.

##  1 Introduction

A _fallacy_ is an invalid or weak argument supported by unsound reasoning
Hinton (2020). The automatic detection of fallacies has important
applications, including providing constructive feedback to learners in
writing. The assessment of argument quality, including fallacy detection, is
considered an important topic in the fields of computational argumentation and
argumentation mining Wachsmuth et al. (2017); Ke and Ng (2019).

Figure 1: Overview of our proposed fallacy logic structure. We extend (b’)
existing argumentative representation Reisert et al. (2018) consisting of
Claim and Premise $P$ by adding (c) Premise $P^{\prime}$, which explains what
makes the argument fallacious. The example annotation shows: (i) the claim
“($A^{\prime}$=_further advanced courses_) are BAD” is supported by “$P$:
($A$=_further advanced courses_) SUPPRESS ($C$= _GPA_), a GOOD thing”, and
(ii) $P$ is then further supported by “$P^{\prime}$: ($A^{\prime}$=_NLP
class_) SUPPRESS ($C$=_GPA_), a GOOD thing”, where $A^{\prime}$=_NLP
class_(their own experience) is implicitly generalized to $A$=_further
advanced courses_(advanced courses in general), which makes the overall
argument fallacious.

Previous work on quality assessment has focused on numerical scoring Carlile
et al. (2018); Ke et al. (2019) and fallacy type-labeling tasks Jin et al.
(2022); Sourati et al. (2023a), without aiming to analyze _fallacy logic
structures_ , namely the representation of _how_ given arguments are weak. In
the field of argumentation theory, a typology of invalid arguments has been
long studied and compiled into an inventory Walton (1987); Bennett (2012). The
inventory typically includes semi-formal definitions and some examples for
each type of fallacy. For example, _Faulty Generalization_ is a widely
recognized fallacy type, characterized by “Drawing a conclusion based on a
small sample size, rather than looking at statistics that are much more in
line with the typical or average situation.” Bennett (2012). The semi-formal
definition is as follows: “(i) Sample $S$ is taken from population $P$. (ii)
Sample $S$ is a very small part of population $P$. (iii) Conclusion $C$ is
drawn from sample $S$ and applied to population $P$”. Although such inventory
provides insights into how the analysis of fallacy logic structure can be
formulated as an NLP task, several important questions remain: (i) How should
the annotation scheme for fallacy logic structure identification be designed?
(ii) Can humans consistently annotate fallacy logic structures? (iii) To what
extent is the automatic identification of fallacy logic structure a
challenging task for machines?

To address this issue, we propose _fallacy logic structure identification_ , a
new task for identifying the underlying logical structure of fallacies. For
this task, we design an annotation scheme and conduct an annotation study to
examine its feasibility. The key idea behind our annotation scheme is to
enrich previous work on the argumentative structure with a fallacy structure
from an inventory of common fallacy types.

Consider the argument in Fig. 1, where the writer persuades people _not_ to
take advanced courses at Stanford because they claim it will hurt their GPA.
The claim is further supported by the writer’s own, single experience based on
their NLP class. This is a faulty generalization caused by the writer
_implicitly_ assuming that their single experience can be generalized to
everyone. Previous work in fallacy identification Sourati et al. (2023b); Jin
et al. (2022) would identify this argument as _Faulty Generalization_ (Fig. 1
(a)), but no additional information such as logical structure or fallacious
reasoning is provided. Argumentation Schemes Walton et al. (2008), a well-
known typology for the representation of arguments, would categorize this
argument as _Argument from Consequence_ (Fig. 1 (b)), and Reisert et al.
(2018)’s Argument Templates, an operationalized version of Argumentation
Schemes, represent this argument with a more fine-grained, logical
representation by structured templates (Fig. 1 (b’)). To represent the
committed fallacy structure, our work further enriches this representation by
adding an additional premise that indicates how the given argument is
fallacious (Fig. 1 (c)).

Our main contributions are as follows:

  * •

We conduct the first study of formulating logical fallacy structure by
creating an inventory of fallacy templates (§3).

  * •

We create the first dataset of fallacy logical structures which consists of
400 arguments from LOGIC Jin et al. (2022) annotated with our templates (§4).
We publicly release both the dataset and
guidelines111https://github.com/itsanonnymous/fallacytemplate. Our dataset
achieves high inter-annotator agreement (Krippendorf’s $\alpha$ of 0.54) and
coverage (0.83%).

  * •

We show that the fallacy logic structure identification task poses a
significant challenge for state-of-the-art language models. (§5).

Figure 2: Our templates for annotating fallacious argument logical structure.
We extend upon existing work Walton et al. (2008); Reisert et al. (2018),
consisting of a conclusion (i.e., $A$ should (not) be brought about) and
supporting premise, by adding an additional supporting premise in bold which
represents the committed fallacy logical structure.

##  2 Related Work

#### Fallacies Annotation Study

Several studies address creating benchmarks for fallacy identification,
including Habernal et al. (2017) for game facilitation and Ruiz-Dolz and
Lawrence (2023) for validating argumentation corpora. Particularly, Jin et al.
(2022) focus on logical fallacies within climate change discourse, emphasizing
the challenges posed by complex scientific data. They developed detailed
annotation guidelines to aid in consistent identification of fallacies within
climate arguments. Similarly, Goffredo et al. (2023) analyzed fallacious
reasoning in U.S. presidential debates, highlighting common fallacies. They
employed advanced computational techniques and the INCEpTION platform for
structured annotation, ensuring reliability through cross-verification and
Krippendorff’s $\alpha$. In addition to the current benchmark establishment,
this research proposes benchmark resources aimed at capturing fallacy
structure rather than solely identifying fallacies. This research fills the
gap, extending previous work by focusing on template annotation to capture the
underlying structure of fallacious arguments.

#### Argumentation Structure

Argumentation theory examines how arguments, including those about daily
exercise, are constructed and evaluated. To begin with, Stab and Gurevych
(2017) establishes methods for parsing argumentation structure in persuasive
essays by identifying and classifying argument components and their
relationships. Toulmin (2003) provides a framework for analyzing arguments by
breaking them down into components like Claim, Grounds, Warrant, and Rebuttal.
Walton (2013) focuses on specific argumentation schemes, such as Argument from
Analogy, which compares similar situations to infer outcomes but risks failure
with irrelevant similarities (false analogy). The Argument from Consequence
Walton et al. (2008) emphasizes potential outcomes of actions, often involving
causality and appeals to consequences. Evaluating it requires considering 1)
the connection between action and consequence, 2) the quality of supporting
evidence, and 3) whether opposing consequences have been addressed. Building
on prior work on argument structure, particularly the Argument from
Consequence scheme (a frequently used scheme by Walton), this research
addresses a gap by using argument templates, inspired byReisert et al. (2018)
to capture the structure of fallacies within this scheme. This choice is
motivated by the scheme’s frequent use and its potential for revealing
fallacious arguments. Building on this potential, and inspired by Reisert et
al. (2018) on templates, we address a gap by using templates to capture the
structure of fallacies within the Argument from the Consequence scheme.
Previous work on Argument from Consequence demonstrates high coverage in
annotation efforts, further supporting this approach.

##  3 Fallacy Logic Structure

###  3.1 Design Principles

To develop an annotation scheme for fallacy logic structure, we adhere to
three key criteria.

First, we require the annotation to be able to explain the underlying
structure of fallacy. We extend the existing representation of arguments (Fig.
1 (b’)) by an additional premise attached with an explanation as to why it
fallaciously supports the original premise (Fig. 1 (c)).

Second, our annotation scheme must cover a majority of fallacy types. We focus
on the fallacies most commonly studied in computational argumentation, such as
those in Alhindi et al. (2023) and Helwe et al. (2023), whose statistics on
fallacy types guide our template design to match the most frequent
occurrences. We develop 20 new templates covering four defective induction
fallacy types–Fallacy of Credibility, False Causality, False Dilemma, and
Faulty Generalization. An example and more detailed explanation regarding four
defective induction fallacy types can be seen in section A.2.

Third, our annotation scheme must utilise Reisert et al. (2018) template
selection and slot-filling approach further simplifying annotation while
remaining computationally friendly. As inspired by the Argument from
Consequence and employing Reisert et al. (2018)’s work as a base scheme, the
template design captures both positive and negative consequences within the
scheme. This results in two templates for each consequence type, along with a
template addressing instances that cannot be directly covered. This approach
aims to provide rich information about fallacy structures while simplifying
the annotation process.

###  3.2 Representation of Core Arguments

The underlying structure of arguments has been represented previously with
Walton et al. (2008)’s Argumentation Schemes, a set of roughly 60 schemes
which provide structure between argumentative components such as a conclusion
(i.e., claim) and premise. An example of a common scheme, Argument from
Negative Consequences, is as follows222For readability, we represent
placeholders in brackets.:

  * •

Premise ($P$): If [$A$] is brought about, bad consequences will plausibly
occur.

  * •

Conclusion: Therefore, [$A$] should not be brought about.

Here, $A$ is a placeholder (i.e., slot-filler) represents an action and $P$
supports conclusion. For the argument in Fig. 1, we represent Argument from
Negative Consequence with [$A$]=“further advanced courses”.

Towards operationalizing Walton et al. (2008)’s Argumentation Schemes into
more fine-grained logical representations, Reisert et al. (2018) developed
argument templates, an inventory of annotation-friendly templates consisting
of ingredients such as placeholders. An example of an argument template built
on top of Argument from Negative Consequences scheme is as follows:

  * •

Premise ($P$): [$A$] SUPPRESS a GOOD [$C$].

  * •

Conclusion: [$A$] is BAD.

Both $A$ and $C$ represent action and consequence placeholders, respectively.
GOOD and BAD represent the sentiment of each placeholder, and SUPPRESS
represents the relation between $A$ and $C$, where SUPPRESS refers to
preventing the consequence Hashimoto et al. (2012). Revisiting the argument in
Fig. 1, we can instantiate the argument template with $A$=“further advanced
courses” and $C$=“GPA”. Such argument templates are a simple, efficient way to
represent underlying logic.

As shown for Faulty Generalization fallacies in Figure 2, argument templates
were handcrafted to allow for both Argument from Positive Consequence ($A$
should be brought about) and Argument from Negative Consequence ($A$ should
not be brought about) with a supporting $P^{\prime}$ (grey) consisting of
positive (e.g., $A$ PROMOTE GOOD($C$)) and negative (e.g., $A$ SUPPRESS
GOOD($C$)) consequences, respectively, where PROMOTE refers to the triggering
of the consequence Hashimoto et al. (2012). We build on top of this for adding
logical structure for fallacies.

###  3.3 Our Fallacy Template Inventory

For representing fallacy logical structure, we extend Walton et al. (2008) and
Reisert et al. (2018) by introducing a new premise $P^{\prime}$ which supports
premise $P$. Consider the following representation for Faulty Generalization:

  * •

Premise ($P$): [$A$] SUPPRESS a GOOD [$C$].

  * •

Premise (P’): [$A^{\prime}$], a subset of $A$, SUPPRESS a GOOD [$C$]

  * •

Conclusion: [$A$] is BAD.

Here, on top of the argument template placeholders $A$ and $C$, $P^{\prime}$
includes a new placeholder $A^{\prime}$, where $A^{\prime}$ is an action and
$A^{\prime}\subseteq{A}$. The faulty generalization is committed as a result
of the argument considering $A^{\prime}$ to represent $A$ as a
whole.Revisiting the argument in Fig. 1, we can instantiate the above with
$A$=“further advanced courses”, $A^{\prime}$=“NLP class”, and $C$=“GPA”.

Figure 3: Examples of template and slot-fillers from FtF for Faulty
Generalization.

Fig. 3 shows additional examples of template instantiation with placeholders
for each target fallacy type, with our new premise $P^{\prime}$. Using this
figure, we exemplify a complex Faulty Generalization argument, where two
subsets $A^{\prime}$ and $C^{\prime}$ are considered. The main point is
symbolized by $A$=“garage” and $C$=“overcharged”, as the narrative implies
that the $A$ is notorious for $C$. Hence, it is implicated that $C$ is BAD and
that $A$] PROMOTE $C$. In $P^{\prime}$,’ $A^{\prime}$=“mechanic” and
$C^{\prime}$=“overcharged her” are identified, where $A^{\prime}\subseteq{A}$
and $C^{\prime}\subseteq{C}$ and $A^{\prime}$ PROMOTE $C^{\prime}$. Therefore,
the relation $A^{\prime}$ PROMOTES $C^{\prime}$ supports the relation $A$
PROMOTE $C$, so template #2 is selected.

##  4 Flee the Flaw (FtF) Dataset

We discuss the creation of our dataset Flee the Flaw (henceforth, FtF). First,
we use an existing dataset of annotated fallacious arguments for creating our
guidelines and building our inventory of fallacy templates. We then conduct a
full-fledged annotation on top of 400 arguments.

###  4.1 Data Collection

To build a dataset of fallacious argument template instantiations, we require
fallacious arguments which cover our target fallacy types. Therefore, we use
LOGIC Jin et al. (2022), an English fallacy dataset consisting of 2,449
fallacious arguments spanned across multiple fallacy types, including our four
target template types. We sampled 400 arguments (100 per target fallacy type)
from LOGIC, equally split between its development (LOGIC-DEV200) and training
sets (LOGIC-TRAIN200), with 200 arguments each. Missing fallacy instances in
the development set were supplemented from the training set, ensuring no
overlap by segmenting the training set before distribution.

Figure 4: The distribution of fallacy templates in our FtF between one
annotator (top row) and the other (bottom row) for all 400 instances in our
train and dev set, where each fallacy type consists of 100 instances. The
x-axis refers to the selected template, and y-axis refers to the frequency.

###  4.2 Guideline Construction

We employed two expert annotators for guideline development and annotation: a
native English-speaking postdoctoral researcher specializing in argument
mining (who led guideline creation), and a non-native English-speaking
graduate student specializing in argumentation.

To create a set of guidelines and test annotation feasibility, we conduct a
multi-round pilot study on top of LOGIC-DEV200. Aside from the pilot study
itself, annotators did not go through any training phase. Given that the LOGIC
dataset has limited fallacious arguments, our pilot study consisted of 200
instances (50 per fallacy type) for creating our final guidelines, where the
study began with an initial set of guidelines for all fallacy types. For each
of the four fallacy types, annotators focused on the 50 instances per each
fallacy. For each type, we split up the instances to annotate (e.g., 10 out of
50) using the latest updated set of guidelines, where results were compared
and discussed after each round. Discussion consisted of findings and whether
annotators agree with each other’s annotation. If there was a new finding or
disagreement, instances were discussed to reach a consensus and guidelines
were updated accordingly. The process was repeated until all 200 instances in
LOGIC-DEV200 were annotated and the final annotation guidelines were
created.333The final guidelines are made publicly available:
https://github.com/itsanonnymous/fallacytemplate

#### Reducing Annotation Complexities

During guideline construction, annotators found that multiple templates could
be instantiated for a single argument. In order to reduce annotation
complexity, the following conditions were created: i) preservation of
argument’s original, explicit intent, ii) paraphrase arguments into Argument
from Consequences, and iii) preference of entities over events.

We demonstrate such conditions with the False Dilemma argument: “ _We either
have to cut taxes or leave a huge debt for our children._ ”. Opposed to
selecting the entity $A$=“taxes” which satisfies the third condition,
annotators were encouraged to select the event $A$=“cut taxes” as it maintains
the explicit intention of the argument, satisfying the first condition. Given
that this is a False Dilemma fallacious argument which follows an either-or,
the annotators satisfied the second condition by considering that the argument
can be thought of in terms of argument from consequence, where the conclusion
“cut taxes should be brought about” is good as it suppresses the premise
“leave a huge debt for our children”, a bad thing.

In addition to the above, it was discovered that the fallacy type provided by
LOGIC could be categorized into other, non-target fallacy types (e.g.,
Slippery Slope instead of Faulty Generalization). In such instances,
annotators were instructed to annotate the instance considering its given type
and encouraged to apply template #5 if the template instantiation could not be
made.

###  4.3 Annotation Procedure

Given a fallacious argument, its fallacy type, and our templates, the
procedure for fallacious template instantiation is as follows. First,
annotators select the appropriate template from the given set of 5 templates.
Next, annotators write in the necessary slot-fillers taken from the input
argument. Afterwards, annotators provide their confidence level for instances
in which they are not 100% confident. Finally, annotators provide any
necessary comments to accompany the annotation. The resulting annotation of
our fallacious templates on top of LOGIC-DEV200 and LOGIC-TRAIN200 resulted in
FtF-DEV and FtF-TRAIN, respectively.

Fallacy Type | GWET AC1 |  Krippendorff’s $\alpha$
---|---|---
False Dilemma | 0.63 |  0.44
Faulty Generalization | 0.40 |  0.36
False Causality | 0.71 |  0.65
Fallacy of Credibility | 0.58 |  0.49
Average | 0.57 |  0.54
Table 1: Template selection Inter-Annotator Agreement.

###  4.4 Statistics and Analysis

#### Inter-Annotator Agreement (IAA)

Table 1 shows our IAA scores for template selection. Our GWET AC1 Gwet (2008)
scores range from 0.40 to 0.71, indicating moderate to the substantial
agreement. We also calculate Krippendorff’s alpha Hayes and Krippendorff
(2007) and achieve a score of 0.54, indicating a high agreement.

Given that Faulty Generalization had the lowest agreement, we conduct an
additional analysis on all disagreements for Faulty Generalization arguments.
We discover that 60% of disagreements were caused when one annotator labeled
’#5’ and the other instantiated a template, where reasons annotators labeled
’#5’ were due to complicated instances and implicitness of the argument.
Lastly, some instances in LOGIC were found to be other types of fallacies,
namely Slippery Slope.

#### Distribution of Templates

Fig. 4 shows the distribution of the fallacy templates for both annotators. We
immediately observe that certain templates were rarely selected by annotators
for LOGIC, such as template #3 for False Dilemma. Regardless of this skewed
distribution, as reported, we still achieved a high IAA and coverage for
template selection.

#### Coverage

Table 2 provides a comparison of annotation coverage forannotators, namely the
percentage of instances where a non-template #5 is annotated. Overall, our
templates achieve a high coverage for both annotators, with scores of 80% and
83%. We observe that fallacy types such as False Dilemma and False Causality
achieve high coverage due to their straightforward reasoning.

Fallacy Type | Annotator 1 | Annotator 2
---|---|---
False Dilemma | 0.90 | 0.91
Faulty Generalization | 0.68 | 0.76
False Causality | 0.95 | 0.96
Fallacy of Credibility | 0.64 | 0.83
Average | 0.80 | 0.83
Table 2: Coverage of fallacy templates for both annotators.

##  5 Experiments

To what extent is the automatic identification of fallacy logic structure
challenging for machines? We evaluate current state-of-the-art LLMs for FtF.

###  5.1 Methodology

The fallacy logic identification task comprises two sub-tasks: (i) template
selection and (ii) slot-filling. As shown in Table 3, the prompt includes this
fallacy-type information, allowing LLM to focus on two key actions. In
template selection, the model chooses the template that best reflects the
fallacious structure. For slot-filling, the model fills in the slots of the
selected template.

It is commonly known that dataset creation in argumentation requires
significant resources (human, time, financial), making it difficult to acquire
highly reliable large-scale annotations. Therefore, we employ LLMs with in-
context learning and fine-tuning to model both sub-tasks jointly. We
experiment with three distinct prompts: (i) NL1, a pure natural language
prompt, (ii) NL2, simplified version of NL1, and (iii) PL, a semi-structured
prompt with propositional logic and mathematical notation. Table 3 summarizes
a general form of these prompts; see Appendix A.6 for an example of the 1-shot
prompt for False Dilemma.444Detailed prompts used in our experiments are
publicly available at
https://github.com/itsanonnymous/fallacytemplate/tree/main/ftf_prompts

# Task
---
Identify the underlying structure of an argument of {fallacy_type}.
Given a list of fallacy templates, your task is to choose a template that best
describes the underlying fallacy structure…
# List of Templates
Template No.1:\n {template_1}
…
Template No.5:\nThere is either no consequence in the argument.
# Output Format
Template No.=[No.]\n{slot_fillers}
# Example
{examples}\n#
# Query
{}
Table 3: Generalized prompt used for our 0, 1, and 5-shot LLM experiments.
{fallacy_type} is either Fallacy of Credibility, False Causality, Faulty
Generalization, or False Dilemma. Depending on the fallacy type, the
appropriate templates and slot-filler choices are provided to the prompt, and
for 1 and 5-shot settings, {examples} are provided. For spacing purposes, we
replace newlines with \n in this prompt and omit templates 2-4.

###  5.2 Setup

#### Models

We employed four state-of-the-art LLMs: GPT-3.5-turbo Abdullah et al. (2022),
GPT-4o Achiam et al. (2023), Llama-3-8BMeta (2024), and Mistral-7BJiang et al.
(2023). We use a temperature of 0, max tokens of 0.6, top_p of 1.0, and both
frequency and presence penalties of 0. Experiments were conducted using zero-
shot, one-shot, and five-shot prompt settings for GPT-3.5-turbo and GPT-4o.
Few-shot examples were sampled from FtF-TRAIN, with the number of shots
reflecting the number of examples provided in the prompt.

For the fine-tuned model, we split Ftf-TRAIN into 150 instances as training
data and 50 instances as validation data. We set the learning rate into 2e-4
and optimizer adamw8bit. All models used Ftf-DEV for testing and evaluating
the results.

#### Evaluation Metrics

We use accuracy for the template section. For the slot-filling, we will target
only instances where the template is correctly identified by the model.
Formally, we define exact-match slot-filling accuracy as follows:
$\frac{|X\cap Y|}{|X|}$, where $X$ is a set of test instances where the
predicted template is correct, and $Y$ is a set of test instances where _all_
predicted slot-fillers must exactly match the gold-standard slot-fillers.555We
lowercased all tokens for word matching. In addition, we use partial-match
slot-filling accuracy, where $Y$ is a set of test instances where _all_
predicted slot-fillers are required to have over 50% word overlap with the
gold standard.

For evaluating overall performance, we define a _joint accuracy_ to be a
multiplication of template selection accuracy and slot-filling accuracy.

###  5.3 Results and Analysis

Tables 4 demonstrate low accuracies across all models. We choose NL2 prompt
for comparing the result with fine-tuned models based on the highest accuracy
in the template selection between GPT4 and GPT3.5 (appendix A.3). Regarding
template selection, the Mistral-7B model generally outperforms every model.
Conversely, in slot-filling, the results show that the GPT4 models with 5-shot
prompting outperform every model. Overall, the low joint accuracy highlights a
significant limitation of state-of-the-art language models in identifying the
logical fallacy structure that best captures the underlying fallacious
structure within FtF. Improving LLMs’ ability to handle slot-filling tasks
remains a significant challenge.

###  5.4 Error Analysis

We conducted an error analysis on 40 instances, aiming to improve the template
and prompt. We focused on the Mistral-7b’s generated results due to their
highest joint accuracy. We discovered the following errors: (i)The model
predicted template 5 despite the argument being able to be instantiated
(32.5%), (ii)The model predicted a different template due to different slot-
fillers (32.5%), (iii)The model predicted a different template despite having
similar slot-fillers as the gold label (17.5%), and (iv)The model instantiated
the template despite no argument from consequence (17.5%).

We found that template 5 was sometimes predicted due to noise in the input
argument. Among all the instances that fell into category (i), four instances
predicted template 5 because of this noise.666See Example 1 in Appendix Table
8 for an example of input question noise leading to template 5

Although the prompts were built off our guidelines, we found that the model
occasionally selected different templates due to many possible terms for slot-
filling. Example 2 in table 8, shows that the model selects a different
template due to the difference in slot-filler $A$. Upon further analysis, the
model’s predicted answer was also correct, as “ban hairspray” suppress “the
world will end” possesses the same semantic meaning as “hairspray” promotes
“the world will end”, but the selected template and slot-filler were both
incorrect.

It still remains a question why such an error that falls into categories (iii)
and (iv) occurred. However, we believe that the performance drop in the model
was attributed to three main factors: noise in the dataset, the presence of
multiple templates that could be selected, and the existence of various
possible terms that could fill the slot-filler. Further details can be seen in
section A.5.

Model | Acc. (TS) | Acc. (SF) | Acc. (Joint)
---|---|---|---
GPT4NL2-0 | 0.36 | 0.06 | 0.02
GPT4NL2-1 | 0.42 | 0.10 | 0.04
GPT4NL2-5 | 0.38 | 0.24 | 0.09
GPT3.5NL2-0 | 0.21 | 0.06 | 0.01
GPT3.5NL2-1 | 0.30 | 0.14 | 0.04
GPT3.5NL2-5 | 0.35 | 0.17 | 0.06
Llama3-7b | 0.34 | 0.16 | 0.05
Mistral-7b | 0.47 | 0.23 | 0.11
Table 4: Model accuracy for template selection (TS) and exact-match accuracy
for slot-filling (SF).

##  6 Conclusion and Future Work

In this work, we conduct the first study to address logical fallacy structure
by creating an inventory of fallacy templates. In total, we created 20 novel
templates spanned across 4 fallacy types (Fallacy of Credibility, False
Causality, False Dilemma, and Faulty Generalization). We created and released
Flee the Flaw, a new dataset consisting of 400 arguments from LOGIC Jin et al.
(2022) annotated with fallacy logic structure and publicly released both the
corpus and guidelines. Our dataset achieved a high inter-annotator agreement
(Krippendorf’s $\alpha$ of 0.54) and coverage (0.83%). We experiment on top of
our new dataset by conducting In-Context Learning and fine-tuning for fallacy
logic structure identification and discover that it is still a significant
challenge for state-of-the-art language models.

Our next step involves studying the underlying patterns and reasoning errors
in arguments by analyzing the logical structure of fallacies. Simultaneously,
we plan to conduct large-scale annotation on top of lengthier, more natural
arguments. Finally, we plan to explore non-consequential topics, allowing for
more Argumentation Schemes to be considered.

## Limitations

In this research, we mainly focus on the proposed explainable fallacy template
for only 4 fallacy types which are all mainly informal fallacies. We do not
address the fallacy of logic which is the extension from the informal fallacy
to formal fallacy. To keep annotation simple, our fallacy templates do not
cover every possible combination of ingredients (e.g. relations such as NOT
PROMOTE, NOT SUPPRESS) which limits the amount of total instantiations we can
acquire. Regardless, we still achieved a coverage score of roughly 80%.
Furthermore, we extend on argument templates Reisert et al. (2018) which were
inspired by Walton (2008)’s Argument from Consequence scheme which is a common
scheme for every day arguments, but may limit the full range of fallacy
instantiations that we can produce.

We limit ourselves to four types of fallacies which only represents a small
subset of all known fallacies. Primarily, we target common informal logical
fallacies as a start for fallacious template structure instantiation. Given
the structure of False Dilemma fallacy, which follows an either-or structure,
we obtain an unbalanced partition for our False Dilemma templates. As shown in
Fig. 4, both annotators mainly annotated with template 2.

## Ethical Considerations

Each author of this paper ensured that all ethical considerations were upheld.
All results are reported as accurately as possible. Given that we conducted an
annotation, we adhere to constructing a high quality dataset as exemplified by
our annotator agreement results.

## References

  * Abdullah et al. (2022) Malak Abdullah, Alia Madain, and Yaser Jararweh. 2022.  Chatgpt: Fundamentals, applications and social impacts.  In _2022 Ninth International Conference on Social Networks Analysis, Management and Security (SNAMS)_ , pages 1–8. IEEE.
  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Alhindi et al. (2023) Tariq Alhindi, Tuhin Chakrabarty, Elena Musi, and Smaranda Muresan. 2023.  Multitask instruction-based prompting for fallacy recognition.  _arXiv preprint arXiv:2301.09992_.
  * Bennett (2012) B. Bennett. 2012.  _Logically Fallacious: The Ultimate Collection of Over 300 Logical Fallacies_.  Ebookit.com.
  * Bonial et al. (2022) Claire Bonial, Austin Blodgett, Taylor Hudson, Stephanie Lukin, Jeffrey Micher, Douglas Summers-Stay, Peter Sutor, and Clare Voss. 2022.  The search for agreement on logical fallacy annotation of an infodemic.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 4430–4438.
  * Carlile et al. (2018) Winston Carlile, Nishant Gurrapadi, Zixuan Ke, and Vincent Ng. 2018.  Give me more feedback: Annotating argument persuasiveness and related attributes in student essays.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 621–631, Melbourne, Australia. Association for Computational Linguistics.
  * Goffredo et al. (2023) Pierpaolo Goffredo, Mariana Espinoza, Serena Villata, and Elena Cabrio. 2023.  Argument-based detection and classification of fallacies in political debates.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 11101–11112. Association for Computational Linguistics.
  * Gwet (2008) Kilem Li Gwet. 2008.  Computing inter-rater reliability and its variance in the presence of high agreement.  _British Journal of Mathematical and Statistical Psychology_ , 61(1):29–48.
  * Habernal et al. (2017) Ivan Habernal, Raffael Hannemann, Christian Pollak, Christopher Klamm, Patrick Pauli, and Iryna Gurevych. 2017.  Argotario: Computational argumentation meets serious games.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 7–12, Copenhagen, Denmark. Association for Computational Linguistics.
  * Hashimoto et al. (2012) Chikara Hashimoto, Kentaro Torisawa, Stijn De Saeger, Jong-Hoon Oh, et al. 2012.  Excitatory or inhibitory: A new semantic orientation extracts contradiction and causality from the web.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_ , pages 619–630.
  * Hayes and Krippendorff (2007) Andrew F Hayes and Klaus Krippendorff. 2007.  Answering the call for a standard reliability measure for coding data.  _Communication methods and measures_ , 1(1):77–89.
  * Helwe et al. (2023) Chadi Helwe, Tom Calamai, Pierre-Henri Paris, Chloé Clavel, and Fabian Suchanek. 2023.  Mafalda: A benchmark and comprehensive study of fallacy detection and classification.  _arXiv preprint arXiv:2311.09761_.
  * Hinton (2020) Martin Hinton. 2020.  _Evaluating the Language of Argument_ , 1 edition, volume 37.  Springer Cham.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Jin et al. (2022) Zhijing Jin, Abhinav Lalwani, Tejas Vaidhya, Xiaoyu Shen, Yiwen Ding, Zhiheng Lyu, Mrinmaya Sachan, Rada Mihalcea, and Bernhard Schoelkopf. 2022.  Logical fallacy detection.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 7180–7198, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Ke et al. (2019) Zixuan Ke, Hrishikesh Inamdar, Hui Lin, and Vincent Ng. 2019.  Give me more feedback II: Annotating thesis strength and related attributes in student essays.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3994–4004, Florence, Italy. Association for Computational Linguistics.
  * Ke and Ng (2019) Zixuan Ke and Vincent Ng. 2019.  Automated essay scoring: A survey of the state of the art.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_ , pages 6300–6308. International Joint Conferences on Artificial Intelligence Organization.
  * Meta (2024) Meta. 2024.  Introducing meta llama 3: The most capable openly available llm to date.
  * Reisert et al. (2018) Paul Reisert, Naoya Inoue, Tatsuki Kuribayashi, and Kentaro Inui. 2018.  Feasible annotation scheme for capturing policy argument reasoning using argument templates.  In _Proceedings of the 5th Workshop on Argument Mining_ , pages 79–89.
  * Ruiz-Dolz and Lawrence (2023) Ramon Ruiz-Dolz and John Lawrence. 2023.  Detecting argumentative fallacies in the wild: Problems and limitations of large language models.  In _Proceedings of the 10th Workshop on Argument Mining_. Association for Computational Linguistics.
  * Sourati et al. (2023a) Zhivar Sourati, Filip Ilievski, Hông-Ân Sandlin, and Alain Mermoud. 2023a.  Case-Based Reasoning with Language Models for Classification of Logical Fallacies.
  * Sourati et al. (2023b) Zhivar Sourati, Vishnu Priya Prasanna Venkatesh, Darshan Deshpande, Himanshu Rawlani, Filip Ilievski, Hông-Ân Sandlin, and Alain Mermoud. 2023b.  Robust and explainable identification of logical fallacies in natural language arguments.  _Knowledge-Based Systems_ , 266:110418.
  * Stab and Gurevych (2017) Christian Stab and Iryna Gurevych. 2017.  Parsing argumentation structures in persuasive essays.  _Computational Linguistics_ , 43(3):619–659.
  * Toulmin (2003) Stephen E Toulmin. 2003.  _The uses of argument_.  Cambridge university press.
  * Wachsmuth et al. (2017) Henning Wachsmuth, Nona Naderi, Yufang Hou, Yonatan Bilu, Vinodkumar Prabhakaran, Tim Alberdingk Thijm, Graeme Hirst, and Benno Stein. 2017.  Computational argumentation quality assessment in natural language.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 176–187, Valencia, Spain. Association for Computational Linguistics.
  * Walton (1987) D.N. Walton. 1987.  _Informal Fallacies: Towards a Theory of Argument Criticisms_.  Companion series. J. Benjamins Publishing Company.
  * Walton (2008) Douglas Walton. 2008.  _Informal logic: A pragmatic approach_.  Cambridge University Press.
  * Walton (2013) Douglas Walton. 2013.  _Argumentation schemes for presumptive reasoning_.  Routledge.
  * Walton et al. (2008) Douglas Walton, Christopher Reed, and Fabrizio Macagno. 2008.  _Argumentation schemes_.  Cambridge University Press.

##  Appendix A Appendix

###  A.1 Template Examples

Figure 5: Examples of template and slot-fillers from FtF for Fallacy of
Credibility.

Shown in Fig. 5 is the example of Fallacy of Credibility. For the Fallacy of
Credibility argument, the fallacy is committed as the $X$=“best friend” is
promoting that $A$=“pizza” has $C$=“health benefits”, resulting in
$P^{\prime}$: $X$=“best friend” promote that $A$=“pizza” promote $C$=“health
benefits”, thus Conclusion is $A$=“pizza” should be brought about. However,
the friend is not an expert in the field of nutrition.

Figure 6: Examples of template and slot-fillers from FtF for False Causality.

For the False Causality argument shown in Fig. 6, the argument is stating that
$A$=“eat yoghurt” has a correlation with people with healthy guts, and thus
the $P$: $A$=“eat yoghurt” suppressing $C$=“sick”. The False Causality is
linked, as it’s implying that $A$=“eating yoghurt” will definitely suppress
$C$=“sick”. In conclusion, $A$=“eating yoghurt” should be brought about.

Figure 7: Examples of template and slot-fillers from FtF for False Dilemma.

The example of argument shown in Fig. 7 is considered as False Dilemma
fallacy. The argument limited the option to $A$=“cut taxes” and negation of
$A$=“cut taxes” for determine the consequence of $C$=“leave a huge debt for
our children”. It conclude $A$=“cut taxes” should be brought about without
considering any possible action except $P$: $A$=“cut taxes” suppress
$C$=“leave a huge debt for our children”, and $P^{\prime}$: negation of
$A$=“cut taxes” promote $C$=“leave a huge debt for our children”.

###  A.2 Fallacy Types

False Dilemma occurs due to the restriction of the choices and ignoring
additional potential options. Faulty Generalization occurs when a belief is
applied to a large population without a sufficient and unbiased sample. False
Causality assumes a cause-and-effect relationship between two events. Finally,
Fallacy of Credibility involves an appeal to ethics, authority, or credibility
that is not directly relevant to the argument. Table 5 provides a definition,
example, and further explanation of the example for False Dilemma, Faulty
Generalization, False Causality, and Fallacy of Credibility.

Fallacy Type |  Definition |  Example |  Explanation
---|---|---|---
False Dilemma |  This fallacy is when incorrect limitations are made on the possible options in a scenario when there could be other options. |  We either have to cut taxes or leave a huge debt for our children |  This argument only limits the options into “cut taxes” or “not cut taxes” for dealing with a “debt” without considering other potential options.
Faulty Generalization |  This fallacy occurs when an argument applies a belief to a large population without having a large enough sample to do so. |  I took an NLP class, an advanced course in Stanford. I suggest not taking further advanced courses because they will hurt your GPA. |  This argument generalizes “further advanced courses” should not be taken due to hurting the person’s "GPA" only because the took one of the advanced courses “NLP class.”
False Causality |  This fallacy occurs when an argument assumes that since two events are correlated,they must also have a cause and effect relationship. |  People who eat yoghurt have healthy guts. If I eat yoghurt I will never get sick |  This argument has a belief that “eat yoghurt” has a strong relation with “never get sick” because of having “healthy guts”. Thus, it believes that by eating yoghurt will “never get sick.”
Fallacy of Credibility |  This fallacy is when an appeal is made to some form of ethics, authority, or credibility. |  My Best friend tweeted about the health benefits of pizza, and so we’re going to out to eat two vegetable pizzas |  The argument has been promoted by the person’s best friend by “tweet about the health benefits of pizza”, but the person best friend is not an expert in the field of nutrition which makes the argument is not credible.
Table 5: Definition and example explanation of four defective induction
fallacy types

###  A.3 Prompt Type for Template Selection and Exact Match Performance

We report the result of template selection accuracy and the average accuracy
of slot-filling for exact match for every three prompt types using GPT-4 in
table 6 and GPT3.5 in table 7.

Template selection performs better for 1-shot prompting for every prompt type
in the GPT4 model. However, not for the slot-filling task, 5-shot prompting
outperforms 1-shot prompting for every prompt type despite not having the
highest accuracy in the template selection task. Different from GPT3.5 where
every task is dominated by 5-shot prompting for every prompt type.

Overall, model performance shows minimal variation based on prompt type,
suggesting that prompt variation has no significant impact on performance.

Pr | $n$ | Acc. (TS) | Acc. (SF) | Acc. (Joint)
---|---|---|---|---
NL1 | 0 | 0.31 | 0.10 | 0.03
NL1 | 1 | 0.36 | 0.12 | 0.04
NL1 | 5 | 0.32 | 0.22 | 0.07
NL2 | 0 | 0.36 | 0.06 | 0.02
NL2 | 1 | 0.42 | 0.10 | 0.04
NL2 | 5 | 0.38 | 0.24 | 0.09
PL | 0 | 0.32 | 0.10 | 0.03
PL | 1 | 0.38 | 0.10 | 0.04
PL | 5 | 0.31 | 0.18 | 0.06
Table 6: GPT-4 accuracy for template selection (TS) and exact-match accuracy for slot-filling (SF). $n$ denotes the number of few-shot examples, and Pr denotes a prompt type.  Pr | $n$ | Acc. (TS) | Acc. (SF) | Acc. (Joint)
---|---|---|---|---
NL1 | 0 | 0.21 | 0.12 | 0.02
NL1 | 1 | 0.31 | 0.14 | 0.04
NL1 | 5 | 0.37 | 0.19 | 0.07
NL2 | 0 | 0.21 | 0.06 | 0.01
NL2 | 1 | 0.30 | 0.14 | 0.04
NL2 | 5 | 0.35 | 0.17 | 0.06
PL | 0 | 0.21 | 0.13 | 0.03
PL | 1 | 0.29 | 0.04 | 0.01
PL | 5 | 0.37 | 0.17 | 0.06
Table 7: GPT-3.5 accuracy for template selection (TS) and exact-match accuracy for slot-filling (SF). |  Fallacy Type |  Example |  Correct Answer |  Predicted Answer
---|---|---|---|---
1 |  False Dilemma |  “America: Love it or leave it. This is an example of which kind of logical fallacy?” |  Template No.=2 $[A]$=Love it $[C]$=leave it |  Template No.=5 $[A]$= $[C]$=
2 |  False Dilemma |  We either ban hairspray or the world will end. |  Template No.=4 $[A]$=hairspray $[C]$=the world will end |  Template No.=2 $[A]$=ban hairspray $[C]$=the world will end
3 |  False Causality |  Iâ€™ve never had the flu because I take my vitamins everyday. |  Template No.=4 $[A]$=vitamins $[C]$=flu |  Template No.=3 $[A]$=vitamins $[C]$=flu
4 |  Faulty Generalization |  This new test seemed so promising, but the 3 studies that supported its validity turned out to have critical methodological flaws, so the test is probably not valid. |  Template No.=2 $[A]$=test $[C]$=critical methodolical flaws $[A^{\prime}]$=3 studies that supported its validity turned out to have critical methodological flaws $[C^{\prime}]$= |  Template No.=5 $[A]$= $[C]$= $[A^{\prime}]$= $[C^{\prime}]$=
5 |  Fallacy of Credibility |  Albert Einstein was extremely impressed with this theory. |  Template No.=5 $[A]$= $[C]$= $[X]$= |  Template No.=2 $[A]$=this theory $[C]$=Albert Einstein $[X]$=extremely impressed
Table 8: False prediction generated by Mistral-7B model

###  A.4 Prompt Type for Template Selection and Partial Match Performance

We report the average accuracy of slot-filling for partial match. The results
are shown in table 9 for GPT-4 and table 10 GPT3.5. Despite NL2 zero-shot
prompt on GPT4 model performance of only 0.06 accuracy for an exact match
slot-filling task in table 6, it performs the best with 0.49 accuracy in the
partial match slot-filling task.

Pr | $n$ | Acc. (TS) | Acc. (SF) | Acc. (Joint)
---|---|---|---|---
NL1 | 0 | 0.31 | 0.24 | 0.07
NL1 | 1 | 0.36 | 0.43 | 0.16
NL1 | 5 | 0.32 | 0.32 | 0.10
NL2 | 0 | 0.36 | 0.49 | 0.17
NL2 | 1 | 0.42 | 0.35 | 0.15
NL2 | 5 | 0.38 | 0.42 | 0.16
PL | 0 | 0.32 | 0.32 | 0.10
PL | 1 | 0.38 | 0.21 | 0.08
PL | 5 | 0.31 | 0.33 | 0.10
Table 9: GPT-4 accuracy for template selection (TS) and partial-match accuracy for slot-filling Pr | $n$ | Acc. (TS) | Acc. (SF) | Acc. (Joint)
---|---|---|---|---
NL1 | 0 | 0.21 | 0.12 | 0.02
NL1 | 1 | 0.31 | 0.33 | 0.10
NL1 | 5 | 0.37 | 0.37 | 0.14
NL2 | 0 | 0.21 | 0.19 | 0.04
NL2 | 1 | 0.29 | 0.36 | 0.11
NL2 | 5 | 0.37 | 0.36 | 0.12
PL | 0 | 0.21 | 0.20 | 0.04
PL | 1 | 0.30 | 0.43 | 0.12
PL | 5 | 0.35 | 0.38 | 0.14
Table 10: GPT-3.5 accuracy for template selection (TS) and partial-match
accuracy for slot-filling

###  A.5 False Template Prediction

Table 8 provides false prediction results using the Mistral-7B model. As
previously mentioned in the section 5.4, we categorize into 4 types of error.
We found that one of the reasons is the noise that causes an error in category
(i), but such an error also occurred even for non-noisy input like in example
4.

In example 3, the model correctly predicts the slot-filler but chooses
different templates. The template conclusion “Vitamins should be brought
about” is correct, but the model incorrectly assigns a good sentiment to “flu”
and creates the premise “Vitamins promote flu” which does not align with the
argument’s intention. It remains unclear whether the model struggles to define
sentiment.

In example 5, we believe the argument cannot be instantiated as it is not an
argument from consequences. However, the model instantiates this argument into
template 2 with the premise “This theory suppresses Albert Einstein” and got
promoted by “extremely impressed” which completely different meaning from the
input. This raises the question of whether the model truly understands
arguments from consequence and the template structure.

###  A.6 Prompt for LLM Experiments

Table 11, Table 12, Table 13 provides an example of the 5-shot prompt for
False Dilemma used during our LLM experiments. Instances used for non-zero-
shot settings are randomly selected from FtF-TRAIN200.

###  A.7 Towards Dataset Expansion

Towards extending FtF with other datasets, using our templates and guidelines,
we conduct a preliminary annotation on top of three existing, pre-labeled
fallacy datasets: LOGICCLIMATE Jin et al. (2022), and Argotario Habernal et
al. (2017), and a Covid dataset Bonial et al. (2022). We randomly sample 40
arguments, where 20 are labeled as “Hasty Generalization” and 20 are labeled
as “Irrelevant Authority”.We obtain a total coverage of 0.50, with Argotario
achieving the highest coverage (0.60) and the Covid dataset achieving the
lowest (0.30). Reasons our templates could not be instantiated included
instances that require evidence to function as the target fallacy type (e.g.,
“Covid vaccines contain aborted babies.” as “Hasty Generalization”), not the
target type fallacy (e.g., “No, because if you start with same sex marriage,
what is next? Marriage with animals?”, a Slippery Slope), of no credible
source information provided (e.g., ““The COVID-19 pandemic is not a real
medical pandemic”; “The COVID-19 vaccine is not proven safe or effective””).

# Task
---
Identify the underlying structure of an argument of False Dilemma.
Given a list of fallacy templates, your task is to choose a template that best
describes the underlying fallacy structure, choosing the template’s
placeholders, $[A]$ and $[C]$, directly from the input text. Additionally, the
text must be a consecutive sequence of one or more terms without any
conjugation.
Please follow the output format.
# Definitions
Entity: a noun phrase in the input.
Event: a verb phrase in the input.
Placeholder: A fill-in-the-blank choice within a template. Each placeholder
may either be an entity or an event.
Please note! Placeholders can ONLY be either an entity (i.e., noun phrase) or
an event (i.e., verb phrase) and may not be any other type of phrase (e.g.,
prepositional phrase).
# List of Templates
Template No.1:
Premise 1: An entity/action [A] promotes a good entity/action [C].
Premise 2: The absence of an entity/action [A] suppresses a good entity/action
[C].
Conclusion: Therefore, both Premise 1 and Premise 2 support that [A] should be
brought about.
Template No.2:
Premise 1: An entity/action [A] suppresses a bad entity/action [C]
Premise 2: The absence of an entity/action [A] promotes a bad entity/action
[C].
Conclusion: Therefore, both Premise 1 and Premise 2 support that [A] should be
brought about.
Template No.3:
Premise 1: An entity/action [A] suppresses a good entity/action [C]
Premise 2: The absence of an entity/action [A] promotes a good entity/action
[C].
Conclusion: Therefore, both Premise 1 and Premise 2 support that [A] should
not be brought about.
Template No.4:
Premise 1: An entity/action [A] promotes a bad entity/action [C].
Premise 2: The absence of an entity/action [A] suppresses a bad entity/action
[C].
Conclusion: Therefore, both Premise 1 and Premise 2 support that [A] should
not be brought about.
Template No.5:
There is either no consequence in the argument, or the argument cannot be
instantiated with one of the templates above.
# Output Format
Template No.=[No.]
$[A]$=
$[C]$=
# Important Criteria: Prioritizing entities over events for placeholder.
For choosing placeholder, please prioritize entities over events in the case
that an entity itself captures the underlying intent of the argument opposed
to the event. However, if the event makes more sense, please choose an event
for the placeholder.
# Correct Example
Input: To get better schools, we have to raise taxes. If we don’t, we can’t
have better schools.
Output:
Template No.=1
$[A]$=raise taxes
$[C]$=schools
Explanation:
Here, there are 2 possible options for [C] which are "schools" (i.e., entity)
and "can’t have better schools" (i.e., event). Since the entity is the top
priority and the second option does not work with template 1 because it is a
suppressed relation, "schools" is cchosen for [C].
Also, [A] and [C] are taken directly from the input text. For example,
"raising taxes" as [A] also sounds correct, but the term "raising" is not
mentioned in the input text. That is why "raise taxes" is chosen for [A].
Because the argument believes that "raise taxes" promote "schools" while not
"raise taxes" suppress "school". he conclusion is implicit that “Premise 1
supports that raise taxes should be brought about.” Thus, Template No.=1 is
selected.
# Wrong Example
Input: To get better schools, we have to raise taxes. If we don’t, we can’t
have better schools.
Output:
Template No.=1
$[A]$=raising taxes
$[C]$=can’t have better schools
Explanation:
Here, there are 2 possible options for [C] which are "schools" (i.e., entity)
and "can’t have better schools" (i.e., event). However, "can’t have better
schools" as [C] is incorrect because it is an event instead of the entity of
"schools" which already makes sense.
Also, "raising taxes" as [A] is incorrect because the placeholder is not taken
directly from the text. Here "raising taxes" is chosen as [A] but the word
"raising" does not appear in the input text. Therefore the correct choice for
[A] is "raise taxes".
# Example1
If you can’t prove that Ken had an affair with the nanny, then he’s been
faithful to his wife.
Template No.=1
$[A]$=prove that Ken had an affair with the nanny
$[C]$=he’s been faithful to his wife
Again, please only select the placeholders directly from the text!
# Query
{}
Table 11: Natural Language (NL1): 1-shot False Dilemma prompt for LLM
experiment # Task
---
Identify the underlying structure of an argument of False Dilemma.
Given a list of fallacy templates, your task is to choose a template that best
describes the underlying fallacy structure, choosing the template’s
placeholders, $[A]$ and $[C]$, directly from the input text. Additionally, the
text must be a consecutive sequence of one or more terms without any
conjugation.
Please follow the output format.
# Definitions
Entity: a noun phrase in the input.
Event: a verb phrase in the input.
Placeholder: A fill-in-the-blank choice within a template. Each placeholder
may either be an entity or an event.
Please note! Placeholders can ONLY be either an entity (i.e., noun phrase) or
an event (i.e., verb phrase) and may not be any other type of phrase (e.g.,
prepositional phrase).
# List of Templates
Template No.1:
Premise 1: An entity/event [A] promotes a good entity/event [C].
Premise 2: An entity/event [¬A] suppresses a good entity/event [C].
Conclusion: Therefore, both Premise 1 and Premise 2 support that [A] should be
brought about.
Template No.2:
Premise 1: An entity/event [A] suppresses a bad entity/event [C]
Premise 2: An entity/event [¬A] promotes a bad entity/event [C].
Conclusion: Therefore, both Premise 1 and Premise 2 support that [A] should be
brought about.
Template No.3:
Premise 1: An entity/event [A] suppresses a good entity/event [C]
Premise 2: An entity/event [¬A] promotes a good entity/event [C].
Conclusion: Therefore, both Premise 1 and Premise 2 support that [A] should
not be brought about.
Template No.4:
Premise 1: An entity/event [A] promotes a bad entity/event [C]
Premise 2: An entity/event [¬A] suppresses a bad entity/event [C].
Conclusion: Therefore, both Premise 1 and Premise 2 support that [A] should
not be brought about.
Template No.5:
There is either no consequence in the argument, or the argument cannot be
instantiated with one of the templates above.
# Output Format
Template No.=[No.]
$[A]$=
$[C]$=
# Important Criteria: Prioritizing entities over events for placeholder.
For choosing placeholder, please prioritize entities over events in the case
that an entity itself captures the underlying intent of the argument opposed
to the event. However, if the event makes more sense, please choose an event
for the placeholder.
# Correct Example
Input: To get better schools, we have to raise taxes. If we don’t, we can’t
have better schools.
Output:
Template No.=1
$[A]$=raise taxes
$[C]$=schools
Explanation:
Here, there are 2 possible options for [C] which are "schools" (i.e., entity)
and "can’t have better schools" (i.e., event). Since the entity is the top
priority and the second option does not work with template 1 because it is a
suppressed relation, "schools" is cchosen for [C].
Also, [A] and [C] are taken directly from the input text. For example,
"raising taxes" as [A] also sounds correct, but the term "raising" is not
mentioned in the input text. That is why "raise taxes" is chosen for [A].
Because the argument believes that "raise taxes" promote "schools" while not
"raise taxes" suppress "school". he conclusion is implicit that “Premise 1
supports that raise taxes should be brought about.” Thus, Template No.=1 is
selected.
# Wrong Example
Input: To get better schools, we have to raise taxes. If we don’t, we can’t
have better schools.
Output:
Template No.=1
$[A]$=raising taxes
$[C]$=can’t have better schools
Explanation:
Here, there are 2 possible options for [C] which are "schools" (i.e., entity)
and "can’t have better schools" (i.e., event). However, "can’t have better
schools" as [C] is incorrect because it is an event instead of the entity of
"schools" which already makes sense.
Also, "raising taxes" as [A] is incorrect because the placeholder is not taken
directly from the text. Here "raising taxes" is chosen as [A] but the word
"raising" does not appear in the input text. Therefore the correct choice for
[A] is "raise taxes".
# Example1
If you can’t prove that Ken had an affair with the nanny, then he’s been
faithful to his wife.
Template No.=1
$[A]$=prove that Ken had an affair with the nanny
$[C]$=he’s been faithful to his wife
Again, please only select the placeholders directly from the text!
# Query
{}
Table 12: Propositional Logic (PL): 1-shot False Dilemma prompt for LLM
experiments. # Task
---
Identify the underlying structure of an argument of False Dilemma.
Given a list of fallacy templates, your task is to choose a template that best
describes the underlying fallacy structure, choosing the template’s
placeholders, $[A]$ and $[C]$, directly from the input text. Additionally, the
text must be a consecutive sequence of one or more terms without any
conjugation.
Please follow the output format.
# Definitions
Entity: a noun phrase in the input.
Event: a verb phrase in the input.
Placeholder: A fill-in-the-blank choice within a template. Each placeholder
may either be an entity or an event.
Please note! Placeholders can ONLY be either an entity (i.e., noun phrase) or
an event (i.e., verb phrase) and may not be any other type of phrase (e.g.,
prepositional phrase).
# List of Templates
Template No.1:
Premise 1: An entity/event [A] promotes a good entity/event [C].
Premise 2: The absence of an entity/event [A] suppresses a good entity/event
[C].
Conclusion: Therefore, [A] should be brought about.
Template No.2:
Premise 1: An entity/event [A] suppresses a bad entity/event [C]
Premise 2: The absence of an entity/event [A] promotes a bad entity/event [C].
Conclusion: Therefore, [A] should be brought about.
Template No.3:
Premise 1: An entity/event [A] suppresses a good entity/event [C]
Premise 2: The absence of an entity/event [A] promotes a good entity/event
[C].
Conclusion: Therefore, [A] should not be brought about.
Template No.4:
Premise 1: An entity/event [A] promotes a bad entity/event [C]
Premise 2: The absence of an entity/event [A] suppresses a bad entity/event
[C].
Conclusion: Therefore, [A] should not be brought about.
Template No.5:
There is either no consequence in the argument, or the argument cannot be
instantiated with one of the templates above.
# Output Format
Template No.=[No.]
$[A]$=
$[C]$=
# Important Criteria: Prioritizing entities over events for placeholder.
For choosing placeholder, please prioritize entities over events in the case
that an entity itself captures the underlying intent of the argument opposed
to the event. However, if the event makes more sense, please choose an event
for the placeholder.
# Correct Example
Input: To get better schools, we have to raise taxes. If we don’t, we can’t
have better schools.
Output:
Template No.=1
$[A]$=raise taxes
$[C]$=schools
Explanation:
Here, there are 2 possible options for [C] which are "schools" (i.e., entity)
and "can’t have better schools" (i.e., event). Since the entity is the top
priority and the second option does not work with template 1 because it is a
suppressed relation, "schools" is cchosen for [C].
Also, [A] and [C] are taken directly from the input text. For example,
"raising taxes" as [A] also sounds correct, but the term "raising" is not
mentioned in the input text. That is why "raise taxes" is chosen for [A].
Because the argument believes that "raise taxes" promote "schools" while not
"raise taxes" suppress "school". he conclusion is implicit that “Premise 1
supports that raise taxes should be brought about.” Thus, Template No.=1 is
selected.
# Wrong Example
Input: To get better schools, we have to raise taxes. If we don’t, we can’t
have better schools.
Output:
Template No.=1
$[A]$=raising taxes
$[C]$=can’t have better schools
Explanation:
Here, there are 2 possible options for [C] which are "schools" (i.e., entity)
and "can’t have better schools" (i.e., event). However, "can’t have better
schools" as [C] is incorrect because it is an event instead of the entity of
"schools" which already makes sense.
Also, "raising taxes" as [A] is incorrect because the placeholder is not taken
directly from the text. Here "raising taxes" is chosen as [A] but the word
"raising" does not appear in the input text. Therefore the correct choice for
[A] is "raise taxes".
#Example1
If you can’t prove that Ken had an affair with the nanny, then he’s been
faithful to his wife.
Template No.=1
$[A]$=prove that Ken had an affair with the nanny
$[C]$=he’s been faithful to his wife
Again, please only select the placeholders directly from the text!
# Query
{}
Table 13: Natural Language2 (NL2): 1-shot False Dilemma prompt for LLM
experiments.