* Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2023.  PaLM: Scaling Language Modeling with Pathways.  _J. Mach. Learn. Res._ 24 (2023), 240:1–240:113.  http://jmlr.org/papers/v24/22-1144.html
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021\.  Training Verifiers to Solve Math Word Problems.  _CoRR_ abs/2110.14168 (2021).  arXiv:2110.14168 https://arxiv.org/abs/2110.14168
  * Deng et al. (2022) Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu. 2022.  RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, 3369–3391.  https://doi.org/10.18653/V1/2022.EMNLP-MAIN.222
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 4171–4186.  https://doi.org/10.18653/V1/N19-1423
  * Dong et al. (2023) Yihong Dong, Jiazheng Ding, Xue Jiang, Zhuo Li, Ge Li, and Zhi Jin. 2023\.  CodeScore: Evaluating Code Generation by Learning Code Execution.  _CoRR_ abs/2301.09043 (2023).  https://doi.org/10.48550/ARXIV.2301.09043 arXiv:2301.09043
  * Dua et al. (2019) Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019\.  DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 2368–2378.  https://doi.org/10.18653/V1/N19-1246
  * Dunn (1961) Olive Jean Dunn. 1961\.  Multiple Comparisons Among Means.  _J. Amer. Statist. Assoc._ 56, 293 (1961), 52–64.  http://www.jstor.org/stable/2282330
  * Durrani et al. (2014) Nadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. 2014.  Edinburgh’s Phrase-based Machine Translation Systems for WMT-14. In _Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT@ACL 2014, June 26-27, 2014, Baltimore, Maryland, USA_. The Association for Computer Linguistics, 97–104.  https://doi.org/10.3115/V1/W14-3309
  * Elnaggar et al. (2021) Ahmed Elnaggar, Wei Ding, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Silvia Severini, Florian Matthes, and Burkhard Rost. 2021.  CodeTrans: Towards Cracking the Language of Silicone’s Code Through Self-Supervised Deep Learning and High Performance Computing.  _CoRR_ abs/2104.02443 (2021).  arXiv:2104.02443 https://arxiv.org/abs/2104.02443
  * Fan et al. (2023) Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. 2023.  Large Language Models for Software Engineering: Survey and Open Problems.  _CoRR_ abs/2310.03533 (2023).  https://doi.org/10.48550/ARXIV.2310.03533 arXiv:2310.03533
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020.  CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In _Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020_ _(Findings of ACL, Vol. EMNLP 2020)_ , Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 1536–1547.
  * Fried et al. (2023) Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2023\.  InCoder: A Generative Model for Code Infilling and Synthesis. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.  https://openreview.net/pdf?id=hQwb-lbM6EL
  * Fu et al. (2023) Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023.  Complexity-Based Prompting for Multi-step Reasoning. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.  https://openreview.net/pdf?id=yf1icZHC-l9
  * Ganguli et al. (2023) Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamile Lukosiute, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, Dawn Drain, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jared Mueller, Joshua Landau, Kamal Ndousse, Karina Nguyen, Liane Lovitt, Michael Sellitto, Nelson Elhage, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robert Lasenby, Robin Larson, Sam Ringer, Sandipan Kundu, Saurav Kadavath, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Christopher Olah, Jack Clark, Samuel R. Bowman, and Jared Kaplan. 2023.  The Capacity for Moral Self-Correction in Large Language Models.  _CoRR_ abs/2302.07459 (2023).  https://doi.org/10.48550/ARXIV.2302.07459 arXiv:2302.07459
  * Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021\.  Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies.  _Trans. Assoc. Comput. Linguistics_ 9 (2021), 346–361.  https://doi.org/10.1162/TACL_A_00370
  * Guo et al. (2021) Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021\.  GraphCodeBERT: Pre-training Code Representations with Data Flow. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.  https://openreview.net/forum?id=jLoC4ez43PZ
  * Han et al. (2023) Chengcheng Han, Liqing Cui, Renyu Zhu, Jianing Wang, Nuo Chen, Qiushi Sun, Xiang Li, and Ming Gao. 2023\.  When Gradient Descent Meets Derivative-Free Optimization: A Match Made in Black-Box Scenario. In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_ , Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 868–880.  https://doi.org/10.18653/V1/2023.FINDINGS-ACL.55
  * Harzing (2016) Anne-Wil Harzing. 2016\.  Publish or Perish.  https://harzing.com/resources/publish-or-perish
  * Hazhirpasand et al. (2019) Mohammadreza Hazhirpasand, Mohammad Ghafari, Stefan Krüger, Eric Bodden, and Oscar Nierstrasz. 2019\.  The Impact of Developer Experience in Using Java Cryptography. In _2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2019, Porto de Galinhas, Recife, Brazil, September 19-20, 2019_. IEEE, 1–6.  https://doi.org/10.1109/ESEM.2019.8870184
  * He and Vechev (2023) Jingxuan He and Martin T. Vechev. 2023.  Large Language Models for Code: Security Hardening and Adversarial Testing. In _Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS 2023, Copenhagen, Denmark, November 26-30, 2023_ , Weizhi Meng, Christian Damsgaard Jensen, Cas Cremers, and Engin Kirda (Eds.). ACM, 1865–1879.  https://doi.org/10.1145/3576915.3623175
  * Hendrycks et al. (2021a) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021a.  Measuring Coding Challenge Competence With APPS. In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_ , Joaquin Vanschoren and Sai-Kit Yeung (Eds.).  https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/c24cd76e1ce41366a4bbe8a49b02a028-Abstract-round2.html
  * Hendrycks et al. (2021b) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b.  Measuring Mathematical Problem Solving With the MATH Dataset. In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_ , Joaquin Vanschoren and Sai-Kit Yeung (Eds.).  https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/be83ab3ecd0db773eb2dc1b0a17836a1-Abstract-round2.html
  * Hosseini et al. (2014) Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014.  Learning to Solve Arithmetic Word Problems with Verb Categorization. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL_ , Alessandro Moschitti, Bo Pang, and Walter Daelemans (Eds.). ACL, 523–533.  https://doi.org/10.3115/V1/D14-1058
  * Huang and Chang (2023) Jie Huang and Kevin Chen-Chuan Chang. 2023.  Towards Reasoning in Large Language Models: A Survey. In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_ , Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 1049–1065.  https://doi.org/10.18653/V1/2023.FINDINGS-ACL.67
  * Ichter et al. (2022) Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang-Huei Lee, Yuheng Kuang, Sally Jesmonth, Nikhil J. Joshi, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu. 2022.  Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. In _Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand_ _(Proceedings of Machine Learning Research, Vol. 205)_ , Karen Liu, Dana Kulic, and Jeffrey Ichnowski (Eds.). PMLR, 287–318.  https://proceedings.mlr.press/v205/ichter23a.html
  * Jain et al. (2022) Naman Jain, Skanda Vaidyanath, Arun Iyer, Nagarajan Natarajan, Suresh Parthasarathy, Sriram Rajamani, and Rahul Sharma. 2022\.  Jigsaw: Large language models meet program synthesis. In _Proceedings of the 44th International Conference on Software Engineering (ICSE)_. 1219–1231.
  * Jain et al. (2021) Paras Jain, Ajay Jain, Tianjun Zhang, Pieter Abbeel, Joseph Gonzalez, and Ion Stoica. 2021\.  Contrastive Code Representation Learning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 5954–5971.  https://doi.org/10.18653/V1/2021.EMNLP-MAIN.482
  * Jesse et al. (2023) Kevin Jesse, Toufique Ahmed, Premkumar T. Devanbu, and Emily Morgan. 2023.  Large Language Models and Simple, Stupid Bugs. In _20th IEEE/ACM International Conference on Mining Software Repositories, MSR 2023, Melbourne, Australia, May 15-16, 2023_. IEEE, 563–575.  https://doi.org/10.1109/MSR59073.2023.00082
  * Jiang et al. (2023) Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. 2023.  Self-planning Code Generation with Large Language Models.  arXiv:2303.06689 [cs.SE]
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers_ , Regina Barzilay and Min-Yen Kan (Eds.). Association for Computational Linguistics, 1601–1611.  https://doi.org/10.18653/V1/P17-1147
  * Khashabi et al. (2020) Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020.  UnifiedQA: Crossing Format Boundaries With a Single QA System. In _Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020_ _(Findings of ACL, Vol. EMNLP 2020)_ , Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 1896–1907.  https://doi.org/10.18653/V1/2020.FINDINGS-EMNLP.171
  * Kim et al. (2023) Geunwoo Kim, Pierre Baldi, and Stephen McAleer. 2023\.  Language Models can Solve Computer Tasks. In _Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023_ , Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.).  http://papers.nips.cc/paper_files/paper/2023/hash/7cc1005ec73cfbaac9fa21192b622507-Abstract-Conference.html
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022\.  Large Language Models are Zero-Shot Reasoners. In _Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022_ , Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.).  http://papers.nips.cc/paper_files/paper/2022/hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html
  * Koncel-Kedziorski et al. (2015) Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015\.  Parsing Algebraic Word Problems into Equations.  _Trans. Assoc. Comput. Linguistics_ 3 (2015), 585–597.  https://doi.org/10.1162/TACL_A_00160
  * Kruskal and Wallis (1952) William H. Kruskal and W. Allen Wallis. 1952.  Use of Ranks in One-Criterion Variance Analysis.  _J. Amer. Statist. Assoc._ 47, 260 (1952), 583–621.  https://doi.org/10.1080/01621459.1952.10483441
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural Questions: a Benchmark for Question Answering Research.  _Trans. Assoc. Comput. Linguistics_ 7 (2019), 452–466.  https://doi.org/10.1162/TACL_A_00276
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017.  RACE: Large-scale ReAding Comprehension Dataset From Examinations. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017_ , Martha Palmer, Rebecca Hwa, and Sebastian Riedel (Eds.). Association for Computational Linguistics, 785–794.  https://doi.org/10.18653/V1/D17-1082
  * Lake and Baroni (2018) Brenden M. Lake and Marco Baroni. 2018.  Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks. In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ _(Proceedings of Machine Learning Research, Vol. 80)_ , Jennifer G. Dy and Andreas Krause (Eds.). PMLR, 2879–2888.  http://proceedings.mlr.press/v80/lake18a.html
  * Lampinen et al. (2022) Andrew K. Lampinen, Ishita Dasgupta, Stephanie C. Y. Chan, Kory W. Mathewson, Michael Henry Tessler, Antonia Creswell, James L. McClelland, Jane Wang, and Felix Hill. 2022.  Can language models learn from explanations in context?. In _Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, 537–563.  https://doi.org/10.18653/V1/2022.FINDINGS-EMNLP.38
  * Le et al. (2022) Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu-Hong Hoi. 2022\.  CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning. In _Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022_ , Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.).  http://papers.nips.cc/paper_files/paper/2022/hash/8636419dea1aa9fbd25fc4248e702da4-Abstract-Conference.html
  * Lieber et al. (2021) Opher Lieber, Or Sharir, Barak Lenz, and Yoav Shoham. 2021\.  JURASSIC-1: TECHNICAL DETAILS AND EVALUATION.  _AI21 Labs Tech. Rep._ (2021).  https://uploadsssl.webflow.com/60fd4503684b466578c0d307/61138924626a6981ee09caf6_jurassic_tech_paper.pdf
  * Lin et al. (2020) Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020\.  CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning. In _Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020_ _(Findings of ACL, Vol. EMNLP 2020)_ , Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 1823–1840.  https://doi.org/10.18653/V1/2020.FINDINGS-EMNLP.165
  * Ling et al. (2017a) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017a.  Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers_ , Regina Barzilay and Min-Yen Kan (Eds.). Association for Computational Linguistics, 158–167.  https://doi.org/10.18653/V1/P17-1015
  * Ling et al. (2017b) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017b.  Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, Volume 1: Long Papers_ , Regina Barzilay and Min-Yen Kan (Eds.). Association for Computational Linguistics, 158–167.  https://doi.org/10.18653/V1/P17-1015
  * Liu et al. (2023) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. 2023\.  Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation. In _Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023_ , Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.).  http://papers.nips.cc/paper_files/paper/2023/hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. 2021\.  CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation. In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_ , Joaquin Vanschoren and Sai-Kit Yeung (Eds.).  https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/c16a5320fa475530d9583c34fd356ef5-Abstract-round1.html
  * Madaan et al. (2023a) Aman Madaan, Alexander Shypula, Uri Alon, Milad Hashemi, Parthasarathy Ranganathan, Yiming Yang, Graham Neubig, and Amir Yazdanbakhsh. 2023a.  Learning Performance-Improving Code Edits.  _CoRR_ abs/2302.07867 (2023).  https://doi.org/10.48550/ARXIV.2302.07867 arXiv:2302.07867
  * Madaan et al. (2023b) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023b.  Self-Refine: Iterative Refinement with Self-Feedback. In _Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023_ , Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.).  http://papers.nips.cc/paper_files/paper/2023/hash/91edff07232fb1b55a505a9e9f6c0ff3-Abstract-Conference.html
  * Marcus et al. (1994) Mitchell P. Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994\.  The Penn Treebank: Annotating Predicate Argument Structure. In _Human Language Technology, Proceedings of a Workshop held at Plainsboro, New Jerey, USA, March 8-11, 1994_. Morgan Kaufmann.  https://aclanthology.org/H94-1020/
  * Mehri and Eskénazi (2020) Shikib Mehri and Maxine Eskénazi. 2020.  Unsupervised Evaluation of Interactive Dialog with DialoGPT. In _Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue, SIGdial 2020, 1st virtual meeting, July 1-3, 2020_ , Olivier Pietquin, Smaranda Muresan, Vivian Chen, Casey Kennington, David Vandyke, Nina Dethlefs, Koji Inoue, Erik Ekstedt, and Stefan Ultes (Eds.). Association for Computational Linguistics, 225–235.  https://aclanthology.org/2020.sigdial-1.28/
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_ , Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (Eds.). Association for Computational Linguistics, 2381–2391.  https://doi.org/10.18653/V1/D18-1260
  * Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James F. Allen. 2016.  A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories.  _CoRR_ abs/1604.01696 (2016).  arXiv:1604.01696 http://arxiv.org/abs/1604.01696
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020\.  Adversarial NLI: A New Benchmark for Natural Language Understanding. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (Eds.). Association for Computational Linguistics, 4885–4901.  https://doi.org/10.18653/V1/2020.ACL-MAIN.441
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  _CoRR_ abs/2303.08774 (2023).  https://doi.org/10.48550/arXiv.2303.08774 arXiv:2303.08774
  * Paperno et al. (2016) Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. 2016.  The LAMBADA dataset: Word prediction requiring a broad discourse context. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_. The Association for Computer Linguistics.  https://doi.org/10.18653/V1/P16-1144
  * Patel et al. (2021) Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021\.  Are NLP Models really able to Solve Simple Math Word Problems?. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tür, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (Eds.). Association for Computational Linguistics, 2080–2094.  https://doi.org/10.18653/V1/2021.NAACL-MAIN.168
  * Pearce et al. (2022) Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022\.  Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions. In _43rd IEEE Symposium on Security and Privacy, SP 2022, San Francisco, CA, USA, May 22-26, 2022_. IEEE, 754–768.  https://doi.org/10.1109/SP46214.2022.9833571
  * Pearce et al. (2023) H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt. 2023.  Examining Zero-Shot Vulnerability Repair with Large Language Models. In _2023 2023 IEEE Symposium on Security and Privacy (SP) (SP)_. IEEE Computer Society, Los Alamitos, CA, USA, 1–18.  https://doi.org/10.1109/SP46215.2023.00001
  * Perry et al. (2023) Neil Perry, Megha Srivastava, Deepak Kumar, and Dan Boneh. 2023.  Do Users Write More Insecure Code with AI Assistants?. In _Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, CCS 2023, Copenhagen, Denmark, November 26-30, 2023_ , Weizhi Meng, Christian Damsgaard Jensen, Cas Cremers, and Engin Kirda (Eds.). ACM, 2785–2799.  https://doi.org/10.1145/3576915.3623157
  * Phan et al. (2021) Long N. Phan, Hieu Tran, Daniel Le, Hieu Nguyen, James T. Anibal, Alec Peltekian, and Yanfang Ye. 2021.  CoTexT: Multi-task Learning with Code-Text Transformer.  _CoRR_ abs/2105.08645 (2021).  arXiv:2105.08645 https://arxiv.org/abs/2105.08645
  * Prasad et al. (2023) Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. 2023\.  GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023_ , Andreas Vlachos and Isabelle Augenstein (Eds.). Association for Computational Linguistics, 3827–3846.  https://doi.org/10.18653/V1/2023.EACL-MAIN.277
  * Puri et al. (2021) Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir R. Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. 2021.  CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks. In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_ , Joaquin Vanschoren and Sai-Kit Yeung (Eds.).  https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/a5bfc9e07964f8dddeb95fc584cd965d-Abstract-round2.html
  * Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019\.  Language Models are Unsupervised Multitask Learners.  https://api.semanticscholar.org/CorpusID:160025533
  * Rajpurkar et al. (2018) Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018.  Know What You Don’t Know: Unanswerable Questions for SQuAD. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers_ , Iryna Gurevych and Yusuke Miyao (Eds.). Association for Computational Linguistics, 784–789.  https://doi.org/10.18653/V1/P18-2124
  * Reddy et al. (2019) Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019\.  CoQA: A Conversational Question Answering Challenge.  _Trans. Assoc. Comput. Linguistics_ 7 (2019), 249–266.  https://doi.org/10.1162/TACL_A_00266
  * Reynolds and McDonell (2021) Laria Reynolds and Kyle McDonell. 2021.  Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. In _CHI ’21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yokohama Japan, May 8-13, 2021, Extended Abstracts_ , Yoshifumi Kitamura, Aaron Quigley, Katherine Isbister, and Takeo Igarashi (Eds.). ACM, 314:1–314:7.  https://doi.org/10.1145/3411763.3451760
  * Roy and Roth (2015) Subhro Roy and Dan Roth. 2015.  Solving General Arithmetic Word Problems. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_ , Lluís Màrquez, Chris Callison-Burch, Jian Su, Daniele Pighin, and Yuval Marton (Eds.). The Association for Computational Linguistics, 1743–1752.  https://doi.org/10.18653/V1/D15-1202
  * Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021.  WinoGrande: an adversarial winograd schema challenge at scale.  _Commun. ACM_ 64, 9 (2021), 99–106.  https://doi.org/10.1145/3474381
  * Sandoval et al. (2023) Gustavo Sandoval, Hammond Pearce, Teo Nys, Ramesh Karri, Siddharth Garg, and Brendan Dolan-Gavitt. 2023.  Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants. In _32nd USENIX Security Symposium, USENIX Security 2023, Anaheim, CA, USA, August 9-11, 2023_ , Joseph A. Calandrino and Carmela Troncoso (Eds.). USENIX Association, 2205–2222.  https://www.usenix.org/conference/usenixsecurity23/presentation/sandoval
  * Sarkar et al. (2022) Advait Sarkar, Carina Negreanu, Ben Zorn, Sruti Srinivasa Ragavan, Christian Pölitz, and Andrew D. Gordon. 2022.  What is it like to program with artificial intelligence?. In _Proceedings of the 33rd Annual Workshop of the Psychology of Programming Interest Group, PPIG 2022, The Open University, Milton Keynes, UK & Online, September 5-9, 2022_, Simon Holland, Marian Petre, Luke Church, and Mariana Marasoiu (Eds.). Psychology of Programming Interest Group, 127–153.  https://ppig.org/papers/2022-ppig-33rd-sarkar/
  * Saunders et al. (2022) William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022\.  Self-critiquing models for assisting human evaluators.  _CoRR_ abs/2206.05802 (2022).  https://doi.org/10.48550/ARXIV.2206.05802 arXiv:2206.05802
  * Siddiq and Santos (2022) Mohammed Latif Siddiq and Joanna C. S. Santos. 2022.  SecurityEval dataset: mining vulnerability examples to evaluate machine learning-based code generation techniques. In _Proceedings of the 1st International Workshop on Mining Software Repositories Applications for Privacy and Security_ (Singapore, Singapore) _(MSR4P &S 2022)_. Association for Computing Machinery, New York, NY, USA, 29–33.  https://doi.org/10.1145/3549035.3561184
  * Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Santilli, Andreas Stuhlmüller, Andrew M. Dai, Andrew La, Andrew K. Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakas, and et al. 2022\.  Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models.  _CoRR_ abs/2206.04615 (2022).  https://doi.org/10.48550/ARXIV.2206.04615 arXiv:2206.04615
  * Sun et al. (2022) Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. 2022.  Black-Box Tuning for Language-Model-as-a-Service. In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ _(Proceedings of Machine Learning Research, Vol. 162)_ , Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (Eds.). PMLR, 20841–20855.  https://proceedings.mlr.press/v162/sun22e.html
  * Suzgun et al. (2023) Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2023\.  Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_ , Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 13003–13051.  https://doi.org/10.18653/V1/2023.FINDINGS-ACL.824
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 4149–4158.  https://doi.org/10.18653/V1/N19-1421
  * Thomas and Harden (2008) James Thomas and Angela Harden. 2008.  Methods for the thematic synthesis of qualitative research in systematic reviews.  _BMC Medical Research Methodology_ 8 (2008).  Issue 45.
  * Tony et al. (2022) Catherine Tony, Nicolás E. Díaz Ferreyra, and Riccardo Scandariato. 2022.  GitHub Considered Harmful? Analyzing Open-Source Projects for the Automatic Generation of Cryptographic API Call Sequences. In _22nd IEEE International Conference on Software Quality, Reliability and Security, QRS 2022, Guangzhou, China, December 5-9, 2022_. IEEE, 896–906.  https://doi.org/10.1109/QRS57517.2022.00094
  * Tony et al. (2023) Catherine Tony, Markus Mutas, Nicolás E. Díaz Ferreyra, and Riccardo Scandariato. 2023.  LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations. In _20th IEEE/ACM International Conference on Mining Software Repositories, MSR 2023, Melbourne, Australia, May 15-16, 2023_. IEEE, 588–592.  https://doi.org/10.1109/MSR59073.2023.00084
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  LLaMA: Open and Efficient Foundation Language Models.  _CoRR_ abs/2302.13971 (2023).  https://doi.org/10.48550/ARXIV.2302.13971 arXiv:2302.13971
  * Tunstall et al. (2022) Lewis Tunstall, Leandro von Werra, and Thomas Wolf. 2022\.  Natural Language Processing with Transformers.  _O’Reilly Media, Inc._ (2022).
  * Turney et al. (2003) Peter D. Turney, Michael L. Littman, Jeffrey Bigham, and Victor Shnayder. 2003.  Combining Independent Modules to Solve Multiple-choice Synonym and Analogy Problems.  _CoRR_ cs.CL/0309035 (2003).  http://arxiv.org/abs/cs/0309035
  * Vaithilingam et al. (2022) Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022\.  Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In _CHI ’22: CHI Conference on Human Factors in Computing Systems, New Orleans, LA, USA, 29 April 2022 - 5 May 2022, Extended Abstracts_ , Simone D. J. Barbosa, Cliff Lampe, Caroline Appert, and David A. Shamma (Eds.). ACM, 332:1–332:7.  https://doi.org/10.1145/3491101.3519665
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019\.  SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 3261–3275.  https://proceedings.neurips.cc/paper/2019/hash/4496bf24afe7fab6f046bf4923da8de6-Abstract.html
  * Wang et al. (2022) Boshi Wang, Xiang Deng, and Huan Sun. 2022.  Iteratively Prompt Pre-trained Language Models for Chain of Thought. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, 2714–2730.  https://aclanthology.org/2022.emnlp-main.174
  * Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023\.  Self-Consistency Improves Chain of Thought Reasoning in Language Models. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.  https://openreview.net/pdf?id=1PL1NIMMrw
  * Wang et al. (2021) Yue Wang, Weishi Wang, Shafiq R. Joty, and Steven C. H. Hoi. 2021\.  CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Computational Linguistics, 8696–8708.  https://doi.org/10.18653/v1/2021.emnlp-main.685
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In _NeurIPS_.  http://papers.nips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html
  * White et al. (2023) Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023.  A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT.  _CoRR_ abs/2302.11382 (2023).  https://doi.org/10.48550/arXiv.2302.11382 arXiv:2302.11382
  * Wickert et al. (2021) Anna-Katharina Wickert, Lars Baumgärtner, Florian Breitfelder, and Mira Mezini. 2021\.  Python Crypto Misuses in the Wild. In _ESEM ’21: ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, Bari, Italy, October 11-15, 2021_ , Filippo Lanubile, Marcos Kalinowski, and Maria Teresa Baldassarre (Eds.). ACM, 31:1–31:6.  https://doi.org/10.1145/3475716.3484195
  * Wickert et al. (2019) Anna-Katharina Wickert, Michael Reif, Michael Eichberg, Anam Dodhy, and Mira Mezini. 2019\.  A dataset of parametric cryptographic misuses. In _Proceedings of the 16th International Conference on Mining Software Repositories, MSR 2019, 26-27 May 2019, Montreal, Canada_ , Margaret-Anne D. Storey, Bram Adams, and Sonia Haiduc (Eds.). IEEE / ACM, 96–100.  https://doi.org/10.1109/MSR.2019.00023
  * Wohlin (2014) Claes Wohlin. 2014\.  Guidelines for snowballing in systematic literature studies and a replication in software engineering. In _18th International Conference on Evaluation and Assessment in Software Engineering, EASE ’14, London, England, United Kingdom, May 13-14, 2014_ , Martin J. Shepperd, Tracy Hall, and Ingunn Myrtveit (Eds.). ACM, 38:1–38:10.  https://doi.org/10.1145/2601248.2601268
  * Xiao and Watson (2019) Yu Xiao and Maria Watson. 2019.  Guidance on Conducting a Systematic Literature Review.  _Journal of Planning Education and Research_ 39 (2019), 93–112.  Issue 1. https://doi.org/10.1177/0739456X17723971
  * Xu et al. (2022a) Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022a.  A systematic evaluation of large language models of code. In _MAPS@PLDI 2022: 6th ACM SIGPLAN International Symposium on Machine Programming, San Diego, CA, USA, 13 June 2022_ , Swarat Chaudhuri and Charles Sutton (Eds.). ACM, 1–10.  https://doi.org/10.1145/3520312.3534862
  * Xu et al. (2022b) Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, and Zhilin Yang. 2022b.  GPS: Genetic Prompt Search for Efficient Few-Shot Learning. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_ , Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (Eds.). Association for Computational Linguistics, 8162–8171.  https://doi.org/10.18653/V1/2022.EMNLP-MAIN.559
  * Yao et al. (2023b) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023b.  Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In _Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023_ , Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.).  http://papers.nips.cc/paper_files/paper/2023/hash/271db9922b8d1f4dd7aaef84ed5ac703-Abstract-Conference.html
  * Yao et al. (2023a) Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Eric Sun, and Yue Zhang. 2023a.  A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly.  _CoRR_ abs/2312.02003 (2023).  https://doi.org/10.48550/ARXIV.2312.02003 arXiv:2312.02003
  * Yetiştiren et al. (2023) Burak Yetiştiren, Işık Özsoy, Miray Ayerdem, and Eray Tüzün. 2023.  Evaluating the Code Quality of AI-Assisted Code Generation Tools: An Empirical Study on GitHub Copilot, Amazon CodeWhisperer, and ChatGPT.  arXiv:2304.10778 [cs.SE]
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  HellaSwag: Can a Machine Really Finish Your Sentence?. In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , Anna Korhonen, David R. Traum, and Lluís Màrquez (Eds.). Association for Computational Linguistics, 4791–4800.  https://doi.org/10.18653/V1/P19-1472
  * Zeng et al. (2022) Zhengran Zeng, Hanzhuo Tan, Haotian Zhang, Jing Li, Yuqun Zhang, and Lingming Zhang. 2022\.  An extensive study on pre-trained models for program understanding and generation. In _ISSTA ’22: 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, Virtual Event, South Korea, July 18 - 22, 2022_ , Sukyoung Ryu and Yannis Smaragdakis (Eds.). ACM, 39–51.  https://doi.org/10.1145/3533767.3534390
  * Zhang et al. (2015) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015.  Character-level Convolutional Networks for Text Classification. In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (Eds.). 649–657.  https://proceedings.neurips.cc/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html
  * Zheng et al. (2023a) Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2023a.  Progressive-Hint Prompting Improves Reasoning in Large Language Models.  _CoRR_ abs/2304.09797 (2023).  https://doi.org/10.48550/ARXIV.2304.09797 arXiv:2304.09797
  * Zheng et al. (2023b) Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. 2023b.  CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X.  _CoRR_ abs/2303.17568 (2023).  https://doi.org/10.48550/ARXIV.2303.17568 arXiv:2303.17568
  * Zhou et al. (2023b) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. 2023b.  Least-to-Most Prompting Enables Complex Reasoning in Large Language Models. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.  https://openreview.net/pdf?id=WZH7099tgfM
  * Zhou et al. (2023a) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023a.  Large Language Models are Human-Level Prompt Engineers. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.  https://openreview.net/pdf?id=92gvk82DE-


# PerPaDa: A Persian Paraphrase Dataset based on Implicit Crowdsourcing Data
Collection

###### Abstract

In this paper we introduce PerPaDa, a Persian paraphrase dataset that is
collected from users’ input in a plagiarism detection system. As an implicit
crowdsourcing experience, we have gathered a large collection of original and
paraphrased sentences from Hamtajoo; a Persian plagiarism detection system, in
which users try to conceal cases of text re-use in their documents by
paraphrasing and re-submitting manuscripts for analysis. The compiled dataset
contains 2446 instances of paraphrasing. In order to improve the overall
quality of the collected data, some heuristics have been used to exclude
sentences that don’t meet the proposed criteria. The introduced corpus is much
larger than the available datasets for the task of paraphrase identification
in Persian. Moreover, there is less bias in the data compared to the similar
datasets, since the users did not try some fixed predefined rules in order to
generate similar texts to their original inputs.

Keywords: Paraphrase, Persian, Implicit crowdsourcing, Plagiarism detection

languageresourceLanguage Resources

PerPaDa: A Persian Paraphrase Dataset based on Implicit Crowdsourcing Data
Collection

Salar Mohtaj1,2, Fatemeh Tavakkoli3, Habibollah Asghari4
---
1 Technische Universität Berlin, Berlin, Germany
2 German Research Centre for Artificial Intelligence (DFKI), Projektbüro
Berlin, Germany
3 Freie Universität Berlin, Berlin, Germany
4 ICT Research Institute of ACECR, Tehran, Iran
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Paraphrase identification is the task of investigating if a piece of text is a
paraphrase of another one [Hunt et al., 2019]. It is a basic task for
different Natural Language Processing (NLP) tasks like Plagiarism Detection
(PD) and question answering. Paraphrase identification could improve the
overall performance of these tasks by enabling systems to detect semantically
similar or related sentences or phrases. For instance, a plagiarism detection
system that empowered by a paraphrase identification module not only can
detect cases of verbatim text re-use, but also the cases in which people try
to conceal plagiarism by using different wordings and the other paraphrasing
techniques.

There are different approaches for identification of paraphrases. A possible
approach for measuring if two sentences are semantically similar (i.e.,
paraphrased) is two measure cosine similarity between two sentences [Mahmoud
and Zrigui, 2017]. However, more recent approaches are using Machine Learning
(ML) models to train a classifier to detect if two pieces of text are
paraphrased. Moreover, pre-trained language models (e.g., BERT [Devlin et al.,
2019]) have been using recently for the task of paraphrase identification
[Wang et al., 2020]. The neural networks could also be used to generate
paraphrases from textual data [Wahle et al., 2021].

A training dataset is an essential part of all supervised learning approaches
in ML to train a model. A paraphrase identification dataset is an important
piece of puzzle to train an outstanding paraphrase detection model. In this
paper we introduced PerPaDa, a new Persian Paraphrase Dataset in Persian. The
data is collected from users’ input to Hamtajoo111www.hamtajoo.ir plagiarism
detection system. Hamtajoo is a Persian plagiarism detection tool that is
being used by journals, editorial board, conferences, faculty members and
students to detect cases of inadvertent or intentional text re-use in
scientific papers.

As an implicit crowdsourcing experience, we have gathered a large collection
of original and paraphrased sentences from Hamtajoo, when users try to conceal
cases of text re-use in their documents by paraphrasing and re-submitting
manuscripts for analysis. The proposed dataset contains 2446 instances of
paraphrased sentences. Based on our knowledge on the current Persian
paraphrase detection dataset, PerPaDa is much larger than the available
datasets. Bias is an important issue in experiments which try to use
crowdsourcing for data curation, annotation and evaluation for ML and NLP .
Similarly, it is a drawback of the available paraphrase datasets based on
crowdsourcing that crowd-workers intend to follow some instructions which
usually are provided by the experiment designers. This leads to a bias on the
applied strategies by people to paraphrase original text. We believe that the
implicit crowdsourcing approach that has been used in this paper results to a
more diverse and general dataset which covers different paraphrasing
strategies.

The paper is organized as follow; Section 2 contains a brief overview of some
of the Persian and English corpora for the task of paraphrase identification.
We explain the raw data collection procedure from Hamtajoo platform in Section
3. The main steps to construct the PerPaDa dataset are described in Section 4.
We represent some statistics on the data and the evaluation of PerPaDa in
Section 5 and finally conclude the paper in Section 6.

## 2\. Related Work

In this section, some of the related datasets for the task of paraphrase
identification will be introduced.

Although the paraphrasing corpora are very limited in Persian, a number of
datasets for the task of text re-use detection (i.e., plagiarism detection)
have been introduced in recent years. Usually they include paraphrased or
obfuscated text that are inserted from source documents into suspicious ones,
and PD systems should automatically detect the position of these re-used
pieces of text in the source and suspicious documents. We will review some of
them in this section.

Khoshnavataher et. al. compiled a Persian plagiarism detection based on
automatically generated cases of paraphrasing [Khoshnavataher et al., 2015].
these automated approaches include shuffling words in sentences, substitution
of some words with their synonyms and addition/deletion of some words to/from
a sentence.

Asghari et. al. from the same team tried to enrich the previous data by
incorporating manually paraphrased sentences [Asghari et al., 2021]. In
addition to the mentioned automated approaches to generate obfuscated
sentences, they also used more than 150 pairs of paraphrased sentences based
on a crowdsourcing experiment.

Mashhadirajab et. al. proposed a text alignment corpus for Persian plagiarism
detection, which includes more than 11000 documents and about 11600 plagiarism
cases in PAN format [Mashhadirajab et al., 2016]. They simulated different
types of plagiarism by exploiting manually, semi-automatically, or
automatically approaches to generate paraphrases in this large-scale corpus
[Mashhadirajab et al., 2016].

To best of our knowledge, there is no Persian paraphrased identification
dataset that is compiled for this standalone task. PerPaDa could be the first
such a dataset that can be used to train Persian paraphrase generation and
identification models.

## 3\. Data Collection

In this section we introduce the data collection procedure, in which we’ve
collected the source data to compile PerPaDa dataset.

### 3.1. Hamtajoo plagiarism detection system

Hamtajoo is a Persian plagiarism detection system for investigating patterns
of text re-use in Persian academic papers [Zarrabi et al., 2021]. The system
works on document level at the first stage and then focuses on paragraph and
sentence level in the second detailed comparison stage. The system was
officially introduced on 2017 and has been using by academicians to prevent
and detect cases of text matching since then.

Two screenshots of Hamtajoo platform are shown in Figures 1 and 2. The first
figure shows the submission page of the platform, where users can either
upload their textual document or directly insert text into related the field.

Figure 1: The submission page of Hamtajoo system (the menu and text are in
Persian) [Zarrabi et al., 2021]

Figure 2 shows the Result page, where the sections with a piece of re-used
text are highlighted. The origin of the re-used texts are also represented in
this page, so users can track reasons behind the system’s decisions.

Figure 2: The Result page of Hamtajoo system (the menu and text are in
Persian) [Zarrabi et al., 2021].

There are two main use cases of Hamtajoo and the other plagiarism detection
tools for the end users; The tool could be used by students, journal editors
and university faculties as dictated by the workflow before publishing papers
or theses. On the other hand, these systems are sometimes used to detect
potential text re-use cases to reduce or conceal those text matching cases by
paraphrasing or removing suspicious parts from manuscripts.

As an implicit crowdsourcing study, in this research we targeted the second
use case of Hamtajoo, where users try to use the system to detect plagiarism
cases and then conceal it by paraphrasing. We believe that the resulting pairs
of original and paraphrased pieces of text are a rich resource to train
paraphrase identification models because:

  1. 1.

There is no bias in the data. In other words, users don’t follow any
instruction to generate paraphrases.

  2. 2.

The dataset is huge, since the system is widely used by academicians in Iran.

  3. 3.

It covers different scientific topics and domains including humanity,
engineering etc.

### 3.2. Raw data gathering

As mentioned earlier, we have focused on those use cases in which users employ
the plagiarism detection system to find case of text re-use and then try to
conceal them by paraphrasing. For this purpose, we excluded organizational
accounts of the system. These users are mainly journal editors who use the
system to check manuscripts for plagiarism before publication. For the
remaining individual users, Figure 3 shows the distribution of number of
checked documents by the system.

Figure 3: The distribution of the number of submitted documents for plagiarism
detection by Hamtajoo users

The number of submitted documents by individual users shows a long tail
distribution. Most of the users (i.e., almost 75%) uploaded fewer than three
documents for checking against plagiarism. On the other hand, there are a few
users who submitted more than 300 documents into the system.

Since the idea is to compare multiple submission of a document to extract
those parts which paraphrased by users, we excluded the users who submitted
just one document in the system. The length distribution of the reminded 18111
documents is shown in Figure 4. The length distribution follow a power law
distribution too. Here again, there are too many documents which are shorter
than 5,000 words and a few documents that are longer than 80,000 words. The
detailed steps to extract paraphrased sentences from these documents are
explained in the succeeding section.

Figure 4: The histogram and length distribution of the target documents

## 4\. Corpus Construction

In this section we elaborate the main steps to extract paraphrased sentences
out of initial selected documents.

A sample scenario in which a user use the plagiarism detection system to find
case of text re-use and then try to conceal them by paraphrasing is depicted
in Figure 5. As it’s highlighted in the image, a user would try to re-write
those colored parts in document A and re-submit the modified document to the
system to check if it can detect those paraphrased parts (Document B).
Regardless of the final results (i.e., whether the system can detect those
paraphrased sections), the matching between the original sentences in Document
A and the re-written ones in Document B is a valuable resource of paraphrased
sentences. It should be noted that a user may re-submit different versions of
the initial document after paraphrasing various parts of the paper, and this
matching could be repeated for each pair.

The main steps for matching the original detected sentences and the
paraphrased ones are as follow:

  1. 1.

Detection of near duplicate documents for each user.

  2. 2.

Ordering documents in the near duplicate clusters based on the time of
submission.

  3. 3.

Extracting detected sentences in the lead documents.

  4. 4.

Searching for the paraphrased sentences at the similar position in the
subsequent documents.

  5. 5.

Post-processing of the extracted pairs by applying some heuristics to exclude
low quality pairs.

These steps are elaborated in the succeeding sub-sections.

Figure 5: A sample scenario in which a user upload a document, and then re-
submit it after paraphrasing to conceal cases of text matching

### 4.1. Near Duplicate Detection

In this step all the submitted documents by a user are clustered into the
group of near duplicate documents. Since users may upload different documents
with totally different contents, this step help us to screen those documents
that potentially includes cases of paraphrasing for the use in the next steps.

For measuring the similarity between submitted documents by a user, we
computed cosine similarity between TF-IDF vectors of pairs of documents. We
set the similarity threshold to [0.9 - 1). So those documents that have cosine
similarity greater or equal to 0.9 and lower than 1 are considered as near
duplicate documents. The lower band threshold is a heuristic that comes after
doing a number of experiments. We don’t consider documents with the cosine
similarly of 1 (i.e., exact match) for further analysis because they
definitely don’t contain cases of paraphrasing. The reason of exact match
documents in the system could be the lack of getting response from plagiarism
detection of a document in time which leads to re-submission of the same
document once more into the system.

Among the clusters of documents of each user, we kept those groups that
include at least two documents for further analysis in the next steps.

### 4.2. Ordering the Documents

In this tiny step, the near duplicate documents are ordered based their
submission date/time. The reason is to be able to track changes on those
documents that submitted later (i.e., re-submissions). So, the comparison of
the documents is always have been done forward and the lead documents compared
to those submitted later (i.e., subsequent documents).

### 4.3. Extraction of the Original Sentences

After ordering near duplicate documents which can potentially include some
cases of paraphrasing, we extracted sentences that detected as text matching
cases in the lead documents by Hamtajoo (i.e., original sentences which are
highlighted parts of Document A in Figure 5).

The text matching cases are separated from normal texts by a specific HTML
tag. So, the pieces of text that detected as cases of plagiarism by the system
are extracted from those tags. Since the extracted pieces of texts would
include several sentences, we tokenized the whole text into sentences, using
Parsivar Persian text pre-processing tool [Mohtaj et al., 2018].

### 4.4. Searching for Paraphrased Sentences

After the extraction of the original sentences from the lead documents, in
this step we search for the paraphrased sentences in the subsequent documents.
For this purpose, we chose the approximate location of the original sentence
(in the lead document) in the subsequent documents. In other words, we focused
on the position in which the original sentence was extracted, in the re-
submitted document. Since most of the users try to conceal plagiarism by just
paraphrasing those sentences that are detected by the system, we expected to
find the paraphrased sentences in the similar position.

However, since the paraphrased sentence would shift a few characters in the
subsequent document compared to the lead document, we also took into account
±100 characters in the subsequent document. It means we looked for the
paraphrased sentences in span of 100 characters before to 100 characters after
the position of the original sentences.

After choosing a span of ±100 characters in the subsequent document, we split
the text within the span into sentences. The resulting sentences are the
potential paraphrased sentences for the original sentence. To detect the true
paraphrased sentence among the potential ones, the original sentence and the
potential paraphrased sentences have been embedded into vectors, using
ParsBERT pre-trained model [Farahani et al., 2021]. ParsBERT is a monolingual
BERT for the Persian language, which shows its state-of-the-art performance
compared to other architectures and multi-lingual models. We used a Bert based
model since it can preserve the semantic representation of the sentences and
as a result, can better identify pairs of sentences that are semantically
similar.

After converting sentences into vector of numbers, the cosine similarity
between the original sentence and each potential paraphrased sentence is
computed as it’s shown in Figure 6. Pairs of sentences with cosine similarity
in the range of [0.8 - 1) are extracted as the cases of paraphrasing. This
process is repeated for all the original sentences in the lead document.

Figure 6: Embedding the original and potential paraphrased sentences into
vectors and computing cosine similarity

### 4.5. Post-Processing

After generating the initial list of pairs of original and paraphrased
sentences in the last step, the low quality candidates are removed in this
stage. Since the whole process of extracting pair of paraphrasing sentences
have been done automatically, some noises may be added into the list. To
remove these noises, we applied a series of heuristics on the list of
candidates. For this purpose, we removed those pairs that at least one of the
sentences contains at least one of the following conditions:

  * •

Sentences that are shorter than 50 characters

  * •

Sentences that are not complete (e.g., no subject or verb)

  * •

Sentences that are not Persian

We used Parsivar [Mohtaj et al., 2018] for Part-of-speech tagging and
langdetect222https://pypi.org/project/langdetect/ to detect the language of
the sentences.

## 5\. Evaluation

In this section we present the dataset statistics and the validation results.

### 5.1. Dataset Statistics

The resulted paraphrased dataset contains 2446 pair of sentences. The length
of sentences vary between almost 50 and 300 characters. The range of lengths
in the original and paraphrased sentences are depicted in a box-plot in Figure
7. As highlighted in the figure, there is no significant difference between
the length of original and paraphrased sentences.

Figure 7: Distribution of text length in the original and paraphrased
sentences

Moreover, the distribution of cosine similarity between the pair of sentences
is shown in Figure 8. As shown in the figure, similarities vary between 0.8 to
0.92, with a peak on 0.87.

Figure 8: Distribution of cosine similarity between the pair of sentences

The PerPaDa dataset is only available for research purposes by academic
researchers. in the corresponding section in Hamtajoo
website333http://hamtajoo.ir/corpus.

### 5.2. Validation Result

To validate the proposed dataset, we compared the similarity of pairs of
sentences in PerPaDa with manually paraphrased Persian text in HAMTA, that is
a Persian plagiarism detection corpus [Asghari et al., 2021], [Asghari et al.,
2016]. HAMTA corpus includes manually and automatically generated paraphrased
pieces of text. We took the manually compiled paraphrases from HAMTA to
compare with PerPaDa. For Comparison, we used the method proposed by Potthast
et. al. that includes 10 different retrieval models. Each model is an n-gram
vector space model (VSM) where n ranges from 1 to 10 words, employing tf-
weighting, and the cosine similarity [Potthast et al., 2010]. The resulting
plot is shown in Figure 9.

Figure 9: Comparison of PerPaDa w.r.t. HAMTA: each box plot shows the range of
cosine similarity between the original and the paraphrased sentences.

As it is expected, the overall similarity of the pair of sentences decreases
by increasing n from 1 to 10 in both data. It means although the original and
paraphrased sentences share a number of terms, these terms usually re-ordered
in the paraphrased sentences. However, the pair of sentences in PerPaDa tends
to have more similarity, comparing to HAMTA. It shows users tend to apply
least changes on the original sentence in the implicit crowdsourcing
experiment, while they have to make more changes based on pre-defined rules in
an explicit crowdsourcing setting.

## 6\. Conclusion

In this paper we introduced PerPaDa, a Persian Paraphrase Dataset based on an
implicit crowdsourcing experiment. The raw data has been collected from
Hamtajoo that is a Persian plagiarism detection system. We tried to extract
those documents from Hamtajoo in which a user tried to conceal text matching
cases by paraphrasing part of text that are detected by system as cases of
plagiarism.

Based on our validation experiments, the proposed data shows similar results
with manually paraphrased corpora. However, an implicitly generated dataset
like PerPaDa is much more cheaper comparing to the explicitly compiled
corpora. It can also better shows the paraphrasing behavior of an ordinary
user.

## 7\. Acknowledgment

We would like to thank all of the members of ITBM and AIS research groups of
ICT research institute for their contribution in developing the Hamtajoo
platform.

## 8\. Bibliographical References

## References

  * Asghari et al., 2016 Asghari, H., Mohtaj, S., Fatemi, O., Faili, H., Rosso, P., and Potthast, M.  (2016).  Algorithms and corpora for persian plagiarism detection - overview of PAN at FIRE 2016.  In Prasenjit Majumder, et al., editors, Text Processing - FIRE 2016 International Workshop, Kolkata, India, December 7-10, 2016, Revised Selected Papers, volume 10478 of Lecture Notes in Computer Science, pages 61–79. Springer.
  * Asghari et al., 2021 Asghari, H., Fatemi, O., Mohtaj, S., and Faili, H.  (2021).  A crowdsourcing approach to construct mono-lingual plagiarism detection corpus.  Int. J. Digit. Libr., 22(1):49–61.
  * Devlin et al., 2019 Devlin, J., Chang, M., Lee, K., and Toutanova, K.  (2019).  BERT: pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, et al., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics.
  * Eickhoff, 2018 Eickhoff, C.  (2018).  Cognitive biases in crowdsourcing.  In Yi Chang, et al., editors, Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM 2018, Marina Del Rey, CA, USA, February 5-9, 2018, pages 162–170. ACM.
  * Farahani et al., 2021 Farahani, M., Gharachorloo, M., Farahani, M., and Manthouri, M.  (2021).  Parsbert: Transformer-based model for persian language understanding.  Neural Process. Lett., 53(6):3831–3847.
  * Hunt et al., 2019 Hunt, E., Dahal, B., Zhan, J., Gewali, L., Oh, P. Y., Janamsetty, R., Kinares, C., Koh, C., Sanchez, A., Zhan, F., Özdemir, M., Waseem, S., and Yolcu, O.  (2019).  Machine learning models for paraphrase identification and its applications on plagiarism detection.  In Yunjun Gao, et al., editors, 2019 IEEE International Conference on Big Knowledge, ICBK 2019, Beijing, China, November 10-11, 2019, pages 97–104. IEEE.
  * Khoshnavataher et al., 2015 Khoshnavataher, K., Zarrabi, V., Mohtaj, S., and Asghari, H.  (2015).  Developing monolingual persian corpus for extrinsic plagiarism detection using artificial obfuscation: Notebook for PAN at CLEF 2015.  In Linda Cappellato, et al., editors, Working Notes of CLEF 2015 - Conference and Labs of the Evaluation forum, Toulouse, France, September 8-11, 2015, volume 1391 of CEUR Workshop Proceedings. CEUR-WS.org.
  * Mahmoud and Zrigui, 2017 Mahmoud, A. and Zrigui, M.  (2017).  Semantic similarity analysis for paraphrase identification in arabic texts.  In Rachel Edita Roxas, editor, Proceedings of the 31st Pacific Asia Conference on Language, Information and Computation, PACLIC 2018, Cebu City, Philippines, November 16-18, 2017, pages 274–281. The National University (Phillippines).
  * Mashhadirajab et al., 2016 Mashhadirajab, F., Shamsfard, M., Adelkhah, R., Shafiee, F., and Saedi, C.  (2016).  A text alignment corpus for persian plagiarism detection.  In Prasenjit Majumder, et al., editors, Working notes of FIRE 2016 - Forum for Information Retrieval Evaluation, Kolkata, India, December 7-10, 2016, volume 1737 of CEUR Workshop Proceedings, pages 184–189. CEUR-WS.org.
  * Mohtaj et al., 2018 Mohtaj, S., Roshanfekr, B., Zafarian, A., and Asghari, H.  (2018).  Parsivar: A language processing toolkit for persian.  In Nicoletta Calzolari, et al., editors, Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018. European Language Resources Association (ELRA).
  * Potthast et al., 2010 Potthast, M., Stein, B., Barrón-Cedeño, A., and Rosso, P.  (2010).  An evaluation framework for plagiarism detection.  In Chu-Ren Huang et al., editors, COLING 2010, 23rd International Conference on Computational Linguistics, Posters Volume, 23-27 August 2010, Beijing, China, pages 997–1005. Chinese Information Processing Society of China.
  * Wahle et al., 2021 Wahle, J. P., Ruas, T., Meuschke, N., and Gipp, B.  (2021).  Are neural language models good plagiarists? A benchmark for neural paraphrase detection.  In J. Stephen Downie, et al., editors, ACM/IEEE Joint Conference on Digital Libraries, JCDL 2021, Champaign, IL, USA, September 27-30, 2021, pages 226–229. IEEE.
  * Wang et al., 2020 Wang, W., Bi, B., Yan, M., Wu, C., Xia, J., Bao, Z., Peng, L., and Si, L.  (2020).  Structbert: Incorporating language structures into pre-training for deep language understanding.  In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
  * Zarrabi et al., 2021 Zarrabi, V., Mohtaj, S., and Asghari, H.  (2021).  Hamtajoo: A persian plagiarism checker for academic manuscripts.  CoRR, abs/2112.13742.