et al., 2023a; Zhuo et al., 2023b). Furthermore, the predominant natural
language in source code and other datasets used is English although other
languages are also present. As such, the model can generate code snippets
provided some non-English context, but the generated code is not guaranteed to
work as intended or equally as well for all languages. This could limit the
model’s fairness and effectiveness across different coding tasks and
environments (Alyafeai et al., 2024).

##### Traceability

Using the SWHID to trace software components is not an easy task and will
challenge most if not all, downstream developers. Future development and
advancement of tools that make it easier to trace software components will be
necessary to enable more transparent and responsible data supply chains (Cosmo
et al., 2020).

##### Job augmentation vs. automation

Code LLMs serve as powerful foundation models that can be fine-tuned to
generate high-quality code, documentation, unit tests, text summaries,
automation workflows, and more. Chen et al. (2023) find a positive correlation
between occupation exposure and wage levels/experience premiums, suggesting
higher-paying and experience-intensive jobs may face greater displacement
risks from LLM-powered software. Goldman Sachs (2024) suggest that AI has the
potential to automate 25% of labor tasks in advanced economies and 10 – 20% in
emerging economies, however, they also state that "those fears should be
counterbalanced, since AI has the potential to create new job tasks or
categories requiring specialized human expertise". Autor et al. (2022) reports
that “Roughly 60% of employment in 2018 is found in job titles that did not
exist in 1940.” and that "augmentation innovations boost occupational labor
demand, while automation innovations erode it". Results from the task-based
analysis in (World Economic Forum, 2024) reveal that jobs with the highest
potential for automation of tasks by LLMs emphasize routine and repetitive
procedures and do not require a high degree of interpersonal communication.
Jobs with the highest potential for augmentation by LLMs emphasize critical
thinking and complex problem-solving skills, especially those in science,
technology, engineering, and mathematics (STEM) fields. Ziegler et al. (2024)
reports the benefits of receiving AI suggestions while coding span the full
range of typically investigated aspects of productivity, such as task time,
product quality, cognitive load, enjoyment, and learning. In (Peng et al.,
2023), a two-year collaboration between Google Core and Google Research (Brain
Team), they find that of the 10k+ Google-internal developers using the code
completion setup in their IDE, they measured user’s code acceptance rate of
25-34%. Yahoo Finance (2024) announced ServiceNow, Inc. (NYSE: NOW) 2024 Q4
Earnings with coverage that the ServiceNow platform Now Assist skills using
text-to-code (ServiceNow, 2024b) and text-to-workflow (ServiceNow, 2024a) LLMs
(based on StarCoder), augment and increased developer productivity and speed
of innovation by 52%.

##  10 Conclusion

We introduced StarCoder2, a family of LLMs designed for code generation, along
with The Stack v2, the largest pre-training corpus for Code LLMs built on the
foundations of the Software Heritage archive. The Stack v2 is ten times larger
than its predecessor, yielding a raw dataset of 67.5 TB. Through extensive
cleaning, filtering, and subsampling of the source code, along with the
incorporation of other high-quality code-related datasets, we created a
training set of approximately 3TB (900B+ tokens). Leveraging this new dataset,
we trained StarCoder2 models with 3B, 7B, and 15B parameters. Our extensive
Code LLM evaluations, assessing code completion, editing, and reasoning
capabilities, revealed that StarCoder2-3B and StarCoder2-15B are state-of-the-
art models within their respective size classes. By not only releasing the
model weights but also ensuring complete transparency regarding the training
data, we hope to increase trust in the developed models and empower other
engineering teams and scientists to build upon our efforts.

##  11 Acknowledgements

This work was made possible by Software Heritage, the great library of source
code: https://www.softwareheritage.org, and all the developers and scientists
that contribute to the open source archives.

We thank Joydeep Biswas (UT Austin), Northeastern Research Computing, and NCSA
Delta for providing computing resources used for evaluation. Carolyn Jane
Anderson and Arjun Guha were partially sponsored by the U.S. National Science
Foundation awards SES-2326173 and SES-2326174. Jiawei Liu, Yuxiang Wei, and
Lingming Zhang were partially sponsored by the U.S. National Science
Foundation award CCF-2131943. Federico Cassano was partly sponsored by Roblox.

We thank Jenny Hui, ServiceNow, for her leadership in executing the StarCoder2
Research Collaboration Agreement between ServiceNow, Hugging Face, and NVIDIA
to enable the training of all 3 models.

We thank the extended members of the BigCode community for the ongoing support
and for their downstream contributions back to the community.

We also thank Hessie Jones and the Privacy Protection Collab that shared
insights and lessons learned from their work in Defining Personal Information
and the Remediation Framework during early exploration and consideration of
PII redaction.

Evgenii Zheltonozhskii is supported by the Adams Fellowships Program of the
Israel Academy of Sciences and Humanities.

## References

  * Abramatic et al. (2018) Jean-François Abramatic, Roberto Di Cosmo, and Stefano Zacchiroli.  Building the universal archive of source code.  _Communications of the ACM_ , 61(10):29–31, 2018.  doi: 10.1145/3183558.  URL https://cacm.acm.org/magazines/2018/10/231366-building-the-universal-archive-of-source-code/fulltext.
  * Ainslie et al. (2023) Joshua Ainslie, James Lee-Thorp, Michiel de Jong, Yury Zemlyanskiy, Federico Lebron, and Sumit Sanghai.  GQA: Training generalized multi-query transformer models from multi-head checkpoints.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pp. 4895–4901, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.298.  URL https://aclanthology.org/2023.emnlp-main.298.
  * Akiki et al. (2022) Christopher Akiki, Giada Pistilli, Margot Mieskes, Matthias Gallé, Thomas Wolf, Suzana Ilic, and Yacine Jernite.  BigScience: a case study in the social construction of a multilingual large language model.  In _Workshop on Broadening Research Collaborations 2022_ , 2022.  URL https://openreview.net/forum?id=2e346l2PPOm.
  * Akiki et al. (2023) Christopher Akiki, Odunayo Ogundepo, Aleksandra Piktus, Xinyu Zhang, Akintunde Oladipo, Jimmy Lin, and Martin Potthast.  Spacerini: Plug-and-play search engines with pyserini and Hugging Face.  In Yansong Feng and Els Lefever (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pp. 140–148, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-demo.12.  URL https://aclanthology.org/2023.emnlp-demo.12.
  * Albalak et al. (2024) Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang.  A survey on data selection for language models.  _arXiv preprint_ , February 2024.  URL https://arxiv.org/abs/2402.16827.
  * Alyafeai et al. (2024) Zaid Alyafeai, Khalid Almubarak, Ahmed Ashraf, Deema Alnuhait, Saied Alshahrani, Gubran A. Q. Abdulrahman, Gamil Ahmed, Qais Gawah, Zead Saleh, Mustafa Ghaleb, Yousef Ali, and Maged S. Al-Shaibani.  CIDAR: culturally relevant instruction dataset for Arabic.  _arXiv preprint_ , February 2024.  URL https://arxiv.org/abs/2402.03177.
  * Arivazhagan et al. (2019) Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu.  Massively multilingual neural machine translation in the wild: Findings and challenges.  _arXiv preprint_ , July 2019.  URL https://arxiv.org/abs/1907.05019.
  * Arxiv (2024) Arxiv, 2024.  URL https://info.arxiv.org/help/bulk_data_s3.html.
  * Austin et al. (2021) Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc V. Le, and Charles Sutton.  Program synthesis with large language models.  _arXiv preprint_ , August 2021.  URL https://arxiv.org/abs/2108.07732.
  * Autor et al. (2022) David Autor, Caroline Chin, Anna M Salomons, and Bryan Seegmiller.  New frontiers: The origins and content of new work, 1940–2018.  Technical Report 30389, National Bureau of Economic Research, August 2022.  URL http://www.nber.org/papers/w30389.
  * Azerbayev et al. (2024) Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck.  Llemma: An open language model for mathematics.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=4WnqRR915j.
  * Bavarian et al. (2022) Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen.  Efficient training of language models to fill in the middle.  _arXiv preprint_ , July 2022.  URL https://arxiv.org/abs/2207.14255.
  * Ben Allal (2023) Loubna Ben Allal.  Big code models leaderboard, 2023.  URL https://huggingface.co/spaces/bigcode/bigcode-models-leaderboard.
  * Ben Allal et al. (2022) Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra.  A framework for the evaluation of code generation models.  https://github.com/bigcode-project/bigcode-evaluation-harness, 2022.
  * Ben Allal et al. (2023) Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo García del Río, Qian Liu, Shamik Bose, Urvashi Bhattacharyya, Terry Yue Zhuo, Ian Yu, Paulo Villegas, Marco Zocca, Sourab Mangrulkar, David Lansky, Huu Nguyen, Danish Contractor, Luis Villa, Jia Li, Dzmitry Bahdanau, Yacine Jernite, Sean Hughes, Daniel Fried, Arjun Guha, Harm de Vries, and Leandro von Werra.  SantaCoder: don’t reach for the stars!  _arXiv preprint_ , August 2023.  URL https://arxiv.org/abs/2301.03988.
  * Bhatt et al. (2023) Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, Sasha Frolov, Ravi Prakash Giri, Dhaval Kapil, Yiannis Kozyrakis, David LeBlanc, James Milazzo, Aleksandar Straumann, Gabriel Synnaeve, Varun Vontimitta, Spencer Whitman, and Joshua Saxe.  Purple llama CyberSecEval: A secure coding benchmark for language models.  _arXiv preprint_ , December 2023.  URL https://arxiv.org/abs/2312.04724.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar Van Der Wal.  Pythia: A suite for analyzing large language models across training and scaling.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pp. 2397–2430. PMLR, 23–29 Jul 2023.  URL https://proceedings.mlr.press/v202/biderman23a.html.
  * BigCode (2024) BigCode.  Models by BigCode on Hugging Face, 2024.  URL https://huggingface.co/api/models?author=bigcode&expand[]=downloadsAllTime.  Accessed: 2024.
  * BigCode collaboration et al. (2023) BigCode collaboration, Sean Hughes, Harm de Vries, Jennifer Robinson, Carlos Muñoz Ferrandis, Loubna Ben Allal, Leandro von Werra, Jennifer Ding, Sebastien Paquet, and Yacine Jernite.  The BigCode project governance card.  _arXiv preprint_ , December 2023.  URL https://arxiv.org/abs/2312.03872.
  * BigCode Project (2023a) BigCode Project.  Bigcode model license agreement, 2023a.  URL https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement.  Accessed: 2023.
  * BigCode Project (2023b) BigCode Project.  BigCode open RAIL: Responsible AI licensing framework, 2023b.  URL https://www.bigcode-project.org/docs/pages/bigcode-openrail/.  Accessed: 2023.
  * BigScience Workshop (2022) BigScience Workshop.  BLOOM (revision 4ab0472), 2022.  URL https://huggingface.co/bigscience/bloom.
  * Blue Oak Council (2024) Blue Oak Council, 2024.  URL https://blueoakcouncil.org/list.
  * Broder (2000) Andrei Z. Broder.  Identifying and filtering near-duplicate documents.  In _Annual symposium on combinatorial pattern matching_ , pp. 1–10. Springer, 2000.  URL https://link.springer.com/chapter/10.1007/3-540-45123-4_1.
  * Caballero et al. (2016) Ethan Caballero, OpenAI, and Ilya Sutskever.  Description2Code dataset, August 2016.  URL https://github.com/ethancaballero/description2code.
  * Cassano et al. (2023a) Federico Cassano, John Gouwar, Francesca Lucchetti, Claire Schlesinger, Carolyn Jane Anderson, Michael Greenberg, Abhinav Jangda, and Arjun Guha.  Knowledge transfer from high-resource to low-resource programming languages for code LLMs.  _arXiv preprint_ , August 2023a.  URL https://arxiv.org/abs/2308.09895.
  * Cassano et al. (2023b) Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda.  MultiPL-E: a scalable and polyglot approach to benchmarking neural code generation.  _IEEE Transactions on Software Engineering_ , 49(7):3675–3691, 2023b.  doi: 10.1109/TSE.2023.3267446.  URL https://www.computer.org/csdl/journal/ts/2023/07/10103177/1MpWUtj7Rwk.
  * Cassano et al. (2024) Federico Cassano, Luisa Li, Akul Sethi, Noah Shinn, Abby Brennan-Jones, Anton Lozhkov, Carolyn Jane Anderson, and Arjun Guha.  Can it edit? evaluating the ability of large language models to follow code editing instructions.  In _The First International Workshop on Large Language Model for Code_ , 2024.  URL https://arxiv.org/abs/2312.12450.
  * Chai et al. (2023) Yekun Chai, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, and Hua Wu.  ERNIE-code: Beyond English-centric cross-lingual pretraining for programming languages.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), _Findings of the Association for Computational Linguistics: ACL 2023_ , pp. 10628–10650, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-acl.676.  URL https://aclanthology.org/2023.findings-acl.676.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code.  _arXiv preprint_ , July 2021.  URL https://arxiv.org/abs/2107.03374.
  * Chen et al. (2023) Qin Chen, Jinfeng Ge, Huaqing Xie, Xingcheng Xu, and Yanqing Yang.  Large language models at work in China’s labor market.  _arXiv preprint_ , August 2023.  URL https://arxiv.org/abs/2308.08776.
  * Chowdhery et al. (2023) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  Palm: Scaling language modeling with pathways.  _Journal of Machine Learning Research_ , 24(240):1–113, 2023.  URL http://jmlr.org/papers/v24/22-1144.html.
  * Christiano et al. (2017) Paul F. Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper_files/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html.
  * ClamAV (2024) ClamAV, 2024.  URL https://www.clamav.net/.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems.  _arXiv preprint_ , October 2021.  URL https://arxiv.org/abs/2110.14168.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 8440–8451, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.747.  URL https://aclanthology.org/2020.acl-main.747.
  * Cosmo & Zacchiroli (2017) Roberto Di Cosmo and Stefano Zacchiroli.  Software heritage: Why and how to preserve software source code.  In _iPRES 2017: 14th International Conference on Digital Preservation_ , Kyoto, Japan, 2017.  URL https://www.softwareheritage.org/wp-content/uploads/2020/01/ipres-2017-swh.pdf.  https://hal.archives-ouvertes.fr/hal-01590958.
  * Cosmo et al. (2020) Roberto Di Cosmo, Morane Gruenpeter, and Stefano Zacchiroli.  Referencing source code artifacts: A separate concern in software citation.  _Computing in Science & Engineering_, 22(2):33–43, 2020.  doi: 10.1109/MCSE.2019.2963148.
  * CrowdStrike (2024) CrowdStrike.  Polymorphic virus.  https://www.crowdstrike.com/cybersecurity-101/malware/polymorphic-virus/, 2024.  Accessed: 2024.
  * CyberArk (2024) CyberArk.  Chatting our way into creating a polymorphic malware.  https://www.cyberark.com/resources/threat-research-blog/chatting-our-way-into-creating-a-polymorphic-malware, 2024.  Accessed: 2024.
  * Cybersecurity & Infrastructure Security Agency (2024) Cybersecurity & Infrastructure Security Agency.  Secure by design, 2024.  URL https://www.cisa.gov/resources-tools/resources/secure-by-design.  Accessed: 2024.
  * Dao (2024) Tri Dao.  FlashAttention-2: faster attention with better parallelism and work partitioning.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=mZn2Xyh9Ec.
  * Dao et al. (2022) Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.  FlashAttention: fast and memory-efficient exact attention with IO-awareness.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), _Advances in Neural Information Processing Systems_ , volume 35, pp. 16344–16359. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/67d57c32e20fd0a7a302cb81d36e40d5-Abstract-Conference.html.
  * de Vries (2023) Harm de Vries.  Go smol or go home.  https://www.harmdevries.com/post/model-size-vs-compute-overhead/, 2023.
  * Dhamala et al. (2021) Jwala Dhamala, Tony Sun, Varun Kumar, Satyapriya Krishna, Yada Pruksachatkun, Kai-Wei Chang, and Rahul Gupta.  BOLD: dataset and metrics for measuring biases in open-ended language generation.  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, pp. 862–872, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450383097.  doi: 10.1145/3442188.3445924.  URL https://doi.org/10.1145/3442188.3445924.
  * Ding et al. (2022) Jennifer Ding, Christopher Akiki, Yacine Jernite, Anne Lee Steele, and Temi Popo.  Towards openness beyond open access: User journeys through 3 open AI collaboratives.  In _Workshop on Broadening Research Collaborations 2022_ , 2022.  URL https://openreview.net/forum?id=slU-5h8rrCz.
  * Ding et al. (2023) Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang.  CrossCodeEval: a diverse and multilingual benchmark for cross-file code completion.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023.  URL https://openreview.net/forum?id=wgDcbBMSfh.
  * Dong et al. (2024) Yi Dong, Ronghui Mu, Gaojie Jin, Yi Qi, Jinwei Hu, Xingyu Zhao, Jie Meng, Wenjie Ruan, and Xiaowei Huang.  Building guardrails for large language models.  _arXiv preprint_ , February 2024.  URL https://arxiv.org/abs/2402.01822.
  * Ethayarajh et al. (2024) Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela.  KTO: model alignment as prospect theoretic optimization.  _arXiv preprint_ , February 2024.  URL https://arxiv.org/abs/2402.01306.
  * Fan et al. (2023) Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang.  Large language models for software engineering: Survey and open problems.  _arXiv preprint_ , October 2023.  URL https://arxiv.org/abs/2310.03533.
  * Gao et al. (2023) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.  PAL: Program-aided language models.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pp. 10764–10799. PMLR, 23–29 Jul 2023.  URL https://proceedings.mlr.press/v202/gao23f.html.
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith.  RealToxicityPrompts: evaluating neural toxic degeneration in language models.  In Trevor Cohn, Yulan He, and Yang Liu (eds.), _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pp. 3356–3369, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.findings-emnlp.301.  URL https://aclanthology.org/2020.findings-emnlp.301.
  * Gemini Team et al. (2023) Gemini Team et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint_ , 2023.  URL https://arxiv.org/abs/2312.11805.
  * Github Archive (2024) Github Archive, 2024.  URL https://gharchive.org.
  * go-enry (2024) go-enry, 2024.  URL https://github.com/go-enry/go-enry.
  * Goldman Sachs (2024) Goldman Sachs.  The generative world order: AI, geopolitics, and power, 2024.  URL https://www.goldmansachs.com/intelligence/pages/the-generative-world-order-ai-geopolitics-and-power.html.
  * Governance AI (2024) Governance AI.  Open sourcing highly capable foundation models, 2024.  URL https://www.governance.ai/research-paper/open-sourcing-highly-capable-foundation-models.  Accessed: 2024.
  * Groeneveld et al. (2024) Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi.  OLMo: accelerating the science of language models.  _arXiv preprint_ , February 2024.  URL https://arxiv.org/abs/2402.00838.
  * Grossman et al. (2023) Aiden Grossman, Ludger Paehler, Konstantinos Parasyris, Tal Ben-Nun, Jacob Hegna, William Moses, Jose M. Monsalve Diaz, Mircea Trofin, and Johannes Doerfert.  ComPile: a large IR dataset from production sources.  _arXiv preprint_ , September 2023.  URL https://arxiv.org/abs/2309.15432.
  * Gu et al. (2024) Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang.  CRUXEval: a benchmark for code reasoning, understanding and execution.  _arXiv preprint_ , January 2024.  URL https://arxiv.org/abs/2401.03065.
  * Guo et al. (2024) Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang.  DeepSeek-Coder: when the large language model meets programming – the rise of code intelligence.  _arXiv preprint_ , 2024.  URL https://arxiv.org/abs/2401.14196.
  * Gupta et al. (2023) Maanak Gupta, Charankumar Akiri, Kshitiz Aryal, Eli Parker, and Lopamudra Praharaj.  From ChatGPT to ThreatGPT: impact of generative AI in cybersecurity and privacy.  _IEEE Access_ , 11:80218–80245, 2023.  ISSN 2169-3536.  doi: 10.1109/access.2023.3300381.  URL http://dx.doi.org/10.1109/ACCESS.2023.3300381.
  * Gutiérrez-Fandiño et al. (2022) Asier Gutiérrez-Fandiño, David Pérez-Fernández, Jordi Armengol-Estapé, David Griol, and Zoraida Callejas.  esCorpius: a massive spanish crawling corpus.  In _IberSPEECH 2022_ , pp. 126–130, 2022.  doi: 10.21437/IberSPEECH.2022-26.  URL https://www.isca-speech.org/archive/pdfs/iberspeech_2022/gutierrezfandino22_iberspeech.pdf.
  * Hendrycks et al. (2021) Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt.  Measuring coding challenge competence with apps.  In J. Vanschoren and S. Yeung (eds.), _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , volume 1. Curran, 2021.  URL https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/c24cd76e1ce41366a4bbe8a49b02a028-Abstract-round2.html.
  * Heritage (2024) Software Heritage.  Software heritage community.  https://www.softwareheritage.org/community/, 2024.  Accessed: 2024.
  * Hou et al. (2023) Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang.  Large language models for software engineering: A systematic literature review.  _arXiv preprint_ , August 2023.  URL https://arxiv.org/abs/2308.10620.
  * Huang et al. (2023) Dong Huang, Qingwen Bu, Jie Zhang, Xiaofei Xie, Junjie Chen, and Heming Cui.  Bias testing and mitigation in LLM-based code generation.  _arXiv preprint_ , 2023.  URL https://arxiv.org/abs/2309.14345.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7B.  _arXiv preprint_ , 2023.  URL https://arxiv.org/abs/2310.06825.
  * Jiang et al. (2021) Albert Qiaochu Jiang, Wenda Li, Jesse Michael Han, and Yuhuai Wu.  LISA: language models of ISAbelle proofs.  In _6th Conference on Artificial Intelligence and Theorem Proving_ , pp. 378–392, 2021.  URL http://aitp-conference.org/2021/abstract/paper_17.pdf.
  * Kingma & Ba (2015) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun (eds.), _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1412.6980.
  * Kocetkov et al. (2023) Denis Kocetkov, Raymond Li, Loubna Ben allal, Jia LI, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro Von Werra, and Harm de Vries.  The stack: 3 TB of permissively licensed source code.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=pxpbTdUEpD.
  * Lacoste et al. (2019) Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres.  Quantifying the carbon emissions of machine learning.  _arXiv preprint_ , October 2019.  URL https://arxiv.org/abs/1910.09700.
  * Lai et al. (2023) Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-Tau Yih, Daniel Fried, Sida Wang, and Tao Yu.  DS-1000: A natural and reliable benchmark for data science code generation.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pp. 18319–18345. PMLR, 23–29 Jul 2023.  URL https://proceedings.mlr.press/v202/lai23b.html.
  * Lattner & Adve (2004) Chris Lattner and Vikram Adve.  LLVM: a compilation framework for lifelong program analysis & transformation.  In _International symposium on code generation and optimization, 2004. CGO 2004._ , pp. 75–86. IEEE, 2004.  URL https://ieeexplore.ieee.org/document/1281665.
  * Li et al. (2023) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries.  StarCoder: may the source be with you!  _arXiv preprint_ , May 2023.  URL https://arxiv.org/abs/2305.06161.
  * Li et al. (2022) Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals.  Competition-level code generation with alphacode.  _Science_ , 378(6624):1092–1097, 2022.  doi: 10.1126/science.abq1158.  URL https://www.science.org/doi/abs/10.1126/science.abq1158.
  * Liu et al. (2023a) Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang.  Is your code generated by chatGPT really correct? rigorous evaluation of large language models for code generation.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023a.  URL https://openreview.net/forum?id=1qvx610Cu7.
  * Liu et al. (2023b) Tianyang Liu, Canwen Xu, and Julian McAuley.  RepoBench: Benchmarking repository-level code auto-completion systems.  _arXiv preprint_ , June 2023b.  URL https://arxiv.org/abs/2306.03091.
  * Longpre et al. (2023) Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, Xinyi Wu, Enrico Shippole, Kurt Bollacker, Tongshuang Wu, Luis Villa, Sandy Pentland, and Sara Hooker.  The data provenance initiative: A large scale audit of dataset licensing & attribution in AI.  _arXiv preprint_ , 2023.  URL https://arxiv.org/abs/2310.16787.
  * Luukkonen et al. (2023) Risto Luukkonen, Ville Komulainen, Jouni Luoma, Anni Eskelinen, Jenna Kanerva, Hanna-Mari Kupari, Filip Ginter, Veronika Laippala, Niklas Muennighoff, Aleksandra Piktus, et al.  Fingpt: Large generative models for a small language.  _arXiv preprint arXiv:2311.05640_ , 2023.  URL https://arxiv.org/abs/2311.05640.
  * Malfa et al. (2023) Emanuele La Malfa, Aleksandar Petrov, Simon Frieder, Christoph Weinhuber, Ryan Burnell, Raza Nazar, Anthony G. Cohn, Nigel Shadbolt, and Michael Wooldridge.  Language models as a service: Overview of a new paradigm and its challenges.  _arXiv preprint_ , 2023.  URL https://arxiv.org/abs/2309.16573.
  * Marone & Van Durme (2023) Marc Marone and Benjamin Van Durme.  Data portraits: Recording foundation model training data.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023.  URL https://arxiv.org/abs/2303.03919.
  * mathlib Community (2020) The mathlib Community.  The lean mathematical library.  In _Proceedings of the 9th ACM SIGPLAN International Conference on Certified Programs and Proofs_ , POPL ’20. ACM, January 2020.  doi: 10.1145/3372885.3373824.  URL http://dx.doi.org/10.1145/3372885.3373824.
  * Mendez et al. (2020) Daniel Mendez, Daniel Graziotin, Stefan Wagner, and Heidi Seibold.  _Open Science in Software Engineering_ , pp. 477–501.  Springer International Publishing, 2020.  doi: 10.1007/978-3-030-32489-6_17.  URL http://dx.doi.org/10.1007/978-3-030-32489-6_17.
  * Merkle (1987) Ralph C. Merkle.  A digital signature based on a conventional encryption function.  In _Conference on the theory and application of cryptographic techniques_ , pp. 369–378. Springer, 1987.
  * Mirakhorli et al. (2024) Mehdi Mirakhorli, Derek Garcia, Schuyler Dillon, Kevin Laporte, Matthew Morrison, Henry Lu, Viktoria Koscinski, and Christopher Enoch.  A landscape study of open source and proprietary tools for software bill of materials (sbom).  _arXiv preprint_ , 2024.  URL https://arxiv.org/abs/2402.11151.
  * Mirzayanov (2020) Mike Mirzayanov.  Codeforces: Results of 2020 [annual report].  https://codeforces.com/blog/entry/89502, 2020.
  * Mozes et al. (2023) Maximilian Mozes, Xuanli He, Bennett Kleinberg, and Lewis D. Griffin.  Use of LLMs for illicit purposes: Threats, prevention measures, and vulnerabilities.  _arXiv preprint_ , 2023.  URL https://arxiv.org/abs/2308.12833.
  * MSFT Q2 Earning Call (2024) MSFT Q2 Earning Call, 2024.  URL https://www.microsoft.com/en-us/investor/events/fy-2024/earnings-fy-2024-q2.aspx.
  * Muennighoff et al. (2022a) Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers.  Mteb: Massive text embedding benchmark.  _arXiv preprint arXiv:2210.07316_ , 2022a.  doi: 10.48550/ARXIV.2210.07316.  URL https://arxiv.org/abs/2210.07316.
  * Muennighoff et al. (2022b) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.  Crosslingual generalization through multitask finetuning, 2022b.  URL https://arxiv.org/abs/2211.01786.
  * Muennighoff et al. (2023) Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Nouamane Tazi, Aleksandra Piktus, Sampo Pyysalo, Thomas Wolf, and Colin Raffel.  Scaling data-constrained language models.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=j5BuTrEj35.
  * Muennighoff et al. (2024a) Niklas Muennighoff, Qian Liu, Armel Randy Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro Von Werra, and Shayne Longpre.  OctoPack: instruction tuning code large language models.  In _The Twelfth International Conference on Learning Representations_ , 2024a.  URL https://openreview.net/forum?id=mw1PWNSWZP.
  * Muennighoff et al. (2024b) Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela.  Generative representational instruction tuning.  _arXiv preprint_ , 2024b.  URL https://arxiv.org/abs/2402.09906.
  * Mökander et al. (2023) J. Mökander, J. Schuett, H.R. Kirk, et al.  Auditing large language models: A three-layered approach.  _AI Ethics_ , 2023.  URL https://doi.org/10.1007/s43681-023-00289-2.
  * Nanz & Furia (2015) Sebastian Nanz and Carlo A. Furia.  A comparative study of programming languages in Rosetta code.  In _2015 IEEE/ACM 37th IEEE International Conference on Software Engineering_ , volume 1, pp. 778–788. IEEE, 2015.  URL https://ieeexplore.ieee.org/document/7194625.
  * Nijkamp et al. (2023) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong.  CodeGen: an open large language model for code with multi-turn program synthesis.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=iaYcJKpY2B_.
  * Nozza et al. (2021) Debora Nozza, Federico Bianchi, and Dirk Hovy.  HONEST: Measuring hurtful sentence completion in language models.  In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 2398–2406, Online, June 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.naacl-main.191.  URL https://aclanthology.org/2021.naacl-main.191.
  * OpenAI et al. (2023) OpenAI et al.  GPT-4 technical report.  _arXiv preprint_ , March 2023.  URL https://arxiv.org/abs/2303.08774.
  * Ortiz Suárez et al. (2019) Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary.  Asynchronous pipelines for processing huge corpora on medium to low resource infrastructures.  In Piotr Bański, Adrien Barbaresi, Hanno Biber, Evelyn Breiteneder, Simon Clematide, Marc Kupietz, Harald Lüngen, and Caroline Iliadi (eds.), _Proceedings of the Workshop on Challenges in the Management of Large Corpora_ , pp. 9 – 16, Mannheim, July 2019. Leibniz-Institut für Deutsche Sprache.  doi: 10.14618/ids-pub-9021.  URL http://nbn-resolving.de/urn:nbn:de:bsz:mh39-90215.
  * Paster et al. (2023) Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, and Jimmy Ba.  OpenWebMath: an open dataset of high-quality mathematical web text.  _arXiv preprint_ , October 2023.  URL https://arxiv.org/abs/2310.06786.
  * Pearce et al. (2022) Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri.  Asleep at the keyboard? assessing the security of github copilot’s code contributions.  In _2022 IEEE Symposium on Security and Privacy (SP)_ , pp. 754–768. IEEE, 2022.
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay.  The RefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data only.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023.  URL https://openreview.net/forum?id=kM5eGcdCzq.
  * Peng et al. (2023) Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer.  The impact of AI on developer productivity: Evidence from GitHub Copilot.  _arXiv preprint_ , 2023.  URL https://arxiv.org/abs/2302.06590.
  * Pietri et al. (2020) Antoine Pietri, Diomidis Spinellis, and Stefano Zacchiroli.  The software heritage graph dataset: Large-scale analysis of public software development history.  In _MSR 2020: The 17th International Conference on Mining Software Repositories_ , pp. 1–5. IEEE, 2020.  doi: 10.1145/3379597.3387510.  URL https://arxiv.org/abs/2011.07824https://www.softwareheritage.org/wp-content/uploads/2021/03/msr-2020-challenge.pdf.
  * Piktus et al. (2023a) Aleksandra Piktus, Christopher Akiki, Paulo Villegas, Hugo Laurençon, Gérard Dupont, Sasha Luccioni, Yacine Jernite, and Anna Rogers.  The ROOTS search tool: Data transparency for LLMs.  In Danushka Bollegala, Ruihong Huang, and Alan Ritter (eds.), _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_ , pp. 304–314, Toronto, Canada, July 2023a. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-demo.29.  URL https://aclanthology.org/2023.acl-demo.29.
  * Piktus et al. (2023b) Aleksandra Piktus, Odunayo Ogundepo, Christopher Akiki, Akintunde Oladipo, Xinyu Zhang, Hailey Schoelkopf, Stella Biderman, Martin Potthast, and Jimmy Lin.  GAIA search: Hugging Face and pyserini interoperability for NLP training data exploration.  In Danushka Bollegala, Ruihong Huang, and Alan Ritter (eds.), _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_ , pp. 588–598, Toronto, Canada, July 2023b. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-demo.57.  URL https://aclanthology.org/2023.acl-demo.57.
  * Pinnaparaju et al. (2024) Nikhil Pinnaparaju, Reshinth Adithyan, Duy Phung, Jonathan Tow, James Baicoianu, , and Nathan Cooper.  Stable code 3B: Coding on the edge.  _Stability AI_ , 2024.  URL https://stability.ai/news/stable-code-2024-llm-code-completion-release.
  * Project (2024) BigCode Project.  The stack v2, 2024.  URL https://huggingface.co/datasets/bigcode/the-stack-v2/.  Accessed: 2024.
  * Puri et al. (2021) Ruchir Puri, David S Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss.  CodeNet: a large-scale AI for code dataset for learning a diversity of coding tasks.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.  URL https://openreview.net/forum?id=6vZVBkCDrHT.
  * RedPajama Wiki (2024) RedPajama Wiki, 2024.  URL https://github.com/togethercomputer/RedPajama-Data/tree/rp_v1/data_prep/wiki.
  * Reimers & Gurevych (2019) Nils Reimers and Iryna Gurevych.  Sentence-BERT: Sentence embeddings using siamese BERT-networks.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 3982–3992, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1410.  URL https://aclanthology.org/D19-1410.
  * Ren et al. (2020) Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma.  Codebleu: a method for automatic evaluation of code synthesis.  _arXiv preprint_ , 2020.  URL https://arxiv.org/abs/2009.10297.
  * Rosetta Code (2023) Rosetta Code, 2023.  URL https://rosettacode.org/.
  * Rozière et al. (2023) Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve.  Code llama: Open foundation models for code.  _arXiv preprint_ , August 2023.  URL https://arxiv.org/abs/2308.12950.
  * Sane Security (2024) Sane Security, 2024.  URL https://sanesecurity.com/usage/signatures.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush.  Multitask prompted training enables zero-shot task generalization.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=9Vrb9D0WI4.
  * Saxton et al. (2019) David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli.  Analysing mathematical reasoning abilities of neural models.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=H1gR5iR5FX.
  * ScanCode (2024) ScanCode, 2024.  URL https://github.com/nexB/scancode-toolkit.
  * ScanCode License Categories (2024) ScanCode License Categories, 2024.  URL https://scancode-licensedb.aboutcode.org/help.html##license-categories.
  * Scao et al. (2022a) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al.  BLOOM: A 176b-parameter open-access multilingual language model.  _CoRR_ , abs/2211.05100, 2022a.  doi: 10.48550/ARXIV.2211.05100.  URL https://doi.org/10.48550/arXiv.2211.05100.
  * Scao et al. (2022b) Teven Le Scao, Thomas Wang, Daniel Hesslow, Lucile Saulnier, Stas Bekman, M Saiful Bari, Stella Biderman, Hady Elsahar, Niklas Muennighoff, Jason Phang, et al.  What language model to train if you have one million gpu hours?  _arXiv preprint arXiv:2210.15424_ , 2022b.
  * ServiceNow (2024a) ServiceNow.  Text2flow LLM: Automating workflow generation from descriptive text.  https://downloads.docs.servicenow.com/resource/enus/infocard/text2flow-llm.pdf, 2024a.
  * ServiceNow (2024b) ServiceNow.  Text-to-code LLM: transforming natural language into executable code, 2024b.  URL https://downloads.docs.servicenow.com/resource/enus/infocard/text-to-code-llm.pdf.
  * Shazeer (2019) Noam Shazeer.  Fast transformer decoding: One write-head is all you need.  _CoRR_ , abs/1911.02150, 2019.  URL http://arxiv.org/abs/1911.02150.
  * Sheng et al. (2019) Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng.  The woman worked as a babysitter: On biases in language generation.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 3407–3412, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1339.  URL https://aclanthology.org/D19-1339.
  * Sholler et al. (2019) Dan Sholler, Igor Steinmacher, Denise Ford, Mara Averick, Mike Hoye, and Greg Wilson.  Ten simple rules for helping newcomers become contributors to open projects.  _PLoS Computational Biology_ , 15(9):e1007296, 2019.  doi: 10.1371/journal.pcbi.1007296.  URL https://doi.org/10.1371/journal.pcbi.1007296.
  * Singh et al. (2024) Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemiński, Hakimeh Fadaei, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzieh Fadaee, and Sara Hooker.  Aya dataset: An open-access collection for multilingual instruction tuning.  _arXiv preprint_ , 2024.  URL https://arxiv.org/abs/2402.06619.
  * Software Heritage (2023) Software Heritage.  Swh statement on llm for code, 2023.  URL https://www.softwareheritage.org/2023/10/19/swh-statement-on-llm-for-code/.
  * Software Heritage (2024a) Software Heritage.  Bulk access terms of use, 2024a.  URL https://www.softwareheritage.org/legal/bulk-access-terms-of-use/.
  * Software Heritage (2024b) Software Heritage, 2024b.  URL https://www.softwareheritage.org.
  * Solaiman (2023) Irene Solaiman.  The gradient of generative AI release: Methods and considerations.  _arXiv preprint_ , 2023.  URL https://arxiv.org/abs/2302.04844.
  * Soldaini et al. (2024) Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Chandu, Jennifer Dumas, Yanai Elazar, Valentin Hofmann, Ananya Harsh Jha, Sachin Kumar, Li Lucy, Xinxi Lyu, Nathan Lambert, Ian Magnusson, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Abhilasha Ravichander, Kyle Richardson, Zejiang Shen, Emma Strubell, Nishant Subramani, Oyvind Tafjord, Pete Walsh, Luke Zettlemoyer, Noah A. Smith, Hannaneh Hajishirzi, Iz Beltagy, Dirk Groeneveld, Jesse Dodge, and Kyle Lo.  Dolma: an open corpus of three trillion tokens for language model pretraining research.  _arXiv preprint_ , 2024.  URL https://arxiv.org/abs/2402.00159.
  * StackExchange Archive (2024) StackExchange Archive, 2024.  URL https://archive.org/details/stackexchange.
  * Su et al. (2021) Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu.  RoFormer: Enhanced transformer with rotary position embedding.  _arXiv preprint_ , April 2021.  URL https://arxiv.org/abs/2104.09864.
  * Szafraniec et al. (2023) Marc Szafraniec, Baptiste Roziere, Hugh James Leather, Patrick Labatut, Francois Charton, and Gabriel Synnaeve.  Code translation with compiler representations.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=XomEU3eNeSQ.
  * Tang et al. (2023) Feiyang Tang, Bjarte M. Østvold, and Magiel Bruntink.  Helping code reviewer prioritize: Pinpointing personal data and its processing.  _Frontiers in Artificial Intelligence and Applications_ , 371:109–124, 2023.  doi: 10.3233/FAIA230228.
  * Tang & Østvold (2024) Feiyang Tang and Bjarte M. Østvold.  Finding privacy-relevant source code.  _arXiv preprint_ , 2024.  URL https://arxiv.org/abs/2401.07316.
  * The SWHID Specification Project (2024) The SWHID Specification Project.  The SWHID specification, 2024.  URL https://www.swhid.org/.
  * Together Computer (2023) Together Computer.  RedPajama: an open dataset for training large language models, October 2023.  URL https://github.com/togethercomputer/RedPajama-Data.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint_ , 2023.  URL https://arxiv.org/abs/2307.09288.
  * Üstün et al. (2024) Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker.  Aya model: An instruction finetuned open-access multilingual language model.  _arXiv preprint_ , 2024.  URL https://arxiv.org/abs/2402.07827.
  * Vidgen et al. (2021) Bertie Vidgen, Tristan Thrush, Zeerak Waseem, and Douwe Kiela.  Learning from the worst: Dynamically generated datasets to improve online hate detection.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 1667–1682, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.132.  URL https://aclanthology.org/2021.acl-long.132.
  * Wang et al. (2024) Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang.  Software testing with large language models: Survey, landscape, and vision.  _IEEE Transactions on Software Engineering_ , pp. 1–27, 2024.  doi: 10.1109/TSE.2024.3368208.  URL https://arxiv.org/abs/2307.07221.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=gEZrGCozdqR.
  * Woelfle et al. (2011) Michael Woelfle, Piero L Olliaro, and Matthew H. Todd.  Open science is a research accelerator.  _Nature chemistry_ , 3 10:745–8, 2011.  URL https://api.semanticscholar.org/CorpusID:205289283.
  * World Economic Forum (2024) World Economic Forum.  Jobs of tomorrow: Large language models and jobs, 2024.  URL https://www.weforum.org/publications/jobs-of-tomorrow-large-language-models-and-jobs/.
  * Xu et al. (2024) Yiheng Xu, Hongjin Su, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbao Xie, Zhoujun Cheng, Siheng Zhao, Lingpeng Kong, Bailin Wang, Caiming Xiong, and Tao Yu.  Lemur: Harmonizing natural language and code for language agents.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=hNhwSmtXRh.
  * Yahoo Finance (2024) Yahoo Finance.  ServiceNow Inc (NYSE: NOW) Q4 earnings: What to expect, 2024.  URL https://finance.yahoo.com/news/servicenow-inc-nyse-now-q4-154816487.html.
  * Yang et al. (2023) Zhou Yang, Zhipeng Zhao, Chenyu Wang, Jieke Shi, Dongsum Kim, Donggyun Han, and David Lo.  Gotcha! this model uses my code! evaluating membership leakage risks in code models.  _arXiv preprint_ , 2023.  URL https://arxiv.org/abs/2310.01166.
  * Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou.  Scaling relationship on learning mathematical reasoning with large language models.  _arXiv preprint_ , August 2023.  URL https://arxiv.org/abs/2308.01825.
  * Zhao et al. (2018) Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang.  Gender bias in coreference resolution: Evaluation and debiasing methods.  In Marilyn Walker, Heng Ji, and Amanda Stent (eds.), _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pp. 15–20, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi: 10.18653/v1/N18-2003.  URL https://aclanthology.org/N18-2003.
  * Zhuo et al. (2023a) Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing.  Red teaming ChatGPT via jailbreaking: Bias, robustness, reliability and toxicity.  _arXiv preprint_ , 2023a.  URL https://arxiv.org/abs/2301.12867.
  * Zhuo et al. (2023b) Terry Yue Zhuo, Zhou Yang, Zhensu Sun, Yufei Wang, Li Li, Xiaoning Du, Zhenchang Xing, and David Lo.  Source code data augmentation for deep learning: A survey.  _arXiv preprint_ , May 2023b.  URL https://arxiv.org/abs/2305.19915.
  * Zhuo et al. (2024) Terry Yue Zhuo, Armel Zebaze, Nitchakarn Suppattarachai, Leandro von Werra, Harm de Vries, Qian Liu, and Niklas Muennighoff.  Astraios: Parameter-efficient instruction tuning code large language models.  _arXiv preprint_ , August 2024.  URL https://arxiv.org/abs/2401.00788.
  * Ziegler et al. (2024) Albert Ziegler, Eirini Kalliamvakou, X. Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian.  Measuring GitHub Copilot’s impact on productivity.  _Commun. ACM_ , 67(3):54–63, feb 2024.  ISSN 0001-0782.  doi: 10.1145/3633453.  URL https://doi.org/10.1145/3633453.

##  Appendix A Data Curation

###  A.1 Excluded Extensions

AL (al), AngelScript (as), AsciiDoc (asc), AspectJ (aj), Bison (bison), Boogie
(bpl),C++ (<empty extension>), Cabal Config (project), ChucK (ck), CODEOWNERS
(<empty extension>),Common Lisp (l, sexp), Common Workflow Language (cwl),
CoNLL-U (conll, conllu), Cue Sheet (cue),CWeb (w), desktop (desktop, in,
service), DIGITAL Command Language (com), DTrace (d), edn (edn),Elixir (lock),
Factor (factor), GAP (g, gd), Gemfile.lock (lock), Gettext Catalog (pot),Git
Config (gitmodules), GLSL (geo), Glyph Bitmap Distribution Format (bdf), GN
(gn),Ignore List (dockerignore, eslintignore, gitignore, npmignore), INI (cfg,
prefs, url),JAR Manifest (mf), Java Properties (properties), Jest Snapshot
(snap), JetBrains MPS (mps),JSONLD (jsonld), LiveScript (ls), Makefile (d,
make), Mathematica (cdf, nb), MAXScript (ms),mIRC Script (mrc), NASL (inc),
nesC (nc), Nunjucks (njk), OpenEdge ABL (p, w),Pascal (<empty extension>, dpr,
inc, pp), Perl (al, ph), PLSQL (pck, pls, tps, trg, vw),Protocol Buffer Text
Format (pbt), Puppet (<empty extension>), PureBasic (pb), Racket (rkt,
rktd),ReScript (res), reStructuredText (rest), Rich Text Format (rtf), Roff
(<empty extension>, 1, 1d, 2,5, 7, 8, 9, in), Roff Manpage (<empty extension>,
1d, 2, 3d, 4, 6, 9, man), Scala (sc), Scilab (tst),SELinux Policy (te), Shell
(env), Slash (sl), Smalltalk (cs), SmPL (cocci), SQL (tab), Standard ML
(sig),Stata (ihlp, sthlp), SuperCollider (sc), SWIG (i), TeX (aux, ltx, toc),
TOML (lock), Turtle (ttl),VBA (frm, frx), Vim Snippet (snippet), Wavefront
Material (mtl), Wikitext (wikitext),Windows Registry Entries (reg), wisp (w),
World of Warcraft Addon Data (toc), X BitMap (xbm),XML (kml, pt, resx, rss),
XML Property List (plist, tmcommand, tmlanguage, tmsnippet, tmtheme), Yacc
(yy).

###  A.2 Excluded Programming Languages

2-Dimensional Array,AGS Script,Bicep,Checksums,COLLADA,CSV,Diff,DirectX 3D
File,E-mail,G-code,Gerber Image,Git Revision List,Gnuplot,Go,Checksums,IRC
log,Jupyter Notebook,KiCad Layout,KiCad Legacy Layout,KiCad
Schematic,Lasso,Linux,Kernel Module,Max,Microsoft Developer Studio
Project,Microsoft Visual Studio Solution,Pickle,PostScript,POV-Ray SDL,Public
Key,Pure Data,Raw token data,robots.txt,STL,SubRip Text,SVG,TSV,Unity3D
Asset,Wavefront Object,WebVTT,X PixMap

###  A.3 License detection

    license_file_names = [
        "li[cs]en[cs]e(s?)",
        "legal",
        "copy(left|right|ing)",
        "unlicense",
        "[al]?gpl([-_ v]?)(\d\.?\d?)?",  # AGPLv3
        "bsd(l?)",  # BSDL
        "mit(x?)",  # MITX
        "apache",
        "artistic",  # Artistic.txt
        "copying(v?)(\d?)",  # COPYING3, COPYINGv3
        "disclaimer",
        "eupl",
        "gfdl",
        "[cm]pl",
        "cc0",
        "al([-_ v]?)(\d\.?\d)?",  # AL2.0
        "about",
        "notice",
        "readme",
        "guidelines",
    ]

    license_file_re = re.compile(
        rf"^(|.*[-_. ])({’|’.join(license_file_names)})(|[-_. ].*)$", re.IGNORECASE
    )

###  A.4 Permissive licenses

##### SPDX-recognized license IDs

0BSD, AAL, Abstyles, AdaCore-doc, Adobe-2006, Adobe-Glyph, ADSL, AFL-1.1,
AFL-1.2, AFL-2.0, AFL-2.1, AFL-3.0, Afmparse, AMDPLPA, AML, AMPAS, ANTLR-PD,
Apache-1.0, Apache-1.1, Apache-2.0, APAFML, App-s2p, Artistic-1.0,
Artistic-1.0-cl8, Artistic-1.0-Perl, Artistic-2.0, Baekmuk, Bahyph, Barr,
Beerware, Bitstream-Charter, Bitstream-Vera, BlueOak-1.0.0, Boehm-GC, Borceux,
Brian-Gladman-3-Clause, BSD-1-Clause, BSD-2-Clause, BSD-2-Clause-Patent,
BSD-2-Clause-Views, BSD-3-Clause, BSD-3-Clause-Attribution, BSD-3-Clause-
Clear, BSD-3-Clause-LBNL, BSD-3-Clause-Modification, BSD-3-Clause-No-Nuclear-
License-2014, BSD-3-Clause-No-Nuclear-Warranty, BSD-3-Clause-Open-MPI,
BSD-4-Clause, BSD-4-Clause-Shortened, BSD-4-Clause-UC, BSD-4.3RENO,
BSD-4.3TAHOE, BSD-Advertising-Acknowledgement, BSD-Attribution-HPND-
disclaimer, BSD-Source-Code, BSL-1.0, bzip2-1.0.6, Caldera, CC-BY-1.0, CC-
BY-2.0, CC-BY-2.5, CC-BY-2.5-AU, CC-BY-3.0, CC-BY-3.0-AT, CC-BY-3.0-DE, CC-
BY-3.0-NL, CC-BY-3.0-US, CC-BY-4.0, CDLA-Permissive-1.0, CDLA-Permissive-2.0,
CECILL-B, CERN-OHL-1.1, CERN-OHL-1.2, CERN-OHL-P-2.0, CFITSIO, checkmk,
ClArtistic, Clips, CMU-Mach, CNRI-Jython, CNRI-Python, CNRI-Python-GPL-
Compatible, COIL-1.0, Community-Spec-1.0, Condor-1.1, Cornell-Lossless-JPEG,
Crossword, CrystalStacker, Cube, curl, DL-DE-BY-2.0, DOC, Dotseqn, DRL-1.0,
DSDP, dtoa, dvipdfm, ECL-1.0, ECL-2.0, EFL-1.0, EFL-2.0, eGenix, Entessa,
EPICS, etalab-2.0, EUDatagrid, Fair, FreeBSD-DOC, FSFAP, FSFULLR, FSFULLRWD,
FTL, GD, Giftware, Glulxe, GLWTPL, Graphics-Gems, GStreamer-exception-2005,
HaskellReport, HP-1986, HPND, HPND-Markus-Kuhn, HPND-sell-variant, HPND-sell-
variant-MIT-disclaimer, HTMLTIDY, IBM-pibs, ICU, IJG, IJG-short, ImageMagick,
iMatix, Info-ZIP, Intel, Intel-ACPI, ISC, Jam, JasPer-2.0, JPNIC, JSON,
Kazlib, Knuth-CTAN, Latex2e, Latex2e-translated-notice, Leptonica, Libpng,
libpng-2.0, libtiff, Linux-OpenIB, LLVM-exception, LOOP, LPL-1.0, LPL-1.02,
LPPL-1.3c, Martin-Birgmeier, metamail, Minpack, MirOS, MIT, MIT-0, MIT-
advertising, MIT-CMU, MIT-enna, MIT-feh, MIT-Festival, MIT-Modern-Variant,
MIT-open-group, MIT-Wu, MITNFA, mpich2, mplus, MS-LPL, MS-PL, MTLL,
MulanPSL-1.0, MulanPSL-2.0, Multics, Mup, NAIST-2003, NASA-1.3, Naumen,
NBPL-1.0, NCSA, Net-SNMP, NetCDF, Newsletr, NICTA-1.0, NIST-PD-fallback, NIST-
Software, NLOD-1.0, NLOD-2.0, NRL, NTP, NTP-0, O-UDA-1.0, ODC-By-1.0, OFFIS,
OFL-1.0, OFL-1.0-no-RFN, OFL-1.0-RFN, OFL-1.1-no-RFN, OFL-1.1-RFN, OGC-1.0,
OGDL-Taiwan-1.0, OGL-Canada-2.0, OGL-UK-1.0, OGL-UK-2.0, OGL-UK-3.0, OGTSL,
OLDAP-1.1, OLDAP-1.2, OLDAP-1.3, OLDAP-1.4, OLDAP-2.0, OLDAP-2.0.1, OLDAP-2.1,
OLDAP-2.2, OLDAP-2.2.1, OLDAP-2.2.2, OLDAP-2.3, OLDAP-2.4, OLDAP-2.5,
OLDAP-2.6, OLDAP-2.7, OLDAP-2.8, OML, OpenSSL, OPUBL-1.0, PHP-3.0, PHP-3.01,
Plexus, PostgreSQL, PSF-2.0, psfrag, psutils, Python-2.0, Python-2.0.1, Qhull,
Rdisc, RSA-MD, Ruby, Saxpath, SCEA, SchemeReport, Sendmail, SGI-B-1.1,
SGI-B-2.0, SGP4, SHL-0.5, SHL-0.51, SHL-2.0, SHL-2.1, SMLNJ, snprintf,
Spencer-86, Spencer-94, Spencer-99, SSH-OpenSSH, SSH-short, SunPro, Swift-
exception, SWL, TCL, TCP-wrappers, TermReadKey, TPDL, TTWL, TU-Berlin-1.0, TU-
Berlin-2.0, UCAR, Unicode-DFS-2015, Unicode-DFS-2016, UnixCrypt, UPL-1.0, Vim,
VSL-1.0, W3C, W3C-19980720, W3C-20150513, w3m, Widget-Workshop, Wsuipa, X11,
X11-distribute-modifications-variant, Xdebug-1.03, Xerox, Xfig, XFree86-1.1,
xinetd, xlock, Xnet, xpp, XSkat, Zed, Zend-2.0, Zlib, zlib-acknowledgement,
ZPL-1.1, ZPL-2.0, ZPL-2.1

##### ScanCode-specific license IDs

LicenseRef-scancode-{3com-microcode, 3dslicer-1.0, 4suite-1.1, accellera-
systemc, adi-bsd, adrian, agere-bsd, alexisisaac-freeware, amd-historical,
ams-fonts, anu-license, apache-patent-exception, apple-attribution, apple-
attribution-1997, apple-excl, apple-sscl, aravindan-premkumar, argouml, arm-
llvm-sga, array-input-method-pl, asmus, asn1, atkinson-hyperlegible-font,
bakoma-fonts-1995, bea-2.1, beal-screamer, beri-hw-sw-1.0, bigdigits, bigelow-
holmes, biopython, bitzi-pd, blas-2017, bohl-0.2, boost-original, boutell-
libgd-2021, bpmn-io, brent-corkum, brian-clapper, brian-gladman, brian-
gladman-3-clause, broadcom-cfe, broadcom-linux-timer, brocade-firmware, bruno-
podetti, bsd-1-clause-build, bsd-1988, bsd-2-clause-plus-advertizing,
bsd-3-clause-devine, bsd-3-clause-fda, bsd-3-clause-jtag, bsd-3-clause-no-
change, bsd-3-clause-no-trademark, bsd-3-clause-sun, bsd-ack-carrot2, bsd-
artwork, bsd-atmel, bsd-axis-nomod, bsd-credit, bsd-dpt, bsd-export, bsd-
innosys, bsd-mylex, bsd-new-derivative, bsd-new-nomod, bsd-new-tcpdump, bsd-
no-disclaimer, bsd-no-disclaimer-unmodified, bsd-original-muscle, bsd-
original-voices, bsd-plus-mod-notice, bsd-simplified-darwin, bsd-simplified-
intel, bsd-simplified-source, bsd-top, bsd-top-gpl-addition, bsd-unchanged,
bsd-unmodified, bsd-x11, bsla-no-advert, bytemark, can-ogl-alberta-2.1, can-
ogl-british-columbia-2.0, can-ogl-nova-scotia-1.0, can-ogl-ontario-1.0, can-
ogl-toronto-1.0, careware, carnegie-mellon, cavium-malloc, cc-by-2.0-uk,
cecill-b-en, cern-attribution-1995, cgic, chicken-dl-0.2, chris-maunder,
chris-stoy, classic-vb, clear-bsd-1-clause, click-license, cmu-mit, cmu-
simple, cmu-template, code-credit-license-1.0.1, code-credit-license-1.1.0,
codeguru-permissions, codesourcery-2004, commonj-timer, compass, componentace-
jcraft, compuphase-linking-exception, cosl, cpm-2022, cpp-core-guidelines,
crcalc, cryptopp, csprng, cve-tou, cwe-tou, cximage, d-zlib, damail, dante-
treglia, dbad-1.1, delorie-historical, dhtmlab-public, dl-de-by-1-0-de, dl-de-
by-1-0-en, dl-de-by-2-0-en, dmalloc, dmtf-2017, docbook, douglas-young,
drl-1.1, dropbear, dropbear-2016, dtree, dwtfnmfpl-3.0, dynamic-drive-tou,
ecfonts-1.0, egenix-1.0.0, ellis-lab, emit, emx-library, energyplus-bsd,
epaperpress, eric-glass, errbot-exception, etalab-2.0-en, fabien-tassin, far-
manager-exception, fastbuild-2012-2020, fatfs, fftpack-2004, filament-group-
mit, flex-2.5, flora-1.1, font-alias, fpl, fplot, fraunhofer-iso-14496-10,
free-art-1.3, freebsd-boot, freebsd-first, freemarker, fsf-notice, fujion-
exception-to-apache-2.0, gareth-mccaughan, gary-s-brown, gdcl, geoff-
kuenning-1993, ghostpdl-permissive, glut, good-boy, greg-roelofs, gregory-
pietsch, gtpl-v1, gtpl-v2, gtpl-v3, happy-bunny, hdf4, hdf5, hdparm, hidapi,
historical-ntp, homebrewed, hp-snmp-pp, html5, httpget, ian-kaplan, ian-
piumarta, ibm-as-is, ibm-dhcp, ibm-icu, ibm-nwsc, ibm-sample, ibpp, icot-free,
idt-notice, ietf, ietf-trust, ilmid, indiana-extreme, infineon-free, info-
zip-1997-10, info-zip-2001-01, info-zip-2002-02, info-zip-2003-05, info-
zip-2004-05, info-zip-2005-02, info-zip-2007-03, info-zip-2009-01, inno-setup,
intel-bsd, intel-bsd-2-clause, intel-osl-1989, intel-osl-1993, intel-royalty-
free, iso-14496-10, iso-8879, itu, ja-sig, jason-mayes, jasper-1.0, java-app-
stub, jdbm-1.00, jdom, jetty, jgraph, jpnic-mdnkit, jpython-1.1, jscheme,
jsfromhell, jython, kalle-kaukonen, keith-rule, kerberos, kevan-stannard,
kevlin-henney, khronos, kumar-robotics, lcs-telegraphics, ldap-sdk-free-use,
libgeotiff, libmib, libmng-2007, libsrv-1.0.2, lil-1, lilo, linux-device-
drivers, linuxbios, linuxhowtos, llnl, logica-1.0, lucre, make-human-
exception, matt-gallagher-attribution, matthew-kwan, mattkruse, mediainfo-lib,
mgopen-font-license, michael-barr, michigan-disclaimer, mit-1995, mit-
license-1998, mit-modification-obligations, mit-nagy, mit-no-advert-export-
control, mit-no-trademarks, mit-old-style, mit-old-style-sparse, mit-readme,
mit-specification-disclaimer, mit-synopsys, mit-taylor-variant, mit-veillard-
variant, mod-dav-1.0, motorola, mpeg-iso, mpeg-ssg, ms-sspl, ms-ws-routing-
spec, msj-sample-code, mulanpsl-1.0-en, mulanpsl-2.0-en, mulle-kybernetik,
musl-exception, mx4j, netcat, netcomponents, netron, newlib-historical,
newran, nice, niels-ferguson, nilsson-historical, nist-srd, node-js,
nonexclusive, nortel-dasa, notre-dame, nrl-permission, ntlm, ntpl-origin,
nvidia, nvidia-2002, nvidia-gov, nwhm, nysl-0.9982, nysl-0.9982-jp, o-young-
jong, oasis-ws-security-spec, object-form-exception-to-mit, odl, odmg, ogc,
ogl-1.0a, ogl-canada-2.0-fr, ogl-wpd-3.0, openmarket-fastcgi, openorb-1.0,
opensaml-1.0, openssl, opml-1.0, opnl-1.0, opnl-2.0, oreilly-notice, oswego-
concurrent, other-permissive, owtchart, ozplb-1.0, ozplb-1.1, paolo-
messina-2000, paraview-1.2, patent-disclaimer, paul-mackerras, paul-mackerras-
binary, paul-mackerras-new, paul-mackerras-simplified, paulo-soares, paypal-
sdk-2013-2016, pcre, pd-mit, pd-programming, perl-1.0, peter-deutsch-document,
philippe-de-muyter, phorum-2.0, php-2.0.2, pine, pngsuite, politepix-pl-1.0,
ppp, protobuf, psf-3.7.2, psytec-freesoft, purdue-bsd, pybench, pycrypto,
pygres-2.2, python-cwi, qlogic-microcode, qpopper, qualcomm-turing,
quirksmode, radvd, red-hat-attribution, red-hat-bsd-simplified, reportbug,
ricebsd, richard-black, robert-hubley, rsa-1990, rsa-cryptoki, rsa-demo, rsa-
md4, rtools-util, rute, ryszard-szopa, saas-mit, saf, sash, sata, sbia-b,
scancode-acknowledgment, scanlogd-license, scansoft-1.2, scintilla, scribbles,
script-asylum, secret-labs-2011, service-comp-arch, sgi-cid-1.0, sgi-glx-1.0,
sglib, shital-shah, simpl-1.1, softfloat, softfloat-2.0, softsurfer, sparky,
speechworks-1.1, ssleay, ssleay-windows, stanford-pvrg, stlport-2000,
stlport-4.5, stream-benchmark, stu-nicholls, sun-rpc, sun-source, sunsoft,
supervisor, svndiff, swig, symphonysoft, synopsys-mit, synthesis-toolkit,
takao-abe, takuya-ooura, tcg-spec-license-v1, tekhvc, tested-software, tex-
live, things-i-made-public-license, tiger-crypto, tigra-calendar-3.2, tigra-
calendar-4.0, tim-janik-2003, timestamp-picker, tso-license, ttcl, ttyp0,
tumbolia, twisted-snmp, ubc, unicode, unicode-icu-58, unicode-mappings,
unlimited-binary-use-exception, unpbook, us-govt-unlimited-rights, usrobotics-
permissive, utopia, vcalendar, vince, visual-idiot, visual-numerics, vixie-
cron, w3c-03-bsd-license, westhawk, whistle, whitecat, wide-license, william-
alexander, wingo, wol, wordnet, wrox, ws-addressing-spec, ws-policy-
specification, ws-trust-specification, wtfnmfpl-1.0, wxwidgets,
wxwindows-u-3.0, x11-acer, x11-adobe, x11-adobe-dec, x11-dec1, x11-dec2,
x11-doc, x11-dsc, x11-hanson, x11-lucent-variant, x11-oar, x11-opengl,
x11-quarterdeck, x11-realmode, x11-sg, x11-stanford, x11-tektronix, x11-x11r5,
x11-xconsortium-veillard, xfree86-1.0, xmldb-1.0, xxd, yale-cas, yensdesign,
zeusbench, zpl-1.0, zsh, zuora-software, zveno-research}

##### Non-licenses

The following contributor license agreements, warranty disclaimers, and other
license amendments were not considered during license labeling: LicenseRef-
scancode-{dco-1.1, generic-cla, google-cla, jetty-ccla-1.1, newton-king-cla,
generic-exception, generic-export-compliance, generic-tos, generic-trademark,
warranty-disclaimer}

###  A.5 Pull Requests

Table 25 shows the volume of PR renderings for various sequence lengths
(measured in characters). We list the volume of the base files for the top 20
languages in Table 25.

###  A.6 StackOverflow

We used the following prompt to

    Below is an instruction from a user and a candidate’s answer. Evaluate whether or not the answer is
    a good example of how AI Assistant should respond to the user’s instruction. Please assign a score
    using the following 10-point scale:

    1: The response is entirely off-topic, contains significant inaccuracies, or is incomprehensible.
    It fails to address the user’s query in any meaningful way.

    2: The answer is largely irrelevant, vague, or controversial. It contains some elements that relate
    to the topic but misses the core of the user’s question or includes substantial misinformation.

    3: The response is somewhat relevant but remains incomplete or contains elements that are
    off-topic or controversial. Key aspects of the user’s query are left unaddressed.

    4: The answer addresses the user’s question to some extent but lacks depth or clarity. It may be
    somewhat helpful but is not comprehensive or detailed.

    5: The response is relevant and offers a basic answer to the user’s question but lacks detail or
    specificity. It’s helpful but not fully developed or insightful.

    6: The answer is moderately helpful and addresses most aspects of the user’s question. It might
    lack some depth or contain minor inaccuracies or irrelevant information.

    7: The response is quite helpful and addresses the user’s query well, but it might not be from an
    AI Assistant’s perspective. It could resemble content from other sources like blog posts or web pages.

    8: The answer is comprehensive and relevant, written from an AI assistant’s perspective. It
    addresses the user’s query effectively but may have minor areas for improvement in focus,
    conciseness, or organization.

    9: The response is almost perfect, providing a clear, comprehensive, and well-organized answer from an
    AI assistant’s perspective. It might have very minor areas for improvement in terms of engagement or
    insight.

    10: The answer is exemplary, perfectly addressing the user’s query from an AI Assistant’s perspective.
    It is highly informative, expertly written, engaging, and insightful, with no discernible areas
    for improvement.

    Please write "Score: <rating>" in the last line, and then provide a brief reasoning you used to derive
    the rating score.

###  A.7 Kaggle Notebooks templates

We remove the following templates if they appear at the beginning of a Kaggle
notebook:

    TEMPLATE_1 = ’# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-
    python
    import numpy as np  # linear algebra
    import pandas as pd  # data processing, CSV file I/O (e.g. pd.read_csv)

    # Input data files are available in the read-only "../input/" directory
    # For example, running this (by clicking run or pressing Shift+Enter) will list all files under the
    input directory
    import os

    for dirname, _, filenames in os.walk("/kaggle/input"):
        for filename in filenames:
            print(os.path.join(dirname, filename))
    # You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output
    when you create a version using "Save & Run All"
    # You can also write temporary files to /kaggle/temp/, but they won’t be saved outside of the current
    session’
    TEMPLATE_2 = ’# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-
    python\n’

Table 24: Volume of the pull requests dataset when we restrict the sequence
length. Seqlen (characters) Volume (GB) `25000` 19.6 `50000` 38.7 `75000`
54.34 `100000` 67.31 `200000` 103.52 `300000` 126.8 `400000` 143.65 `500000`
156.76 `600000` 167.21 `700000` 175.94 `800000` 183.18 `900000` 189.32
`1000000` 194.58

Table 25: Size of base files range of changes for top 20 languages in Pull
Requests. Language Volume (GB) `Python` 13.46 `JavaScript` 9.55 `Java` 8.37
`Markdown` 7.34 `C++` 5.89 `Go` 5.59 `JSON` 4.13 `TypeScript` 3.96 `C#` 3.76
`YAML` 3.1 `XML` 2.55 `C` 2.34 `HTML` 2.31 `Rust` 2.27 `PHP` 2.09 `Ruby` 1.73
`project.pbxproj` 1.51 `Scala` 1.25 `TSX` 1.2 `Swift` 0.9

##  Appendix B Processing Pipeline

###  B.1 Malware removal

We show the top-10 detected malware signatures in Table 27 and the top-10
languages by potentially malicous files in Table 27.

Table 26: Top 10 detected malware signatures. Signature | Count
---|---
Sanesecurity.Malware.28845.BadVBS | 11876
winnow.compromised.ts.jsexploit.5 | 2251
Sanesecurity.Malware.26492.JsHeur | 2247
Sanesecurity.Spam.8879 | 1597
Sanesecurity.Malware.25834.JsHeur | 1560
Sanesecurity.Malware.27112.JsHeur | 1258
Sanesecurity.Malware.26222.JsHeur | 888
Porcupine.Malware.52833 | 814
Sanesecurity.SpamL.8887 | 792
Sanesecurity.Malware.26557.JsHeur | 728

Table 27: Top 10 languages by the number of potentially malicious files. Language | Count
---|---
Text | 13281
HTML | 11336
JavaScript | 10210
VBScript | 7947
Logos | 3283
Markdown | 2736
Linker Script | 1390
XML | 1260
VBA | 990
JSON | 547

##  Appendix C Data Composition

###  C.1 TheStackV2-train-smol

  * •

Configuration languages

    * –

Ant Build System – CMake – Dockerfile – Go Module

    * –

Gradle – INI – Java Properties

    * –

Makefile – Maven POM – TOML

  * •

Configuration files:

    * –

CMakeLists.txt – Cargo.toml – DESCRIPTION – Gemfile – Makefile – Makefile.am –
NAMESPACE – Package.swift – Pipfile – build.gradle

    * –

build.gradle.kts – composer.json – conda.yml – configure.ac – docker-
compose.yaml – docker-compose.yml – go.mod – package.json – pom.xml

    * –

pyproject.toml – requirements-dev.txt – requirements-prod.txt –
requirements.in – requirements.test.txt – requirements.txt – setup.cfg –
tsconfig.json – yarn.lock

###  C.2 TheStackV2-train-full

In Table 28, we summarize the data volume for the subsamples languages.

Table 28: Subsampling volumes for languages in the Stack v2 dataset. Final volume | Languages
---|---
200GB | Java, JavaScript
100GB | HTML
8GB | CSS, Java Server Pages, JSON,
| SCSS, Smali, XML, YAML
1GB | BibTeX, Gettext Catalog, Graphviz (DOT),
| Java Properties, Roff, Roff Manpage,
| Web Ontology Language