* Chen et al. (2023a) Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao.  Fireact: Toward language agent fine-tuning, 2023a.
  * Chen et al. (2023b) Hao Chen, Yiming Zhang, Qi Zhang, Hantao Yang, Xiaomeng Hu, Xuetao Ma, Yifan Yanggong, and Junbo Zhao.  Maybe only 0.5% data is needed: A preliminary exploration of low training data instruction tuning.  _arXiv preprint arXiv:2305.09246_ , 2023b.
  * Chen et al. (2023c) Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin.  Alpagasus: Training a better alpaca with fewer data, 2023c.  URL https://arxiv.org/abs/2307.08701.
  * Chen et al. (2024b) Shu Chen, Xinyan Guan, Yaojie Lu, Hongyu Lin, Xianpei Han, and Le Sun.  Reinstruct: Building instruction data from unlabelled corpus.  In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics_ , 2024b.
  * Chen et al. (2024c) Weixin Chen, Dawn Song, and Bo Li.  Grath: Gradual self-truthifying for large language models, 2024c.
  * Chen et al. (2024d) Xiaoyang Chen, Ben He, Hongyu Lin, Xianpei Han, Tianshu Wang, Boxi Cao, Le Sun, and Yingfei Sun.  Spiral of silences: How is large language model killing information retrieval?–a case study on open domain question answering.  _arXiv preprint arXiv:2404.10496_ , 2024d.
  * Chen et al. (2023d) Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou.  Teaching large language models to self-debug, 2023d.
  * Chen et al. (2024e) Xiusi Chen, Hongzhi Wen, Sreyashi Nag, Chen Luo, Qingyu Yin, Ruirui Li, Zheng Li, and Wei Wang.  Iteralign: Iterative constitutional alignment of large language models.  _ArXiv preprint_ , abs/2403.18341, 2024e.  URL https://arxiv.org/abs/2403.18341.
  * Chen et al. (2023e) Yongrui Chen, Haiyun Jiang, Xinting Huang, Shuming Shi, and Guilin Qi.  Tegit: Generating high-quality instruction-tuning data with text-grounded task design, 2023e.
  * Chen et al. (2024f) Zhipeng Chen, Kun Zhou, Wayne Xin Zhao, Junchen Wan, Fuzheng Zhang, Di Zhang, and Ji-Rong Wen.  Improving large language models via fine-grained reinforcement learning with minimum editing constraint, 2024f.
  * Chen et al. (2024g) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu.  Self-play fine-tuning converts weak language models to strong language models.  _ArXiv preprint_ , abs/2401.01335, 2024g.  URL https://arxiv.org/abs/2401.01335.
  * Cheng et al. (2024) Pengyu Cheng, Tianhao Hu, Han Xu, Zhisong Zhang, Yong Dai, Lei Han, and Nan Du.  Self-playing adversarial language game enhances llm reasoning.  _ArXiv preprint_ , abs/2404.10642, 2024.  URL https://arxiv.org/abs/2404.10642.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023.  URL https://lmsys.org/blog/2023-03-30-vicuna/.
  * Christiano et al. (2018) Paul Christiano, Buck Shlegeris, and Dario Amodei.  Supervising strong learners by amplifying weak experts.  _ArXiv preprint_ , abs/1810.08575, 2018.  URL https://arxiv.org/abs/1810.08575.
  * Christiano et al. (2017) Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , pages 4299–4307, 2017.  URL https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems, 2021.
  * Cui et al. (2024) Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun.  Ultrafeedback: Boosting language models with high-quality feedback, 2024.  URL https://openreview.net/forum?id=pNkOx3IVWI.
  * Dai et al. (2023) Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei.  Why can GPT learn in-context? language models secretly perform gradient descent as meta-optimizers.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 4005–4019, Toronto, Canada, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-acl.247.  URL https://aclanthology.org/2023.findings-acl.247.
  * Deng and Raffel (2023) Haikang Deng and Colin Raffel.  Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 11781–11791, Singapore, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.721.  URL https://aclanthology.org/2023.emnlp-main.721.
  * DiGiovanni and Zell (2021) Anthony DiGiovanni and Ethan C Zell.  Survey of self-play in reinforcement learning.  _ArXiv preprint_ , abs/2107.02850, 2021.  URL https://arxiv.org/abs/2107.02850.
  * Ding et al. (2023) Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou.  Enhancing chat language models by scaling high-quality instructional conversations.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 3029–3051, Singapore, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.183.  URL https://aclanthology.org/2023.emnlp-main.183.
  * Dong et al. (2023) Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang.  RAFT: Reward ranked finetuning for generative foundation model alignment.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=m7p5O7zblY.
  * Duan et al. (2023) Hanyu Duan, Yixuan Tang, Yi Yang, Ahmed Abbasi, and Kar Yan Tam.  Exploring the relationship between in-context learning and instruction tuning, 2023.  URL https://arxiv.org/abs/2311.10367.
  * Fernandes et al. (2023) Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat.  The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation.  In Philipp Koehn, Barry Haddow, Tom Kocmi, and Christof Monz, editors, _Proceedings of the Eighth Conference on Machine Translation_ , pages 1066–1083, Singapore, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.wmt-1.100.  URL https://aclanthology.org/2023.wmt-1.100.
  * Fernando et al. (2023) Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel.  Promptbreeder: Self-referential self-improvement via prompt evolution.  _ArXiv preprint_ , abs/2309.16797, 2023.  URL https://arxiv.org/abs/2309.16797.
  * Ferreira et al. (2023) Rafael EP Ferreira, Yong Jae Lee, and João RR Dórea.  Using pseudo-labeling to improve performance of deep neural networks for animal identification.  _Scientific Reports_ , 13(1):13875, 2023.  URL https://doi.org/10.1038/s41598-023-40977-x.
  * Forbes et al. (2020) Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi.  Social chemistry 101: Learning to reason about social and moral norms.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 653–670, Online, 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.48.  URL https://aclanthology.org/2020.emnlp-main.48.
  * Fränken et al. (2024) Jan-Philipp Fränken, Eric Zelikman, Rafael Rafailov, Kanishk Gandhi, Tobias Gerstenberg, and Noah D. Goodman.  Self-supervised alignment with mutual information: Learning to follow principles without preference labels, 2024.  URL https://arxiv.org/abs/2404.14313.
  * Fu et al. (2023a) Yao Fu, Hao Peng, Tushar Khot, and Mirella Lapata.  Improving language model negotiation with self-play and in-context learning from ai feedback.  _ArXiv preprint_ , abs/2305.10142, 2023a.  URL https://arxiv.org/abs/2305.10142.
  * Fu et al. (2023b) Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot.  Specializing smaller language models towards multi-step reasoning, 2023b.
  * Ganguli et al. (2022) Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al.  Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.  _ArXiv preprint_ , abs/2209.07858, 2022.  URL https://arxiv.org/abs/2209.07858.
  * Gao et al. (2023) Leo Gao, John Schulman, and Jacob Hilton.  Scaling laws for reward model overoptimization.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 10835–10866. PMLR, 2023.  URL https://proceedings.mlr.press/v202/gao23h.html.
  * Gao et al. (2024) Yang Gao, Dana Alon, and Donald Metzler.  Impact of preference noise on the alignment performance of generative language models, 2024.
  * Garrabrant and Demski (2018) Scott Garrabrant and Abram Demski.  Embedded agency, 2018.  URL https://www.alignmentforum.org/s/Rm6oQRJJmhGCcLvxh/p/i3BTagvt3HbPMx6PN.
  * Gekhman et al. (2024) Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig.  Does fine-tuning llms on new knowledge encourage hallucinations?, 2024.  URL https://arxiv.org/abs/2405.05904.
  * Gim et al. (2023) In Gim, Guojun Chen, Seung-seob Lee, Nikhil Sarda, Anurag Khandelwal, and Lin Zhong.  Prompt cache: Modular attention reuse for low-latency inference.  _ArXiv preprint_ , abs/2311.04934, 2023.  URL https://arxiv.org/abs/2311.04934.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio.  Generative adversarial nets.  In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors, _Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada_ , pages 2672–2680, 2014.  URL https://proceedings.neurips.cc/paper/2014/hash/5ca3e9b122f61f8f06494c97b1afccf3-Abstract.html.
  * Gou et al. (2024) Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen.  Critic: Large language models can self-correct with tool-interactive critiquing, 2024.  URL https://arxiv.org/abs/2305.11738.
  * Grandvalet and Bengio (2004) Yves Grandvalet and Yoshua Bengio.  Semi-supervised learning by entropy minimization.  In _Advances in Neural Information Processing Systems 17 [Neural Information Processing Systems, NIPS 2004, December 13-18, 2004, Vancouver, British Columbia, Canada]_ , pages 529–536, 2004.  URL https://proceedings.neurips.cc/paper/2004/hash/96f2b50b5d3613adf9c27049b2a888c7-Abstract.html.
  * Gu et al. (2024) Shangding Gu, Alois Knoll, and Ming Jin.  Teaching LLMs to teach themselves better instructions via reinforcement learning, 2024.  URL https://openreview.net/forum?id=wlRp8IdLkN.
  * Gudibande et al. (2023) Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song.  The false promise of imitating proprietary llms, 2023.  URL https://arxiv.org/abs/2305.15717.
  * Guo et al. (2024a) Hongyi Guo, Yuanshun Yao, Wei Shen, Jiaheng Wei, Xiaoying Zhang, Zhaoran Wang, and Yang Liu.  Human-instruction-free llm self-alignment with limited samples, 2024a.
  * Guo et al. (2024b) Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al.  Direct language model alignment from online ai feedback.  _ArXiv preprint_ , abs/2402.04792, 2024b.  URL https://arxiv.org/abs/2402.04792.
  * Hase et al. (2024) Peter Hase, Mohit Bansal, Peter Clark, and Sarah Wiegreffe.  The unreasonable effectiveness of easy training data for hard tasks, 2024.  URL https://arxiv.org/abs/2401.06751.
  * Havrilla et al. (2024) Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Railneau.  Glore: When, where, and how to improve llm reasoning via global and local refinements, 2024.
  * Hazra and Anjaria (2022) Tanmoy Hazra and Kushal Anjaria.  Applications of game theory in deep learning: a survey.  _Multimedia Tools and Applications_ , 81(6):8963–8994, 2022.
  * He et al. (2023) Guande He, Peng Cui, Jianfei Chen, Wenbo Hu, and Jun Zhu.  Investigating uncertainty calibration of aligned language models under the multiple-choice setting.  _ArXiv preprint_ , abs/2310.11732, 2023.  URL https://arxiv.org/abs/2310.11732.
  * He et al. (2020) Junxian He, Jiatao Gu, Jiajun Shen, and Marc’Aurelio Ranzato.  Revisiting self-training for neural sequence generation.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.  URL https://openreview.net/forum?id=SJgdnAVKDH.
  * Ho et al. (2023) Namgyu Ho, Laura Schmid, and Se-Young Yun.  Large language models are reasoning teachers.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 14852–14882, Toronto, Canada, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.830.  URL https://aclanthology.org/2023.acl-long.830.
  * Hoare (1961) Charles Antony Richard Hoare.  Algorithm 64: quicksort.  _Communications of the ACM_ , 4(7):321, 1961.
  * Hong et al. (2023) Jixiang Hong, Quan Tu, Changyu Chen, Xing Gao, Ji Zhang, and Rui Yan.  Cyclealign: Iterative distillation from black-box llm to white-box models for better human alignment, 2023.
  * Honovich et al. (2023) Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick.  Unnatural instructions: Tuning language models with (almost) no human labor.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 14409–14428, Toronto, Canada, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.806.  URL https://aclanthology.org/2023.acl-long.806.
  * Hosseini et al. (2024) Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal.  V-star: Training verifiers for self-taught reasoners, 2024.
  * Hsieh et al. (2023) Cheng-Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister.  Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 8003–8017, Toronto, Canada, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-acl.507.  URL https://aclanthology.org/2023.findings-acl.507.
  * Huang et al. (2023a) Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han.  Large language models can self-improve.  In _The 2023 Conference on Empirical Methods in Natural Language Processing_ , 2023a.  URL https://openreview.net/forum?id=uuUQraD4XX.
  * Huang et al. (2023b) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou.  Large language models cannot self-correct reasoning yet, 2023b.  URL https://arxiv.org/abs/2310.01798.
  * Irving et al. (2018) Geoffrey Irving, Paul Christiano, and Dario Amodei.  Ai safety via debate.  _ArXiv preprint_ , abs/1805.00899, 2018.  URL https://arxiv.org/abs/1805.00899.
  * Jacob et al. (2024) Athul Paul Jacob, Yikang Shen, Gabriele Farina, and Jacob Andreas.  The consensus game: Language model generation via equilibrium search.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=n9xeGcI4Yg.
  * Ji et al. (2023) Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, Fanzhi Zeng, Kwan Yee Ng, Juntao Dai, Xuehai Pan, Aidan O’Gara, Yingshan Lei, Hua Xu, Brian Tse, Jie Fu, Stephen McAleer, Yaodong Yang, Yizhou Wang, Song-Chun Zhu, Yike Guo, and Wen Gao.  AI Alignment: A Comprehensive Survey, 2023.  URL https://arxiv.org/abs/2310.19852.
  * Ji et al. (2024) Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, and Yaodong Yang.  Aligner: Achieving efficient alignment through weak-to-strong correction, 2024.
  * Jiang et al. (2023) Shuyang Jiang, Yuhao Wang, and Yu Wang.  Selfevolve: A code evolution framework via large language models, 2023.
  * Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al.  Language models (mostly) know what they know.  _ArXiv preprint_ , abs/2207.05221, 2022.  URL https://arxiv.org/abs/2207.05221.
  * Khalifa et al. (2023) Muhammad Khalifa, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, and Lu Wang.  GRACE: Discriminator-guided chain-of-thought reasoning.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 15299–15328, Singapore, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-emnlp.1022.  URL https://aclanthology.org/2023.findings-emnlp.1022.
  * Khan et al. (2024) Akbir Khan, John Hughes, Dan Valentine, Laura Ruis, Kshitij Sachan, Ansh Radhakrishnan, Edward Grefenstette, Samuel R Bowman, Tim Rocktäschel, and Ethan Perez.  Debating with more persuasive llms leads to more truthful answers.  _ArXiv preprint_ , abs/2402.06782, 2024.  URL https://arxiv.org/abs/2402.06782.
  * Khot et al. (2023) Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal.  Decomposed prompting: A modular approach for solving complex tasks.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=_nGgzQjzaRy.
  * Kim et al. (2023) Sungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak, Kang Yoo, and Minjoon Seo.  Aligning large language models through synthetic feedback.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 13677–13700, Singapore, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.844.  URL https://aclanthology.org/2023.emnlp-main.844.
  * Klingefjord et al. (2024) Oliver Klingefjord, Ryan Lowe, and Joe Edelman.  What are human values, and how do we align ai to them?, 2024.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  _Advances in neural information processing systems_ , 35:22199–22213, 2022.
  * Koutcheme et al. (2024) Charles Koutcheme, Nicola Dainese, Sami Sarsa, Arto Hellas, Juho Leinonen, and Paul Denny.  Open source language models can provide feedback: Evaluating llms’ ability to help students using gpt-4-as-a-judge, 2024.
  * Köksal et al. (2024) Abdullatif Köksal, Timo Schick, Anna Korhonen, and Hinrich Schütze.  Longform: Effective instruction tuning with reverse instructions, 2024.
  * Lan et al. (2024) Tian Lan, Wenwei Zhang, Chen Xu, Heyan Huang, Dahua Lin, Kai Chen, and Xian-ling Mao.  Criticbench: Evaluating large language models as critic, 2024.  URL https://arxiv.org/abs/2402.13764.
  * Lang et al. (2024) Hunter Lang, David Sontag, and Aravindan Vijayaraghavan.  Theoretical analysis of weak-to-strong generalization, 2024.
  * Lango and Dusek (2023) Mateusz Lango and Ondrej Dusek.  Critic-driven decoding for mitigating hallucinations in data-to-text generation.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 2853–2862, Singapore, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.172.  URL https://aclanthology.org/2023.emnlp-main.172.
  * Le et al. (2022) Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven C. H. Hoi.  Coderl: Mastering code generation through pretrained models and deep reinforcement learning, 2022.
  * Lee et al. (2013) Dong-Hyun Lee et al.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In _Workshop on challenges in representation learning, ICML_ , volume 3, page 896. Atlanta, 2013.
  * Lee et al. (2023) Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash.  Rlaif: Scaling reinforcement learning from human feedback with ai feedback, 2023.
  * Lee et al. (2024) Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipali, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami.  Llm2llm: Boosting llms with novel iterative data enhancement, 2024.
  * Li et al. (2023a) Chengpeng Li, Zheng Yuan, Hongyi Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou.  Query and response augmentation cannot help out-of-domain math reasoning generalization, 2023a.
  * Li et al. (2024a) Dexun Li, Cong Zhang, Kuicai Dong, Derrick Goh Xin Deik, Ruiming Tang, and Yong Liu.  Aligning crowd feedback via distributional preference reward modeling, 2024a.
  * Li et al. (2023b) Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, and Tianyi Zhou.  Reflection-tuning: Recycling data for better instruction-tuning.  In _NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following_ , 2023b.  URL https://openreview.net/forum?id=xaqoZZqkPU.
  * Li et al. (2024b) Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, and Tianyi Zhou.  Selective reflection-tuning: Student-selected data recycling for llm instruction-tuning, 2024b.
  * Li et al. (2024c) Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou.  Superfiltering: Weak-to-strong data filtering for fast instruction-tuning, 2024c.
  * Li et al. (2024d) Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao.  From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning, 2024d.  URL https://arxiv.org/abs/2308.12032.
  * Li et al. (2022) Shiyang Li, Jianshu Chen, Yelong Shen, Zhiyu Chen, Xinlu Zhang, Zekun Li, Hong Wang, Jing Qian, Baolin Peng, Yi Mao, Wenhu Chen, and Xifeng Yan.  Explanations from large language models make small reasoners better, 2022.
  * Li et al. (2024e) Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis.  Self-alignment with instruction backtranslation.  In _The Twelfth International Conference on Learning Representations_ , volume abs/2308.06259, 2024e.  URL https://openreview.net/forum?id=1oijHJBRsT.
  * Li et al. (2024f) Xiang Lisa Li, Vaishnavi Shrivastava, Siyan Li, Tatsunori Hashimoto, and Percy Liang.  Benchmarking and improving generator-validator consistency of language models.  In _The Twelfth International Conference on Learning Representations_ , volume abs/2310.01846, 2024f.  URL https://openreview.net/forum?id=phBS6YpTzC.
  * Li et al. (2023c) Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen.  Making language models better reasoners with step-aware verifier.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5315–5333, Toronto, Canada, 2023c. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.291.  URL https://aclanthology.org/2023.acl-long.291.
  * Li et al. (2023d) Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee.  Textbooks are all you need ii: phi-1.5 technical report, 2023d.
  * Li et al. (2024g) Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang.  RAIN: Your language models can align themselves without finetuning.  In _The Twelfth International Conference on Learning Representations_ , volume abs/2309.07124, 2024g.  URL https://openreview.net/forum?id=pETSfWMUzy.
  * Li et al. (2023e) Zongjie Li, Chaozheng Wang, Pingchuan Ma, Daoyuan Wu, Shuai Wang, Cuiyun Gao, and Yang Liu.  Split and merge: Aligning position biases in large language model based evaluators, 2023e.  URL https://arxiv.org/abs/2310.01432.
  * Liao et al. (2024) Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan.  Mario: Math reasoning with code interpreter output – a reproducible pipeline, 2024.
  * Lightman et al. (2023) Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.  Let’s verify step by step, 2023.
  * Lin et al. (2024a) Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi.  The unlocking spell on base LLMs: Rethinking alignment via in-context learning.  In _The Twelfth International Conference on Learning Representations_ , volume abs/2312.01552, 2024a.  URL https://openreview.net/forum?id=wxJ0eXwwda.
  * Lin et al. (2024b) Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujiu Yang.  Criticbench: Benchmarking llms for critique-correct reasoning.  _ArXiv preprint_ , abs/2402.14809, 2024b.  URL https://arxiv.org/abs/2402.14809.
  * Liu et al. (2024a) Aiwei Liu, Haoping Bai, Zhiyun Lu, Xiang Kong, Simon Wang, Jiulong Shan, Meng Cao, and Lijie Wen.  Direct large language model alignment through self-rewarding contrastive prompt distillation, 2024a.  URL https://arxiv.org/abs/2402.11907.
  * Liu et al. (2023a) Ruibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M. Dai, Diyi Yang, and Soroush Vosoughi.  Training socially aligned language models on simulated social interactions, 2023a.
  * Liu et al. (2024b) Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, and Mathieu Blondel.  Decoding-time realignment of language models, 2024b.
  * Liu et al. (2024c) Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He.  What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning.  In _The Twelfth International Conference on Learning Representations_ , 2024c.  URL https://openreview.net/forum?id=BTKAeLqLMw.
  * Liu et al. (2023b) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.  G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023b.  URL https://arxiv.org/abs/2303.16634.
  * Liu et al. (2023c) Yixin Liu, Avi Singh, C. Daniel Freeman, John D. Co-Reyes, and Peter J. Liu.  Improving large language model fine-tuning for solving math problems, 2023c.
  * Liu and Alahi (2024) Yuejiang Liu and Alexandre Alahi.  Co-supervised learning: Improving weak-to-strong generalization with hierarchical mixture of experts, 2024.
  * Liu et al. (2023d) Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, and Qi Zhang.  Calibrating llm-based evaluator, 2023d.  URL https://arxiv.org/abs/2309.13308.
  * Lu et al. (2022) Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi.  Quark: Controllable text generation with reinforced unlearning, 2022.
  * Lu et al. (2024) Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li.  Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms, 2024.
  * Luo et al. (2023a) Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang.  Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct, 2023a.
  * Luo et al. (2023b) Liangchen Luo, Zi Lin, Yinxiao Liu, Lei Shu, Yun Zhu, Jingbo Shang, and Lei Meng.  Critique ability of large language models, 2023b.
  * Luo et al. (2024) Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang.  Wizardcoder: Empowering code large language models with evol-instruct.  In _The Twelfth International Conference on Learning Representations_ , 2024.  URL https://openreview.net/forum?id=UnUwSIgK5W.
  * Ma et al. (2023a) Chengdong Ma, Ziran Yang, Minquan Gao, Hai Ci, Jun Gao, Xuehai Pan, and Yaodong Yang.  Red teaming game: A game-theoretic framework for red teaming language models.  _ArXiv preprint_ , abs/2310.00322, 2023a.  URL https://arxiv.org/abs/2310.00322.
  * Ma et al. (2023b) Qianli Ma, Haotian Zhou, Tingkai Liu, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang.  Let’s reward step by step: Step-level reward model as the navigators for reasoning, 2023b.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark.  Self-refine: Iterative refinement with self-feedback.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , volume 36, pages 46534–46594, 2023.  URL https://openreview.net/forum?id=S37hOerQLB.
  * Magister et al. (2023) Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn.  Teaching small language models to reason.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1773–1781, Toronto, Canada, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-short.151.  URL https://aclanthology.org/2023.acl-short.151.
  * Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark Gales.  SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 9004–9017, Singapore, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.557.  URL https://aclanthology.org/2023.emnlp-main.557.
  * Mecklenburg et al. (2024) Nick Mecklenburg, Yiyou Lin, Xiaoxiao Li, Daniel Holstein, Leonardo Nunes, Sara Malvar, Bruno Silva, Ranveer Chandra, Vijay Aski, Pavan Kumar Reddy Yannam, Tolga Aktas, and Todd Hendry.  Injecting new knowledge into large language models via supervised fine-tuning, 2024.  URL https://arxiv.org/abs/2404.00213.
  * Mitchell (1980) Tom M Mitchell.  The need for biases in learning generalizations.  1980\.
  * Mitra et al. (2023) Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, and Ahmed Awadallah.  Orca 2: Teaching small language models how to reason, 2023.
  * Mudgal et al. (2024) Sidharth Mudgal, Jong Lee, Harish Ganapathy, YaGuang Li, Tao Wang, Yanping Huang, Zhifeng Chen, Heng-Tze Cheng, Michael Collins, Trevor Strohman, Jilin Chen, Alex Beutel, and Ahmad Beirami.  Controlled decoding from language models, 2024.
  * Mukherjee et al. (2023) Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah.  Orca: Progressive learning from complex explanation traces of gpt-4, 2023.
  * Nash et al. (1950) John F Nash et al.  The bargaining problem.  _Econometrica_ , 18(2):155–162, 1950.
  * Nash et al. (1951) John F Nash et al.  Non-cooperative games.  page 286–295, 1951.
  * Nigam and Ghani (2000) Kamal Nigam and Rayid Ghani.  Analyzing the effectiveness and applicability of co-training.  In _Proceedings of the ninth international conference on Information and knowledge management_ , pages 86–93, 2000.
  * OpenAI (2023a) OpenAI.  Preparedness framework (beta), 2023a.  URL https://cdn.openai.com/openai-preparedness-framework-beta.pdf.
  * OpenAI (2023b) OpenAI.  GPT-4 Technical Report, 2023b.  URL https://arxiv.org/abs/2303.08774.
  * Ought (2017) Ought.  Factored cognition, 2017.  https://ought.org/research/factored-cognition.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback, 2022.  URL https://arxiv.org/abs/2203.02155.
  * Pace et al. (2024) Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn.  West-of-n: Synthetic preference generation for improved reward modeling.  _ArXiv preprint_ , abs/2401.12086, 2024.  URL https://arxiv.org/abs/2401.12086.
  * Padmanabhan et al. (2023) Shankar Padmanabhan, Yasumasa Onoe, Michael JQ Zhang, Greg Durrett, and Eunsol Choi.  Propagating knowledge updates to LMs through distillation.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=DFaGf3O7jf.
  * Pan et al. (2023) Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Hanlin Zhang, Scott Emmons, and Dan Hendrycks.  Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark.  In _International Conference on Machine Learning_ , pages 26837–26867. PMLR, 2023.
  * Park et al. (2023) Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein.  Generative agents: Interactive simulacra of human behavior, 2023.
  * Patil et al. (2023) Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez.  Gorilla: Large language model connected with massive apis, 2023.
  * Peng et al. (2023) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao.  Instruction tuning with gpt-4, 2023.
  * Polu and Sutskever (2020) Stanislas Polu and Ilya Sutskever.  Generative language modeling for automated theorem proving.  _ArXiv preprint_ , abs/2009.03393, 2020.  URL https://arxiv.org/abs/2009.03393.
  * Qiao et al. (2024) Shuofei Qiao, Honghao Gui, Chengfei Lv, Qianghuai Jia, Huajun Chen, and Ningyu Zhang.  Making language models better tool learners with execution feedback, 2024.
  * Qin et al. (2023) Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun.  Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn.  Direct Preference Optimization: Your Language Model is Secretly a Reward Model, 2023.  URL https://arxiv.org/abs/2305.18290.
  * Rame et al. (2023) Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord.  Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=lSbbC2VyCu.
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International conference on machine learning_ , pages 8821–8831. Pmlr, 2021.
  * Ren et al. (2024) Mengjie Ren, Boxi Cao, Hongyu Lin, Cao Liu, Xianpei Han, Ke Zeng, Guanglu Wan, Xunliang Cai, and Le Sun.  Learning or self-aligning? rethinking instruction fine-tuning, 2024.  URL https://arxiv.org/abs/2402.18243.
  * Roy et al. (2021) Nicholas Roy, Ingmar Posner, Tim D. Barfoot, Philippe Beaudoin, Yoshua Bengio, Jeannette Bohg, Oliver Brock, Isabelle Depatie, Dieter Fox, Daniel E. Koditschek, Tomás Lozano-Pérez, Vikash Mansinghka, Christopher J. Pal, Blake A. Richards, Dorsa Sadigh, Stefan Schaal, Gaurav S. Sukhatme, Denis Thérien, Marc Toussaint, and Michiel van de Panne.  From machine learning to robotics: Challenges and opportunities for embodied intelligence.  _ArXiv preprint_ , abs/2110.15245, 2021.  URL https://arxiv.org/abs/2110.15245.
  * Saunders et al. (2022) William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike.  Self-critiquing models for assisting human evaluators.  _ArXiv preprint_ , abs/2206.05802, 2022.  URL https://arxiv.org/abs/2206.05802.
  * Scheurer et al. (2022) Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez.  Training language models with language feedback, 2022.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms, 2017.
  * Scudder (1965) H. Scudder.  Probability of error of some adaptive pattern-recognition machines.  _IEEE Transactions on Information Theory_ , 11(3):363–371, 1965.  doi: 10.1109/TIT.1965.1053799.
  * Shao et al. (2024) Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo.  Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
  * Shapley (1971) Lloyd S Shapley.  Cores of convex games.  _International journal of game theory_ , 1:11–26, 1971.
  * Sharma et al. (2024) Archit Sharma, Sedrick Keh, Eric Mitchell, Chelsea Finn, Kushal Arora, and Thomas Kollar.  A critical evaluation of ai feedback for aligning large language models.  _ArXiv preprint_ , abs/2402.12366, 2024.  URL https://arxiv.org/abs/2402.12366.
  * Shavit et al. (2023) Yonadav Shavit, Sandhini Agarwal, Miles Brundage, Steven Adler, Cullen O’Keefe, Rosie Campbell, Teddy Lee, Pamela Mishkin, Tyna Eloundou, Alan Hickey, et al.  Practices for governing agentic ai systems.  2023\.
  * Shen et al. (2023a) Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing.  Large language models are not yet human-level evaluators for abstractive summarization.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 4215–4233, Singapore, 2023a. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-emnlp.278.  URL https://aclanthology.org/2023.findings-emnlp.278.
  * Shen et al. (2023b) Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, and Deyi Xiong.  Large Language Model Alignment: A Survey, 2023b.  URL https://arxiv.org/abs/2309.15025.
  * Shi et al. (2023) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou.  Large language models can be easily distracted by irrelevant context.  In _International Conference on Machine Learning_ , pages 31210–31227. PMLR, 2023.
  * Shi et al. (2024) Taiwei Shi, Kai Chen, and Jieyu Zhao.  Safer-instruct: Aligning language models with automated preference data, 2024.
  * Shinn et al. (2023) Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.  Reflexion: Language agents with verbal reinforcement learning, 2023.
  * Shridhar et al. (2023) Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan.  Distilling reasoning capabilities into smaller language models.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 7059–7073, Toronto, Canada, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-acl.441.  URL https://aclanthology.org/2023.findings-acl.441.
  * Silver et al. (2018) David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  _Science_ , 362(6419):1140–1144, 2018.
  * Singh et al. (2024) Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron T Parisi, Abhishek Kumar, Alexander A Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Fathy Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura A Culp, Lechao Xiao, Maxwell Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel.  Beyond human data: Scaling self-training for problem-solving with language models.  _Transactions on Machine Learning Research_ , 2024.  ISSN 2835-8856.  URL https://openreview.net/forum?id=lNAyUngGFK.  Expert Certification.
  * Singhal et al. (2023) Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Nenad Tomasev, Yun Liu, Renee Wong, Christopher Semturs, S. Sara Mahdavi, Joelle Barral, Dale Webster, Greg S. Corrado, Yossi Matias, Shekoofeh Azizi, Alan Karthikesalingam, and Vivek Natarajan.  Towards expert-level medical question answering with large language models, 2023.  URL https://arxiv.org/abs/2305.09617.
  * Somerstep et al. (2024) Seamus Somerstep, Felipe Maia Polo, Moulinath Banerjee, Ya’acov Ritov, Mikhail Yurochkin, and Yuekai Sun.  A statistical framework for weak-to-strong generalization, 2024.
  * Song et al. (2023) Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang.  Preference Ranking Optimization for Human Alignment, 2023.  URL https://arxiv.org/abs/2306.17492.
  * Song et al. (2024a) Feifan Song, Bowen Yu, Hao Lang, Haiyang Yu, Fei Huang, Houfeng Wang, and Yongbin Li.  Scaling data diversity for fine-tuning language models in human alignment.  In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_ , pages 14358–14369, Torino, Italia, 2024a. ELRA and ICCL.  URL https://aclanthology.org/2024.lrec-main.1251.
  * Song et al. (2024b) Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin.  Trial and error: Exploration-based trajectory optimization for llm agents, 2024b.
  * Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano.  Learning to summarize with human feedback.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/1f89885d556929e98d3ef9b86448f951-Abstract.html.
  * StockFish (2023) StockFish.  Stockfish - open source chess engine, 2023.  https://stockfishchess.org/.
  * Sun et al. (2023a) Hao Sun, Zhexin Zhang, Fei Mi, Yasheng Wang, Wei Liu, Jianwei Cui, Bin Wang, Qun Liu, and Minlie Huang.  MoralDial: A framework to train and evaluate moral dialogue systems via moral discussions.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2213–2230, Toronto, Canada, 2023a. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.123.  URL https://aclanthology.org/2023.acl-long.123.
  * Sun et al. (2024a) Shichao Sun, Junlong Li, Weizhe Yuan, Ruifeng Yuan, Wenjie Li, and Pengfei Liu.  The critique of critique.  _ArXiv preprint_ , abs/2401.04518, 2024a.  URL https://arxiv.org/abs/2401.04518.
  * Sun et al. (2023b) Yuchong Sun, Che Liu, Jinwen Huang, Ruihua Song, Fuzheng Zhang, Di Zhang, Zhongyuan Wang, and Kun Gai.  Parrot: Enhancing multi-turn chat models by learning to ask questions, 2023b.
  * Sun et al. (2023c) Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan.  Salmon: Self-alignment with principle-following reward models, 2023c.  URL https://openreview.net/forum?id=xJbsmB8UMx.
  * Sun et al. (2023d) Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan.  Principle-driven self-alignment of language models from scratch with minimal human supervision.  In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, _Advances in Neural Information Processing Systems_ , volume 36, pages 2511–2565. Curran Associates, Inc., 2023d.  URL https://proceedings.neurips.cc/paper_files/paper/2023/file/0764db1151b936aca59249e2c1386101-Paper-Conference.pdf.
  * Sun et al. (2024b) Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan.  Easy-to-hard generalization: Scalable alignment beyond human supervision, 2024b.  URL https://arxiv.org/abs/2403.09472.
  * Tan et al. (2024) Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An.  True knowledge comes from practice: Aligning llms with embodied environments via reinforcement learning, 2024.
  * Tan et al. (2023) Xiaoyu Tan, Shaojie Shi, Xihe Qiu, Chao Qu, Zhenting Qi, Yinghui Xu, and Yuan Qi.  Self-criticism: Aligning large language models with their understanding of helpfulness, honesty, and harmlessness.  In Mingxuan Wang and Imed Zitouni, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track_ , pages 650–662, Singapore, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-industry.62.  URL https://aclanthology.org/2023.emnlp-industry.62.
  * Tang et al. (2023) Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun.  Toolalpaca: Generalized tool learning for language models with 3000 simulated cases.  _ArXiv preprint_ , abs/2306.05301, 2023.  URL https://arxiv.org/abs/2306.05301.
  * Tao et al. (2024) Zhengwei Tao, Ting-En Lin, Xiancai Chen, Hangyu Li, Yuchuan Wu, Yongbin Li, Zhi Jin, Fei Huang, Dacheng Tao, and Jingren Zhou.  A survey on self-evolution of large language models.  _arXiv preprint arXiv:2404.14387_ , 2024.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Taubenfeld et al. (2024) Amir Taubenfeld, Yaniv Dover, Roi Reichart, and Ariel Goldstein.  Systematic biases in llm simulations of debates, 2024.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _ArXiv preprint_ , abs/2307.09288, 2023.  URL https://arxiv.org/abs/2307.09288.
  * Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf.  Zephyr: Direct distillation of lm alignment, 2023.
  * Ubani et al. (2023) Solomon Ubani, Suleyman Olcay Polat, and Rodney Nielsen.  Zeroshotdataaug: Generating and augmenting training data with chatgpt, 2023.
  * Uesato et al. (2022) Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins.  Solving math word problems with process- and outcome-based feedback, 2022.
  * Ulmer et al. (2024) Dennis Ulmer, Elman Mansimov, Kaixiang Lin, Justin Sun, Xibin Gao, and Yi Zhang.  Bootstrapping llm-based task-oriented dialogue agents via self-talk, 2024.
  * von Oswald et al. (2023) Johannes von Oswald, Eyvind Niklasson, Maximilian Schlegel, Seijin Kobayashi, Nicolas Zucchet, Nino Scherrer, Nolan Miller, Mark Sandler, Max Vladymyrov, Razvan Pascanu, et al.  Uncovering mesa-optimization algorithms in transformers.  _ArXiv preprint_ , abs/2309.05858, 2023.  URL https://arxiv.org/abs/2309.05858.
  * Wang et al. (2024a) Boshi Wang, Hao Fang, Jason Eisner, Benjamin Van Durme, and Yu Su.  Llms in the imaginarium: Tool learning through simulated trial and error, 2024a.
  * Wang et al. (2024b) Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, and Yang Liu.  Openchat: Advancing open-source language models with mixed-quality data.  In _The Twelfth International Conference on Learning Representations_ , 2024b.  URL https://openreview.net/forum?id=AOJyfhWYHf.
  * Wang et al. (2023a) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An open-ended embodied agent with large language models, 2023a.
  * Wang et al. (2024c) Jiahao Wang, Bolin Zhang, Qianlong Du, Jiajun Zhang, and Dianhui Chu.  A survey on data selection for llm instruction tuning, 2024c.
  * Wang et al. (2024d) Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li.  Mathcoder: Seamless code integration in LLMs for enhanced mathematical reasoning.  In _The Twelfth International Conference on Learning Representations_ , 2024d.  URL https://openreview.net/forum?id=z8TW0ttBPp.
  * Wang et al. (2023b) Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim.  Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2609–2634, Toronto, Canada, 2023b. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.147.  URL https://aclanthology.org/2023.acl-long.147.
  * Wang et al. (2023c) Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui.  Large language models are not fair evaluators, 2023c.  URL https://arxiv.org/abs/2305.17926.
  * Wang et al. (2024e) Peiyi Wang, Lei Li, Zhihong Shao, R. X. Xu, Damai Dai, Yifei Li, Deli Chen, Y. Wu, and Zhifang Sui.  Math-shepherd: Verify and reinforce llms step-by-step without human annotations, 2024e.
  * Wang et al. (2024f) Ruiyi Wang, Haofei Yu, Wenxin Zhang, Zhengyang Qi, Maarten Sap, Graham Neubig, Yonatan Bisk, and Hao Zhu.  Sotopia-$\pi$: Interactive learning of socially intelligent language agents, 2024f.
  * Wang et al. (2021) Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng.  Want to reduce labeling cost? GPT-3 can help.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4195–4205, Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.findings-emnlp.354.  URL https://aclanthology.org/2021.findings-emnlp.354.
  * Wang et al. (2023d) Tianlu Wang, Ping Yu, Xiaoqing Ellen Tan, Sean O’Brien, Ramakanth Pasunuru, Jane Dwivedi-Yu, Olga Golovneva, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz.  Shepherd: A critic for language model generation, 2023d.  URL https://arxiv.org/abs/2308.04592.
  * Wang et al. (2024g) Xinpeng Wang, Shitong Duan, Xiaoyuan Yi, Jing Yao, Shanlin Zhou, Zhihua Wei, Peng Zhang, Dongkuan Xu, Maosong Sun, and Xing Xie.  On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models, 2024g.  URL https://arxiv.org/abs/2403.04204.
  * Wang and Zhou (2024) Xuezhi Wang and Denny Zhou.  Chain-of-thought reasoning without prompting.  _ArXiv preprint_ , abs/2402.10200, 2024.  URL https://arxiv.org/abs/2402.10200.
  * Wang et al. (2023e) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou.  Self-consistency improves chain of thought reasoning in language models.  In _The Eleventh International Conference on Learning Representations_ , 2023e.  URL https://openreview.net/forum?id=1PL1NIMMrw.
  * Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen.  Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5085–5109, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics.  URL https://aclanthology.org/2022.emnlp-main.340.
  * Wang et al. (2023f) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-instruct: Aligning language models with self-generated instructions.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13484–13508, Toronto, Canada, 2023f. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.754.  URL https://aclanthology.org/2023.acl-long.754.
  * Wang et al. (2023g) Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu.  Aligning Large Language Models with Human: A Survey, 2023g.  URL https://arxiv.org/abs/2307.12966.
  * Wang et al. (2024h) Zihan Wang, Yunxuan Li, Yuexin Wu, Liangchen Luo, Le Hou, Hongkun Yu, and Jingbo Shang.  Multi-step problem solving through a verifier: An empirical analysis on model-induced process supervision, 2024h.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837, 2022.
  * Wei et al. (2024) Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, et al.  Long-form factuality in large language models.  _arXiv preprint arXiv:2403.18802_ , 2024.
  * Wei et al. (2023) Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang.  Magicoder: Source code is all you need, 2023.
  * Weng et al. (2023) Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao.  Large language models are better reasoners with self-verification.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 2550–2575, Singapore, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.findings-emnlp.167.  URL https://aclanthology.org/2023.findings-emnlp.167.
  * West et al. (2023) Peter West, Ximing Lu, Nouha Dziri, Faeze Brahman, Linjie Li, Jena D. Hwang, Liwei Jiang, Jillian Fisher, Abhilasha Ravichander, Khyathi Chandu, Benjamin Newman, Pang Wei Koh, Allyson Ettinger, and Yejin Choi.  The generative ai paradox: ”what it can create, it may not understand”, 2023.  URL https://arxiv.org/abs/2311.00059.
  * Weyssow et al. (2024) Martin Weyssow, Aton Kamanda, and Houari Sahraoui.  Codeultrafeedback: An llm-as-a-judge dataset for aligning large language models to coding preferences, 2024.
  * Wilf (2002) Herbert S Wilf.  _Algorithms and complexity_.  AK Peters/CRC Press, 2002.
  * Wu et al. (2021a) Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano.  Recursively summarizing books with human feedback.  _ArXiv preprint_ , abs/2109.10862, 2021a.  URL https://arxiv.org/abs/2109.10862.
  * Wu and Aji (2023) Minghao Wu and Alham Fikri Aji.  Style over substance: Evaluation biases for large language models, 2023.  URL https://arxiv.org/abs/2307.03025.
  * Wu et al. (2024) Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Aji.  LaMini-LM: A diverse herd of distilled models from large-scale instructions.  In Yvette Graham and Matthew Purver, editors, _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 944–964, St. Julian’s, Malta, 2024. Association for Computational Linguistics.  URL https://aclanthology.org/2024.eacl-long.57.
  * Wu et al. (2021b) Qingyang Wu, Lei Li, and Zhou Yu.  Textgail: Generative adversarial imitation learning for text generation.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , pages 14067–14075. AAAI Press, 2021b.  URL https://ojs.aaai.org/index.php/AAAI/article/view/17656.
  * Wu et al. (2023a) Xuansheng Wu, Wenlin Yao, Jianshu Chen, Xiaoman Pan, Xiaoyang Wang, Ninghao Liu, and Dong Yu.  From language modeling to instruction following: Understanding the behavior shift in llms after instruction tuning, 2023a.  URL https://arxiv.org/abs/2310.00492.
  * Wu et al. (2023b) Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi.  Fine-grained human feedback gives better rewards for language model training.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023b.  URL https://openreview.net/forum?id=CSbGXyCswu.
  * Xiang et al. (2023) Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu.  Language models meet world models: Embodied experiences enhance language models, 2023.
  * Xie et al. (2023) Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min-Yen Kan, Junxian He, and Qizhe Xie.  Self-evaluation guided beam search for reasoning.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=Bw82hwg5Q3.
  * Xu et al. (2023a) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.  Wizardlm: Empowering large language models to follow complex instructions.  _ArXiv preprint_ , abs/2304.12244, 2023a.  URL https://arxiv.org/abs/2304.12244.
  * Xu et al. (2024a) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang.  WizardLM: Empowering large pre-trained language models to follow complex instructions.  In _The Twelfth International Conference on Learning Representations_ , 2024a.  URL https://openreview.net/forum?id=CfXh93NDgH.
  * Xu et al. (2023b) Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley.  Baize: An open-source chat model with parameter-efficient tuning on self-chat data.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 6268–6278, Singapore, 2023b. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.385.  URL https://aclanthology.org/2023.emnlp-main.385.
  * Xu et al. (2023c) Chunpu Xu, Steffi Chern, Ethan Chern, Ge Zhang, Zekun Wang, Ruibo Liu, Jing Li, Jie Fu, and Pengfei Liu.  Align on the fly: Adapting chatbot behavior to established norms.  _ArXiv preprint_ , abs/2312.15907, 2023c.  URL https://arxiv.org/abs/2312.15907.
  * Xu et al. (2024b) Jie Xu, Hanbo Zhang, Xinghang Li, Huaping Liu, Xuguang Lan, and Tao Kong.  Sinvig: A self-evolving interactive visual agent for human-robot interaction, 2024b.
  * Xu et al. (2024c) Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou.  A survey on knowledge distillation of large language models, 2024c.
  * Xue et al. (2023) Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You.  To repeat or not to repeat: Insights from scaling llm under token-crisis.  In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, _Advances in Neural Information Processing Systems_ , volume 36, pages 59304–59322. Curran Associates, Inc., 2023.  URL https://proceedings.neurips.cc/paper_files/paper/2023/file/b9e472cd579c83e2f6aa3459f46aac28-Paper-Conference.pdf.
  * Yang et al. (2023a) Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen.  Large language models as optimizers.  _ArXiv preprint_ , abs/2309.03409, 2023a.  URL https://arxiv.org/abs/2309.03409.
  * Yang et al. (2024a) Kailai Yang, Zhiwei Liu, Qianqian Xie, Tianlin Zhang, Nirui Song, Jimin Huang, Ziyan Kuang, and Sophia Ananiadou.  Metaaligner: Conditional weak-to-strong correction for generalizable multi-objective alignment of language models, 2024a.
  * Yang et al. (2024b) Kevin Yang, Dan Klein, Asli Celikyilmaz, Nanyun Peng, and Yuandong Tian.  RLCD: Reinforcement learning from contrastive distillation for LM alignment.  In _The Twelfth International Conference on Learning Representations_ , 2024b.  URL https://openreview.net/forum?id=v3XXtxWKi6.
  * Yang et al. (2023b) Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan.  Gpt4tools: Teaching large language model to use tools via self-instruction.  In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, _Advances in Neural Information Processing Systems_ , volume 36, pages 71995–72007. Curran Associates, Inc., 2023b.  URL https://proceedings.neurips.cc/paper_files/paper/2023/file/e393677793767624f2821cec8bdd02f1-Paper-Conference.pdf.
  * Yang and Wang (2020) Yaodong Yang and Jun Wang.  An overview of multi-agent reinforcement learning from game theoretical perspective.  _ArXiv preprint_ , abs/2011.00583, 2020.  URL https://arxiv.org/abs/2011.00583.
  * Yang et al. (2024c) Zonghan Yang, Peng Li, Ming Yan, Ji Zhang, Fei Huang, and Yang Liu.  React meets actre: When language agents enjoy training data autonomy, 2024c.
  * Yang et al. (2024d) Zonghan Yang, An Liu, Zijun Liu, Kaiming Liu, Fangzhou Xiong, Yile Wang, Zeyuan Yang, Qingyuan Hu, Xinrui Chen, Zhenhe Zhang, et al.  Towards unified alignment between agents, humans, and environment.  _ArXiv preprint_ , abs/2402.07744, 2024d.  URL https://arxiv.org/abs/2402.07744.
  * Yao et al. (2023a) Jing Yao, Xiaoyuan Yi, Xiting Wang, Jindong Wang, and Xing Xie.  From Instructions to Intrinsic Human Values – A Survey of Alignment Goals for Big Models, 2023a.  URL https://arxiv.org/abs/2308.12014.
  * Yao et al. (2022) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao.  React: Synergizing reasoning and acting in language models.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Yao et al. (2023b) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan.  Tree of thoughts: Deliberate problem solving with large language models.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023b.  URL https://openreview.net/forum?id=5Xc1ecxO1h.
  * Yao et al. (2021) Yuan Yao, Haoxi Zhong, Zhengyan Zhang, Xu Han, Xiaozhi Wang, Kai Zhang, Chaojun Xiao, Guoyang Zeng, Zhiyuan Liu, and Maosong Sun.  Adversarial language games for advanced natural language intelligence.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , pages 14248–14256. AAAI Press, 2021.  URL https://ojs.aaai.org/index.php/AAAI/article/view/17676.
  * Yin et al. (2023) Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, and Kai-Wei Chang.  Dynosaur: A dynamic growth paradigm for instruction-tuning data curation.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 4031–4047, Singapore, 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.245.  URL https://aclanthology.org/2023.emnlp-main.245.
  * Ying et al. (2024) Huaiyuan Ying, Shuo Zhang, Linyang Li, Zhejian Zhou, Yunfan Shao, Zhaoye Fei, Yichuan Ma, Jiawei Hong, Kuikun Liu, Ziyi Wang, Yudong Wang, Zijian Wu, Shuaibin Li, Fengzhe Zhou, Hongwei Liu, Songyang Zhang, Wenwei Zhang, Hang Yan, Xipeng Qiu, Jiayu Wang, Kai Chen, and Dahua Lin.  Internlm-math: Open math large language models toward verifiable reasoning, 2024.
  * Yu et al. (2024a) Fei Yu, Anningzhe Gao, and Benyou Wang.  Ovm, outcome-supervised value models for planning in mathematical reasoning, 2024a.
  * Yu et al. (2024b) Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.  Metamath: Bootstrap your own mathematical questions for large language models.  In _The Twelfth International Conference on Learning Representations_ , 2024b.  URL https://openreview.net/forum?id=N8N0hgNDRt.
  * Yu et al. (2023) Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang.  Large language model as attributed training data generator: A tale of diversity and bias.  In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, _Advances in Neural Information Processing Systems_ , volume 36, pages 55734–55784. Curran Associates, Inc., 2023.  URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ae9500c4f5607caf2eff033c67daa9d7-Paper-Datasets_and_Benchmarks.pdf.
  * Yu et al. (2024c) Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin.  Wavecoder: Widespread and versatile enhanced instruction tuning with refined data generation, 2024c.
  * Yuan et al. (2023a) Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang.  RRHF: Rank responses to align language models with human feedback.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023a.  URL https://openreview.net/forum?id=EdIGMCHk4l.
  * Yuan et al. (2024) Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston.  Self-rewarding language models.  _ArXiv preprint_ , abs/2401.10020, 2024.  URL https://arxiv.org/abs/2401.10020.
  * Yuan et al. (2023b) Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou.  Scaling relationship on learning mathematical reasoning with large language models, 2023b.
  * Yue et al. (2024a) Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen.  MAmmoTH: Building math generalist models through hybrid instruction tuning.  In _The Twelfth International Conference on Learning Representations_ , 2024a.  URL https://openreview.net/forum?id=yLClGs770I.
  * Yue et al. (2024b) Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen.  Mammoth2: Scaling instructions from the web.  _ArXiv preprint_ , abs/2405.03548, 2024b.  URL https://arxiv.org/abs/2405.03548.
  * Zelikman et al. (2022) Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman.  Star: Bootstrapping reasoning with reasoning.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems_ , volume 35, pages 15476–15488. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/639a9a172c044fbb64175b5fad42e9a5-Paper-Conference.pdf.
  * Zelikman et al. (2024) Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman.  Quiet-star: Language models can teach themselves to think before speaking.  _ArXiv preprint_ , abs/2403.09629, 2024.  URL https://arxiv.org/abs/2403.09629.
  * Zeng et al. (2023) Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang.  Agenttuning: Enabling generalized agent abilities for llms, 2023.
  * Zeng et al. (2024) Dun Zeng, Yong Dai, Pengyu Cheng, Longyue Wang, Tianhao Hu, Wanshun Chen, Nan Du, and Zenglin Xu.  On diversified preferences of large language model alignment, 2024.
  * Zhang and Parkes (2023) Hugh Zhang and David Parkes.  Chain-of-thought reasoning is a policy improvement operator.  In _NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following_ , 2023.  URL https://openreview.net/forum?id=bH64KCBzqS.
  * Zhang (2023) Jiawei Zhang.  Graph-toolformer: To empower llms with graph reasoning ability via prompt augmented by chatgpt, 2023.
  * Zhang et al. (2024) Mozhi Zhang, Mianqiu Huang, Rundong Shi, Linsen Guo, Chong Peng, Peng Yan, Yaqian Zhou, and Xipeng Qiu.  Calibrating the confidence of large language models by eliciting fidelity.  _ArXiv preprint_ , abs/2404.02655, 2024.  URL https://arxiv.org/abs/2404.02655.
  * Zhang et al. (2016) Yizhe Zhang, Zhe Gan, and Lawrence Carin.  Generating text via adversarial training.  In _NIPS workshop on Adversarial Training_ , volume 21, pages 21–32. Academia. edu, 2016.
  * Zheng et al. (2024a) Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, and Nanyun Peng.  Weak-to-strong extrapolation expedites alignment, 2024a.
  * Zheng et al. (2024b) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _Advances in Neural Information Processing Systems_ , 36:46595–46623, 2024b.  URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html.
  * Zheng et al. (2023) Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang.  Secrets of rlhf in large language models part i: Ppo, 2023.
  * Zheng et al. (2024c) Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue.  Opencodeinterpreter: Integrating code generation with execution and refinement, 2024c.
  * Zhou et al. (2023a) Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy.  Lima: Less is more for alignment.  In A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, _Advances in Neural Information Processing Systems_ , volume 36, pages 55006–55021. Curran Associates, Inc., 2023a.  URL https://proceedings.neurips.cc/paper_files/paper/2023/file/ac662d74829e4407ce1d126477f4a03a-Paper-Conference.pdf.
  * Zhou et al. (2023b) Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi.  Least-to-most prompting enables complex reasoning in large language models.  In _The Eleventh International Conference on Learning Representations_ , 2023b.  URL https://openreview.net/forum?id=WZH7099tgfM.
  * Zhu et al. (2023a) Banghua Zhu, Michael Jordan, and Jiantao Jiao.  Principled reinforcement learning with human feedback from pairwise or k-wise comparisons.  In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 43037–43067. PMLR, 2023a.  URL https://proceedings.mlr.press/v202/zhu23f.html.
  * Zhu et al. (2023b) Xinyu Zhu, Junjie Wang, Lin Zhang, Yuxiang Zhang, Yongfeng Huang, Ruyi Gan, Jiaxing Zhang, and Yujiu Yang.  Solving math word problems via cooperative reasoning induced language models.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4471–4485, Toronto, Canada, 2023b. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.245.  URL https://aclanthology.org/2023.acl-long.245.
  * Ziems et al. (2022) Caleb Ziems, Jane Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang.  The moral integrity corpus: A benchmark for ethical dialogue systems.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3755–3773, Dublin, Ireland, 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.261.  URL https://aclanthology.org/2022.acl-long.261.