M. Cheng, J. Yi, P.-Y. Chen, H. Zhang, and C.-J. Hsieh.
Seq2sick: Evaluating the robustness of sequence-to-sequence models
with adversarial examples.
In Proceedings of the AAAI conference on artificial
intelligence, volume 34, pages 3601–3608, 2020.

[79]
Y. Cheng, L. Jiang, and W. Macherey.
Robust neural machine translation with doubly adversarial inputs.
arXiv preprint arXiv:1906.02443, 2019.

[80]
W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang,
Y. Zhuang, J. E. Gonzalez, et al.
Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt
See https://vicuna. lmsys. org (accessed 14 April 2023), 2023.

[81]
H. Cho, C. Park, J. Kang, K. M. Yoo, T. Kim, and S.-g. Lee.
Enhancing out-of-distribution detection in natural language
understanding via implicit layer ensemble.
In Findings of the Association for Computational Linguistics:
EMNLP 2022, pages 783–798, Abu Dhabi, United Arab Emirates, Dec. 2022.
Association for Computational Linguistics.

[82]
A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham,
H. W. Chung, C. Sutton, S. Gehrmann, et al.
Palm: Scaling language modeling with pathways.
arXiv preprint arXiv:2204.02311, 2022.

[83]
P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei.
Deep reinforcement learning from human preferences.
In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus,
S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information
Processing Systems, volume 30. Curran Associates, Inc., 2017.

[84]
P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei.
Deep reinforcement learning from human preferences.
Advances in neural information processing systems, 30, 2017.

[85]
K. Clark, M.-T. Luong, Q. V. Le, and C. D. Manning.
Electra: Pre-training text encoders as discriminators rather than
arXiv preprint arXiv:2003.10555, 2020.

[86]
K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert,
J. Tworek, J. Hilton, R. Nakano, et al.
Training verifiers to solve math word problems.
arXiv preprint arXiv:2110.14168, 2021.

[87]
J. Cohen, E. Rosenfeld, and Z. Kolter.
Certified adversarial robustness via randomized smoothing.
In international conference on machine learning, pages
1310–1320. PMLR, 2019.

[88]
F. Croce and M. Hein.
Reliable evaluation of adversarial robustness with an ensemble of
diverse parameter-free attacks.
In International conference on machine learning, pages
2206–2216. PMLR, 2020.

[89]
J. Dai, C. Chen, and Y. Li.
A backdoor attack against lstm-based text classification systems.
IEEE Access, 7:138872–138878, 2019.

[90]
S. Dan and D. Roth.
On the effects of transformer size on in-and out-of-domain
In Findings of the Association for Computational Linguistics:
EMNLP 2021, pages 2096–2101, 2021.

[91]
M. Davies, N. Srinivasa, T.-H. Lin, G. Chinya, Y. Cao, S. H. Choday, G. Dimou,
P. Joshi, N. Imam, S. Jain, et al.
Loihi: A neuromorphic manycore processor with on-chip learning.
IEEE Micro, 38(1):82–99, 2018.

[92]
L. De Moura and N. Bjørner.
Z3: An efficient smt solver.
In Tools and Algorithms for the Construction and Analysis of
Systems: 14th International Conference, TACAS 2008, Held as Part of the Joint
European Conferences on Theory and Practice of Software, ETAPS 2008,
Budapest, Hungary, March 29-April 6, 2008. Proceedings 14, pages 337–340.
Springer, 2008.

[93]
A. De Vries, U. Gallersdörfer, L. Klaaßen, and C. Stoll.
Revisiting bitcoin’s carbon footprint.
Joule, 6(3):498–502, 2022.

[94]
S. Desai and G. Durrett.
Calibration of pre-trained transformers.
In Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP), pages 295–302, Online, Nov. 2020.
Association for Computational Linguistics.

[95]
A. Deshpande, V. Murahari, T. Rajpurohit, A. Kalyan, and K. Narasimhan.
Toxicity in chatgpt: Analyzing persona-assigned language models.
arXiv preprint arXiv:2304.05335, 2023.

[96]
T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer.
Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.
Advances in Neural Information Processing Systems,
35:30318–30332, 2022.

[97]
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.
Bert: Pre-training of deep bidirectional transformers for language
arXiv preprint arXiv:1810.04805, 2018.

[98]
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.
BERT: Pre-training of deep bidirectional transformers for language
In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers), pages 4171–4186,
Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.

[99]
T. DeVries and G. W. Taylor.
Learning confidence for out-of-distribution detection in neural
arXiv preprint 1802.04865, 2018.

[100]
N. Dey.
Gpt: A family of open, compute-efficient, large language models.
Apr 2023.
Accessed: 2023-08-20.

[101]
J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith.
Fine-tuning pretrained language models: Weight initializations, data
orders, and early stopping.
arXiv preprint arXiv:2002.06305, 2020.

[102]
N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou,
A. W. Yu, O. Firat, et al.
Glam: Efficient scaling of language models with mixture-of-experts.
In International Conference on Machine Learning, pages
5547–5569. PMLR, 2022.

[103]
T. Du, S. Ji, L. Shen, Y. Zhang, J. Li, J. Shi, C. Fang, J. Yin, R. Beyah, and
T. Wang.
Cert-rnn: Towards certifying the robustness of recurrent neural
CCS, 21(2021):15–19, 2021.

[104]
H. Duan, Y. Yang, A. Abbasi, and K. Y. Tam.
BARLE: Background-aware representation learning for background
shift out-of-distribution detection.
In Findings of the Association for Computational Linguistics:
EMNLP 2022, pages 750–764, Abu Dhabi, United Arab Emirates, Dec. 2022.
Association for Computational Linguistics.

[105]
J. Duan, F. Kong, S. Wang, X. Shi, and K. Xu.
Are diffusion models vulnerable to membership inference attacks?
arXiv preprint arXiv:2302.01316, 2023.

[106]
J. J. Dudley and P. O. Kristensson.
A review of user interface design for interactive machine learning.
ACM Transactions on Interactive Intelligent Systems (TiiS),
8(2):1–37, 2018.

[107]
Gpt-4: Everything you want to know about openai's new ai model.
Mar 2023.
Accessed: 2023-08-20.

[108]
J. Ebrahimi, A. Rao, D. Lowd, and D. Dou.
Hotflip: White-box adversarial examples for text classification.
arXiv preprint arXiv:1712.06751, 2017.

[109]
B. EDWARDS.
Study claims chatgpt is losing capability, but some experts aren’t
Accessed: 2023-08-20.

[110]
D. Eppstein.
Zonohedra and zonotopes.
Mathematica in Education and Research, 5(4):15–21, 1996.

[111]
P. Esser, R. Rombach, and B. Ommer.
Taming transformers for high-resolution image synthesis.
In Proceedings of the IEEE/CVF conference on computer vision and
pattern recognition, pages 12873–12883, 2021.

[112]
F. Farhat, S. Sohail, and D. Madsen.
How trustworthy is chatgpt? the case of bibliometric analyses.
Cogent Engineering, 10, 06 2023.

[113]
W. Fedus, B. Zoph, and N. Shazeer.
Switch transformers: Scaling to trillion parameter models with simple
and efficient sparsity.
J. Mach. Learn. Res, 23:1–40, 2021.

[114]
R. Feinman, R. R. Curtin, S. Shintre, and A. B. Gardner.
Detecting adversarial samples from artifacts.
arXiv preprint arXiv:1703.00410, 2017.

[115]
M. Fitting.
First-Order Logic and Automated Theorem Proving, Second
Graduate Texts in Computer Science. Springer, 1996.

[116]
E. Frantar and D. Alistarh.
Optimal brain compression: A framework for accurate post-training
quantization and pruning.
arXiv preprint arXiv:2208.11580, 2022.

[117]
E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh.
Gptq: Accurate quantization for generative pre-trained transformers.
In International Conference on Learning Representations, 2023.

[118]
S. Frieder, L. Pinchetti, R.-R. Griffiths, T. Salvatori, T. Lukasiewicz, P. C.
Petersen, A. Chevalier, and J. Berner.
Mathematical capabilities of chatgpt.
arXiv preprint arXiv:2301.13867, 2023.

[119]
V. Gangal, A. Arora, A. Einolghozati, and S. Gupta.
Likelihood ratios and generative classifiers for unsupervised
out-of-domain detection in task oriented dialog.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 34, pages 7764–7771, 2020.

[120]
D. Ganguli, A. Askell, N. Schiefer, T. Liao, K. Lukošiūtė,
A. Chen, A. Goldie, A. Mirhoseini, C. Olsson, D. Hernandez, et al.
The capacity for moral self-correction in large language models.
arXiv preprint arXiv:2302.07459, 2023.

[121]
J. Gao, J. Lanchantin, M. L. Soffa, and Y. Qi.
Black-box generation of adversarial text sequences to evade deep
learning classifiers.
In 2018 IEEE Security and Privacy Workshops (SPW), pages
50–56. IEEE, 2018.

[122]
L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig.
Pal: Program-aided language models, 2023.

[123]
J. Garcıa and F. Fernández.
A comprehensive survey on safe reinforcement learning.
Journal of Machine Learning Research, 16(1):1437–1480, 2015.

[124]
I. Goodfellow and N. Papernot.
The challenge of verification and testing of machine learning.
Cleverhans-blog, 2017.

[125]
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair,
A. Courville, and Y. Bengio.
Generative adversarial networks.
Communications of the ACM, 63(11):139–144, 2020.

[126]
I. J. Goodfellow, J. Shlens, and C. Szegedy.
Explaining and harnessing adversarial examples.
arXiv preprint arXiv:1412.6572, 2014.

[127]
D. GOODIN.
Hackers are selling a service that bypasses chatgpt restrictions on
Accessed: 2023-08-20.

[128]
D. Gopinath, K. Wang, M. Zhang, C. S. Pasareanu, and S. Khurshid.
Symbolic execution for deep neural networks.
arXiv preprint arXiv:1807.10439, 2018.

[129]
J. Gou, B. Yu, S. J. Maybank, and D. Tao.
Knowledge distillation: A survey.
International Journal of Computer Vision, 129:1789–1819, 2021.

[130]
S. Gowal, K. Dvijotham, R. Stanforth, R. Bunel, C. Qin, J. Uesato,
R. Arandjelovic, T. Mann, and P. Kohli.
On the effectiveness of interval bound propagation for training
verifiably robust models.
arXiv preprint arXiv:1810.12715, 2018.

[131]
S. Goyal, S. Doddapaneni, M. M. Khapra, and B. Ravindran.
A survey in adversarial defences and robustness in nlp.
arXiv preprint arXiv:2203.06414, 2022.

[132]
K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz.
More than you've asked for: A comprehensive analysis of novel prompt
injection threats to application-integrated large language models.
arXiv preprint arXiv:2302.12173, 2023.

[133]
J.-C. Gu, T. Li, Q. Liu, Z.-H. Ling, Z. Su, S. Wei, and X. Zhu.
Speaker-aware bert for multi-turn response selection in
retrieval-based chatbots.
In Proceedings of the 29th ACM International Conference on
Information & Knowledge Management, CIKM '20, page 2041–2044, New
York, NY, USA, 2020. Association for Computing Machinery.

[134]
S. Gu, A. Kshirsagar, Y. Du, G. Chen, Y. Yang, J. Peters, and A. Knoll.
A human-centered safe robot reinforcement learning framework with
interactive behaviors.
arXiv preprint arXiv:2302.13137, 2023.

[135]
S. Gu, L. Yang, Y. Du, G. Chen, F. Walter, J. Wang, Y. Yang, and A. Knoll.
A review of safe reinforcement learning: Methods, theory and
arXiv preprint arXiv:2205.10330, 2022.

[136]
T. Gu, K. Liu, B. Dolan-Gavitt, and S. Garg.
Badnets: Evaluating backdooring attacks on deep neural networks.
IEEE Access, 7:47230–47244, 2019.

[137]
Y. Gu, L. Dong, F. Wei, and M. Huang.
Knowledge distillation of large language models.
arXiv preprint arXiv:2306.08543, 2023.

[138]
D. Gunning, M. Stefik, J. Choi, T. Miller, S. Stumpf, and G.-Z. Yang.
Xai—explainable artificial intelligence.
Science robotics, 4(37):eaay7120, 2019.

[139]
B. Guo, X. Zhang, Z. Wang, M. Jiang, J. Nie, Y. Ding, J. Yue, and Y. Wu.
How close is chatgpt to human experts? comparison corpus, evaluation,
and detection.
CoRR, abs/2301.07597, 2023.

[140]
D. Hendrycks and K. Gimpel.
A baseline for detecting misclassified and out-of-distribution
examples in neural networks.
In International Conference on Learning Representations, 2016.

[141]
D. Hendrycks, X. Liu, E. Wallace, A. Dziedzic, R. Krishnan, and D. Song.
Pretrained transformers improve out-of-distribution robustness.
In Proceedings of the 58th Annual Meeting of the Association for
Computational Linguistics, pages 2744–2751, 2020.

[142]
T. A. Henzinger, A. Lukina, and C. Schilling.
Outside the box: Abstraction-based monitoring of neural networks.
In ECAI2020.

[143]
G. Hinton, O. Vinyals, and J. Dean.
Distilling the knowledge in a neural network.
arXiv preprint arXiv:1503.02531, 2015.

[144]
A. Hintze.
Chatgpt believes it is conscious.
arXiv preprint arXiv:2304.12898, 2023.

[145]
J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford,
D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, et al.
Training compute-optimal large language models.
arXiv preprint arXiv:2203.15556, 2022.

[146]
J. Holmes, Z. Liu, L. Zhang, Y. Ding, T. T. Sio, L. A. McGee, J. B. Ashman,
X. Li, T. Liu, J. Shen, et al.
Evaluating large language models on a highly-specialized topic,
radiation oncology physics.
arXiv preprint arXiv:2304.01938, 2023.

[147]
H. Hosseini, S. Kannan, B. Zhang, and R. Poovendran.
Deceiving google's perspective api built for detecting toxic
arXiv preprint arXiv:1702.08138, 2017.

[148]
N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe,
A. Gesmundo, M. Attariyan, and S. Gelly.
Parameter-efficient transfer learning for nlp.
In International Conference on Machine Learning, pages
2790–2799. PMLR, 2019.

[149]
O. Hrinchuk, M. Popova, and B. Ginsburg.
Correction of automatic speech recognition with transformer
sequence-to-sequence model.
In Icassp 2020-2020 IEEE international conference on acoustics,
speech and signal processing (icassp), pages 7074–7078. IEEE, 2020.

[150]
E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and
W. Chen.
Lora: Low-rank adaptation of large language models.
arXiv preprint arXiv:2106.09685, 2021.

[151]
E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and
W. Chen.
LoRA: Low-rank adaptation of large language models.
In International Conference on Learning Representations, 2022.

[152]
Z. Hu, Z. Yang, X. Liang, R. Salakhutdinov, and E. P. Xing.
Toward controlled generation of text.
In International conference on machine learning, pages
1587–1596. PMLR, 2017.

[153]
H. Huang, Z. Li, L. Wang, S. Chen, B. Dong, and X. Zhou.
Feature space singularity for out-of-distribution detection.
arXiv preprint arXiv:2011.14654, 2020.

[154]
P.-S. Huang, R. Stanforth, J. Welbl, C. Dyer, D. Yogatama, S. Gowal,
K. Dvijotham, and P. Kohli.
Achieving verified robustness to symbol substitutions via interval
bound propagation.
arXiv preprint arXiv:1909.01492, 2019.

[155]
W. Huang, Y. Sun, X. Zhao, J. Sharp, W. Ruan, J. Meng, and X. Huang.
Coverage-guided testing for recurrent neural networks.
IEEE Transactions on Reliability, 71(3):1191–1206, 2021.

[156]
W. Huang, X. Zhao, A. Banks, V. Cox, and X. Huang.
Hierarchical distribution-aware testing of deep learning.
arXiv preprint arXiv:2205.08589, 2022.

[157]
W. Huang, X. Zhao, G. Jin, and X. Huang.
Safari: Versatile and efficient evaluations for robustness of
arXiv preprint arXiv:2208.09418, 2022.

[158]
X. Huang, M. Alzantot, and M. Srivastava.
Neuroninspect: Detecting backdoors in neural networks via output
arXiv preprint arXiv:1911.07399, 2019.

[159]
X. Huang, G. Jin, and W. Ruan.
Machine learning basics.
In Machine Learning Safety, pages 3–13. Springer, 2012.

[160]
X. Huang, D. Kroening, W. Ruan, J. Sharp, Y. Sun, E. Thamo, M. Wu, and X. Yi.
A survey of safety and trustworthiness of deep neural networks:
Verification, testing, adversarial attack and defence, and interpretability.
Computer Science Review, 37:100270, 2020.

[161]
X. Huang, M. Kwiatkowska, S. Wang, and M. Wu.
Safety verification of deep neural networks.
In R. Majumdar and V. Kuncak, editors, Computer Aided
Verification - 29th International Conference, CAV 2017, Heidelberg,
Germany, July 24-28, 2017, Proceedings, Part I, volume 10426 of Lecture Notes in Computer Science, pages 3–29. Springer, 2017.

[162]
X. Huang, W. Ruan, Q. Tang, and X. Zhao.
Bridging formal methods and machine learning with global
In Formal Methods and Software Engineering: 23rd International
Conference on Formal Engineering Methods, ICFEM 2022, Madrid, Spain, October
24–27, 2022, Proceedings, page 1–19, Berlin, Heidelberg, 2022.

[163]
A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry.
Adversarial examples are not bugs, they are features.
Advances in neural information processing systems, 32, 2019.

[164]
A. Ivankay, I. Girardi, C. Marchiori, and P. Frossard.
Fooling explanations in text classifiers.
arXiv preprint arXiv:2206.03178, 2022.

[165]
M. Iyyer, J. Wieting, K. Gimpel, and L. Zettlemoyer.
Adversarial example generation with syntactically controlled
paraphrase networks.
arXiv preprint arXiv:1804.06059, 2018.

[166]
A. Jaiswal, A. R. Babu, M. Z. Zadeh, D. Banerjee, and F. Makedon.
A survey on contrastive self-supervised learning.
Technologies, 9(1):2, 2020.

[167]
M. Jang and T. Lukasiewicz.
Consistency analysis of chatgpt.
arXiv preprint arXiv:2303.06273, 2023.

[168]
N. Jansen, B. Könighofer, J. Junges, A. Serban, and R. Bloem.
Safe reinforcement learning using probabilistic shields.
Dagstuhl: Schloss Dagstuhl, 2020.

[169]
N. Jansen, B. Könighofer, S. Junges, and R. Bloem.
Shielded decision-making in mdps.
arXiv preprint arXiv:1807.06096, 2018.

[170]
Y. Ji, Y. Gong, Y. Peng, C. Ni, P. Sun, D. Pan, B. Ma, and X. Li.
Exploring chatgpt's ability to rank content: A preliminary study on
consistency with human preferences, 2023.

[171]
R. Jia and P. Liang.
Adversarial examples for evaluating reading comprehension systems.
arXiv preprint arXiv:1707.07328, 2017.

[172]
R. Jia, A. Raghunathan, K. Göksel, and P. Liang.
Certified robustness to adversarial word substitutions.
arXiv preprint arXiv:1909.00986, 2019.

[173]
A. Q. Jiang, S. Welleck, J. P. Zhou, W. Li, J. Liu, M. Jamnik, T. Lacroix,
Y. Wu, and G. Lample.
Draft, sketch, and prove: Guiding formal theorem provers with
informal proofs.
arXiv preprint arXiv:2210.12283, 2022.

[174]
W. Jiao, W. Wang, J.-t. Huang, X. Wang, and Z. Tu.
Is chatgpt a good translator? a preliminary study.
arXiv preprint arXiv:2301.08745, 2023.

[175]
D. Jin, Z. Jin, J. T. Zhou, and P. Szolovits.
Is bert really robust? a strong baseline for natural language attack
on text classification and entailment.
In Proceedings of the AAAI conference on artificial
intelligence, volume 34, pages 8018–8025, 2020.

[176]
K. S. Kalyan, A. Rajasekharan, and S. Sangeetha.
Ammus: A survey of transformer-based pretrained models in natural
language processing.
arXiv preprint arXiv:2108.05542, 2021.

[177]
S. Kambhampati.
Changing the nature of ai research.
Commun. ACM, 65(9):8–9, aug 2022.

[178]
R. Kande, H. Pearce, B. Tan, B. Dolan-Gavitt, S. Thakur, R. Karri, and
J. Rajendran.
Llm-assisted generation of hardware assertions.
CoRR, abs/2306.14027, 2023.

[179]
D. Kang, X. Li, I. Stoica, C. Guestrin, M. Zaharia, and T. Hashimoto.
Exploiting programmatic behavior of llms: Dual-use through standard
security attacks.
arXiv preprint arXiv:2302.05733, 2023.

[180]
Y. Kang, Q. Zhang, and R. Roth.
The ethics of ai-generated maps: A study of dalle 2 and implications
for cartography.
arXiv preprint arXiv:2304.10743, 2023.

[181]
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child,
S. Gray, A. Radford, J. Wu, and D. Amodei.
Scaling laws for neural language models.
arXiv preprint arXiv:2001.08361, 2020.

[182]
D. M. Katz, M. J. Bommarito, S. Gao, and P. Arredondo.
Gpt-4 passes the bar exam.
Available at SSRN 4389233, 2023.

[183]
R. Khoury, A. R. Avila, J. Brunelle, and B. M. Camara.
How secure is code generated by chatgpt?
arXiv preprint arXiv:2304.09655, 2023.

[184]
Y.-M. Kim.
Data and fair use.
Korea Copyright Commission, 141:5–53, 03 2023.

[185]
C.-Y. Ko, Z. Lyu, L. Weng, L. Daniel, N. Wong, and D. Lin.
Popqorn: Quantifying robustness of recurrent neural networks.
In International Conference on Machine Learning, pages
3468–3477. PMLR, 2019.

[186]
J. Y. Koh, D. Fried, and R. Salakhutdinov.
Generating images with multimodal language models.
arXiv preprint arXiv:2305.17216, 2023.

[187]
V. Kuleshov, S. Thakoor, T. Lau, and S. Ermon.
Adversarial examples for natural language classification problems.
arXiv preprint, 2018.

[188]
A. Kumar, K. Ahuja, R. Vadapalli, and P. Talukdar.
Syntax-guided controlled generation of paraphrases.
Transactions of the Association for Computational Linguistics,
8:330–345, 2020.

[189]
K. Kurita, P. Michel, and G. Neubig.
Weight poisoning attacks on pretrained models.
In Proceedings of the 58th Annual Meeting of the Association for
Computational Linguistics, pages 2793–2806, 2020.

[190]
E. La Malfa, M. Wu, L. Laurenti, B. Wang, A. Hartshorn, and M. Kwiatkowska.
Assessing robustness of text classification through maximal safe
radius computation.
arXiv preprint arXiv:2010.02004, 2020.

[191]
M. Lam, R. Sethi, J. D. Ullman, and A. Aho.
Compilers: principles, techniques, and tools.
Pearson Education, 2006.

[192]
N. Lambert, L. Castricato, L. von Werra, and A. Havrilla.
Illustrating reinforcement learning from human feedback (rlhf).
Hugging Face Blog, 2022.

[193]
Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut.
Albert: A lite bert for self-supervised learning of language
arXiv preprint arXiv:1909.11942, 2019.

[194]
C. Lee, K. Cho, and W. Kang.
Mixout: Effective regularization to finetune large-scale pretrained
language models.
arXiv preprint arXiv:1909.11299, 2019.

[195]
J. Y. Lee.
Can an artificial intelligence chatbot be the author of a scholarly
journal of Educational Evaluation for Health Professions, 20,

[196]
K. Lee, H. Liu, M. Ryu, O. Watkins, Y. Du, C. Boutilier, P. Abbeel,
M. Ghavamzadeh, and S. S. Gu.
Aligning text-to-image models using human feedback.
arXiv preprint arXiv:2302.12192, 2023.

[197]
N. Lee, Y. Bang, A. Madotto, and P. Fung.
Misinformation has high perplexity.
arXiv preprint arXiv:2006.04666, 2020.

[198]
P. Lee.
Learning from tay’s introduction.
Accessed: 2023-08-20.

[199]
Y. Lei, Y. Cao, D. Li, T. Zhou, M. Fang, and M. Pechenizkiy.
Phrase-level textual adversarial attack with label preservation.
arXiv preprint arXiv:2205.10710, 2022.

[200]
D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer,
and Z. Chen.
Gshard: Scaling giant models with conditional computation and
automatic sharding.
arXiv preprint arXiv:2006.16668, 2020.

[201]
M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov,
and L. Zettlemoyer.
BART: Denoising sequence-to-sequence pre-training for natural
language generation, translation, and comprehension.
In Proceedings of the 58th Annual Meeting of the Association for
Computational Linguistics, pages 7871–7880, Online, July 2020. Association
for Computational Linguistics.

[202]
H. Li, D. Guo, W. Fan, M. Xu, and Y. Song.
Multi-step jailbreaking privacy attacks on chatgpt.
arXiv preprint arXiv:2304.05197, 2023.

[203]
J. Li, X. Cheng, W. X. Zhao, J.-Y. Nie, and J.-R. Wen.
Halueval: A large-scale hallucination evaluation benchmark for large
language models.
arXiv e-prints, pages arXiv–2305, 2023.

[204]
J. Li, S. Ji, T. Du, B. Li, and T. Wang.
Textbugger: Generating adversarial text against real-world
arXiv preprint arXiv:1812.05271, 2018.

[205]
J. Li, T. Tang, W. X. Zhao, J.-Y. Nie, and J.-R. Wen.
Pretrained language models for text generation: A survey.
arXiv preprint arXiv:2201.05273, 2022.

[206]
S. Li, H. Liu, T. Dong, B. Z. H. Zhao, M. Xue, H. Zhu, and J. Lu.
Hidden backdoors in human-centric language models.
In CCS '21: 2021 ACM SIGSAC Conference on Computer and
Communications Security, Virtual Event, Republic of Korea, November 15 - 19,
2021, pages 3123–3140. ACM, 2021.

[207]
X. Li, J. Li, X. Sun, C. Fan, T. Zhang, F. Wu, Y. Meng, and J. Zhang.
kfolden: k-fold ensemble for out-of-distribution detection-fold
ensemble for out-of-distribution detection.
In Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing, pages 3102–3115, 2021.

[208]
Y. Li, L. Ding, and X. Gao.
On the decision boundary of deep neural networks.
arXiv preprint arXiv:1808.05385, 2018.

[209]
B. Liang, H. Li, M. Su, P. Bian, X. Li, and W. Shi.
Deep text classification can be fooled.
arXiv preprint arXiv:1704.08006, 2017.

[210]
S. Liang, Y. Li, and R. Srikant.
Enhancing the reliability of out-of-distribution image detection in
neural networks.
In 6th International Conference on Learning Representations,
ICLR 2018, 2018.

[211]
T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan,
P. Dollár, and C. L. Zitnick.
Microsoft coco: Common objects in context.
In Computer Vision–ECCV 2014: 13th European Conference, Zurich,
Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755.
Springer, 2014.

[212]
Z. Lin, P. Xu, G. I. Winata, F. B. Siddique, Z. Liu, J. Shin, and P. Fung.
Caire: An empathetic neural chatbot.
arXiv preprint arXiv:1907.12108, 2019.

[213]
C. Liu, T. Arnon, C. Lazarus, C. Strong, C. Barrett, M. J. Kochenderfer, et al.
Algorithms for verifying deep neural networks.
Foundations and Trends® in Optimization,
4(3-4):244–404, 2021.

[214]
H. Liu, R. Ning, Z. Teng, J. Liu, Q. Zhou, and Y. Zhang.
Evaluating the logical reasoning ability of chatgpt and gpt-4.
arXiv preprint arXiv:2304.03439, 2023.

[215]
J. Liu, C. S. Xia, Y. Wang, and L. Zhang.
Is your code generated by chatgpt really correct? rigorous evaluation
of large language models for code generation.
arXiv preprint arXiv:2305.01210, 2023.

[216]
W. Liu, X. Wang, J. Owens, and Y. Li.
Energy-based out-of-distribution detection.
Advances in neural information processing systems,
33:21464–21475, 2020.

[217]
X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang.
Self-supervised learning: Generative or contrastive.
IEEE Transactions on Knowledge and Data Engineering,
35(1):857–876, 2021.

[218]
Y. Liu, T. Han, S. Ma, J. Zhang, Y. Yang, J. Tian, H. He, A. Li, M. He, Z. Liu,
et al.
Summary of chatgpt/gpt-4 research and perspective towards the future
of large language models.
arXiv preprint arXiv:2304.01852, 2023.

[219]
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis,
L. Zettlemoyer, and V. Stoyanov.
Roberta: A robustly optimized bert pretraining approach.
arXiv preprint arXiv:1907.11692, 2019.

[220]
Z. Liu, Y. Wang, K. Han, W. Zhang, S. Ma, and W. Gao.
Post-training quantization for vision transformer.
Advances in Neural Information Processing Systems,
34:28092–28103, 2021.

[221]
Z. Liu, X. Yu, L. Zhang, Z. Wu, C. Cao, H. Dai, L. Zhao, W. Liu, D. Shen,
Q. Li, et al.
Deid-gpt: Zero-shot medical text de-identification by gpt-4.
arXiv preprint arXiv:2303.11032, 2023.

[222]
R. Lou, K. Zhang, and W. Yin.
Is prompt all you need? no. a comprehensive and broader view of
instruction learning.
arXiv preprint arXiv:2303.10475, 2023.

[223]
N. Madaan, I. Padhi, N. Panwar, and D. Saha.
Generate your counterfactuals: Towards controlled counterfactual
generation for text.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 35, pages 13516–13524, 2021.

[224]
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu.
Towards deep learning models resistant to adversarial attacks.
arXiv preprint arXiv:1706.06083, 2017.

[225]
K. Malinka, M. Peresíni, A. Firc, O. Hujnák, and F. Janus.
On the educational impact of chatgpt: Is artificial intelligence
ready to obtain a university degree?
In Proceedings of the 2023 Conference on Innovation and
Technology in Computer Science Education V. 1, pages 47–53, 2023.

[226]
Z. Manna and A. Pnueli.
The temporal logic of reactive and concurrent systems:
Springer Science & Business Media, 2012.

[227]
N. Maus, P. Chao, E. Wong, and J. Gardner.
Adversarial prompting for black box foundation models.
arXiv preprint arXiv:2302.04237, 2023.

[228]
W. McCune.
Prover9 and mace4.
<https://www.cs.unm.edu/ mccune/prover9/>, 2005.
Accessed: 2023-08-20.

[229]
Y. Mehdi.
Announcing the next wave of ai innovation with microsoft bing and
edge, May 2023.

[230]
S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and
L. Zettlemoyer.
Rethinking the role of demonstrations: What makes in-context learning
arXiv preprint arXiv:2202.12837, 2022.

[231]
M. Mirman, T. Gehr, and M. Vechev.
Differentiable abstract interpretation for provably robust neural
In J. Dy and A. Krause, editors, Proceedings of the 35th
International Conference on Machine Learning, volume 80 of Proceedings
of Machine Learning Research, pages 3578–3586. PMLR, 10–15 Jul 2018.

[232]
S. Mitrović, D. Andreoletti, and O. Ayoub.
Chatgpt or human? detect and explain. explaining decisions of machine
learning model for detecting short chatgpt-generated text, 2023.

[233]
J. Monteiro, I. Albuquerque, Z. Akhtar, and T. H. Falk.
Generalizable adversarial examples detection based on bi-model
decision mismatch.
In 2019 IEEE International Conference on Systems, Man and
Cybernetics (SMC), pages 2839–2844. IEEE, 2019.

[234]
M. Nagel, R. A. Amjad, M. Van Baalen, C. Louizos, and T. Blankevoort.
Up or down? adaptive rounding for post-training quantization.
In International Conference on Machine Learning, pages
7197–7206. PMLR, 2020.

[235]
B. Nelson, M. Barreno, F. J. Chi, A. D. Joseph, B. I. P. Rubinstein, U. Saini,
C. Sutton, J. D. Tygar, and K. Xia.
Exploiting machine learning to subvert your spam filter.
In Proceedings of the 1st Usenix Workshop on Large-Scale
Exploits and Emergent Threats, LEET'08, USA, 2008. USENIX Association.

[236]
T. H. News.
Wormgpt: New ai tool allows cybercriminals to launch sophisticated
cyber attacks.
Accessed: 2023-08-20.

[237]
A. Ni, S. Iyer, D. Radev, V. Stoyanov, W.-t. Yih, S. Wang, and X. V. Lin.
Lever: Learning to verify language-to-code generation with execution.
In International Conference on Machine Learning, pages
26106–26128. PMLR, 2023.

[238]
A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew,
I. Sutskever, and M. Chen.
Glide: Towards photorealistic image generation and editing with
text-guided diffusion models.
arXiv preprint arXiv:2112.10741, 2021.

[239]
Y. Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela.
Adversarial nli: A new benchmark for natural language understanding.
arXiv preprint arXiv:1910.14599, 2019.

[240]
GPT-4 Technical Report.
arXiv e-prints 2303.08774, 2023.

[241]
L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang,
S. Agarwal, K. Slama, A. Ray, et al.
Training language models to follow instructions with human feedback.
Advances in Neural Information Processing Systems,
35:27730–27744, 2022.

[242]
S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang, and X. Wu.
Unifying large language models and knowledge graphs: A roadmap, 2023.

[243]
G. Pang, C. Shen, L. Cao, and A. V. D. Hengel.
Deep learning for anomaly detection: A review.
ACM computing surveys (CSUR), 54(2):1–38, 2021.

[244]
G. Park, B. Park, S. J. Kwon, B. Kim, Y. Lee, and D. Lee.
nuqmm: Quantized matmul for efficient inference of large-scale
generative language models.
arXiv preprint arXiv:2206.09557, 2022.

[245]
D. Patterson, J. Gonzalez, U. Holzle, Q. Le, C. Liang, L.-M. Munguia,
D. Rothchild, D. R. So, M. Texier, and J. Dean.
The carbon footprint of machine learning training will plateau, then
Computer, 55(7):18–28, 2022.

[246]
H. Pearce, B. Tan, B. Ahmad, R. Karri, and B. Dolan-Gavitt.
Examining zero-shot vulnerability repair with large language models.
In 2023 IEEE Symposium on Security and Privacy (SP), pages
2339–2356. IEEE, 2023.

[247]
A. Pegoraro, K. Kumari, H. Fereidooni, and A.-R. Sadeghi.
To chatgpt, or not to chatgpt: That is the question!
arXiv preprint arXiv:2304.01487, 2023.

[248]
B. Peng, C. Li, P. He, M. Galley, and J. Gao.
Instruction tuning with gpt-4.
arXiv preprint arXiv:2304.03277, 2023.

[249]
J. Pennington, R. Socher, and C. D. Manning.
Glove: Global vectors for word representation.
In Proceedings of the 2014 conference on empirical methods in
natural language processing (EMNLP), pages 1532–1543, 2014.

[250]
F. Perez and I. Ribeiro.
Ignore previous prompt: Attack techniques for language models.
arXiv preprint arXiv:2211.09527, 2022.

[251]
A. Podolskiy, D. Lipin, A. Bout, E. Artemova, and I. Piontkovskaya.
Revisiting mahalanobis distance for transformer-based out-of-domain
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 35, pages 13675–13682, 2021.

[252]
Y. Qi, X. Zhao, and X. Huang.
safety analysis in the era of large language models: a case study of
stpa using chatgpt.
arXiv preprint arXiv:2304.01246, 2023.

[253]
A. Radford, R. Jozefowicz, and I. Sutskever.
Learning to generate reviews and discovering sentiment.
arXiv preprint arXiv:1704.01444, 2017.

[254]
A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al.
Improving language understanding by generative pre-training.
OpenAI, 2018.

[255]
J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song,
J. Aslanides, S. Henderson, R. Ring, S. Young, et al.
Scaling language models: Methods, analysis & insights from training
arXiv preprint arXiv:2112.11446, 2021.

[256]
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou,
W. Li, and P. J. Liu.
Exploring the limits of transfer learning with a unified text-to-text
The Journal of Machine Learning Research, 21(1):5485–5551,

[257]
R. Ramamurthy, P. Ammanabrolu, K. Brantley, J. Hessel, R. Sifa, C. Bauckhage,
H. Hajishirzi, and Y. Choi.
Is reinforcement learning (not) for natural language processing?:
Benchmarks, baselines, and building blocks for natural language policy
arXiv preprint arXiv:2210.01241, 2022.

[258]
A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen.
Hierarchical text-conditional image generation with clip latents.
arXiv preprint arXiv:2204.06125, 2022.

[259]
A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and
I. Sutskever.
Zero-shot text-to-image generation.
In International Conference on Machine Learning, pages
8821–8831. PMLR, 2021.

[260]
M. V. Reiss.
Testing the reliability of chatgpt for text annotation and
classification: A cautionary remark.
arXiv preprint arXiv:2304.11085, 2023.

[261]
J. Ren, P. J. Liu, E. Fertig, J. Snoek, R. Poplin, M. Depristo, J. Dillon, and
B. Lakshminarayanan.
Likelihood ratios for out-of-distribution detection.
Advances in neural information processing systems, 32, 2019.

[262]
S. Ren, Y. Deng, K. He, and W. Che.
Generating natural language adversarial examples through probability
weighted word saliency.
In Proceedings of the 57th annual meeting of the association for
computational linguistics, pages 1085–1097, 2019.

[263]
X. Ren, P. Zhou, X. Meng, X. Huang, Y. Wang, W. Wang, P. Li, X. Zhang,
A. Podolskiy, G. Arshinov, et al.
Pangu-$\sigma$: Towards trillion parameter language model with sparse
heterogeneous computing.
arXiv preprint arXiv:2303.10845, 2023.

[264]
M. T. Ribeiro, S. Singh, and C. Guestrin.
"why should I trust you?": Explaining the predictions of any
In HLT-NAACL Demos, 2016.

[265]
J. T. Rolfe.
Discrete variational autoencoders.
arXiv preprint arXiv:1609.02200, 2016.

[266]
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer.
High-resolution image synthesis with latent diffusion models.
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 10684–10695, 2022.

[267]
W. Ruan, X. Huang, and M. Kwiatkowska.
Reachability analysis of deep neural networks with provable
In IJCAI2018, pages 2651–2659, 2018.

[268]
W. Ruan, M. Wu, Y. Sun, X. Huang, D. Kroening, and M. Kwiatkowska.
Global robustness evaluation of deep neural networks with provable
guarantees for the hamming distance.
In IJCAI2019, pages 5944–5952, 2019.

[269]
S. Ruder, M. E. Peters, S. Swayamdipta, and T. Wolf.
Transfer learning in natural language processing.
In Proceedings of the 2019 Conference of the North American
Chapter of the Association for Computational Linguistics: Tutorials, pages
15–18, 2019.

[270]
B. Rueckauer, I.-A. Lungu, Y. Hu, M. Pfeiffer, and S.-C. Liu.
Conversion of continuous-valued deep networks to efficient
event-driven networks for image classification.
Frontiers in neuroscience, 11:682, 2017.

[271]
J. Rutinowski, S. Franke, J. Endendyk, I. Dormuth, and M. Pauly.
The self-perception and political biases of chatgpt.
arXiv preprint arXiv:2304.07333, 2023.

[272]
W. Ryou, J. Chen, M. Balunovic, G. Singh, A. Dan, and M. Vechev.
Scalable polyhedral verification of recurrent neural networks.
In International Conference on Computer Aided Verification,
pages 225–248. Springer, 2021.

[273]
C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour,
R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al.
Photorealistic text-to-image diffusion models with deep language
Advances in Neural Information Processing Systems,
35:36479–36494, 2022.

[274]
S. Samanta and S. Mehta.
Towards crafting text adversarial samples.
arXiv preprint arXiv:1707.02812, 2017.

[275]
G. Sandoval, H. Pearce, T. Nys, R. Karri, S. Garg, and B. Dolan-Gavitt.
Lost at c: A user study on the security implications of large
language model code assistants.
arXiv preprint arXiv:2208.09727, 2023.

[276]
T. L. Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow,
R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, et al.
Bloom: A 176b-parameter open-access multilingual language model.
arXiv preprint arXiv:2211.05100, 2022.

[277]
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov.
Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.06347, 2017.

[278]
U. Senate.
Senate judiciary subcommittee hearing on oversight of ai.
Accessed: 2023-08-20.

[279]
S. A. Seshia, D. Sadigh, and S. S. Sastry.
Towards verified artificial intelligence.
arXiv preprint arXiv:1606.08514, 2016.

[280]
M. Shanahan.
Talking about large language models.
arXiv preprint arXiv:2212.03551, 2022.

[281]
L. Shen, S. Ji, X. Zhang, J. Li, J. Chen, J. Shi, C. Fang, J. Yin, and T. Wang.
Backdoor pre-trained models can transfer to all.
In Proceedings of the 2021 ACM SIGSAC Conference on Computer and
Communications Security, pages 3141–3158, 2021.

[282]
X. Shen, Z. Chen, M. Backes, and Y. Zhang.
In chatgpt we trust? measuring and characterizing the reliability of
arXiv preprint arXiv:2304.08979, 2023.

[283]
Y. Shen, Y.-C. Hsu, A. Ray, and H. Jin.
Enhancing the generalization for intent classification and
out-of-domain detection in slu.
In Proceedings of the 59th Annual Meeting of the Association for
Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers), pages 2443–2453, 2021.

[284]
Z. Shi, H. Zhang, K.-W. Chang, M. Huang, and C.-J. Hsieh.
Robustness verification for transformers.
In International Conference on Learning Representations, 2019.

[285]
K. Shuster, M. Komeili, L. Adolphs, S. Roller, A. Szlam, and J. Weston.
Language models that seek for knowledge: Modular search & generation
for dialogue and prompt completion.
arXiv preprint arXiv:2203.13224, 2022.

[286]
K. Shuster, S. Poff, M. Chen, D. Kiela, and J. Weston.
Retrieval augmentation reduces hallucination in conversation.
arXiv preprint arXiv:2104.07567, 2021.

[287]
A. Sinha, H. Namkoong, R. Volpi, and J. Duchi.
Certifying some distributional robustness with principled adversarial
arXiv preprint arXiv:1710.10571, 2017.

[288]
L. Smith and Y. Gal.
Understanding measures of uncertainty for adversarial example
arXiv preprint arXiv:1803.08533, 2018.

[289]
S. Smith, M. Patwary, B. Norick, P. LeGresley, S. Rajbhandari, J. Casper,
Z. Liu, S. Prabhumoye, G. Zerveas, V. Korthikanti, et al.
Using deepspeed and megatron to train megatron-turing nlg 530b, a
large-scale generative language model.
arXiv preprint arXiv:2201.11990, 2022.

[290]
D. Sobania, M. Briesch, C. Hanna, and J. Petke.
An analysis of the automatic bug fixing performance of chatgpt.
arXiv preprint arXiv:2301.08653, 2023.

[291]
S. Soltan, S. Ananthakrishnan, J. FitzGerald, R. Gupta, W. Hamza, H. Khan,
C. Peris, S. Rawls, A. Rosenbaum, A. Rumshisky, et al.
Alexatm 20b: Few-shot learning using a large-scale multilingual
seq2seq model.
arXiv preprint arXiv:2208.01448, 2022.

[292]
L. Struppek, D. Hintersdorf, and K. Kersting.
Rickrolling the artist: Injecting invisible backdoors into
text-guided image generation models.
arXiv preprint arXiv:2211.02408, 2022.

[293]
H. Sun, Z. Zhang, J. Deng, J. Cheng, and M. Huang.
Safety assessment of chinese large language models.
arXiv preprint arXiv:2304.10436, 2023.

[294]
Y. Sun, X. Huang, D. Kroening, J. Sharp, M. Hill, and R. Ashmore.
Testing deep neural networks.
arXiv preprint arXiv:1803.04792, 2018.

[295]
Y. Sun, X. Huang, D. Kroening, J. Sharp, M. Hill, and R. Ashmore.
Structural test coverage criteria for deep neural networks.
ACM Trans. Embed. Comput. Syst., 18(5s), Oct. 2019.

[296]
Y. Sun, S. Wang, S. Feng, S. Ding, C. Pang, J. Shang, J. Liu, X. Chen, Y. Zhao,
Y. Lu, et al.
Ernie 3.0: Large-scale knowledge enhanced pre-training for language
understanding and generation.
arXiv preprint arXiv:2107.02137, 2021.

[297]
Y. Sun, M. Wu, W. Ruan, X. Huang, M. Kwiatkowska, and D. Kroening.
Concolic testing for deep neural networks.
In ASE2018, 2018.

[298]
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and
R. Fergus.
Intriguing properties of neural networks.
arXiv preprint arXiv:1312.6199, 2013.

[299]
L. Tanguy, N. Tulechki, A. Urieli, E. Hermann, and C. Raynal.
Natural language processing for aviation safety reports: From
classification to interactive analysis.
Computers in Industry, 78:80–95, 2016.

[300]
R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and
T. B. Hashimoto.
Stanford alpaca: An instruction-following llama model, 2023.

[301]
R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia,
A. Poulton, V. Kerkez, and R. Stojnic.
Galactica: A large language model for science.
arXiv preprint arXiv:2211.09085, 2022.

[302]
A. Tejankar, M. Sanjabi, Q. Wang, S. Wang, H. Firooz, H. Pirsiavash, and
L. Tan.
Defending against patch-based backdoor attacks on self-supervised
In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pages 12239–12249, 2023.

[303]
S. Thakur, B. Ahmad, Z. Fan, H. Pearce, B. Tan, R. Karri, B. Dolan-Gavitt, and
S. Garg.
Benchmarking large language models for automated verilog rtl code
In 2023 Design, Automation & Test in Europe Conference &
Exhibition (DATE), pages 1–6. IEEE, 2023.

[304]
R. Thoppilan, D. De Freitas, J. Hall, N. Shazeer, A. Kulshreshtha, H.-T. Cheng,
A. Jin, T. Bos, L. Baker, Y. Du, et al.
Lamda: Language models for dialog applications.
arXiv preprint arXiv:2201.08239, 2022.

[305]
J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal.
Fever: A large-scale dataset for fact extraction and verification.
In 2018 Conference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies, NAACL
HLT 2018, pages 809–819. Association for Computational Linguistics (ACL),

[306]
H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix,
B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al.
Llama: Open and efficient foundation language models.
arXiv preprint arXiv:2302.13971, 2023.

[307]
A. S. Tulshan and S. N. Dhage.
Survey on virtual assistant: Google assistant, siri, cortana, alexa.
In Advances in Signal Processing and Intelligent Recognition
Systems: 4th International Symposium SIRS 2018, Bangalore, India, September
19–22, 2018, Revised Selected Papers 4, pages 190–201. Springer, 2019.

[308]
A. Uchendu, J. Lee, H. Shen, T. Le, T. K. Huang, and D. Lee.
Understanding individual and team-based human factors in detecting
deepfake texts.
CoRR, abs/2304.01002, 2023.

[309]
M. Y. Vardi and P. Wolper.
An automata-theoretic approach to automatic program verification.
In 1st Symposium in Logic in Computer Science (LICS). IEEE
Computer Society, 1986.

[310]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,
Ł. Kaiser, and I. Polosukhin.
Attention is all you need.
Advances in neural information processing systems, 30, 2017.

[311]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u.
Kaiser, and I. Polosukhin.
Attention is all you need.
In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus,
S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information
Processing Systems, volume 30. Curran Associates, Inc., 2017.

[312]
M. Wallace, R. Khandelwal, and B. Tang.
Does ibp scale?
arXiv preprint, 2022.

[313]
B. Wang, C. Xu, S. Wang, Z. Gan, Y. Cheng, J. Gao, A. H. Awadallah, and B. Li.
Adversarial glue: A multi-task benchmark for robustness evaluation of
language models.
arXiv preprint arXiv:2111.02840, 2021.

[314]
F. Wang, P. Xu, W. Ruan, and X. Huang.
Towards verifying the geometric robustness of large-scale neural
arXiv preprint arXiv:2301.12456, 2023.

[315]
G. Wang, Y. Lin, and W. Yi.
Kernel fusion: An effective method for better power efficiency on
multithreaded gpu.
In 2010 IEEE/ACM Int'l Conference on Green Computing and
Communications & Int'l Conference on Cyber, Physical and Social Computing,
pages 344–350. IEEE, 2010.

[316]
J. Wang, X. Hu, W. Hou, H. Chen, R. Zheng, Y. Wang, L. Yang, H. Huang, W. Ye,
X. Geng, et al.
On the robustness of chatgpt: An adversarial and out-of-distribution
arXiv preprint arXiv:2302.12095, 2023.

[317]
J. Wang, X. Hu, W. Hou, H. Chen, R. Zheng, Y. Wang, L. Yang,
H. Huang, W. Ye, X. Geng, B. Jiao, Y. Zhang, and X. Xie.
On the Robustness of ChatGPT: An Adversarial and Out-of-distribution
arXiv e-prints, page arXiv:2302.12095, Feb. 2023.

[318]
W. Wang, P. Tang, J. Lou, and L. Xiong.
Certified robustness to word substitution attack with differential
In Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, pages 1102–1112, 2021.

[319]
X. Wang, J. Wei, D. Schuurmans, Q. V. Le, E. H. Chi, S. Narang, A. Chowdhery,
and D. Zhou.
Self-consistency improves chain of thought reasoning in language
In The Eleventh International Conference on Learning
Representations, 2023.

[320]
Y. Wang and M. Bansal.
Robust machine comprehension models via adversarial training.
arXiv preprint arXiv:1804.06473, 2018.

[321]
J. Wei, S. Kim, H. Jung, and Y.-H. Kim.
Leveraging large language models to power chatbots for collecting
user self-reported data.
arXiv preprint arXiv:2301.05843, 2023.

[322]
J. Wei, X. Wang, D. Schuurmans, M. Bosma, brian ichter, F. Xia, E. H. Chi,
Q. V. Le, and D. Zhou.
Chain of thought prompting elicits reasoning in large language
In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.

[323]
T.-W. Weng, H. Zhang, P.-Y. Chen, J. Yi, D. Su, Y. Gao, C.-J. Hsieh, and
L. Daniel.
Evaluating the robustness of neural networks: An extreme value theory
arXiv preprint arXiv:1801.10578, 2018.

[324]
Y. Weng, M. Zhu, S. He, K. Liu, and J. Zhao.
Large language models are reasoners with self-verification.
arXiv preprint arXiv:2212.09561, 2022.

[325]
Y. Weng, M. Zhu, F. Xia, B. Li, S. He, K. Liu, and J. Zhao.
Neural comprehension: Language models with compiled neural networks.
arXiv preprint arXiv:2304.01665, 2023.

[326]
M. Wicker, X. Huang, and M. Kwiatkowska.
Feature-guided black-box safety testing of deep neural networks.
In Tools and Algorithms for the Construction and Analysis of
Systems: 24th International Conference, TACAS 2018, Held as Part of the
European Joint Conferences on Theory and Practice of Software, ETAPS 2018,
Thessaloniki, Greece, April 14-20, 2018, Proceedings, Part I 24, pages
408–426, 2018.

[327]
Y. Wolf, N. Wies, Y. Levine, and A. Shashua.
Fundamental limitations of alignment in large language models.
arXiv preprint arXiv:2304.11082, 2023.

[328]
E. Wong, L. Rice, and J. Z. Kolter.
Fast is better than free: Revisiting adversarial training.
arXiv preprint arXiv:2001.03994, 2020.

[329]
D. Wu, G. Jin, H. Yu, X. Yi, and X. Huang.
Optimising event-driven spiking neural network with regularisation
and cutoff.
arXiv preprint arXiv:2301.09522, 2023.

[330]
D. Wu, X. Yi, and X. Huang.
A little energy goes a long way: Build an energy-efficient, accurate
spiking neural network from convolutional neural network.
Frontiers in neuroscience, 16, 2022.

[331]
H. Wu, W. Wang, Y. Wan, W. Jiao, and M. Lyu.
Chatgpt or grammarly? evaluating chatgpt on grammatical error
correction benchmark.
arXiv preprint arXiv:2303.13648, 2023.

[332]
M. Wu, A. Waheed, C. Zhang, M. Abdul-Mageed, and A. F. Aji.
Lamini-lm: A diverse herd of distilled models from large-scale
arXiv preprint arXiv:2304.14402, 2023.

[333]
M. Wu, M. Wicker, W. Ruan, X. Huang, and M. Kwiatkowska.
A game-based approximate verification of deep neural networks with
provable guarantees.
Theoretical Computer Science, 807:298–329, 2020.

[334]
S. Wu, O. Irsoy, S. Lu, V. Dabravolski, M. Dredze, S. Gehrmann, P. Kambadur,
D. Rosenberg, and G. Mann.
Bloomberggpt: A large language model for finance.
arXiv preprint arXiv:2303.17564, 2023.

[335]
X. Wu, K. Sun, F. Zhu, R. Zhao, and H. Li.
Better aligning text-to-image models with human preference.
arXiv preprint arXiv:2303.14420, 2023.

[336]
Y. Wu, A. Q. Jiang, W. Li, M. N. Rabe, C. E. Staats, M. Jamnik, and C. Szegedy.
Autoformalization with large language models.
In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.

[337]
F. Xu, H. Uszkoreit, Y. Du, W. Fan, D. Zhao, and J. Zhu.
Explainable ai: A brief survey on history, research areas, approaches
and challenges.
In Natural Language Processing and Chinese Computing: 8th CCF
International Conference, NLPCC 2019, Dunhuang, China, October 9–14, 2019,
Proceedings, Part II 8, pages 563–574. Springer, 2019.

[338]
H. Xu, K. He, Y. Yan, S. Liu, Z. Liu, and W. Xu.
A deep generative distance-based classifier for out-of-domain
detection with mahalanobis space.
In Proceedings of the 28th International Conference on
Computational Linguistics, pages 1452–1460, 2020.

[339]
H. Xu, Y. Ma, H.-C. Liu, D. Deb, H. Liu, J.-L. Tang, and A. K. Jain.
Adversarial attacks and defenses in images, graphs and text: A
International Journal of Automation and Computing, 17:151–178,

[340]
J. Xu, X. Liu, Y. Wu, Y. Tong, Q. Li, M. Ding, J. Tang, and Y. Dong.
Imagereward: Learning and evaluating human preferences for
text-to-image generation.
arXiv preprint arXiv:2304.05977, 2023.

[341]
P. Xu, W. Ruan, and X. Huang.
Quantifying safety risks of deep neural networks.
Complex & Intelligent Systems, pages 1–18, 2022.

[342]
Yandex/yalm-100b: Pretrained language model with 100b parameters.
Accessed: 2023-08-20.

[343]
J. Yang, H. Jin, R. Tang, X. Han, Q. Feng, H. Jiang, B. Yin, and X. Hu.
Harnessing the power of llms in practice: A survey on chatgpt and
arXiv preprint arXiv:2304.13712, 2023.

[344]
J. Yang, K. Zhou, Y. Li, and Z. Liu.
Generalized out-of-distribution detection: A survey.
arXiv preprint arXiv:2110.11334, 2021.

[345]
W. Yang, L. Li, Z. Zhang, X. Ren, X. Sun, and B. He.
Be careful about poisoned word embeddings: Exploring the
vulnerability of the embedding layers in nlp models.
In Proceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, pages 2048–2058, 2021.

[346]
Z. Yang.
Chinese tech giant baidu just released its answer to chatgpt, Mar

[347]
Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le.
Xlnet: Generalized autoregressive pretraining for language
Advances in Neural Information Processing Systems, 32, 2019.

[348]
S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao.
React: Synergizing reasoning and acting in language models.
In The Eleventh International Conference on Learning
Representations, 2023.

[349]
Z. Yao, R. Yazdani Aminabadi, M. Zhang, X. Wu, C. Li, and Y. He.
Zeroquant: Efficient and affordable post-training quantization for
large-scale transformers.
Advances in Neural Information Processing Systems,
35:27168–27183, 2022.

[350]
M. Ye, C. Gong, and Q. Liu.
Safer: A structure-free approach for certified robustness to
adversarial word substitutions.
arXiv preprint arXiv:2005.14424, 2020.

[351]
X. Ye, S. Iyer, A. Celikyilmaz, V. Stoyanov, G. Durrett, and R. Pasunuru.
Complementary explanations for effective in-context learning.
arXiv preprint arXiv:2211.13892, 2022.

[352]
E. Yilmaz and C. Toraman.
D2u: Distance-to-uniform learning for out-of-scope detection.
In Proceedings of the 2022 Conference of the North American
Chapter of the Association for Computational Linguistics: Human Language
Technologies, pages 2093–2108, 2022.

[353]
J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku,
Y. Yang, B. K. Ayan, et al.
Scaling autoregressive models for content-rich text-to-image
arXiv preprint arXiv:2206.10789, 2022.

[354]
J. Zeng, X. Zheng, J. Xu, L. Li, L. Yuan, and X. Huang.
Certified robustness to text adversarial attacks by randomized
arXiv preprint arXiv:2105.03743, 2021.

[355]
W. Zeng, X. Ren, T. Su, H. Wang, Y. Liao, Z. Wang, X. Jiang, Z. Yang, K. Wang,
X. Zhang, et al.
Pangu-$\alpha$: Large-scale autoregressive pretrained chinese
language models with auto-parallel computation.
arXiv preprint arXiv:2104.12369, 2021.

[356]
W. Zeng, X. Ren, T. Su, H. Wang, Y. Liao, Z. Wang, X. Jiang, Z. Yang, K. Wang,
X. Zhang, et al.
Pangu-$\alpha$: Large-scale autoregressive pretrained chinese
language models with auto-parallel computation.
arXiv preprint arXiv:2104.12369, 2021.

[357]
Z. Zeng, K. He, Y. Yan, Z. Liu, Y. Wu, H. Xu, H. Jiang, and W. Xu.
Modeling discriminative representations for out-of-domain detection
with supervised contrastive learning.
In Proceedings of the 59th Annual Meeting of the Association for
Computational Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 2: Short Papers), pages 870–878, 2021.

[358]
C. Zhang, W. Ruan, F. Wang, P. Xu, G. Min, and X. Huang.
Model-agnostic reachability analysis on deep neural networks.
arXiv preprint arXiv:2304.00813, 2023.

[359]
C. Zhang, W. Ruan, and P. Xu.
Reachability analysis of neural network control systems.
arXiv preprint arXiv:2301.12100, 2023.

[360]
J. Zhang, Y. Zhao, M. Saleh, and P. Liu.
PEGASUS: Pre-training with extracted gap-sentences for abstractive
In H. D. III and A. Singh, editors, Proceedings of the 37th
International Conference on Machine Learning, volume 119 of Proceedings
of Machine Learning Research, pages 11328–11339. PMLR, 13–18 Jul 2020.

[361]
S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab,
X. Li, X. V. Lin, et al.
Opt: Open pre-trained transformer language models.
arXiv preprint arXiv:2205.01068, 2022.

[362]
T. Zhang, F. Ladhak, E. Durmus, P. Liang, K. McKeown, and T. B. Hashimoto.
Benchmarking large language models for news summarization.
arXiv preprint arXiv:2301.13848, 2023.

[363]
Y. Zhang, A. Albarghouthi, and L. D'Antoni.
Certified robustness to programmable transformations in lstms.
arXiv preprint arXiv:2102.07818, 2021.

[364]
R. Zhao, X. Li, Y. K. Chia, B. Ding, and L. Bing.
Can chatgpt-like generative models guarantee factual accuracy? on the
mistakes of new generation search engines.
arXiv preprint arXiv:2304.11076, 2023.

[365]
W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang,
J. Zhang, Z. Dong, et al.
A survey of large language models.
arXiv preprint arXiv:2303.18223, 2023.

[366]
X. Zhao, W. Huang, X. Huang, V. Robu, and D. Flynn.
Baylime: Bayesian local interpretable model-agnostic explanations.
In C. de Campos and M. H. Maathuis, editors, Proceedings of the
Thirty-Seventh Conference on Uncertainty in Artificial Intelligence, volume
161 of Proceedings of Machine Learning Research, pages 887–896. PMLR,
27–30 Jul 2021.

[367]
X. Zhao, W. Huang, S. Schewe, Y. Dong, and X. Huang.
Detecting operational adversarial examples for reliable deep
In 2021 51st Annual IEEE/IFIP International Conference on
Dependable Systems and Networks - Supplemental Volume (DSN-S), pages 5–6,

[368]
Z. Zhao, D. Dua, and S. Singh.
Generating natural adversarial examples.
arXiv preprint arXiv:1710.11342, 2017.

[369]
Q. Zhong, L. Ding, J. Liu, B. Du, and D. Tao.
Can chatgpt understand too? a comparative study on chatgpt and
fine-tuned bert.
arXiv preprint arXiv:2302.10198, 2023.

[370]
C. Zhou, Q. Li, C. Li, J. Yu, Y. Liu, G. Wang, K. Zhang, C. Ji, Q. Yan, L. He,
et al.
A comprehensive survey on pretrained foundation models: A history
from bert to chatgpt.
arXiv preprint arXiv:2302.09419, 2023.

[371]
W. Zhou, F. Liu, and M. Chen.
Contrastive out-of-distribution detection for pretrained
In Proceedings of the 2021 Conference on Empirical Methods in
Natural Language Processing (EMNLP), 2021.

[372]
Y. Zhou, P. Liu, and X. Qiu.
Knn-contrastive learning for out-of-domain intent classification.
In Proceedings of the 60th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), pages 5129–5141, 2022.

[373]
R.-J. Zhu, Q. Zhao, and J. K. Eshraghian.
Spikegpt: Generative pre-trained language model with spiking neural
arXiv preprint arXiv:2302.13939, 2023.

[374]
D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei,
P. Christiano, and G. Irving.
Fine-tuning language models from human preferences.
arXiv preprint arXiv:1909.08593, 2019.