* Ram et al. (2018) Ram A, Prasad R, Khatri C, Venkatesh A, Gabriel R, Liu Q, Nunn J, Hedayatnia B, Cheng M, Nagar A, et al. (2018) Conversational ai: The science behind the alexa prize. arXiv preprint arXiv:180103604
  * Rameshkumar and Bailey (2020) Rameshkumar R, Bailey P (2020) Storytelling with dialogue: A critical role dungeons and dragons dataset. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp 5121–5134
  * Ramshaw and Marcus (1999) Ramshaw LA, Marcus MP (1999) Text chunking using transformation-based learning. In: Natural language processing using very large corpora, Springer, pp 157–176
  * Rashkin et al. (2018) Rashkin H, Smith EM, Li M, Boureau YL (2018) Towards empathetic open-domain conversation models: A new benchmark and dataset. arXiv preprint arXiv:181100207
  * Rastogi et al. (2020) Rastogi A, Zang X, Sunkara S, Gupta R, Khaitan P (2020) Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 34, no 5, pp 8689–8696
  * Ravuri and Stolcke (2015) Ravuri S, Stolcke A (2015) Recurrent neural network and lstm models for lexical utterance classification. In: Sixteenth Annual Conference of the International Speech Communication Association
  * Ravuri and Stolcke (2016) Ravuri S, Stolcke A (2016) A comparative study of recurrent neural network models for lexical domain classification. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 6075–6079
  * Rawat and Wang (2017) Rawat W, Wang Z (2017) Deep convolutional neural networks for image classification: A comprehensive review. Neural computation 29(9):2352–2449
  * Reiter (1994) Reiter E (1994) Has a consensus nl generation architecture appeared, and is it psycholinguistically plausible? arXiv preprint cmp-lg/9411032
  * Ren et al. (2013) Ren H, Xu W, Zhang Y, Yan Y (2013) Dialog state tracking using conditional random fields. In: Proceedings of the SIGDIAL 2013 Conference, pp 457–461
  * Ren et al. (2018) Ren L, Xie K, Chen L, Yu K (2018) Towards universal dialogue state tracking. arXiv preprint arXiv:181009587
  * Ren et al. (2020) Ren P, Chen Z, Ren Z, Kanoulas E, Monz C, de Rijke M (2020) Conversations with search engines. arXiv preprint arXiv:200414162
  * Ritter et al. (2011) Ritter A, Cherry C, Dolan WB (2011) Data-driven response generation in social media. In: Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp 583–593
  * Rodriguez et al. (2020) Rodriguez P, Crook P, Moon S, Wang Z (2020) Information seeking in the spirit of learning: a dataset for conversational curiosity. arXiv preprint arXiv:200500172
  * Saha et al. (2018) Saha A, Khapra M, Sankaranarayanan K (2018) Towards building large scale multimodal domain-aware conversation systems. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 32, no 1
  * Saha et al. (2020) Saha T, Patra A, Saha S, Bhattacharyya P (2020) Towards emotion-aided multi-modal dialogue act classification. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp 4361–4372
  * Sankar et al. (2019) Sankar C, Subramanian S, Pal C, Chandar S, Bengio Y (2019) Do neural dialog systems use the conversation history effectively? an empirical study. arXiv preprint arXiv:190601603
  * Santhanam and Shaikh (2019) Santhanam S, Shaikh S (2019) A survey of natural language generation techniques with a focus on dialogue systems-past, present and future directions. arXiv preprint arXiv:190600500
  * Sarikaya et al. (2011) Sarikaya R, Hinton GE, Ramabhadran B (2011) Deep belief nets for natural language call-routing. In: 2011 IEEE International conference on acoustics, speech and signal processing (ICASSP), IEEE, pp 5680–5683
  * Sarikaya et al. (2014) Sarikaya R, Hinton GE, Deoras A (2014) Application of deep belief networks for natural language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing 22(4):778–784
  * Sato et al. (2020) Sato S, Akama R, Ouchi H, Suzuki J, Inui K (2020) Evaluating dialogue generation systems via response selection. arXiv preprint arXiv:200414302
  * Schatzmann and Young (2009) Schatzmann J, Young S (2009) The hidden agenda user simulation model. IEEE transactions on audio, speech, and language processing 17(4):733–747
  * Schuster and Paliwal (1997) Schuster M, Paliwal KK (1997) Bidirectional recurrent neural networks. IEEE transactions on Signal Processing 45(11):2673–2681
  * Serban et al. (2016) Serban I, Sordoni A, Bengio Y, Courville A, Pineau J (2016) Building end-to-end dialogue systems using generative hierarchical neural network models. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 30, no 1
  * Serban et al. (2017a) Serban I, Sordoni A, Lowe R, Charlin L, Pineau J, Courville A, Bengio Y (2017a) A hierarchical latent variable encoder-decoder model for generating dialogues. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 31, no 1
  * Serban et al. (2017b) Serban IV, Sankar C, Germain M, Zhang S, Lin Z, Subramanian S, Kim T, Pieper M, Chandar S, Ke NR, et al. (2017b) A deep reinforcement learning chatbot. arXiv preprint arXiv:170902349
  * Serras et al. (2019) Serras M, Torres MI, del Pozo A (2019) Goal-conditioned user modeling for dialogue systems using stochastic bi-automata. In: ICPRAM, pp 128–134
  * Shah et al. (2018) Shah P, Hakkani-Tür D, Tür G, Rastogi A, Bapna A, Nayak N, Heck L (2018) Building a conversational agent overnight with dialogue self-play. arXiv preprint arXiv:180104871
  * Shan et al. (2020) Shan Y, Li Z, Zhang J, Meng F, Feng Y, Niu C, Zhou J (2020) A contextual hierarchical attention network with adaptive objective for dialogue state tracking. arXiv preprint arXiv:200601554
  * Shao et al. (2017) Shao L, Gouws S, Britz D, Goldie A, Strope B, Kurzweil R (2017) Generating long and diverse responses with neural conversation models. ArXiv abs/1701.03185
  * Shao and Nakashole (2020) Shao Y, Nakashole N (2020) Chartdialogs: Plotting from natural language instructions. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp 3559–3574
  * Shen et al. (2019) Shen L, Feng Y, Zhan H (2019) Modeling semantic relationship in multi-turn conversations with hierarchical latent variables. arXiv preprint arXiv:190607429
  * Shi and Weninger (2017) Shi B, Weninger T (2017) Proje: Embedding projection for knowledge graph completion. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 31, no 1
  * Shuster et al. (2018) Shuster K, Humeau S, Bordes A, Weston J (2018) Image chat: Engaging grounded conversations. arXiv preprint arXiv:181100945
  * Shuster et al. (2019) Shuster K, Ju D, Roller S, Dinan E, Boureau YL, Weston J (2019) The dialogue dodecathlon: Open-domain knowledge and image grounded conversational agents. arXiv preprint arXiv:191103768
  * Shuster et al. (2020) Shuster K, Humeau S, Bordes A, Weston J (2020) Image-chat: Engaging grounded conversations. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, pp 2414–2429, DOI 10.18653/v1/2020.acl-main.219, URL https://www.aclweb.org/anthology/2020.acl-main.219
  * Siddharthan (2001) Siddharthan A (2001) Ehud reiter and robert dale. building natural language generation systems. Natural Language Engineering 7(3):271
  * Silver et al. (2016) Silver D, Huang A, Maddison CJ, Guez A, Sifre L, Van Den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, et al. (2016) Mastering the game of go with deep neural networks and tree search. nature 529(7587):484–489
  * Simonyan and Zisserman (2014) Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14091556
  * Singh et al. (2002) Singh S, Litman D, Kearns M, Walker M (2002) Optimizing dialogue management with reinforcement learning: Experiments with the njfun system. Journal of Artificial Intelligence Research 16:105–133
  * Singla et al. (2020) Singla K, Chen Z, Atkins D, Narayanan S (2020) Towards end-2-end learning for predicting behavior codes from spoken utterances in psychotherapy conversations. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp 3797–3803
  * Sinha et al. (2020) Sinha K, Parthasarathi P, Wang J, Lowe R, Hamilton WL, Pineau J (2020) Learning an unreferenced metric for online dialogue evaluation. arXiv preprint arXiv:200500583
  * Smith et al. (2020) Smith EM, Williamson M, Shuster K, Weston J, Boureau YL (2020) Can you put it all together: Evaluating conversational agents’ ability to blend skills. arXiv preprint arXiv:200408449
  * Socher et al. (2013) Socher R, Chen D, Manning CD, Ng A (2013) Reasoning with neural tensor networks for knowledge base completion. In: Advances in neural information processing systems, Citeseer, pp 926–934
  * Song et al. (2020a) Song H, Wang Y, Zhang WN, Liu X, Liu T (2020a) Generate, delete and rewrite: A three-stage framework for improving persona consistency of dialogue generation. arXiv preprint arXiv:200407672
  * Song et al. (2020b) Song H, Wang Y, Zhang WN, Zhao Z, Liu T, Liu X (2020b) Profile consistency identification for open-domain dialogue agents. arXiv preprint arXiv:200909680
  * Song et al. (2016) Song Y, Yan R, Li X, Zhao D, Zhang M (2016) Two are better than one: An ensemble of retrieval-and generation-based dialog systems. arXiv preprint arXiv:161007149
  * Song et al. (2019) Song Z, Zheng X, Liu L, Xu M, Huang XJ (2019) Generating responses with a specific emotion in dialog. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp 3685–3695
  * Sordoni et al. (2015a) Sordoni A, Bengio Y, Vahabi H, Lioma C, Grue Simonsen J, Nie JY (2015a) A hierarchical recurrent encoder-decoder for generative context-aware query suggestion. In: Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, pp 553–562
  * Sordoni et al. (2015b) Sordoni A, Galley M, Auli M, Brockett C, Ji Y, Mitchell M, Nie JY, Gao J, Dolan B (2015b) A neural network approach to context-sensitive generation of conversational responses. arXiv preprint arXiv:150606714
  * Stasaski et al. (2020) Stasaski K, Yang GH, Hearst MA (2020) More diverse dialogue datasets via diversity-informed data collection. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp 4958–4968
  * Stent et al. (2005) Stent A, Marge M, Singhai M (2005) Evaluating evaluation methods for generation in the presence of variation. In: international conference on intelligent text processing and computational linguistics, Springer, pp 341–351
  * Su et al. (2019a) Su H, Shen X, Zhang R, Sun F, Hu P, Niu C, Zhou J (2019a) Improving multi-turn dialogue modelling with utterance rewriter. arXiv preprint arXiv:190607004
  * Su et al. (2020) Su H, Shen X, Zhao S, Zhou X, Hu P, Zhong R, Niu C, Zhou J (2020) Diversifying dialogue generation with non-conversational text. arXiv preprint arXiv:200504346
  * Su et al. (2015) Su PH, Vandyke D, Gasic M, Kim D, Mrksic N, Wen TH, Young S (2015) Learning from real users: Rating dialogue success with neural networks for reinforcement learning in spoken dialogue systems. arXiv preprint arXiv:150803386
  * Su et al. (2016) Su PH, Gasic M, Mrksic N, Rojas-Barahona L, Ultes S, Vandyke D, Wen TH, Young S (2016) Continuously learning neural dialogue management. arXiv preprint arXiv:160602689
  * Su et al. (2019b) Su SY, Huang CW, Chen YN (2019b) Dual supervised learning for natural language understanding and generation. arXiv preprint arXiv:190506196
  * Sukhbaatar et al. (2015) Sukhbaatar S, Szlam A, Weston J, Fergus R (2015) End-to-end memory networks. arXiv preprint arXiv:150308895
  * Sutskever et al. (2014) Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. arXiv preprint arXiv:14093215
  * Sutton (1988) Sutton RS (1988) Learning to predict by the methods of temporal differences. Machine learning 3(1):9–44
  * Sutton et al. (1999) Sutton RS, McAllester DA, Singh SP, Mansour Y, et al. (1999) Policy gradient methods for reinforcement learning with function approximation. In: NIPs, Citeseer, vol 99, pp 1057–1063
  * Szegedy et al. (2015) Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2015) Going deeper with convolutions. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1–9
  * Takanobu et al. (2020) Takanobu R, Liang R, Huang M (2020) Multi-agent task-oriented dialog policy learning with role-aware reward decomposition. arXiv preprint arXiv:200403809
  * Takmaz et al. (2020) Takmaz E, Giulianelli M, Pezzelle S, Sinclair A, Fernández R (2020) Refer, reuse, reduce: Generating subsequent references in visual and conversational contexts. arXiv preprint arXiv:201104554
  * Tamar et al. (2016) Tamar A, Wu Y, Thomas G, Levine S, Abbeel P (2016) Value iteration networks. arXiv preprint arXiv:160202867
  * Tanana et al. (2016) Tanana M, Hallgren KA, Imel ZE, Atkins DC, Srikumar V (2016) A comparison of natural language processing methods for automated coding of motivational interviewing. Journal of substance abuse treatment 65:43–50
  * Tang et al. (2015) Tang D, Qin B, Liu T (2015) Learning semantic representations of users and products for document level sentiment classification. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp 1014–1023
  * Tang et al. (2019) Tang J, Zhao T, Xiong C, Liang X, Xing EP, Hu Z (2019) Target-guided open-domain conversation. arXiv preprint arXiv:190511553
  * Tao et al. (2018) Tao C, Mou L, Zhao D, Yan R (2018) Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 32, no 1
  * Tao et al. (2019) Tao C, Wu W, Xu C, Hu W, Zhao D, Yan R (2019) One time of interaction may not be enough: Go deep with an interaction-over-interaction network for response selection in dialogues. In: Proceedings of the 57th annual meeting of the association for computational linguistics, pp 1–11
  * Tay et al. (2019) Tay Y, Wang S, Tuan LA, Fu J, Phan MC, Yuan X, Rao J, Hui SC, Zhang A (2019) Simple and effective curriculum pointer-generator networks for reading comprehension over long narratives. arXiv preprint arXiv:190510847
  * Teng et al. (2021) Teng D, Qin L, Che W, Zhao S, Liu T (2021) Injecting word information with multi-level word adapter for chinese spoken language understanding. In: ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, pp 8188–8192
  * Theune (2003) Theune M (2003) Natural language generation for dialogue: system survey. Centre for Telematics and Information Technology, University of Twente
  * Thomas et al. (2006) Thomas M, Pang B, Lee L (2006) Get out the vote: Determining support or opposition from congressional floor-debate transcripts. arXiv preprint cs/0607062
  * Tian et al. (2019) Tian Z, Bi W, Li X, Zhang NL (2019) Learning to abstract for memory-augmented conversational response generation. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp 3816–3825
  * Tiedemann (2012) Tiedemann J (2012) Parallel data, tools and interfaces in opus. In: Lrec, vol 2012, pp 2214–2218
  * Tonelli et al. (2010) Tonelli S, Riccardi G, Prasad R, Joshi AK (2010) Annotation of discourse relations for conversational spoken dialogs. In: LREC
  * Tran and Nguyen (2017) Tran VK, Nguyen LM (2017) Semantic refinement gru-based neural language generation for spoken dialogue systems. In: International Conference of the Pacific Association for Computational Linguistics, Springer, pp 63–75
  * Tur et al. (2010) Tur G, Hakkani-Tür D, Heck L (2010) What is left to be understood in atis? In: 2010 IEEE Spoken Language Technology Workshop, IEEE, pp 19–24
  * Tur et al. (2012) Tur G, Deng L, Hakkani-Tür D, He X (2012) Towards deeper understanding: Deep convex networks for semantic utterance classification. In: 2012 IEEE international conference on acoustics, speech and signal processing (ICASSP), IEEE, pp 5045–5048
  * Ultes et al. (2017) Ultes S, Barahona LMR, Su PH, Vandyke D, Kim D, Casanueva I, Budzianowski P, Mrkšić N, Wen TH, Gasic M, et al. (2017) Pydial: A multi-domain statistical dialogue system toolkit. In: Proceedings of ACL 2017, System Demonstrations, pp 73–78
  * Urbanek et al. (2019) Urbanek J, Fan A, Karamcheti S, Jain S, Humeau S, Dinan E, Rocktäschel T, Kiela D, Szlam A, Weston J (2019) Learning to speak and act in a fantasy text adventure game. arXiv preprint arXiv:190303094
  * Vaswani et al. (2017) Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Lu, Polosukhin I (2017) Attention is all you need. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in Neural Information Processing Systems, Curran Associates, Inc., vol 30, URL https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * Vijayakumar et al. (2016) Vijayakumar AK, Cogswell M, Selvaraju RR, Sun Q, Lee S, Crandall D, Batra D (2016) Diverse beam search: Decoding diverse solutions from neural sequence models. arXiv preprint arXiv:161002424
  * Vinyals and Le (2015) Vinyals O, Le Q (2015) A neural conversational model. arXiv preprint arXiv:150605869
  * Viterbi (1967) Viterbi A (1967) Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE transactions on Information Theory 13(2):260–269
  * Vougiouklis et al. (2016) Vougiouklis P, Hare J, Simperl E (2016) A neural network approach for knowledge-driven response generation. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, The COLING 2016 Organizing Committee, Osaka, Japan, pp 3370–3380, URL https://www.aclweb.org/anthology/C16-1318
  * Walker et al. (1997) Walker MA, Litman DJ, Kamm CA, Abella A (1997) Paradise: A framework for evaluating spoken dialogue agents. arXiv preprint cmp-lg/9704004
  * Wan and McAuley (2016) Wan M, McAuley J (2016) Modeling ambiguity, subjectivity, and diverging viewpoints in opinion question answering systems. In: 2016 IEEE 16th international conference on data mining (ICDM), IEEE, pp 489–498
  * Wang et al. (2020a) Wang H, Peng B, Wong KF (2020a) Learning efficient dialogue policy from demonstrations through shaping. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp 6355–6365
  * Wang et al. (2020b) Wang K, Tian J, Wang R, Quan X, Yu J (2020b) Multi-domain dialogue acts and response co-generation. arXiv preprint arXiv:200412363
  * Wang et al. (2020c) Wang L, Li J, Zeng X, Zhang H, Wong KF (2020c) Continuity of topic, interaction, and query: Learning to quote in online conversations. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp 6640–6650
  * Wang et al. (2020d) Wang S, Zhou K, Lai K, Shen J (2020d) Task-completion dialogue policy learning via Monte Carlo tree search with dueling network. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online, pp 3461–3471, DOI 10.18653/v1/2020.emnlp-main.278, URL https://www.aclweb.org/anthology/2020.emnlp-main.278
  * Wang et al. (2019a) Wang W, Zhang J, Li Q, Hwang MY, Zong C, Li Z (2019a) Incremental learning from scratch for task-oriented dialogue systems. arXiv preprint arXiv:190604991
  * Wang and Yuan (2016) Wang X, Yuan C (2016) Recent advances on human-computer dialogue. CAAI Transactions on Intelligence Technology 1(4):303–312
  * Wang et al. (2019b) Wang X, Shi W, Kim R, Oh Y, Yang S, Zhang J, Yu Z (2019b) Persuasion for good: Towards a personalized persuasive dialogue system for social good. arXiv preprint arXiv:190606725
  * Wang et al. (2018) Wang Y, Shen Y, Jin H (2018) A bi-model based rnn semantic frame parsing model for intent detection and slot filling. arXiv preprint arXiv:181210235
  * Wang et al. (2020e) Wang Y, Guo Y, Zhu S (2020e) Slot attention with value normalization for multi-domain dialogue state tracking. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp 3019–3028
  * Wang et al. (2020f) Wang Y, Joty S, Lyu M, King I, Xiong C, Hoi SC (2020f) VD-BERT: A Unified Vision and Dialog Transformer with BERT. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online, pp 3325–3338, DOI 10.18653/v1/2020.emnlp-main.269, URL https://www.aclweb.org/anthology/2020.emnlp-main.269
  * Wang et al. (2014) Wang Z, Zhang J, Feng J, Chen Z (2014) Knowledge graph embedding by translating on hyperplanes. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 28, no 1
  * Wang et al. (2016) Wang Z, Schaul T, Hessel M, Hasselt H, Lanctot M, Freitas N (2016) Dueling network architectures for deep reinforcement learning. In: International conference on machine learning, PMLR, pp 1995–2003
  * Welleck et al. (2018) Welleck S, Weston J, Szlam A, Cho K (2018) Dialogue natural language inference. arXiv preprint arXiv:181100671
  * Wen et al. (2018) Wen H, Liu Y, Che W, Qin L, Liu T (2018) Sequence-to-sequence learning for task-oriented dialogue with dialogue state representation. In: Proceedings of the 27th International Conference on Computational Linguistics, Association for Computational Linguistics, Santa Fe, New Mexico, USA, pp 3781–3792, URL https://www.aclweb.org/anthology/C18-1320
  * Wen et al. (2015a) Wen TH, Gasic M, Kim D, Mrksic N, Su PH, Vandyke D, Young S (2015a) Stochastic language generation in dialogue using recurrent neural networks with convolutional sentence reranking. arXiv preprint arXiv:150801755
  * Wen et al. (2015b) Wen TH, Gasic M, Mrksic N, Su PH, Vandyke D, Young S (2015b) Semantically conditioned lstm-based natural language generation for spoken dialogue systems. arXiv preprint arXiv:150801745
  * Wen et al. (2016a) Wen TH, Gasic M, Mrksic N, Rojas-Barahona LM, Su PH, Ultes S, Vandyke D, Young S (2016a) Conditional generation and snapshot learning in neural dialogue systems. arXiv preprint arXiv:160603352
  * Wen et al. (2016b) Wen TH, Gasic M, Mrksic N, Rojas-Barahona LM, Su PH, Vandyke D, Young S (2016b) Multi-domain neural network language generation for spoken dialogue systems. arXiv preprint arXiv:160301232
  * Wen et al. (2016c) Wen TH, Vandyke D, Mrksic N, Gasic M, Rojas-Barahona LM, Su PH, Ultes S, Young S (2016c) A network-based end-to-end trainable task-oriented dialogue system. arXiv preprint arXiv:160404562
  * Weston et al. (2014) Weston J, Chopra S, Bordes A (2014) Memory networks. arXiv preprint arXiv:14103916
  * Williams et al. (2013) Williams J, Raux A, Ramachandran D, Black A (2013) The dialog state tracking challenge. In: Proceedings of the SIGDIAL 2013 Conference, Association for Computational Linguistics, Metz, France, pp 404–413, URL https://www.aclweb.org/anthology/W13-4065
  * Williams (2007) Williams JD (2007) Partially observable markov decision processes for spoken dialogue management. PhD thesis, University of Cambridge
  * Williams (2013) Williams JD (2013) Multi-domain learning and generalization in dialog state tracking. In: Proceedings of the SIGDIAL 2013 Conference, pp 433–441
  * Williams (2014) Williams JD (2014) Web-style ranking and slu combination for dialog state tracking. In: Proceedings of the 15th Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pp 282–291
  * Williams and Zweig (2016) Williams JD, Zweig G (2016) End-to-end lstm-based dialog control optimized with supervised and reinforcement learning. arXiv preprint arXiv:160601269
  * Williams et al. (2017) Williams JD, Asadi K, Zweig G (2017) Hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning. arXiv preprint arXiv:170203274
  * Williams (1992) Williams RJ (1992) Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8(3-4):229–256
  * Williams and Zipser (1989) Williams RJ, Zipser D (1989) A learning algorithm for continually running fully recurrent neural networks. Neural computation 1(2):270–280
  * Wu and Xiong (2020) Wu CS, Xiong C (2020) Probing task-oriented dialogue representation from language models. arXiv preprint arXiv:201013912
  * Wu et al. (2019a) Wu CS, Madotto A, Hosseini-Asl E, Xiong C, Socher R, Fung P (2019a) Transferable multi-domain state generator for task-oriented dialogue systems. arXiv preprint arXiv:190508743
  * Wu et al. (2020a) Wu CS, Hoi S, Socher R, Xiong C (2020a) Tod-bert: Pre-trained natural language understanding for task-oriented dialogues. arXiv preprint arXiv:200406871
  * Wu et al. (2019b) Wu J, Wang X, Wang WY (2019b) Self-supervised dialogue learning. arXiv preprint arXiv:190700448
  * Wu et al. (2020b) Wu S, Li Y, Zhang D, Zhou Y, Wu Z (2020b) Diverse and informative dialogue generation with context-specific commonsense knowledge awareness. In: Proceedings of the 58th annual meeting of the association for computational linguistics, pp 5811–5820
  * Wu et al. (2019c) Wu W, Guo Z, Zhou X, Wu H, Zhang X, Lian R, Wang H (2019c) Proactive human-machine conversation with explicit conversation goals. arXiv preprint arXiv:190605572
  * Wu et al. (2016) Wu Y, Wu W, Xing C, Zhou M, Li Z (2016) Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. arXiv preprint arXiv:161201627
  * Xiao et al. (2015) Xiao H, Huang M, Hao Y, Zhu X (2015) Transg: A generative mixture model for knowledge graph embedding. arXiv preprint arXiv:150905488
  * Xiao et al. (2017) Xiao H, Huang M, Meng L, Zhu X (2017) Ssp: semantic space projection for knowledge graph embedding with text descriptions. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 31, no 1
  * Xie et al. (2016) Xie R, Liu Z, Jia J, Luan H, Sun M (2016) Representation learning of knowledge graphs with entity descriptions. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 30, no 1
  * Xing et al. (2017) Xing C, Wu W, Wu Y, Liu J, Huang Y, Zhou M, Ma WY (2017) Topic aware neural response generation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 31, no 1
  * Xing et al. (2018) Xing C, Wu Y, Wu W, Huang Y, Zhou M (2018) Hierarchical recurrent attention network for response generation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 32, no 1
  * Xu et al. (2019) Xu C, Wu W, Tao C, Hu H, Schuerman M, Wang Y (2019) Neural response generation with meta-words. arXiv preprint arXiv:190606050
  * Xu et al. (2020a) Xu J, Wang H, Niu ZY, Wu H, Che W, Liu T (2020a) Conversational graph grounded policy learning for open-domain conversation generation. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp 1835–1845
  * Xu et al. (2020b) Xu K, Tan H, Song L, Wu H, Zhang H, Song L, Yu D (2020b) Semantic role labeling guided multi-turn dialogue rewriter. arXiv preprint arXiv:201001417
  * Yadollahi et al. (2017) Yadollahi A, Shahraki AG, Zaiane OR (2017) Current state of text sentiment analysis from opinion to emotion mining. ACM Computing Surveys (CSUR) 50(2):1–33
  * Yang et al. (2020) Yang S, Zhang R, Erfani S (2020) Graphdialog: Integrating graph knowledge into end-to-end task-oriented dialogue systems. arXiv preprint arXiv:201001447
  * Yann et al. (2014) Yann D, Tur G, Hakkani-Tur D, Heck L (2014) Zero-shot learning and clustering for semantic utterance classification using deep learning. In: International Conference on Learning Representations (cited on page 28)
  * Yao et al. (2013) Yao K, Zweig G, Hwang MY, Shi Y, Yu D (2013) Recurrent neural networks for language understanding. In: Interspeech, pp 2524–2528
  * Yao et al. (2014) Yao K, Peng B, Zhang Y, Yu D, Zweig G, Shi Y (2014) Spoken language understanding using long short-term memory neural networks. In: 2014 IEEE Spoken Language Technology Workshop (SLT), IEEE, pp 189–194
  * Yao et al. (2016) Yao K, Peng B, Zweig G, Wong KF (2016) An attentional neural conversation model with improved specificity. arXiv preprint arXiv:160601292
  * Yih et al. (2015) Yih Wt, He X, Gao J (2015) Deep learning and continuous representations for natural language processing. In: Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorial Abstracts, pp 6–8
  * Yin et al. (2015) Yin J, Jiang X, Lu Z, Shang L, Li H, Li X (2015) Neural generative question answering. arXiv preprint arXiv:151201337
  * Yoshino et al. (2018) Yoshino K, Hori C, Perez J, D’Haro LF, Polymenakos L, Gunasekara C, Lasecki WS, Kummerfeld J, Galley M, Brockett C, et al. (2018) The 7th dialog system technology challenge. arXiv preprint
  * Young et al. (2010) Young S, Gašić M, Keizer S, Mairesse F, Schatzmann J, Thomson B, Yu K (2010) The hidden information state model: A practical framework for pomdp-based spoken dialogue management. Computer Speech & Language 24(2):150–174
  * Young et al. (2018) Young T, Cambria E, Chaturvedi I, Zhou H, Biswas S, Huang M (2018) Augmenting end-to-end dialogue systems with commonsense knowledge. In: AAAI, pp 4970–4977
  * Young et al. (2020) Young T, Pandelea V, Poria S, Cambria E (2020) Dialogue systems with audio context. Neurocomputing 388:102–109
  * Yu and Joty (2020) Yu T, Joty S (2020) Online conversation disentanglement with pointer networks. arXiv preprint arXiv:201011080
  * Zahiri and Choi (2017) Zahiri SM, Choi JD (2017) Emotion detection on tv show transcripts with sequence-based convolutional neural networks. arXiv preprint arXiv:170804299
  * Zeiler and Fergus (2014) Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: European conference on computer vision, Springer, pp 818–833
  * Zhang et al. (2018a) Zhang C, Li Y, Du N, Fan W, Yu PS (2018a) Joint slot filling and intent detection via capsule neural networks. arXiv preprint arXiv:181209471
  * Zhang et al. (2019a) Zhang H, Lan Y, Pang L, Guo J, Cheng X (2019a) Recosa: Detecting the relevant contexts with self-attention for multi-turn dialogue generation. arXiv preprint arXiv:190705339
  * Zhang et al. (2019b) Zhang H, Liu Z, Xiong C, Liu Z (2019b) Grounded conversation generation as guided traverses in commonsense knowledge graphs. arXiv preprint arXiv:191102707
  * Zhang and Danescu-Niculescu-Mizil (2020) Zhang J, Danescu-Niculescu-Mizil C (2020) Balancing objectives in counseling conversations: Advancing forwards or looking backwards. arXiv preprint arXiv:200504245
  * Zhang et al. (2018b) Zhang S, Dinan E, Urbanek J, Szlam A, Kiela D, Weston J (2018b) Personalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:180107243
  * Zhang and Wang (2016) Zhang X, Wang H (2016) A joint model of intent determination and slot filling for spoken language understanding. In: IJCAI, vol 16, pp 2993–2999
  * Zhang and Wallace (2017) Zhang Y, Wallace B (2017) A sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification. In: Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Asian Federation of Natural Language Processing, Taipei, Taiwan, pp 253–263, URL https://www.aclweb.org/anthology/I17-1026
  * Zhang et al. (2020) Zhang Y, Ou Z, Wang H, Feng J (2020) A probabilistic end-to-end task-oriented dialog model with latent belief states towards semi-supervised learning. arXiv preprint arXiv:200908115
  * Zhang et al. (2018c) Zhang Z, Li J, Zhu P, Zhao H, Liu G (2018c) Modeling multi-turn conversation with deep utterance aggregation. arXiv preprint arXiv:180609102
  * Zhang et al. (2019c) Zhang Z, Li X, Gao J, Chen E (2019c) Budgeted policy learning for task-oriented dialogue systems. arXiv preprint arXiv:190600499
  * Zhao and Eskenazi (2016) Zhao T, Eskenazi M (2016) Towards end-to-end learning for dialog state tracking and management using deep reinforcement learning. arXiv preprint arXiv:160602560
  * Zhao and Eskenazi (2018) Zhao T, Eskenazi M (2018) Zero-shot dialog generation with cross-domain latent actions. arXiv preprint arXiv:180504803
  * Zhao et al. (2020a) Zhao T, Lala D, Kawahara T (2020a) Designing precise and robust dialogue response evaluators. arXiv preprint arXiv:200404908
  * Zhao et al. (2020b) Zhao X, Wu W, Xu C, Tao C, Zhao D, Yan R (2020b) Knowledge-grounded dialogue generation with pre-trained language models. arXiv preprint arXiv:201008824
  * Zhong et al. (2020) Zhong P, Zhang C, Wang H, Liu Y, Miao C (2020) Towards persona-based empathetic conversational models. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp 6556–6566, DOI 10.18653/v1/2020.emnlp-main.531
  * Zhou et al. (2016) Zhou H, Huang M, Zhu X (2016) Context-aware natural language generation for spoken dialogue systems. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp 2032–2041
  * Zhou et al. (2020) Zhou H, Zheng C, Huang K, Huang M, Zhu X (2020) Kdconv: a chinese multi-domain dialogue dataset towards multi-turn knowledge-driven conversation. arXiv preprint arXiv:200404100
  * Zhou et al. (2018) Zhou K, Prabhumoye S, Black AW (2018) A dataset for document grounded conversations. arXiv preprint arXiv:180907358
  * Zhou and Wang (2017) Zhou X, Wang WY (2017) Mojitalk: Generating emotional responses at scale. arXiv preprint arXiv:171104090
  * Zhu et al. (2018) Zhu Q, Cui L, Zhang W, Wei F, Liu T (2018) Retrieval-enhanced adversarial training for neural response generation. arXiv preprint arXiv:180904276
  * Zhu et al. (2020) Zhu Q, Zhang W, Liu T, Wang WY (2020) Counterfactual off-policy training for neural dialogue generation. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp 3438–3448


# Towards Large Language Model driven Reference-less Translation Evaluation
for English and Indian Languages

Vandan Mujadia  Pruthwik Mishra  Arafat Ahsan  Dipti Misra Sharma

###### Abstract

With the primary focus on evaluating the effectiveness of large language
models for automatic reference-less translation assessment, this work presents
our experiments on mimicking human direct assessment to evaluate the quality
of translations in English and Indian languages. We constructed a translation
evaluation task where we performed zero-shot learning, in-context example-
driven learning, and fine-tuning of large language models to provide a score
out of 100, where 100 represents a perfect translation and 1 represents a poor
translation. We compared the performance of our trained systems with existing
methods such as COMET, BERT-Scorer, and LABSE, and found that the LLM-based
evaluator (LLaMA-2-13B) achieves a comparable or higher overall correlation
with human judgments for the considered Indian language pairs (Refer figure
1).

Towards Large Language Model driven Reference-less Translation Evaluation for
English and Indian Languages

##  1 Introduction

The field of natural language processing (NLP) and artificial intelligence
(AI) has been transformed by the rapid advancements in Large Language Models
(LLMs), as they have demonstrated their capabilities in a wide range of
natural language processing tasks, including open/close question answering,
summarization (Chang et al., 2023; Min et al., 2023), code completion, and
code debugging (Wang et al., 2023; Zan et al., 2023; Surameery and Shakor,
2023), etc. These models have significantly impacted NLP applications by
enhancing various aspects of language understanding, generation, and analysis
(Zhao et al., 2023). As LLMs continue to evolve and improve, they hold immense
potential for further advancements in NLP, paving the way for more
sophisticated and intelligent solutions (Hadi et al., 2023).

Translation, on the other hand, plays a vital role in bridging the gap between
different languages, enabling effective communication. With the advancement of
machine translation systems (Ranathunga et al., 2023; Gala et al., 2023),
there has been a significant shift in how translations are produced
(Arivazhagan et al., 2019). Translation engines have made great progress in
understanding languages and generating translations, but they still face
numerous challenges in accurately conveying the intended meaning (Al Sharou
and Specia, 2022; Wan et al., 2022; Hayakawa and Arase, 2020). While these
systems have made remarkable progress, their evaluation remains an
indispensable component in ensuring the accuracy and quality of translations
produced by them (Sangal, 2022).

Figure 1: Spearman co-relation: Human translation evaluation vs different
reference-less translation evaluation metrics. Llama-2-7b-Adapt (lora),
Llama-2-13b-Adapt (lora), Mistral-7b-Adpt (lora), COMET-QE
(https://github.com/Unbabel/COMET) are fine-tuned models using WMT-23 MT-QE
corpora and evaluated on development corpora. BERTScore
(https://github.com/Tiiiger/bert_score) and LABSE
(https://huggingface.co/sentence-transformers/LaBSE) represents direct cosine-
similarity scores.

To date, human involvement has been considered one of the most reliable and
effective ways to evaluate translations (Freitag et al., 2021; Rivera-
Trigueros, 2022; Guzmán et al., 2015). However, this requires a significant
investment of time, expertise, and financial resources. The cost and effort
involved can limit the frequency and scale at which human involved evaluation
can be conducted. Therefore, it is crucial to develop a scalable, replicable,
and efficient automatic method that mimics human evaluation to ensure reliable
and effective translation assessment.

Recent developments in the field of automatic translation evaluation have
demonstrated that techniques utilizing multilingual embeddings have a tendency
to outperform other traditional approaches and display the strongest
correlation with human assessments (Zerva et al., 2022). Notable instances of
these techniques include BERTScore (Zhang et al., 2019) and COMET (Rei et al.,
2020, 2022; Kocmi et al., 2022). To further improve the effectiveness of
automatic translation evaluation, it is reasonable to investigate
methodologies that leverage large language models, considering their notable
capability for comprehension.

In this work, we aim to assess the capability of LLMs and utilize them for
reference-less translation evaluation involving English and Indian languages.
The research questions that we pose are as follows:

  * •

Do LLMs possess zero-shot or in-context translation evaluation capabilities?

  * •

How do fine-tuned LLMs compare with existing state-of-the-art translation
evaluation methods such as COMET, BertScore, and LABSE?

    * –

Does translation evaluation fine-tuning for LLMs enhance their translation
evaluation capabilities? Does fine-tuning improve their performance when
applied to other unseen Indian languages in fine-tuning?

    * –

Does fine-tuning LLMs for both translation and translation evaluation increase
their evaluation capabilities?

In this study, our objective is to gain a clearer understanding of the
reference-less translation evaluation capabilities of several popular large
language models. Specifically, we focus on Opt (Zhang et al., 2022), bloom
Scao et al. (2022), LLaMA-1 111https://huggingface.co/decapoda-
research/llama-7b-hf, MPT 222https://huggingface.co/mosaicml/mpt-7b, Falcon
(Penedo et al., 2023), LLaMA-2 Touvron et al. (2023), and Mistral (Jiang et
al., 2023) for translating English to five Indian languages: Hindi, Gujarati,
Marathi, Tamil, and Telugu.

Reference-less translation evaluation generally involves assessing the
translation of a given source language sentence without a specific reference
translation. In this evaluation, an automatic reference-less evaluation system
assigns a score to the translation, which is similar to the direct assessment
(DA) score (Akhbardeh et al., 2021), commonly used in human translation
evaluation. This score ranges from 1 to 100, with 1 representing a poor
translation and 100 representing a perfect translation. In our work, we
initially evaluate the reference-less translation evaluation capabilities of
the raw large language models mentioned above in both zero-shot (Kojima et
al., 2022) and in-context learning
333https://ai.stanford.edu/blog/understanding-incontext/ (Brown et al., 2020)
scenarios. Subsequently, we employ the LoRa parameter-efficient fine-tuning
technique (Hu et al., 2021), as well as full fine-tuning, to refine the
selected base LLM models. Additionally, considering that LLMs are known for
their multi-task learning abilities (Radford et al., 2019), we explore the
fine-tuning of LLMs on translation corpora alongside translation evaluation
corpora to determine if translation-based fine-tuning improves overall
translation evaluation performance.

The key findings of our study, presented in Figure 1, highlight the
performance of our fine-tuned LLM-based reference-less translation evaluation
models compared to various well-known reference-less evaluation methods, such
as COMET, LABSE, and Bert-scorer, based on their Spearman correlation with
human judgments.

Our findings emphasize the significant potential of large language models for
reference-less translation evaluation tasks involving English and Indian
languages. Raw LLMs do not inherently possess the capabilities for translation
evaluation as they do not provide a score as an evaluation outcome. However,
our multi-lingual LLM based LORa-fine-tuned models (LLaMA-2-7b, LLaMA-2-13b
and Mistral-7b) demonstrate competitive or superior correlation with human
judgments compared to existing reference-less methods like COMET under same
training and evaluation configurations. We did not observe any additional
benefits when we perform translation evaluation fine-tuning with translation
fine-tuning under the multi-task setting.

The results suggest that fine-tuned LLMs hold promise for translation
evaluation in the targeted reference-less translation evaluation task. Our
study represents an essential and pioneering milestone in assessing and
enhancing the reference-less translation evaluation capabilities of LLMs,
involving English and Indian languages.

##  2 Related Work

There are two main categories of automatic machine translation (MT) metrics:
string-based metrics and pretrained models based metrics. In the following
sub-section, we discuss them briefly one by one.

###  2.1 String-based metrics:

String-based metrics involve comparing the coverage of various substrings
between the human-generated reference and MT translations. This includes
metrics such as ChrF (Popović, 2015), BLEU (Papineni et al., 2002), METEOR
(Gupta et al., 2010), or TER (Snover et al., 2009). String-based methods
heavily rely on the quality of reference translations. However, they have the
advantage of predictable and faster performance, as it is computationally easy
to identify which substrings have the most impact on the score.

###  2.2 Pretrained models based metrics:

This category consists of metrics that utilize pretrained models to evaluate
the quality of machine translation (MT) outputs given the source sentence, the
human reference, or both. Evaluation metrics in this category include COMET
(Rei et al., 2020), BLEURT (Sellam et al., 2020), and BERTScore (Zhang et al.,
2019). These models are based on pretrained models such as XLM-RoBERTa
(Conneau et al., 2020) and MBERT (Devlin et al., 2018). These metrics are not
strictly dependent on reference quality and can better evaluate synonyms or
paraphrases. Several studies (Mathur et al., 2020; Kocmi et al., 2022;
Akhbardeh et al., 2021) have demonstrated their superiority over string-based
metrics. However, it is important to note that the performance of these
metrics is influenced by the data they have been trained on, which can
introduce bias. Pretrained model-based metrics can be further categorized as
reference-based and reference-free metrics. As the name suggests, reference-
free metrics (also known as quality estimation metrics) do not require a human
reference for evaluation. COMET-QE (Chimoto and Bassett, 2022; Rei et al.,
2020), BERTScore (Zhang et al., 2019), and LABSE (Feng et al., 2022) fall
under this category of metrics. In this direction, we develop a reference-less
metric utilizing the large language model (LLM) and human-labeled data for
fine-tuning. In the following section, we discuss our considered LLM models
briefly.

##  3 Large Language Models

Language modeling, a well-established task in the field of natural language
processing, has garnered significant attention over the years (Bellegarda,
2004; Bengio et al., 2000). This task involves predicting the probability of
the next token in a sequence of words. Transformers have emerged as the
fundamental architecture underlying many existing Large Language Models
(Vaswani et al., 2017). Transformers based auto-regressive models like GPT
Brown et al. (2020); Radford et al. (2018, 2019) have played a crucial role in
advancing Natural Language Processing (NLP). In this work, we used following
base LLM models to check how we can utilise them for machine translation
evaluation involving English and Indian Languages.
opt-6.7b444https://huggingface.co/facebook/opt-6.7b : The OPT-6.7b (Zhang et
al., 2022) model has been extensively trained on the objective of causal
language modeling (CLM) using English text with 6.7 billion parameters.
Bloom-7B555https://huggingface.co/bigscience/bloom-7b1 : BLOOM Scao et al.
(2022) was the first largest multilingual large language model with causal
language modeling objective and supports 46 languages and 13 programming
languages. It has 7,069,016,064 parameters.
LLaMA-7B666https://huggingface.co/decapoda-research/llama-7b-hf: LLaMA is a
collection of foundation language models ranging from 7B to 65B parameters. In
our experiments we evaluated LLaMA model with 7B parameters where 4096 is the
embedding dimensions and 32 layers and 32 attention head.
MPT-7B777https://huggingface.co/mosaicml/mpt-7b : Similar to above models
MPT-7B model is trained on a large amount of data 1T tokens on causal language
modeling objective.
Falcon888https://huggingface.co/tiiuae/falcon-7b : Falcon (Penedo et al.,
2023) is another large language model trained on causal language modeling
(CLM) objective. Here, we utilised Falcon-7B model which is a 7B parameters
for our experiments.
LLaMA-2-7B999https://huggingface.co/meta-llama/Llama-2-7b-hf and
LLaMA-2-13B101010https://huggingface.co/meta-llama/Llama-2-13b-hf : LLaMA 2
based models Touvron et al. (2023) are also trained on causal language
modeling (CLM) objective and pretrained on 2 trillion tokens of data from
publicly available sources. In our experiments we have experimented with 7B
and 13B LLaMA-2 models. LLaMA-2-7B network has 32 layers and 32 attention
heads while LLaMA-2-13B has 40 layers and 40 attention heads.
Mistral-7B111111https://huggingface.co/mistralai/Mistral-7B-v0.1 : Mistral-7B
Large Language Model (LLM) (Jiang et al., 2023) is a pre-trained on causal
language modeling (CLM) objective with 7 billion parameters. It uses Sliding
Window Attention (SWA) to handle longer sequences at smaller cost and Grouped-
query attention (GQA) for faster inference which reduces the memory
requirement during decoding. It has 4096 embedding dimension, 32 layers and 32
attention heads with context length of 8192.

Figure 2: Prompting Mechanism for reference-less translation evaluation

##  4 Reference-less Translation Evaluation on Raw LLM

To evaluate the effectiveness of the LLMs mentioned above for translation
evaluation tasks without reference, we conducted two different experiments.
The first involved assessing the performance of the pre-trained (raw) LLM for
reference-less translation evaluation. In the second experiment, we performed
example-based in-context learning for the same purpose. Both experiments were
carried out using translation evaluation data mentioned in the section 5.1.

As part of our experimental setup, we configured a prompting pipeline depicted
in Figure 2. This pipeline involved using a Prompt Generator to generate
specific prompts for the source and target language pairs for source and
translation text. Subsequently, an LLM call is triggered to generate a
response, which was then processed by a reply parser to obtain the actual
translation. To ensure high-throughput and memory-efficient inference and
serving for LLMs, we utilized the vLLM library121212https://github.com/vllm-
project/vllm (Kwon et al., 2023). We conducted all experiments using a
temperature parameter of 0, which ensures that the model behaves
deterministically. By setting the temperature to 0, the model is constrained
to select the word with the highest probability, effectively limiting its
choice to the most likely option (Aksitov et al., 2023). All of our
experiments are conducted using this library, and the models are deployed on
A100, 40GB GPUs.

###  4.1 Zero-shot

We performed manual trials to determine the optimal prompt for zero-shot
reference-less translation evaluation. These trials revealed that instructing
an LLM to mimic a human evaluator and provide scores out of 100, combined with
presenting the text in JSON format, resulted in better results (prompt
presented below).

Zero-shot Translation Evaluation Prompt Example:
You are an experienced translation evaluator and you need to evaluate a
translation for <Source Language> language to <Target Language> language.
<Source Language>: <Source Language Text>
<Target Language>: <Target Translated Text>
The evaluation score out of 100 is

###  4.2 Example-based (In-Context Learning - ICL)

In a similar manner, we identified and adjusted the prompt for example-based
in-context learning with LLM. This specific prompt is outlined in the Example
above (ICL Translation Prompt). In all of our experiments, specific to
language pairs, we used a single and identical human translation evaluation
with a score as a contextual learning example before executing the actual
translation evaluation command using the training data mentioned in sub-
section 5.1.

Example-based ICL Translation Evaluation Prompt Example:
If the <Source Language> to <Target Language> translation score by human for
’<Source Language Text>’ to ’<Target Language Translation>’ is <Human Score>
from 100 then following that, if you are an experienced translation evaluator
and you need to evaluate a translation for <Source Language> language to
<Target Language> language.
<Source Language>: <Source Language Text>
<Target Language>: <Target Translated Text>
The evaluation score out of 100 is

##  5 Fine-tuning LLM for Reference-less Translation Evaluation

To assess the potential enhancement in reference-less translation evaluation
performance of LLMs beyond the zero-shot LLM baseline, we performed fine-
tuning using the training corpus and method described below.

Method | Hyper-para | Value
---|---|---
LoRA/Full | LoRA | PEFT131313https://github.com/huggingface/peft; FSDP141414for Full Fine-Tuning
rank | 8
dropout | 0.05
learning rate | 1e-4
batch size | 4
epochs | 5
Table 1: Hyper-parameter configurations of LoRA based and full fine-tuning for
4*A100 40GB GPUs

###  5.1 Human Judgement Train and Development Corpora

For this task, we take the publicly available datasets released under the
quality estimation shared task 151515https://wmt-qe-task.github.io/ (MTQE) in
WMT for Indian languages. English to five Indian languages are included in the
shared task this year. The dataset comprises of 2 predominant language
families: 3 from Indo-Aryan family and 2 Dravidian languages. The details of
the train and development sets are presented in the table 2.

Lang Pair | #Train_Snts | #Dev_Snts
---|---|---
English-Hindi | 7000 | 1000
English-Gujarati | 7000 | 1000
English-Marathi | 26000 | 1000
English-Tamil | 7000 | 1000
English-Telugu | 7000 | 1000
Table 2: Train and Development Set Details for Reference-less Translation
Evaluation

####  5.1.1 Using Min-Max scaling on z-scores

In order to fine-tune large language models on the aforementioned training
data, we utilized the mean of
z-scores161616https://www.investopedia.com/terms/z/zscore.asp, followed by
language-specific min-max-based re-scaling
171717https://en.wikipedia.org/wiki/Feature_scaling from 1 to 100 for the
entire corpora.

Model/Language Pair EN- | Hindi | Gujarati | Marathi | Tamil | Telugu
---|---|---|---|---|---
Co-relation scores | S | P | K | S | P | K | S | P | K | S | P | K | S | P | K
Llama-2-7b-Adpt | 0.4268 | 0.5303 | 0.3056 | 0.4795 | 0.585 | 0.3757 | 0.4481 | 0.4939 | 0.3182 | 0.5345 | 0.6632 | 0.3967 | 0.3772 | 0.3324 | 0.2893
Llama-2-13b-Adpt | 0.3967 | 0.587 | 0.2824 | 0.5226 | 0.604 | 0.419 | 0.5062 | 0.5485 | 0.3689 | 0.5057 | 0.6501 | 0.3774 | 0.3408 | 0.2976 | 0.2708
Mistral-7b-Adpt | 0.3923 | 0.5295 | 0.2822 | 0.4908 | 0.5724 | 0.3994 | 0.4736 | 0.5518 | 0.3364 | 0.5372 | 0.615 | 0.3969 | 0.333 | 0.2685 | 0.2662
Llama-2-7b-Adpt-Full | 0.4003 | 0.5249 | 0.3068 | 0.4959 | 0.5503 | 0.3958 | 0.4524 | 0.4938 | 0.3402 | 0.5413 | 0.6535 | 0.4138 | 0.2940 | 0.2500 | 0.2360
Llama-2-13b-Adpt-Full | 0.3004 | 0.2164 | 0.2144 | 0.3619 | 0.3294 | 0.2994 | 0.4424 | 0.4229 | 0.3245 | 0.4408 | 0.484 | 0.3454 | 0.2849 | 0.1690 | 0.2271
Llama-2-7b-Adpt-Trans | 0.3273 | 0.4602 | 0.2316 | 0.4717 | 0.5389 | 0.3824 | 0.4196 | 0.4554 | 0.2993 | 0.4921 | 0.6404 | 0.359 | 0.2800 | 0.2266 | 0.2137
Llama-2-13b-Adpt-Trans | 0.4011 | 0.5165 | 0.2888 | 0.5185 | 0.6039 | 0.4088 | 0.4451 | 0.4956 | 0.3146 | 0.5234 | 0.6287 | 0.3892 | 0.3264 | 0.2714 | 0.2544
Mistral-7b-Adpt-Trans | 0.3507 | 0.4611 | 0.2525 | 0.4638 | 0.5105 | 0.3671 | 0.4308 | 0.4849 | 0.3174 | 0.468 | 0.5796 | 0.3425 | 0.3748 | 0.3403 | 0.3042

Table 3: Correlation between Human judgement and Fine-tuned LLM with
reference-less translation evaluation task for English to 4 Indian Languages
is shown. Here, S, P, and K represent Spearman’s Rank Correlation Coefficient,
Pearson Correlation Coefficient, and Kendall’s Rank Correlation Coefficient,
respectively.

###  5.2 Training Data for Translation Fine-Tuning

To fine-tune LLMs for translation along with translation evaluation task under
multi-task setting, we used the publicly available Bharat Parallel Corpus
Collection (BPCC) designed for English to 22 Indic languages. We used BPCC-
Human dataset, containing 2.2 million English-Indic pairs for this fine-tuning
purpose.

###  5.3 LLM Fine-tuning Details

Considering the raw LLM performance, model parameters, and resource
constraints, we selected a subset of LLMs for the fine-tuning process.
Specifically, we opted for LLaMA-2-7b, LLaMA-2-13b, and Mistral-7B for the
fine-tuning experiment. For these selected LLMs, we decided to perform low-
rank adaptation-based fine-tuning (Hu et al., 2021) as well as full fine-
tuning. In the fine-tuning process, we conducted a multi-lingual approach by
considering corpora from all languages. Additionally, we conducted another
experiment under multi-task settings, where we fine-tuned both LLaMA-2-7b and
LLaMA-2-13b for reference-less translation evaluation and translation tasks.
For the translation task, we utilized the translation dataset mentioned
earlier (in subsection-5.2), along with a simple JSON prompt instructing the
model to translate a given sentence from the source language to the target
language.

For both types of fine-tuning LLMs, we utilized the llama-recipes
codebase181818https://github.com/facebookresearch/llama-recipes/ which
provides an efficient implementation for LoRa-based adaptor fine-tuning with
PEFT (Mangrulkar et al., 2022). For more details, please refer to the llama-
recipes documentation 191919https://github.com/facebookresearch/llama-
recipes/blob/main/docs/LLM_finetuning.md. The hyperparameters for the fine-
tuning process are specified in Table 1.

##  6 Results and Discussion

In order to determine the performance of the LLM models, we examined the
correlation between human judgments and metric output scores. This correlation
served as the primary evaluation factor for identifying the most effective
fine-tuned model. To conduct this analysis on the aforementioned development
corpora, we utilized Spearman’s Rank Correlation Coefficient, Pearson
Correlation Coefficient, and Kendall’s Rank Correlation Coefficient. To
calculate these correlations, we employed the SciPy
library202020https://scipy.org/.

###  6.1 Zero shot vs ICL based Reference-less Translation Evaluation over Raw
LLMs

Score Bin | Hindi | Gujarati | Marathi | Tamil | Telugu
---|---|---|---|---|---
99 | 2 | 0 | 0 | 1 | 2
95 | 5 | 43 | 32 | 9 | 0
90 | 943 | 788 | 940 | 941 | 930
80 | 50 | 155 | 27 | 45 | 68
70-80 | 0 | 14 | 1 | 4 | 0

Table 4: LLAMA-2-7B zero-shot performance on reference-less translation
evaluation for English to 5 languages. Here, the ‘Score Bin’ represents the
score (out of 100) produced by LLAMA-2-7B. The corresponding language columns
represent the respective sentence pairs that got that particular score bin. A
total of 1000 samples were involved per language pair.

Overall, the correlation scores with human evaluation for both Raw LLMs and In
Context Learning (ICL) based LLMs in reference-less translation evaluation are
substandard. All LLMs exhibited poor performance, which could be attributed to
a lack of understanding or knowledge of the translation evaluation task. In
our manual analysis, we observed that LLMs consistently provide a common
number close to 100, regardless of the translation quality, for zero-shot
evaluation (refer Table 4). For in-context learning, LLMs simply mimic the
example translation evaluation scores. Therefore, in response to our initial
question, it can be concluded that Raw LLMs do not possess inherent
translation evaluation capabilities, either with zero-shot or example-driven
contextual learning. Refer to the appendix for examples and correlation
details.

###  6.2 Fine-Tuned LLM driven Reference-less Translation Evaluation

We conducted an evaluation to compare the performance of our Fine-Tuned LLM
models. The comparison results for English to 5 Indian language reference-less
translation evaluation correlation with human judgments are presented in Table
3. It is evident that the highest correlations are achieved with LoRa-based
fine-tuning methods for LLaMa-2-7b and LLaMa-2-13b models (indicated by
-Adpt). On the other hand, Full Fine-tuning (indicated by -Adpt-Full)
demonstrates maximum correlation scores for Tamil. However, overall, there is
a low correlation for Telugu, indicating the need for further exploration, as
Telugu language may have less representation compared to other languages in
base LLM models. Additionally, it is worth noting that multi-task learning-
based fine-tuning (indicated by -Adpt-Trans) does not lead to performance
improvement compared to single-task fine-tuning for translation evaluation.

English to INs | Average Across Languages
---|---
Co-relation scores | S | P | K
COMET-QE | 0.4568 | 0.5236 | 0.32042
Llama-2-7b-Adpt | 0.45322 | 0.52096 | 0.3371
Llama-2-13b-Adpt | 0.4574 | 0.53744 | 0.3437
Mistral-7b-Adpt | 0.44538 | 0.50744 | 0.33622
Llama-2-7b-Adpt-Full | 0.43678 | 0.4945 | 0.33852
Llama-2-13b-Adpt-Full | 0.36608 | 0.32434 | 0.28216
Llama-2-7b-Adpt-Trans | 0.39814 | 0.4643 | 0.2972
Llama-2-13b-Adpt-Trans | 0.4429 | 0.50322 | 0.33116
Mistral-7b-Adpt-Trans | 0.41762 | 0.47528 | 0.31674

Table 5: The correlation between Human judgement and Fine-tuned LLM and COMET-
QE with the reference-less translation evaluation task is shown, averaged
across English to 4 Indian Languages (Hindi, Gujarati, Marathi, Tamil and
Telugu). Here, S, P, and K represent Spearman’s Rank Correlation Coefficient,
Pearson Correlation Coefficient, and Kendall’s Rank Correlation Coefficient,
respectively.

Table 5 displays the average performance across all English to 4 language
pairs for different fine-tuned LLMs and COMET-QE. We trained COMET-QE
following the COMET architecture as described in COMET by
Unbabel212121https://github.com/Unbabel/COMET, using the same training data
configuration as our LLM fine-tuning. It is worth noting that the
Llama-2-13b-Adpt model, which is an adapted model using the LORa method,
achieves a high overall average human correlation across language pairs. This
indicates the superior performance of LLM-driven reference-less translation
evaluation. Hence, it highlights the potential of LLMs for the translation
evaluation task involving English and Indian languages.

In answer to the question posed in the introduction, Fine-tuning LLMs does
indeed improve translation reference-less evaluation capabilities. However,
when it comes to multi-task fine-tuning, including the translation task does
not result in better performance compared to fine-tuning focused solely on the
translation evaluation task.

Based on our manual analysis of outputs, it became evident that the scores
provided by the Llama-2-13b-Adpt model indicate superior quality. The model
demonstrates the ability to detect translation differences, which is indeed
reflected in its scores. Refer to the appendix for examples and correlation
details.

##  7 Limitations

We conducted all our experiments using high-performance GPUs, specifically the
A100-40GB, which may not be readily available for everyone to reproduce these
experiments and obtain the same results due to compute limitations. To address
this constraint, our goal is to make all outputs, including model outputs and
results, openly accessible 222222https://github.com/vmujadia/LLMT-Eval for
further research.

##  8 Conclusion and Future Work

Our experiments and results indicate that fine-tuned LLMs show promise for
translation evaluation in the targeted reference-less translation evaluation
task. The findings call for further analysis and understanding of translation
evaluation tasks involving large language models.

As part of future work, we plan to incorporate additional Indian languages for
the reference-less translation evaluation task. Furthermore, we aim to utilize
large language models for reference-driven translation evaluation,
encompassing English and a wider range of Indian languages.

Overall, our study represents a significant milestone in assessing and
enhancing the reference-less translation evaluation capabilities of LLMs,
specifically in the context of English and Indian languages. For the future of
automatic translation evaluation, we strongly advocate for solutions based on
large language models as the primary method of evaluation. We have also
released our models and predictions for future research.

## Acknowledgement

We would like to extend our sincere appreciation to Palash Gupta and Khoushik
Ananth for their invaluable contributions at various stages of this project.
The Ministry of Electronics and In- formation Technology, Government of India,
has generously funded this endeavor, as part of the Sanction Order :
11(1)/2022-HCC(TDIL)-Part(2)/A/B/C and the Administrative Approval:
11(1)/2022-HCC(TDIL)-Part(2).

## References

  * Akhbardeh et al. (2021) Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondřej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. 2021.  Findings of the 2021 conference on machine translation (WMT21).  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 1–88, Online. Association for Computational Linguistics.
  * Aksitov et al. (2023) Renat Aksitov, Chung-Ching Chang, David Reitter, Siamak Shakeri, and Yunhsuan Sung. 2023.  Characterizing attribution and fluency tradeoffs for retrieval-augmented large language models.  _arXiv preprint arXiv:2302.05578_.
  * Al Sharou and Specia (2022) Khetam Al Sharou and Lucia Specia. 2022.  A taxonomy and study of critical errors in machine translation.  In _Proceedings of the 23rd Annual Conference of the European Association for Machine Translation_ , pages 171–180.
  * Arivazhagan et al. (2019) Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, et al. 2019.  Massively multilingual neural machine translation in the wild: Findings and challenges.  _arXiv preprint arXiv:1907.05019_.
  * Bellegarda (2004) Jerome R Bellegarda. 2004.  Statistical language model adaptation: review and perspectives.  _Speech communication_ , 42(1):93–108.
  * Bengio et al. (2000) Yoshua Bengio, Réjean Ducharme, and Pascal Vincent. 2000.  A neural probabilistic language model.  _Advances in neural information processing systems_ , 13.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chang et al. (2023) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023.  A survey on evaluation of large language models.  _arXiv preprint arXiv:2307.03109_.
  * Chimoto and Bassett (2022) Everlyn Chimoto and Bruce Bassett. 2022.  COMET-QE and active learning for low-resource machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 4735–4740, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  BERT: pre-training of deep bidirectional transformers for language understanding.  _CoRR_ , abs/1810.04805.
  * Feng et al. (2022) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022.  Language-agnostic BERT sentence embedding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 878–891, Dublin, Ireland. Association for Computational Linguistics.
  * Freitag et al. (2021) Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021.  Experts, errors, and context: A large-scale study of human evaluation for machine translation.  _Transactions of the Association for Computational Linguistics_ , 9:1460–1474.
  * Gala et al. (2023) Jay Gala, Pranjal A Chitale, Raghavan AK, Sumanth Doddapaneni, Varun Gumma, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, et al. 2023.  Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages.  _arXiv preprint arXiv:2305.16307_.
  * Gupta et al. (2010) Ankush Gupta, Sriram Venkatapathy, and Rajeev Sangal. 2010.  Meteor-hindi: automatic mt evaluation metric for hindi as a target.  In _Proceedings of ICON-2010: 8th international conference on natural language processing, Macmillan Publishers. India_.
  * Guzmán et al. (2015) Francisco Guzmán, Ahmed Abdelali, Irina Temnikova, Hassan Sajjad, and Stephan Vogel. 2015.  How do humans evaluate machine translation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 457–466, Lisbon, Portugal. Association for Computational Linguistics.
  * Hadi et al. (2023) Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muhammad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. 2023.  Large language models: A comprehensive survey of its applications, challenges, limitations, and future prospects.
  * Hayakawa and Arase (2020) Takeshi Hayakawa and Yuki Arase. 2020.  Fine-grained error analysis on English-to-Japanese machine translation in the medical domain.  In _Proceedings of the 22nd Annual Conference of the European Association for Machine Translation_ , pages 155–164, Lisboa, Portugal. European Association for Machine Translation.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Kocmi et al. (2022) Tom Kocmi, Hitokazu Matsushita, and Christian Federmann. 2022.  MS-COMET: More and better human judgements improve metric performance.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 541–548, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Kojima et al. (2022) Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022.  Large language models are zero-shot reasoners.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 22199–22213. Curran Associates, Inc.
  * Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023.  Efficient memory management for large language model serving with pagedattention.  In _Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles_.
  * Mangrulkar et al. (2022) Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022.  Peft: State-of-the-art parameter-efficient fine-tuning methods.  https://github.com/huggingface/peft.
  * Mathur et al. (2020) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020.  Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4984–4997, Online. Association for Computational Linguistics.
  * Min et al. (2023) Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. 2023.  Recent advances in natural language processing via large pre-trained language models: A survey.  _ACM Computing Surveys_ , 56(2):1–40.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023.  The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only.  _arXiv preprint arXiv:2306.01116_.
  * Popović (2015) Maja Popović. 2015.  chrf: character n-gram f-score for automatic mt evaluation.  In _Proceedings of the tenth workshop on statistical machine translation_ , pages 392–395.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. 2018.  Improving language understanding by generative pre-training.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Ranathunga et al. (2023) Surangika Ranathunga, En-Shiun Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen Alam, and Rishemjit Kaur. 2023.  Neural machine translation for low-resource languages: A survey.  _ACM Computing Surveys_ , 55(11):1–37.
  * Rei et al. (2020) Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020.  COMET: A neural framework for MT evaluation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2685–2702, Online. Association for Computational Linguistics.
  * Rei et al. (2022) Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022.  CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Rivera-Trigueros (2022) Irene Rivera-Trigueros. 2022.  Machine translation systems and quality assessment: a systematic review.  _Language Resources and Evaluation_ , 56(2):593–619.
  * Sangal (2022) Rajeev Sangal. 2022.  Evaluating MT Systems: A Theoretical Framework.  _arXiv preprint arXiv:2202.05806_.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Snover et al. (2009) Matthew G Snover, Nitin Madnani, Bonnie Dorr, and Richard Schwartz. 2009.  Ter-plus: paraphrase, semantic, and alignment enhancements to translation edit rate.  _Machine Translation_ , 23:117–127.
  * Surameery and Shakor (2023) Nigar M Shafiq Surameery and Mohammed Y Shakor. 2023.  Use chat gpt to solve programming bugs.  _International Journal of Information Technology & Computer Engineering (IJITC) ISSN: 2455-5290_, 3(01):17–22.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wan et al. (2022) Yu Wan, Baosong Yang, Derek Fai Wong, Lidia Sam Chao, Liang Yao, Haibo Zhang, and Boxing Chen. 2022.  Challenges of Neural Machine Translation for Short Texts.  _Computational Linguistics_ , 48(2):321–342.
  * Wang et al. (2023) Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2023.  A survey on large language model based autonomous agents.  _arXiv preprint arXiv:2308.11432_.
  * Zan et al. (2023) Daoguang Zan, Bei Chen, Fengji Zhang, Dianjie Lu, Bingchao Wu, Bei Guan, Wang Yongji, and Jian-Guang Lou. 2023.  Large language models meet NL2Code: A survey.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7443–7464, Toronto, Canada. Association for Computational Linguistics.
  * Zerva et al. (2022) Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvittayakumjorn, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Orăsan, Marina Fomicheva, André F. T. Martins, and Lucia Specia. 2022.  Findings of the WMT 2022 shared task on quality estimation.  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , pages 69–99, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_.

##  Appendix A Appendix

###  A.1 Examples (English-Hindi)

###  A.2 Results

Model/Language Pair: English- | Hindi | Gujarati | Marathi | Tamil | Telugu
---|---|---|---|---|---
Co-relation scores | S | P | K | S | P | K | S | P | K | S | P | K | S | P | K
bloom-7b1 | -0.0058 | -0.0116 | -0.0047 | 0.0154 | 0.0216 | 0.0129 | 0.0122 | 0.0222 | 0.0098 | -0.0256 | -0.015 | -0.0208 | 0.0268 | 0.0091 | 0.0222
falcon-7b | 0.0211 | 0.0466 | 0.0162 | -0.0108 | 0.0072 | -0.0079 | -0.1052 | -0.0083 | -0.0811 | 0.0651 | 0.0787 | 0.0496 | 0.1151 | 0.0775 | 0.0882
llama-7b | 0.02 | 0.019 | 0.0178 | -0.035 | -0.014 | -0.0276 | -0.0239 | -0.0243 | -0.019 | 0.053 | 0.0311 | 0.0441 | 0.0519 | 0.0292 | 0.0473
Llama-2-7b | 0.022 | 0.0223 | 0.0177 | -0.0301 | -0.0135 | -0.0249 | -0.0222 | -0.0212 | -0.018 | 0.0674 | 0.0328 | 0.0547 | 0.0606 | 0.0304 | 0.0504
Mistral-7B-v0.1 | -0.0474 | -0.0582 | -0.0382 | 0.0741 | 0.063 | 0.0615 | 0.077 | 0.0766 | 0.0628 | -0.0487 | -0.054 | -0.0367 | -0.0143 | -0.0062 | -0.0116
opt-6.7b | NAN | NAN | NAN | NAN | NAN | NAN | 0.0826 | 0.0229 | 0.0673 | NAN | NAN | NAN | NAN | NAN | NAN
mpt-7b | -0.0168 | -0.0003 | -0.0137 | 0.0098 | 0.0204 | 0.0082 | 0.0234 | -0.0095 | 0.0191 | NAN | NAN | NAN | 0.0405 | 0.0312 | 0.0337
Llama-2-13b | 0.0347 | -0.0038 | 0.0267 | 0.0862 | 0.0965 | 0.0685 | -0.0691 | -0.0698 | -0.0551 | 0.0748 | 0.0276 | 0.0591 | 0.0407 | -0.0077 | 0.0323

Table 6: Correlation between Human judgement and Zero-shot LLM with reference-
less translation evaluation task for English to 4 Indian Languages is shown.
Here, S, P, and K represent Spearman’s Rank Correlation Coefficient, Pearson
Correlation Coefficient, and Kendall’s Rank Correlation Coefficient,
respectively.

Model/Language Pair: English- | Hindi | Gujarati | Marathi | Tamil | Telugu
---|---|---|---|---|---
Co-relation scores | S | P | K | S | P | K | S | P | K | S | P | K | S | P | K
bloom-7b1 | -0.0496 | -0.0398 | -0.0405 | 0.0093 | 0.0075 | 0.0078 | 0.1191 | 0.1094 | 0.0973 | 0.0233 | -0.0143 | 0.0191 | -0.0461 | -0.0381 | -0.0384
falcon-7b | 0.1374 | 0.0175 | 0.1112 | 0.03496 | 0.0497 | 0.0284 | 0.0008 | 0.0009 | 0.0008 | 0.127 | 0.1015 | 0.102 | -0.1161 | -0.0825 | -0.0938
llama-7b | -0.0492 | -0.0615 | -0.0402 | NAN | NAN | NAN | NAN | NAN | NAN | -0.0416 | -0.0262 | -0.034 | NAN | NAN | NAN
Llama-2-7b | NAN | NAN | NAN | NAN | NAN | NAN | NAN | NAN | NAN | -0.0416 | -0.0262 | -0.034 | NAN | NAN | NAN
Mistral-7B-v0.1 | 0.0382 | 0.0144 | 0.0312 | 0.0287 | 0.0382 | 0.0239 | NAN | NAN | NAN | -0.0023 | -0.0117 | -0.0018 | 0.008 | 0.0348 | 0.0066
opt-6.7b | -0.0576 | -0.0436 | -0.0471 | -0.0095 | 0.0005 | -0.0079 | 0.0278 | 0.0129 | 0.0227 | 0.1141 | 0.0936 | 0.0933 | 0.0918 | 0.0791 | 0.0764
mpt-7b | 0.0661 | 0.0454 | 0.054 | -0.0539 | -0.0498 | -0.0449 | 0.0639 | 0.0431 | 0.0522 | 0.1546 | 0.137 | 0.1263 | -0.1268 | -0.0826 | -0.1054
Llama-2-13b | NAN | NAN | NAN | NAN | NAN | NAN | NAN | NAN | NAN | -0.0006 | -0.0167 | -0.0005 | 0.0159 | 0.0214 | 0.0132

Table 7: Correlation between Human judgement and In Context Learning LLM with
reference-less translation evaluation task for English to 4 Indian Languages
is shown. Here, S, P, and K represent Spearman’s Rank Correlation Coefficient,
Pearson Correlation Coefficient, and Kendall’s Rank Correlation Coefficient,
respectively.