the current state and potential of world models but also illuminates the path
toward their future development and application. We hope this survey can
inspire the community toward novel solutions, thereby broadening the horizon
for world models and their applications in shaping the future of AGI.

## References

  * [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [2] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell.  Localizing moments in video with natural language.  In ICCV, 2017.
  * [3] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid.  Vivit: A video vision transformer.  In ICCV, 2021.
  * [4] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas.  Self-supervised learning from images with a joint-embedding predictive architecture.  In CVPR, 2023.
  * [5] Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros.  Sequential modeling enables scalable learning for large vision models.  arXiv preprint arXiv:2312.00785, 2023.
  * [6] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman.  Frozen in time: A joint video and image encoder for end-to-end retrieval.  In ICCV, 2021.
  * [7] Yogesh Balaji, Martin Renqiang Min, Bing Bai, Rama Chellappa, and Hans Peter Graf.  Conditional gan with discriminative filter generation for text-to-video synthesis.  In IJCAI, 2019.
  * [8] Andrea Banino, Adrià Puigdomènech Badia, Raphael Köster, Martin J. Chadwick, Vinícius Flores Zambaldi, Demis Hassabis, Caswell Barry, Matthew M. Botvinick, Dharshan Kumaran, and Charles Blundell.  MEMO: A deep network for flexible combination of episodic memories.  In ICLR, 2020.
  * [9] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu.  All are worth words: A vit backbone for diffusion models.  In CVPR, 2023.
  * [10] Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu.  One transformer fits all distributions in multi-modal diffusion at scale.  In ICML, 2023.
  * [11] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  Beit: Bert pre-training of image transformers.  arXiv preprint arXiv:2106.08254, 2021.
  * [12] Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mahmoud Assran, and Nicolas Ballas.  Revisiting feature prediction for learning visual representations from video, 2024.
  * [13] Adrien Bardes, Jean Ponce, and Yann LeCun.  Mc-jepa: A joint-embedding predictive architecture for self-supervised learning of motion and content features, 2023.
  * [14] Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.  The arcade learning environment: An evaluation platform for general agents.  J. Artif. Int. Res., 2013.
  * [15] Gedas Bertasius, Heng Wang, and Lorenzo Torresani.  Is space-time attention all you need for video understanding?  In ICML, 2021.
  * [16] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al.  Improving image generation with better captions.  Computer Science, 2023.
  * [17] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al.  Stable video diffusion: Scaling latent video diffusion models to large datasets.  arXiv preprint arXiv:2311.15127, 2023.
  * [18] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis.  Align your latents: High-resolution video synthesis with latent diffusion models.  In CVPR, 2023.
  * [19] Daniel Bogdoll, Yitian Yang, and J Marius Zöllner.  Muvo: A multimodal generative world model for autonomous driving with geometric representations.  arXiv preprint arXiv:2311.11762, 2023.
  * [20] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  arXiv preprint arXiv:1809.11096, 2018.
  * [21] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh.  Video generation models as world simulators.  2024\.
  * [22] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  NeurIPS, 2020.
  * [23] Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, and Tim Rocktäschel.  Genie: Generative interactive environments, 2024.
  * [24] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.  Activitynet: A large-scale video benchmark for human activity understanding.  In CVPR, 2015.
  * [25] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In CVPR, 2020.
  * [26] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In ECCV, 2020.
  * [27] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In CVPR, 2017.
  * [28] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman.  Maskgit: Masked generative image transformer.  In CVPR, 2022.
  * [29] Chang Chen, Yi-Fu Wu, Jaesik Yoon, and Sungjin Ahn.  Transdreamer: Reinforcement learning with transformer world models, 2022.
  * [30] David Chen and William B Dolan.  Collecting highly parallel data for paraphrase evaluation.  In ACL, 2011.
  * [31] Junsong Chen, Yue Wu, Simian Luo, Enze Xie, Sayak Paul, Ping Luo, Hang Zhao, and Zhenguo Li.  Pixart-$\\{$$\backslash$delta$\\}$: Fast and controllable image generation with latent consistency models.  arXiv preprint arXiv:2401.05252, 2024.
  * [32] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al.  Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis.  arXiv preprint arXiv:2310.00426, 2023.
  * [33] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In ICML, 2020.
  * [34] Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, et al.  Panda-70m: Captioning 70m videos with multiple cross-modality teachers.  arXiv preprint arXiv:2402.19479, 2024.
  * [35] Silvia Chiappa, Sébastien Racanière, Daan Wierstra, and Shakir Mohamed.  Recurrent environment simulators.  In ICLR, 2017.
  * [36] Joseph Cho, Fachrina Dewi Puspitasari, Sheng Zheng, Jingyao Zheng, Lik-Hang Lee, Tae-Ho Kim, Choong Seon Hong, and Chaoning Zhang.  Sora as an agi world model? a complete survey on text-to-video generation.  arXiv preprint arXiv:2403.05131, 2024.
  * [37] OpenScene Contributors.  Openscene: The largest up-to-date 3d occupancy prediction benchmark in autonomous driving.  https://github.com/OpenDriveLab/OpenScene, 2023.
  * [38] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov.  Transformer-XL: Attentive language models beyond a fixed-length context.  In ACL, 2019.
  * [39] Marc Peter Deisenroth and Carl Edward Rasmussen.  Pilco: a model-based and data-efficient approach to policy search.  In ICML, 2011.
  * [40] Ugur Demir and Gozde Unal.  Patch-based image inpainting with generative adversarial networks.  arXiv preprint arXiv:1803.07422, 2018.
  * [41] Fei Deng, Ingook Jang, and Sungjin Ahn.  Dreamerpro: Reconstruction-free model-based reinforcement learning with prototypical representations.  In ICML, 2022.
  * [42] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [43] Kangle Deng, Tianyi Fei, Xin Huang, and Yuxin Peng.  Irc-gan: Introspective recurrent convolutional gan for text-to-video generation.  In IJCAI, 2019.
  * [44] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [45] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  NeurIPS, 2021.
  * [46] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al.  Cogview: Mastering text-to-image generation via transformers.  NeurIPS, 2021.
  * [47] Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang.  Cogview2: Faster and better text-to-image generation via hierarchical transformers.  NeurIPS, 2022.
  * [48] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [49] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.  Carla: An open urban driving simulator.  In CoRL, 2017.
  * [50] Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel.  Learning universal policies via text-guided video generation.  NeurIPS, 2024.
  * [51] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu.  IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures.  In ICML, 2018.
  * [52] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis.  Structure and content-guided video synthesis with diffusion models.  In ICCV, 2023.
  * [53] Gongfan Fang, Xinyin Ma, and Xinchao Wang.  Structural pruning for diffusion models.  NeurIPS, 2024.
  * [54] Zhengcong Fei, Mingyuan Fan, and Junshi Huang.  A-jepa: Joint-embedding predictive architecture can listen, 2024.
  * [55] Christoph Feichtenhofer.  X3d: Expanding architectures for efficient video recognition.  In CVPR, 2020.
  * [56] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  In ICCV, 2019.
  * [57] Craig R Fox and Gülden Ülkümen.  Distinguishing two dimensions of uncertainty.  SSRN Electronic Journal, 2011.
  * [58] Stan Franklin and Art Graesser.  Is it an agent, or just a program?: A taxonomy for autonomous agents.  In Intelligent Agents III Agent Theories, Architectures, and Languages, 1997.
  * [59] Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen.  Improving PILCO with Bayesian neural network dynamics models.  In Data-Efficient Machine Learning workshop, ICML, 2016.
  * [60] Zeyu Gao, Yao Mu, Ruoyan Shen, Chen Chen, Yangang Ren, Jianyu Chen, Shengbo Eben Li, Ping Luo, and Yanfeng Lu.  Enhance sample efficiency and robustness of end-to-end urban autonomous driving via semantic masked world model.  arXiv preprint arXiv:2210.04017, 2022.
  * [61] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In CVPR, 2012.
  * [62] Anastasis Germanidis.  Introducing general world models.  2023\.
  * [63] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra.  Emu video: Factorizing text-to-video generation by explicit image conditioning.  arXiv preprint arXiv:2311.10709, 2023.
  * [64] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  NeurIPS, 2014.
  * [65] Jake Grigsby and Yanjun Qi.  Measuring visual generalization in continuous control from pixels, 2020.
  * [66] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo.  Vector quantized diffusion model for text-to-image synthesis.  In CVPR, 2022.
  * [67] Yanchen Guan, Haicheng Liao, Zhenning Li, Guohui Zhang, and Chengzhong Xu.  World models for autonomous driving: An initial survey.  arXiv preprint arXiv:2403.02622, 2024.
  * [68] Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama.  Photorealistic video generation with diffusion models.  arXiv preprint arXiv:2312.06662, 2023.
  * [69] David Ha and J”urgen Schmidhuber.  Recurrent world models facilitate policy evolution.  In NeurIPS, 2018.
  * [70] Jeongsoo Ha, Kyungsoo Kim, and Yusung Kim.  Dream to generalize: Zero-shot model-based reinforcement learning for unseen visual distractions.  AAAI, 2023.
  * [71] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson.  Learning latent dynamics for planning from pixels.  In ICML, 2019.
  * [72] Danijar Hafner, Timothy P. Lillicrap, Jimmy Ba, and Mohammad Norouzi.  Dream to control: Learning behaviors by latent imagination.  In ICLR, 2020.
  * [73] Danijar Hafner, Timothy P. Lillicrap, Mohammad Norouzi, and Jimmy Ba.  Mastering atari with discrete world models.  In ICLR, 2021.
  * [74] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap.  Mastering diverse domains through world models, 2023.
  * [75] Ali Hatamizadeh, Jiaming Song, Guilin Liu, Jan Kautz, and Arash Vahdat.  Diffit: Diffusion vision transformers for image generation.  arXiv preprint arXiv:2312.02139, 2023.
  * [76] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In CVPR, 2022.
  * [77] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, 2017.
  * [78] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [79] Mikael Henaff, William F. Whitney, and Yann LeCun.  Model-based planning with discrete and continuous actions, 2018.
  * [80] Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Streamingt2v: Consistent, dynamic, and extendable long video generation from text.  arXiv preprint arXiv:2403.14773, 2024.
  * [81] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning.  arXiv preprint arXiv:2104.08718, 2021.
  * [82] Matteo Hessel, Joseph Modayil, Hado van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver.  Rainbow: combining improvements in deep reinforcement learning.  In AAAI, 2018.
  * [83] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  NeurIPS, 2017.
  * [84] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al.  Imagen video: High definition video generation with diffusion models.  arXiv preprint arXiv:2210.02303, 2022.
  * [85] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  NeurIPS, 2020.
  * [86] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet.  Video diffusion models.  NeurIPS, 2022.
  * [87] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 1997.
  * [88] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang.  Cogvideo: Large-scale pretraining for text-to-video generation via transformers.  arXiv preprint arXiv:2205.15868, 2022.
  * [89] Anthony Hu.  Neural World Models for Computer Vision.  PhD thesis, University of Cambridge, 2022.
  * [90] Anthony Hu, Gianluca Corrado, Nicolas Griffiths, Zachary Murez, Corina Gurau, Hudson Yeo, Alex Kendall, Roberto Cipolla, and Jamie Shotton.  Model-based imitation learning for urban driving.  NeurIPS, 2022.
  * [91] Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado.  Gaia-1: A generative world model for autonomous driving.  arXiv preprint arXiv:2309.17080, 2023.
  * [92] Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu.  Ella: Equip diffusion models with llm for enhanced semantic alignment.  arXiv preprint arXiv:2403.05135, 2024.
  * [93] Binyuan Huang, Yuqing Wen, Yucheng Zhao, Yaosi Hu, Yingfei Liu, Fan Jia, Weixin Mao, Tiancai Wang, Chi Zhang, Chang Wen Chen, et al.  Subjectdrive: Scaling generative data in autonomous driving via subject control.  arXiv preprint arXiv:2403.19438, 2024.
  * [94] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In CVPR, 2017.
  * [95] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou.  Composer: Creative and controllable image synthesis with composable conditions.  arXiv preprint arXiv:2302.09778, 2023.
  * [96] Weidong Huang, Jiaming Ji, Borong Zhang, Chunhe Xia, and Yaodong Yang.  Safedreamer: Safe reinforcement learning with world models.  In ICLR, 2024.
  * [97] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al.  Vbench: Comprehensive benchmark suite for video generative models.  arXiv preprint arXiv:2311.17982, 2023.
  * [98] Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl.  Peekaboo: Interactive video generation via masked-diffusion.  arXiv preprint arXiv:2312.07509, 2023.
  * [99] Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon.  A survey on contrastive self-supervised learning.  Technologies, 2020.
  * [100] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison.  Rlbench: The robot learning benchmark & learning environment.  RAL, 2020.
  * [101] Jiaming Ji, Borong Zhang, Jiayi Zhou, Xuehai Pan, Weidong Huang, Ruiyang Sun, Yiran Geng, Yifan Zhong, Josef Dai, and Yaodong Yang.  Safety gymnasium: A unified safe reinforcement learning benchmark.  In NeurIPS, 2023.
  * [102] Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu.  3d convolutional neural networks for human action recognition.  TPAMI, 2012.
  * [103] Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao, Yuqing Wen, Chi Zhang, Xiangyu Zhang, and Tiancai Wang.  Adriver-i: A general world model for autonomous driving.  arXiv preprint arXiv:2311.13549, 2023.
  * [104] Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu.  Tinybert: Distilling bert for natural language understanding.  arXiv preprint arXiv:1909.10351, 2019.
  * [105] Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H. Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski.  Model based reinforcement learning for atari.  In ICLR, 2020.
  * [106] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In CVPR, 2019.
  * [107] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  ToG, 2023.
  * [108] Doyeon Kim, Donggyu Joo, and Junmo Kim.  Tivgan: Text to image to video generation with step-by-step evolutionary generator.  IEEE Access, 2020.
  * [109] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [110] Jing Yu Koh, Honglak Lee, Yinfei Yang, Jason Baldridge, and Peter Anderson.  Pathdreamer: A world model for indoor navigation.  In ICCV, 2021.
  * [111] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al.  Videopoet: A large language model for zero-shot video generation.  arXiv preprint arXiv:2312.14125, 2023.
  * [112] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  NeurIPS, 25, 2012.
  * [113] Michael Laskin, Aravind Srinivas, and Pieter Abbeel.  CURL: Contrastive unsupervised representations for reinforcement learning.  In ICML, 2020.
  * [114] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner.  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 1998.
  * [115] Yann LeCun and Courant.  A path towards autonomous machine intelligence version 0.9.2, 2022-06-27.  2022\.
  * [116] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han.  Autoregressive image generation using residual quantization.  In CVPR, 2022.
  * [117] Lik-Hang Lee, Tristan Braud, Pengyuan Zhou, Lin Wang, Dianlei Xu, Zijun Lin, Abhishek Kumar, Carlos Bermejo, and Pan Hui.  All one needs to know about metaverse: A complete survey on technological singularity, virtual ecosystem, and research agenda.  arXiv preprint arXiv:2110.05352, 2021.
  * [118] Sangho Lee, Jiwan Chung, Youngjae Yu, Gunhee Kim, Thomas Breuel, Gal Chechik, and Yale Song.  Acav100m: Automatic curation of large-scale datasets for audio-visual video representation learning.  In ICCV, 2021.
  * [119] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  arXiv preprint arXiv:1910.13461, 2019.
  * [120] Bowen Li.  Word-level fine-grained story visualization.  In ECCV, 2022.
  * [121] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In ICML, 2022.
  * [122] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  NeurIPS, 2021.
  * [123] Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao.  Uniformer: Unified transformer for efficient spatiotemporal representation learning.  arXiv preprint arXiv:2201.04676, 2022.
  * [124] Qifeng Li, Xiaosong Jia, Shaobo Wang, and Junchi Yan.  Think2drive: Efficient reinforcement learning by thinking in latent world model for quasi-realistic autonomous driving (in carla-v2).  arXiv preprint arXiv:2402.16720, 2024.
  * [125] Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan.  Mage: Masked generative encoder to unify representation learning and image synthesis.  In CVPR, 2023.
  * [126] Tianhong Li, Dina Katabi, and Kaiming He.  Self-conditioned image generation via generating representations.  arXiv preprint arXiv:2312.03701, 2023.
  * [127] Xiaofan Li, Yifu Zhang, and Xiaoqing Ye.  Drivingdiffusion: Layout-guided multi-view driving scene video generation with latent diffusion model.  arXiv preprint arXiv:2310.07771, 2023.
  * [128] Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, and Jianfeng Gao.  Storygan: A sequential conditional gan for story visualization.  In CVPR, 2019.
  * [129] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee.  Gligen: Open-set grounded text-to-image generation.  In ICCV, 2023.
  * [130] Yitong Li, Martin Min, Dinghan Shen, David Carlson, and Lawrence Carin.  Video generation from text.  In AAAI, 2018.
  * [131] Yiyi Liao, Jun Xie, and Andreas Geiger.  Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d.  TPAMI, 2022.
  * [132] Hongyu Liu, Ziyu Wan, Wei Huang, Yibing Song, Xintong Han, and Jing Liao.  Pd-gan: Probabilistic diverse gan for image inpainting.  In CVPR, 2021.
  * [133] Yaofang Liu, Xiaodong Cun, Xuebo Liu, Xintao Wang, Yong Zhang, Haoxin Chen, Yang Liu, Tieyong Zeng, Raymond Chan, and Ying Shan.  Evalcrafter: Benchmarking and evaluating large video generation models.  arXiv preprint arXiv:2310.11440, 2023.
  * [134] Yuanxin Liu, Lei Li, Shuhuai Ren, Rundong Gao, Shicheng Li, Sishuo Chen, Xu Sun, and Lu Hou.  Fetv: A benchmark for fine-grained evaluation of open-domain text-to-video generation.  NeurIPS, 2024.
  * [135] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [136] Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujie Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al.  Sora: A review on background, technology, limitations, and opportunities of large vision models.  arXiv preprint arXiv:2402.17177, 2024.
  * [137] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In ICCV, 2021.
  * [138] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu.  Video swin transformer.  In CVPR, 2022.
  * [139] Jiachen Lu, Ze Huang, Jiahui Zhang, Zeyu Yang, and Li Zhang.  Wovogen: World volume-aware diffusion for controllable multi-camera driving scene generation.  arXiv preprint arXiv:2312.02934, 2023.
  * [140] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao.  Latent consistency models: Synthesizing high-resolution images with few-step inference.  arXiv preprint arXiv:2310.04378, 2023.
  * [141] Haoyu Ma, Jialong Wu, Ningya Feng, Chenjun Xiao, Dong Li, Jianye Hao, Jianmin Wang, and Mingsheng Long.  Harmonydream: Task harmonization inside world models, 2024.
  * [142] Xinyin Ma, Gongfan Fang, and Xinchao Wang.  Deepcache: Accelerating diffusion models for free.  arXiv preprint arXiv:2312.00858, 2023.
  * [143] Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao.  Latte: Latent diffusion transformer for video generation.  arXiv preprint arXiv:2401.03048, 2024.
  * [144] Adyasha Maharana, Darryl Hannan, and Mohit Bansal.  Storydall-e: Adapting pretrained text-to-image transformers for story continuation.  In ECCV, 2022.
  * [145] Russell Mendonca, Shikhar Bahl, and Deepak Pathak.  Structured World Models from Human Videos.  In RSS, 2023.
  * [146] Vincent Micheli, Eloi Alonso, and François Fleuret.  Transformers are sample-efficient world models.  In ICLR, 2023.
  * [147] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic.  Howto100m: Learning a text-video embedding by watching hundred million narrated video clips.  In ICCV, 2019.
  * [148] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 2021.
  * [149] Chen Min, Liang Xiao, Dawei Zhao, Yiming Nie, and Bin Dai.  Occupancy-mae: Self-supervised pre-training large-scale lidar point clouds with masked occupancy autoencoders.  IEEE Transactions on Intelligent Vehicles, 2023.
  * [150] Chen Min, Liang Xiao, Dawei Zhao, Yiming Nie, and Bin Dai.  Multi-camera unified pre-training via 3d scene reconstruction.  RAL, 2024.
  * [151] Chen Min, Dawei Zhao, Liang Xiao, Yiming Nie, and Bin Dai.  Uniworld: Autonomous driving pre-training via world models.  arXiv preprint arXiv:2308.07234, 2023.
  * [152] Chen Min, Dawei Zhao, Liang Xiao, Jian Zhao, Xinli Xu, Zheng Zhu, Lei Jin, Jianshu Li, Yulan Guo, Junliang Xing, Liping Jing, Yiming Nie, and Bin Dai.  Driveworld: 4d pre-trained scene understanding via world models for autonomous driving.  In CVPR, 2024.
  * [153] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.  Human-level control through deep reinforcement learning.  Nature, 2015.
  * [154] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  In AAAI, 2024.
  * [155] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [156] World Model of Tesla.  [online].  http://https://www.https://www.youtube.com/watch?v=svgGsnBkl_o.
  * [157] Masashi Okada and Tadahiro Taniguchi.  Dreaming: Model-based reinforcement learning by latent imagination without reconstruction.  In ICRA, 2021.
  * [158] Masashi Okada and Tadahiro Taniguchi.  Dreamingv2: Reinforcement learning with discrete world models without reconstruction.  In IROS, 2022.
  * [159] Minting Pan, Xiangming Zhu, Yunbo Wang, and Xiaokang Yang.  Iso-dream: Isolating and leveraging noncontrollable visual dynamics in world models.  NeurIPS, 2022.
  * [160] Yingwei Pan, Zhaofan Qiu, Ting Yao, Houqiang Li, and Tao Mei.  To create what you tell: Generating videos from captions.  In ACM ICM, 2017.
  * [161] Zizheng Pan, Bohan Zhuang, De-An Huang, Weili Nie, Zhiding Yu, Chaowei Xiao, Jianfei Cai, and Anima Anandkumar.  T-stitch: Accelerating sampling in pre-trained diffusion models with trajectory stitching.  arXiv preprint arXiv:2402.14167, 2024.
  * [162] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski.  Styleclip: Text-driven manipulation of stylegan imagery.  In ICCV, 2021.
  * [163] Judea Pearl and Dana Mackenzie.  The book of why: the new science of cause and effect.  Basic books, 2018.
  * [164] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  In ICCV, 2023.
  * [165] Guim Perarnau, Joost Van De Weijer, Bogdan Raducanu, and Jose M Álvarez.  Invertible conditional gans for image editing.  arXiv preprint arXiv:1611.06355, 2016.
  * [166] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville.  Film: Visual reasoning with a general conditioning layer.  In AAAI, 2018.
  * [167] Aj Piergiovanni, Alan Wu, and Michael S. Ryoo.  Learning real-world robot policies by dreaming.  In IROS, 2019.
  * [168] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  arXiv preprint arXiv:2307.01952, 2023.
  * [169] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In ICML, 2021.
  * [170] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * [171] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  OpenAI blog, 2019.
  * [172] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  JMLR, 2020.
  * [173] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [174] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  NeurIPS, 2015.
  * [175] Jan Robine, Marc H”oftmann, Tobias Uelwer, and Stefan Harmeling.  Transformer-based world models are happy with 100k interactions.  In ICLR, 2023.
  * [176] Anna Rohrbach, Marcus Rohrbach, Niket Tandon, and Bernt Schiele.  A dataset for movie description.  In CVPR, 2015.
  * [177] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In CVPR, 2022.
  * [178] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In MICCAI, 2015.
  * [179] Reuven Rubinstein.  The Cross-Entropy method for combinatorial and continuous optimization.  Methodology And Computing In Applied Probability, 1999.
  * [180] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  NeurIPS, 2022.
  * [181] Ryo Sakagami, Florian S. Lay, Andreas Dömel, Martin J. Schuster, Alin Albu-Schäffer, and Freek Stulp.  Robotic world models—conceptualization, review, and engineering best practices.  Frontiers in Robotics and AI, 2023.
  * [182] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  arXiv preprint arXiv:2202.00512, 2022.
  * [183] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms, 2017.
  * [184] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak.  Planning to explore via self-supervised world models.  In ICML, 2020.
  * [185] Yuzhang Shang, Zhihang Yuan, Bin Xie, Bingzhe Wu, and Yan Yan.  Post-training quantization on diffusion models.  In CVPR, 2023.
  * [186] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [187] Kihyuk Sohn, Honglak Lee, and Xinchen Yan.  Learning structured output representation using deep conditional generative models.  NeurIPS, 2015.
  * [188] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  arXiv preprint arXiv:2010.02502, 2020.
  * [189] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models.  arXiv preprint arXiv:2303.01469, 2023.
  * [190] Robin Staab, Mark Vero, Mislav Balunović, and Martin Vechev.  Beyond memorization: Violating privacy via inference with large language models.  arXiv preprint arXiv:2310.07298, 2023.
  * [191] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid.  Segmenter: Transformer for semantic segmentation.  In ICCV, 2021.
  * [192] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In CVPR, 2020.
  * [193] Rui Sun, Yumin Zhang, Tejal Shah, Jiaohao Sun, Shuoying Zhang, Wenqi Li, Haoran Duan, and Bo Wei.  From sora what we can see: A survey of text-to-video generation.
  * [194] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In CVPR, 2015.
  * [195] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller.  Deepmind control suite, 2018.
  * [196] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [197] Emanuel Todorov, Tom Erez, and Yuval Tassa.  Mujoco: A physics engine for model-based control.  In ICIRS, 2012.
  * [198] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In ICML, 2021.
  * [199] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [200] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [201] Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri.  Learning spatiotemporal features with 3d convolutional networks.  In ICCV, 2015.
  * [202] Du Tran, Jamie Ray, Zheng Shou, Shih-Fu Chang, and Manohar Paluri.  Convnet architecture search for spatiotemporal feature learning.  arXiv preprint arXiv:1708.05038, 2017.
  * [203] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly.  Towards accurate generative models of video: A new metric & challenges.  arXiv preprint arXiv:1812.01717, 2018.
  * [204] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  NeurIPS, 2017.
  * [205] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  NeurIPS, 2017.
  * [206] Sai H Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor.  Chatgpt for robotics: Design principles and model abilities.  IEEE Access, 2024.
  * [207] Tengfei Wang, Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen.  High-fidelity gan inversion for image attribute editing.  In CVPR, 2022.
  * [208] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang.  Vatex: A large-scale, high-quality multilingual dataset for video-and-language research.  In ICCV, 2019.
  * [209] Xiaofeng Wang, Zheng Zhu, Guan Huang, Xinze Chen, and Jiwen Lu.  Drivedreamer: Towards real-world-driven world models for autonomous driving.  arXiv preprint arXiv:2309.09777, 2023.
  * [210] Xiaofeng Wang, Zheng Zhu, Guan Huang, Boyuan Wang, Xinze Chen, and Jiwen Lu.  Worlddreamer: Towards general world models for video generation via predicting masked tokens, 2024.
  * [211] Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al.  Lavie: High-quality video generation with cascaded latent diffusion models.  arXiv preprint arXiv:2309.15103, 2023.
  * [212] Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang.  Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving.  arXiv preprint arXiv:2311.17918, 2023.
  * [213] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al.  Internvid: A large-scale video-text dataset for multimodal understanding and generation.  arXiv preprint arXiv:2307.06942, 2023.
  * [214] Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al.  Internvideo2: Scaling video foundation models for multimodal video understanding.  arXiv preprint arXiv:2403.15377, 2024.
  * [215] Zengran Wang, Chen Min, Zheng Ge, Yinhao Li, Zeming Li, Hongyu Yang, and Di Huang.  Sts: Surround-view temporal stereo for multi-view 3d detection.  arXiv preprint arXiv:2208.10145, 2022.
  * [216] Zizhuang Wei, Qingtian Zhu, Chen Min, Yisong Chen, and Guoping Wang.  Aa-rmvsnet: Adaptive aggregation recurrent multi-view stereo network.  In ICCV, pages 6187–6196, 2021.
  * [217] Zizhuang Wei, Qingtian Zhu, Chen Min, Yisong Chen, and Guoping Wang.  Bidirectional hybrid lstm based recurrent neural network for multi-view stereo.  TVCG, 2022.
  * [218] Yuqing Wen, Yucheng Zhao, Yingfei Liu, Fan Jia, Yanhui Wang, Chong Luo, Chi Zhang, Tiancai Wang, Xiaoyan Sun, and Xiangyu Zhang.  Panacea: Panoramic and controllable video generation for autonomous driving.  arXiv preprint arXiv:2311.16813, 2023.
  * [219] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al.  Argoverse 2: Next generation datasets for self-driving perception and forecasting.  arXiv preprint arXiv:2301.00493, 2023.
  * [220] Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan.  Godiva: Generating open-domain videos from natural descriptions.  arXiv preprint arXiv:2104.14806, 2021.
  * [221] Jay Zhangjie Wu, Guian Fang, Haoning Wu, Xintao Wang, Yixiao Ge, Xiaodong Cun, David Junhao Zhang, Jia-Wei Liu, Yuchao Gu, Rui Zhao, et al.  Towards a better metric for text-to-video generation.  arXiv preprint arXiv:2401.07781, 2024.
  * [222] Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg.  Daydreamer: World models for physical robot learning.  In CoRL, 2023.
  * [223] Weijia Wu, Zhuang Li, Yefei He, Mike Zheng Shou, Chunhua Shen, Lele Cheng, Yan Li, Tingting Gao, Di Zhang, and Zhongyuan Wang.  Paragraph-to-image generation with information-enriched diffusion model.  arXiv preprint arXiv:2311.14284, 2023.
  * [224] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  NeurIPS, 2021.
  * [225] Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi, Jiabo Ye, Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, et al.  mplug-2: A modularized multi-modal foundation model across text, image and video.  In ICML, 2023.
  * [226] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  Msr-vtt: A large video description dataset for bridging video and language.  In CVPR, 2016.
  * [227] Mingxing Xu, Wenrui Dai, Chunmiao Liu, Xing Gao, Weiyao Lin, Guo-Jun Qi, and Hongkai Xiong.  Spatial-temporal transformer networks for traffic flow forecasting, 2021.
  * [228] Hongwei Xue, Tiankai Hang, Yanhong Zeng, Yuchong Sun, Bei Liu, Huan Yang, Jianlong Fu, and Baining Guo.  Advancing high-resolution video-language representation with large-scale video transcriptions.  In CVPR, 2022.
  * [229] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas.  Videogpt: Video generation using vq-vae and transformers.  arXiv preprint arXiv:2104.10157, 2021.
  * [230] Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou.  Temporal pyramid network for action recognition.  In CVPR, 2020.
  * [231] Jiazhi Yang, Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng, Ping Luo, et al.  Generalized predictive model for autonomous driving.  arXiv preprint arXiv:2403.09630, 2024.
  * [232] Sherry Yang, Yilun Du, Seyed Kamyar Seyed Ghasemipour, Jonathan Tompson, Leslie Pack Kaelbling, Dale Schuurmans, and Pieter Abbeel.  Learning interactive real-world simulators.  In ICLR, 2024.
  * [233] Zetong Yang, Li Chen, Yanan Sun, and Hongyang Li.  Visual point cloud forecasting enables scalable autonomous driving.  In CVPR, 2024.
  * [234] Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun.  Unisim: A neural closed-loop sensor simulator.  In CVPR, 2023.
  * [235] Zhuyu Yao, Jiangbo Ai, Boxun Li, and Chi Zhang.  Efficient detr: improving end-to-end object detector with dense prior.  arXiv preprint arXiv:2104.01318, 2021.
  * [236] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  arXiv preprint arXiv:2206.10789, 2022.
  * [237] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  arXiv preprint arXiv:2206.10789, 2022.
  * [238] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al.  Magvit: Masked generative video transformer.  In CVPR, 2023.
  * [239] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al.  Language model beats diffusion–tokenizer is key to visual generation.  arXiv preprint arXiv:2310.05737, 2023.
  * [240] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.  In CoRL, 2020.
  * [241] Yingchen Yu, Fangneng Zhan, Rongliang Wu, Jianxiong Pan, Kaiwen Cui, Shijian Lu, Feiying Ma, Xuansong Xie, and Chunyan Miao.  Diverse image inpainting with bidirectional and autoregressive transformers.  In ACM ICM, 2021.
  * [242] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi.  Merlot: Multimodal neural script knowledge models.  NeurIPS, 2021.
  * [243] Yan Zeng, Guoqiang Wei, Jiani Zheng, Jiaxin Zou, Yang Wei, Yuchen Zhang, and Hang Li.  Make pixels dance: High-dynamic video generation.  arXiv preprint arXiv:2311.10982, 2023.
  * [244] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum.  Dino: Detr with improved denoising anchor boxes for end-to-end object detection.  arXiv preprint arXiv:2203.03605, 2022.
  * [245] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models, 2023.
  * [246] Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun.  Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion.  In ICLR, 2024.
  * [247] Weipu Zhang, Gang Wang, Jian Sun, Yetian Yuan, and Gao Huang.  Storm: Efficient stochastic transformer based world models for reinforcement learning.  In NeurIPS, 2023.
  * [248] Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Luc Van Gool.  Trafficbots: Towards world models for autonomous driving simulation and motion prediction.  In ICRA, 2023.
  * [249] Guosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen, Guan Huang, Xiaoyi Bao, and Xingang Wang.  Drivedreamer-2: Llm-enhanced world models for diverse driving video generation.  arXiv preprint arXiv:2403.06845, 2024.
  * [250] Hongkai Zheng, Weili Nie, Arash Vahdat, Kamyar Azizzadenesheli, and Anima Anandkumar.  Fast sampling of diffusion models via operator learning.  In ICML, 2023.
  * [251] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In CVPR, 2021.
  * [252] Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu.  Occworld: Learning a 3d occupancy world model for autonomous driving.  arXiv preprint arXiv:2311.16038, 2023.
  * [253] Luowei Zhou, Chenliang Xu, and Jason Corso.  Towards automatic learning of procedures from web instructional videos.  In AAAI, 2018.
  * [254] Mi Zhou, Vibhanshu Abhishek, Timothy Derdenger, Jaymo Kim, and Kannan Srinivasan.  Bias in generative ai.  arXiv preprint arXiv:2403.02726, 2024.
  * [255] Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan.  Robodreamer: Learning compositional world models for robot imagination, 2024.
  * [256] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou.  In-domain gan inversion for real image editing.  In ECCV, 2020.
  * [257] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In ICCV, 2017.
  * [258] Qingtian Zhu, Chen Min, Zizhuang Wei, Yisong Chen, and Guoping Wang.  Deep learning for multi-view stereo via plane sweep: A survey.  arXiv preprint arXiv:2106.15328, 2021.
  * [259] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  arXiv preprint arXiv:2010.04159, 2020.
  * [260] Yuke Zhu, Josiah Wong, Ajay Mandlekar, Roberto Martín-Martín, Abhishek Joshi, Soroush Nasiriany, and Yifeng Zhu.  robosuite: A modular simulation framework and benchmark for robot learning, 2022.
  * [261] Vlas Zyrianov, Henry Che, Zhijian Liu, and Shenlong Wang.  Lidardm: Generative lidar simulation in a generated world.  arXiv preprint arXiv:2404.02903, 2024.


# RV4JaCa – Runtime Verification for
Multi-Agent Systems

Debora C. Engelmann School of Technology – PUCRS, BrazilDIBRIS – University of
Genoa, Italy<EMAIL_ADDRESS>DIBRIS – University of Genoa,
ItalyDepartment of Computing – UFSC, BrazilDIBRIS – University of Genoa,
ItalySchool of Technology – PUCRS, BrazilDIBRIS – University of Genoa, Italy
Angelo Ferrando DIBRIS – University of Genoa, Italy<EMAIL_ADDRESS>Department of Computing – UFSC, BrazilDIBRIS – University of Genoa,
ItalySchool of Technology – PUCRS, BrazilDIBRIS – University of Genoa, Italy
Alison R. Panisson Department of Computing – UFSC, Brazil
<EMAIL_ADDRESS>DIBRIS – University of Genoa, ItalySchool of
Technology – PUCRS, BrazilDIBRIS – University of Genoa, Italy Davide Ancona
DIBRIS – University of Genoa, Italy<EMAIL_ADDRESS>School of
Technology – PUCRS, BrazilDIBRIS – University of Genoa, Italy Rafael H.
Bordini School of Technology – PUCRS, Brazil<EMAIL_ADDRESS>DIBRIS –
University of Genoa, Italy Viviana Mascardi DIBRIS – University of Genoa,
Italy<EMAIL_ADDRESS>
###### Abstract

This paper presents a Runtime Verification (RV) approach for Multi-Agent
Systems (MAS) using the JaCaMo framework. Our objective is to bring a layer of
security to the MAS. This layer is capable of controlling events during the
execution of the system without needing a specific implementation in the
behaviour of each agent to recognise the events. MAS have been used in the
context of hybrid intelligence. This use requires communication between
software agents and human beings. In some cases, communication takes place via
natural language dialogues. However, this kind of communication brings us to a
concern related to controlling the flow of dialogue so that agents can prevent
any change in the topic of discussion that could impair their reasoning. We
demonstrate the implementation of a monitor that aims to control this dialogue
flow in a MAS that communicates with the user through natural language to aid
decision-making in hospital bed allocation.

##  1 Introduction

A characteristic identified as essential in Artificial Intelligence (AI) is
explainability, as it provides users with the necessary inputs to make it
possible to understand the system’s behaviour, and inspire confidence in the
final outcome. Explainability becomes an essential feature in Multi-Agent
Systems (MAS), as it is one of the most powerful paradigms for implementing
complex distributed systems powered by artificial intelligence techniques. MAS
are built upon core concepts such as distribution, reactivity, and individual
rationality. Agents have been widely studied, and an extensive range of tools
have been developed, such as agent-oriented programming languages and
methodologies [13]. Thus, practical applications of multi-agent technologies
have become a reality to solve complex and distributed problems [34]. In
addition, it also allows the execution of various tasks and makes it possible
the integration with various technologies.

Even though MAS solutions can be a natural choice for developing complex and
distributed systems, like any other software development technique they are
prone to errors and bugs (whether at the implementation or description level).
Standard techniques such as testing and debugging can be deployed to tackle
this problem. However, in case of MAS, the process of testing [36], debugging
[37], and verifying [17] such systems can be quite complex. For this and other
reasons, more lightweight approaches to guarantee the correct execution of the
system are valuable. One technique that can be applied in such cases is
Runtime Verification (RV) [10, 26]. Differently from other verification
techniques, RV is lightweight because it only concerns the analysis of the
runtime execution of the system under analysis; which makes RV very similar to
testing. However, rather than testing, RV is based on a specification
formalism, as it happens in formal verification, to express the properties to
be checked against the system’s behaviour, and is particularly suitable to
monitor control-oriented properties [3].

In this paper, we present RV4JaCa, an approach to perform RV of multi-agent
systems developed using the JaCaMo framework [12]. RV4JaCa is obtained by
extending MASs implemented in JaCaMo with a monitoring feature. In a nutshell,
RV4JaCa handles all the engineering pipeline to introduce RV in JaCaMo, and to
extract and check runs of the MAS against formal properties. In particular,
RV4JaCa enables RV of agents’ interactions (i.e., messages). As a proof of
concept, we demonstrate a case study in a hospital bed allocation domain where
RV is used to verify two different agent interaction protocols.

The paper is structured as follows. Section 2 reports the background needed to
fully understand the contribution; here, the JaCaMo framework and the notion
of RV are introduced. Section 3 presents RV4JaCa and its application in a bed
allocation case study. Section 4 positions the contribution w.r.t. the state
of the art. Finally, Section 5 summarises the contribution’s results, and
points out future developments.

##  2 Background

###  2.1 Multi-agent systems

Multi-agent systems (MAS) are systems composed of multiple agents. They seem
to be a natural metaphor for building and understanding a wide range of
artificial social systems and can be applied in several different domains
[38]. There are two interlocking strands of work in multi-agent systems: the
one that concerns individual agents and the one that concerns itself with the
collections of these agents. In practice, agents rarely act alone, they
usually inhabit an environment that contains other agents. Each agent can
control, or partially control, parts of the environment, which is called its
“sphere of influence”. It may happen that these spheres of influence overlap,
hence (parts of) the environment may be controlled jointly by more than one
agent. In this case, to achieve the desired result, an agent must also
consider how other agents may act. These agents will have some knowledge,
possibly incomplete, about the other agents [14].

Wooldridge [38] believes that to be able to understand a multi-agent domain it
is essential to understand the type of interaction that occurs between agents.
For intelligent autonomous agents, the ability to reach agreements is
extremely important, and for this, negotiation and argumentation skills are
often necessary. In [21], the authors mention two types of studies in multi-
agent systems; the first one is Agent-Centred Multi-Agent Systems (ACMAS),
which study, at the level of an agent, states and the relationship between
those states and their general behaviour and are projected in terms of the
agent’s states of mind. The second one is Organisation-Centred Multi-Agent
Systems (OCMAS), which are systems whose foundations reside in the concepts of
organisations, groups, communities, roles, and functions, among others. An
OCMAS is not considered in terms of mental states but in capacities and
constraints, which are considered organisational concepts, as well as
functions, tasks, groups, and interaction protocols.

In a multi-agent system, the organisation is the collection of roles,
relationships, and authority structures that govern agents’ behaviour. Every
multi-agent system has some form of organisation, even if it is implicit and
informal. Organisations guide the mode of interaction between agents, which
may influence data flows, resource allocation, authority relationships, and
various other features of the system [24].

In [8], the authors argue that systems which heavily adopt AI techniques are
increasingly available, and making them explainable is a priority. Explainable
Artificial Intelligence (XAI) is a research field that aims “to make AI
systems results more understandable to humans” [2]. These results must be
clear (in non-technical terms) and provide justifications about decisions made
[18]. In order to achieve a satisfactory level of explainability in multi-
agent systems, much communication between agents and humans needs to be
performed. However, this can generate an additional point of failure in the
execution of the system since if, for example, the predefined communication
protocol is not followed, or even if the human decides to change the
conversation topic unexpectedly, this can lead to unexpected and probably
inappropriate behaviour of the system.

###  2.2 JaCaMo framework

JaCaMo is a framework that allows for multi-agent oriented programming. This
framework consists of the integration of three previously existing platforms:
Jason – for programming autonomous agents, CArtAgO – for programming
environmental artifacts, and Moise – for programming multi-agent organisations
[12]. A multi-agent system programmed in JaCaMo has Jason agents that are
organised and follow roles according to Moise’s hierarchical structure. These
agents work in environments based on distributed artifacts programmed using
CArtAgO. Figure 1 shows an overview of the three dimensions of JaCaMo.

Figure 1: Overview of the three dimensions of JaCaMo [12]

Jason (Agent dimension) is an agent-oriented programming language that is an
interpreter for an extended version of the AgentSpeak [14] language. Agents
programmed in Jason use the Belief-Desired-Intention (BDI) model [31]. The
main idea of this approach is to model the process of deciding which action to
take to achieve certain objectives [31]. Moise [25, 12] is related to the
organisation dimension, where agents can be part of groups and follow specific
roles. Also, with Moise, _schemes_ are defined, that is, the structure of
organisation goals is decomposed into sub-goals and grouped into missions. An
organisation is specified in an XML (Extensible Markup Language) file. CArtAgO
[32], the environment dimension, is used to simulate an environment or
interface with a real one; this is where _artifacts_ are defined. These
artifacts define the environment’s structure and behaviour, representing all
resources that agents need. Agents can discover, create, and use artifacts at
runtime [12]. Artifacts are programmed in Java. Combining these dimensions
provides us with a complete framework for developing multi-agent systems
through agents, organisations, and environments.

###  2.3 Runtime verification

Runtime Verification (RV) [11] is a kind of formal verification technique that
focuses on checking the behaviour of software/hardware systems. With respect
to other formal verification techniques, such as Model Checking [16] and
Theorem Provers [28], RV is considered more dynamic and lightweight. This is
mainly due to its being completely focused on checking how the system behaves,
while the latter is currently running. This is important from a complexity
perspective. RV does not need to simulate the system in order to check all
possible execution scenarios; but, it only analyses what the system produces
(i.e., everything that can be observed of the system). This is usually
obtained through monitors, which are automatically generated from the
specifications of the properties to be checked, and are nothing more than
validation engines which, given a trace of events generated by the system
execution, conclude the satisfaction (resp. violation) of the corresponding
properties. In turn, a formal property is the formal representation of how we
expect the system should behave. The monitor’s job is to verify at runtime
whether such a property holds.

Since monitors are usually deployed together with the system under analysis,
they are well suited for checking properties that require to be continuously
monitored. This is especially true in safety-critical scenarios, where a
system’s fault can cause injuries, loss of money and even deaths. A key
example is autonomous and robotic systems, where _reliability_ is vital [23],
and the addition of monitors ensuring a correct behaviour is a valuable
feature.

In the scenario envisaged in this contribution, we aim to use RV as a safety
net for message exchange in JaCaMo. As pointed out elsewhere, the protocols
involved in the communication amongst agents and human beings can be very
complex, and hard to track. Moreover, agents are usually particularly focused
on the reasoning and reactive aspects, while the consistency of the protocols
is given for granted. However, above all in case of human being in the loop,
such assumption cannot be made. RV is a suitable candidate to keep track of
the protocols, to check whether the current agents’ enactment is consistent
(or not) with the expected protocol. Such consistency checking is extremely
important in safety-critical scenarios, as in the healthcare domain, where a
protocol violation can be costly.

###  2.4 Runtime Monitoring Language

Runtime Monitoring Language111https://rmlatdibris.github.io/ (RML [7]) is a
Domain-Specific Language (DSL) for specifying highly expressive properties in
RV (such as non context-free ones). We use RML in this paper for its support
of parametric specifications and its native use for defining interaction
protocols. In fact, the low-level language on which RML is based upon was born
for specifying communication protocols [5, 6].

Since RML is just a means for our purposes, indeed other formalisms can be as
easily integrated into RV4JaCa, we only provide a simplified and abstracted
view of its syntax and semantics. However, the complete presentation can be
found in [7].

In RML, a property is expressed as a tuple $\langle t,\mathit{ETs}\rangle$,
with $t$ a term and $\mathit{ETs}=\\{ET_{\\_}1,\ldots,ET_{\\_}n\\}$ a set of
event types. An event type $ET$ is represented as a set of pairs
$\\{k_{\\_}1:v_{\\_}1,\ldots,k_{\\_}n:v_{\\_}n\\}$, where each pair identifies
a specific piece of information ($k_{\\_}i$) and its value ($v_{\\_}i$). An
event $Ev$ is denoted as a set of pairs
$\\{k_{\\_}1^{\prime}:v_{\\_}1^{\prime},\ldots,k_{\\_}m^{\prime}:v_{\\_}m^{\prime}\\}$.
Given an event type $ET$, an event $Ev$ matches $ET$ if $ET\subseteq Ev$,
which means $\forall(k_{\\_}i:v_{\\_}i)\in
ET\cdot\exists(k_{\\_}j:v_{\\_}j)\in Ev\cdot k_{\\_}i=k_{\\_}j\land
v_{\\_}i=v_{\\_}j$. In other words, an event type $ET$ specifies the
requirements that an event $Ev$ has to satisfy to be considered valid.

An RML term $t$, with $t_{\\_}1$, $t_{\\_}2$ and $t^{\prime}$ as other RML
terms, can be:

  * •

$ET$, denoting a set of singleton traces containing the events $Ev$ s.t.
$ET\subseteq Ev$;

  * •

$t_{\\_}1\;\;t_{\\_}2$, denoting the sequential composition of two sets of
traces;

  * •

$t_{\\_}1\;|\;t_{\\_}2$, denoting the unordered composition of two sets of
traces (also called shuffle or interleaving);

  * •

$t_{\\_}1\land t_{\\_}2$, denoting the intersection of two sets of traces;

  * •

$t_{\\_}1\lor t_{\\_}2$, denoting the union of two sets of traces;

  * •

$\\{let\;x;\;t^{\prime}\\}$, denoting the set of traces $t^{\prime}$ where the
variable $x$ can be used (i.e., the variable $x$ can appear in event types in
$t^{\prime}$, and can be unified with values).

  * •

$t^{\prime}*$, denoting the set of chains of concatenations of traces in
$t^{\prime}$

Event types can contain variables. For example,
$ET(ag1,ag2)=\\{sender:ag1,receiver:ag2\\}$, where we do not force any
specific value for the sender (resp., receiver) of a message (in this case the
events of interest would be messages). This event type matches all events
containing sender and receiver. When an event matches an event type with
variables, such as in this case, the variables get the values from the event.
For instance, if the event observed would be
$Ev=\\{sender:``Alice",receiver:``Bob"\\}$, it would match $ET$ by unifying
its variables as follows: $ag1=``Alice"$, and $ag2=``Bob"$. This aspect is
important because, as we are going to show in the bed allocation domain, we
can use variables in RML terms to enforce a specific order of messages. For
instance, in this very high-level example, we could say that when a message
from $ag1$ to $ag2$ is observed, the only possible consequent message can be a
message from $ag2$ to $ag1$. Since the first event has unified the two
variables, the second event will have to be a message from $Bob$ to $Alice$
(otherwise this would be considered a violation). Naturally, this is only the
intuition behind it, but it should help to grasp the expressiveness of RML and
how variables can be exploited at the protocol level to enforce specific
orders amongst the messages.

##  3 Engineering runtime verification for multi-agent systems

Our approach named RV4JaCa222The source code is available at
https://github.com/DeboraEngelmann/RV4JaCa allows the runtime verification of
multi-agent systems based on the JaCaMo platform. In Figure 2, we present an
overview of the entire approach. RV4JaCa is composed of: (i) a Sniffer class,
developed in Java, responsible for observing all communication between agents
in the MAS; (ii) a CArtAgO artifact named RV4JaCa Artifact responsible for
analysing the messages observed by the Sniffer, transforming them into a JSON
(JavaScript Object Notation) object and sending it as a REST (Representational
State Transfer) request to the RML Monitor. Note that RV4JaCa is not in any
way limited to a specific kind of monitor; we used RML simply because it was a
most suitable candidate for specifying the protocols of our interest.
Nonetheless, a different monitor could be as easily integrated as the RML one.
In addition, when the RV4JaCa Artifact receives the response of the request
made to the RML Monitor saying that there was a violation, it can add a belief
in the Monitor agent belief base; (iii) the RML Monitor responsible for
analysing the events sent by RV4JaCa Artifact and verifying the satisfaction
or violation of a formal property of interest; and (iv) a Monitor agent, which
can be added to the system if it is necessary to interfere with agents’
behaviour at runtime. In this case, if there is a violation, the RV4JaCa
Artifact adds a belief to the Monitor’s belief base. When the agent perceives
this addition, it can react by sending a message to the interested agents
warning about the violation. This may trigger some consequent recovery
mechanism, which usually is fully domain dependent. On the other hand, the
Monitor agent can also perform different activities depending on the needs of
the system. To clarify how our approach works, we developed a case study in
the field of bed allocation.

Figure 2: Approach overview for RV in MAS.

###  3.1 Bed allocation case study

We are working on a framework that supports the development of multi-agent
applications to assist humans in decision making. One of the domains for which
we are building an instance of this framework is hospital bed allocation. As
it is shown in Figure 3, to provide interface with natural language processing
platforms, such as Dialogflow333https://cloud.google.com/dialogflow/es/docs,
our framework relies on the use of Dial4JaCa [20]. The Human user can interact
with the chatbot through text or voice. Dialogflow classifies the interaction
intents and sends it to Dial4JaCa, making the request available to the
Communication expert agent assigned to that specific user. One or more
Communication expert agents can be instantiated, each one responsible for
representing one particular Human user. It uses natural language templates
[29] to translates the Assistant responses (the result of the MAS reasoning)
into natural language and send them to its corresponding Human user. The
Assistant agent performs argumentation reasoning [30] and is responsible for
communicating with other agents in search of information. Several Ontology
expert agents can be instantiated, allowing the MAS to consult different
ontologies simultaneously given that each of these agents are able to
interface with a specific ontology through a CArtAgO artifact. Such agents can
also perform ontological reasoning. Dial4JaCa together with these three types
of agents make up our General approach.

We can add domain-specific agents to the system to address the specificity of
different application domains. For example, in the instance shown in Figure 3,
we added specific agents for the bed allocation domain444This scenario is
detailed in [19]. Among those domain-specific agents, the Validator agent is
responsible for validating bed allocation plans made by the user (via our
system interface) using a PDDL (Planning Domain Definition Language) plan
validator; the Optimiser agent is responsible for making suggestions for
optimised allocations using the GLPSol solver of
GLPK555http://winglpk.sourceforge.net/ (GNU Linear Programming Kit), which is
a free open source software for solving linear programming problems; and the
Database agent is responsible for querying and updating the bed allocation
system database.

RV4JaCa has been added to that MAS for collecting information about all
messages exchanged between agents and sending them through a REST request to
the RML monitor (where the properties that need to be checked are defined).
After processing a received message, the monitor returns a result that states
whether the message sent from one agent to the other violates any of the
properties being checked by it. If a property is violated, RV4JaCa sends the
information to the Monitor agent so it can add that information to its belief
base and warns the agents involved in the exchange of messages that there has
been a violation. This makes it possible for our agents to take action to
recover from the failure that the breach caused.

Figure 3: RV4JaCa in a bed allocation domain

The ability to monitor the messages exchanged between the agents in this case
study is used for two different purposes. The first one is, according to the
performative and content of each move, to verify whether the agents are
following the predefined communication protocol. This aspect may be crucial
for safety-critical and privacy-preserving aspects. For instance, in a
healthcare domain, such as bed allocation, the agents might be expected to
follow some specific medical guidelines for the communication of personal
information (even amongst themselves). Moreover, when in presence of multiple
agents, each one having its own goals, it is common to specify the ideal
expected outcome at a more abstract level, where is more natural to reason
upon.

The second one is to check if a human participant changed the topic of the
conversation without the proper conclusion of the previous topic. When we add
humans to the agent-to-agent communication loop, the developer has limited
control over the human’s interactions in the dialogue. For example, in some
cases, when the MAS is performing some specific task, it is important to be
able to finish it before starting a new one. But when the completion of this
task depends on some human interaction, we have no guarantee that the human
will complete the necessary exchange. Naturally, it would be possible to do
this check within each agent. However, when we have agents specialised in
specific tasks, it is preferable to have all the agent plans related to the
specific topic rather that other concerns such as verification, which
significantly facilitates implementation and code maintenance.

Below, we report two example properties, written in RML, that have been
checked for the bed allocation case study using RV4JaCa.

###  3.2 RML properties for the bed allocation domain

The first property, which is presented in Listing LABEL:list:result, concerns
checking that the user does not change the topic before completing the one
currently processed by the agents. In particular, the reported property cares
about the ’getValidationResult’ topic. Such a topic relates to the user asking
the assistant agent to validate a suggested bed allocation (the corresponding
event type is expressed in lines 1-5). When this message is received by the
assistant, the protocol goes on, causing a sequence of messages exchanged
amongst the assistant, the optimiser, and the validator agents. Note that this
part is not reported because it is not of interest for checking the property.
After this step, the assistant agent sends back an answer to the user (the
event type is in lines 6-11). If this answer is not empty (i.e., the event
contains fields arg1 and arg2), then the user is expected to conclude the
communication with a certain content (listed in lines 12-17). For instance,
the user could reply with an additional message containing ’allocValPatients’,
meaning that the user is fine with the result of the validation, and he/she
wants to allocate the corresponding patient to the proposed bed. Naturally,
the user might decline the allocation, in that scenario the message would have
content ’dontAllocValPatients’. Similar reasoning goes for the other possible
options listed in lines 12-17.

Once the events corresponding to the messages previously mentioned have been
specified (lines 1-27), the actual property can be expressed following the RML
syntax (lines 28-31). In more detail, we may find in line 28 the definition of
the main term denoting the property to check (which in RML is always called
Main). In this scenario, the principal term corresponds to a sequence of
subterms, named Question. Such term is defined in line 29, and starts with a
question (as defined in lines 1-5). This means that, to comply with the
protocol, the first event has to be a message containing a ’question’ from the
’operator’ to the ’assistant’ (in this case regarding the validation of a bed
allocation). After that, the property goes on with the Answer term (line 30).
Inside it, we find a disjunction between two possible alternatives in the
protocol. On the left, we may observe an answer_with_constraint event, which
means, according to lines 6-11, that the ’assistant’ replied to the ’operator’
with a validated result that the latter has to decide upon. On the right, we
may observe an event corresponding to any other answer, which in this specific
case denotes the case where the result is empty, meaning that no result is
available to be sent to the ’operator’. In the latter case (the right branch),
the property ends this cycle, since the communication between the two agents
is concluded and new messages concerning new topics can be exchanged in the
future. Instead, in the former case (the left branch), the current cycle is
not ended, because the ’assistant’ is still waiting for an answer from the
’operator’ regarding the result sent. This last aspect is handled in the term
in line 31, where no ’question’ is admissible from the ’operator’, only one
from those listed in lines 12-17. Upon receiving an event matching one such
listed event types, the cycle of the property ends, and as for the right
branch, the protocol can move on.

Now, before presenting another property of interest that we have analysed
through RV4JaCa, it is important to explicit how a property can be violated.
As we mentioned before, we presented which are the events that in certain
points of the property are accepted, and why. An RML property is violated
whenever given the current term denoting the current state of the property,
and a new event, the property does not accept such event. For instance, in the
property presented in Listing LABEL:list:result, an event which is different
from an answer, after having observed a question, is not acceptable. This can
be seen in line 30, where after consuming an event denoting a question, the
only possible following events can be an answer requiring additional info
(left branch), or a general answer (right branch). Thus, if the observed event
is neither of the two, the term is stuck and cannot move on. In RML this
translates into a violation of the property, which is then reported back to
the monitor agent that in turn will trigger all mechanisms for the agents
involved to properly react.

⬇

1question matches {

2 performative:’question’,

3 sender:’operator’, receiver:’assistant’,

4 content:{name:’getValidationResult’}

5};

6answer_with_constraint matches

7{

8 performative:’assert’,

9 sender:’assistant’, receiver:’operator’,

10 content:{name:’answer’, name:’result’, arg1:_, arg2:_}

11};

12constrained_question matches

13 {performative:’question’, sender:’operator’, receiver:’assistant’,
content:{name:’allocValPatients’}} |

14 {performative:’question’, sender:’operator’, receiver:’assistant’,
content:{name:’getOptimisedAllocation’}} |

15 {performative:’question’, sender:’operator’, receiver:’assistant’,
content:{name:’dontAllocValPatients’}} |

16 {performative:’question’, sender:’operator’, receiver:’assistant’,
content:{name:’allocValidValPatients’}} |

17 {performative:’question’, sender:’operator’, receiver:’assistant’,
content:{name:’allocValPatients’}};

18a_question matches

19{

20 performative:’question’,

21 sender:’operator’, receiver:’assistant’

22};

23an_answer matches

24{

25 performative:’assert’,

26 sender:’assistant’, receiver:’operator’

27};

28Main = Question*;

29Question = (question Answer);

30Answer = (answer_with_constraint ConstrainedQuestion) \/ (an_answer);

31ConstrainedQuestion = constrained_question Answer;

The second RML property we tested in the bed allocation domain is reported in
Listing LABEL:list:question-answer. Differently from the property reported in
Listing LABEL:list:result, here we do not check the consistency amongst
topics; instead, we care about checking that an agent always replies to a
question, before posing a new one. As before, the first part of Listing
LABEL:list:question-answer concerns the definition of which events are of
interest for the property (lines 1-10). In this specific case, we have
questions (lines 1-5), and answers (lines 6-10). Note that, differently from
the previous RML property, here we exploit parameters inside the
specification. In fact, the event types reported in lines 1-10 are all
parametric w.r.t. the agents involved in the communication. This means that
such event types do not focus on specific messages exchanged between
predefined agents, as in the previous case, but are kept free (through RML
parameters, we have late binding on the agents involved in the interaction).
This makes the definition of the RML property in line 11 highly parametric,
avoiding the need to update the property for each new agent added to the
system. The property is defined in line 11, through the standard Main term in
RML. Since the property is parametric, it starts with the let operator which
defines the variables used in the term. In this case, the variables used are
ag1 and ag2 (naturally any other name would have sufficed). After that, the
property goes on expecting a question, followed by a corresponding answer.
Here, note that in the first event (i.e., the question), the variables are
bound to the agents involved in the communication, while in the second event
(i.e., the answer), such variables are ground to the previously initialised
values. In this way, a question is free to be sent by any possible agent ag1,
to any possible agent ag2 in the system (where ag1 and ag2 are bound to the
observed agents involved in the communication); instead, an answer is
constrained to be sent by agent ag2 to agent ag1 (with both variables already
bound to the respective values through the previously observed question).

As before, also with this property we can ponder on which events can cause a
violation. In particular, the property expressed in Listing
LABEL:list:question-answer is violated when after a question between two
agents (ag1$\rightarrow$ag2), the following event is not the corresponding
answer (ag2$\rightarrow$ag1), but another message (for instance another
question).

⬇

1question(ag1, ag2) matches

2{

3 performative:’question’,

4 sender:ag1, receiver:ag2

5};

6answer(ag1, ag2) matches

7{

8 performative:’assert’,

9 sender:ag1, receiver:ag2

10};

11Main = {let ag1, ag2; question(ag1, ag2) answer(ag2, ag1)}*;

##  4 Related Work

In past years, some work has focused on formal verification from a more
dynamic viewpoint. In [6], the authors presented a framework to verify Agent
Interaction Protocols (AIP) at runtime. The formalism used in this work allows
the introduction of variables, that are then used to constrain the expected
behaviour in a more expressive way. In [22], the same authors proposed an
approach to verify AIPs at runtime using multiple monitors. This is obtained
by decentralising the global specification (specified as a Trace Expression
[5]), which is used to represent the global protocol, into partial
specifications denoting the single agents’ perspective. Both those approaches
are based on the formalism serving as building block for RML’s semantics. From
this perspective, RV4JaCa is an evolution of these approaches in two ways: (i)
it allows general-purpose verification since no constraint is assumed on the
monitor side, except for being capable of receiving and sending JSON messages;
(ii) to be self-contained, RV4JaCa natively supports RML monitors, and because
of that it allows a more intuitive and high-level protocol specification. In
[9, 33], other approaches to runtime verification of agent interactions are
proposed, and in [27] a framework for dynamic adaptive MAS (DAMS-RV) based on
an adaptive feedback loop is presented. Other approaches to MAS RV include the
spin-off proposals from the SOCS project where the SCIFF computational logic
framework [4] is used to provide the semantics of social integrity
constraints. To model MAS interaction, an expectation-based semantics
specifies the links between observed and expected events, providing a means to
test runtime conformance of an actual conversation with respect to a given
interaction protocol [35]. Similar work has been performed using commitments
[15].

To the best of our knowledge, RV4JaCa is the first approach that integrates RV
within JaCaMo for verifying agent interaction protocols.

##  5 Conclusions and Future Work

Communications between agents play a key role in the functioning of a multi-
agent system since, in practice, agents rarely act alone, they usually inhabit
an environment that contains other agents. Therefore, an extra layer of
security that allows us to verify key aspects of this message exchange adds
great value, in addition to great possibilities for improvement, since certain
aspects do not need to be considered when developing each of the agents. Using
this type of formal verification at runtime allows us to standardise the
interaction between agents through previously defined protocols that all
agents must follow and, if they do not, react in a way that the execution is
not negatively affected by the effects that were caused by this protocol
deviation.

On the other hand, the checks done with RV are not limited to protocol
validation. More specific properties of each application domain can also be
verified once the monitor has access to the content of the exchanged messages.
Even the execution of certain routines or functions according to the direction
in which the conversations between the agents go can be done. For example,
recording the results obtained during the agents’ reasoning in a database
without agents having the responsibility to carry out the registrations
themselves. Or even sending an automatic email to a supervisor if any property
identified by an agent and communicated to another is outside certain
parameters. Therefore, depending on the MAS’s domain, there is a range of
possibilities in which RV can be used.

Based on that, we proposed RV4JaCa, an approach to integrate multi-agent
systems and runtime verification. Our approach was built using JaCaMo and RML,
and it provides significant progress toward obtaining guarantees of the
correct execution of the MAS. The case study presented in this paper
demonstrates the use of RV4JaCa in practice, also showing promising
preliminary results. In addition, it is important to note that our approach
can be applied to different scenarios in different MAS. Based on the presented
case study, we created two distinct properties to be checked at runtime in the
system. The first one is to allow agents to be alerted if there is an
unexpected change in the topic of conversation by the human user who is
interacting with the system. And the second one is capable of verifying that
the previously defined communication protocol between agents is being followed
correctly (in particular, question-answer relations).

As MAS have been used to build systems focused on explainability, RV’s extra
safety layer is certainly useful, since to achieve explainability in MAS much
communication between agents and humans needs to be carried out. Also, we need
to take into account that the developer does not have complete control over
the interactions that human users can make during a dialogue. In this sense,
RV allows us to avoid unexpected and probably inappropriate system behaviour.

As future work, we plan to further extend RV4JaCa to more than interaction
protocols. For instance, it would be relevant to check the agents’ state of
mind as well. Moreover, through RV4JaCa we also want to create a library of
verifiably correct agent interaction protocols to be used in different
scenarios involving both agents and humans in the loop.

## References

  * [1]
  * [2] Amina Adadi & Mohammed Berrada (2018): _Peeking inside the black-box: A survey on Explainable Artificial Intelligence (XAI)_.  IEEE Access 6, pp. 52138–52160, 10.1109/ACCESS.2018.2870052.
  * [3] Wolfgang Ahrendt, Jesús Mauricio Chimento, Gordon J. Pace & Gerardo Schneider (2017): _Verifying data- and control-oriented properties combining static and runtime verification: theory and tools_.  Formal Methods Syst. Des. 51(1), pp. 200–265, 10.1007/s10703-017-0274-y.
  * [4] Marco Alberti, Marco Gavanelli, Evelina Lamma, Paola Mello & Paolo Torroni (2005): _The _S_ CIFF Abductive Proof-Procedure_.  In Stefania Bandini & Sara Manzoni, editors: AI*IA 2005: Advances in Artificial Intelligence, 9th Congress of the Italian Association for Artificial Intelligence, Milan, Italy, September 21-23, 2005, Proceedings, Lecture Notes in Computer Science 3673, Springer, pp. 135–147, 10.1007/11558590_14.
  * [5] Davide Ancona, Angelo Ferrando & Viviana Mascardi (2016): _Comparing Trace Expressions and Linear Temporal Logic for Runtime Verification_.  In Erika Ábrahám, Marcello M. Bonsangue & Einar Broch Johnsen, editors: Theory and Practice of Formal Methods - Essays Dedicated to Frank de Boer on the Occasion of His 60th Birthday, Lecture Notes in Computer Science 9660, Springer, pp. 47–64, 10.1007/978-3-319-30734-3_6.
  * [6] Davide Ancona, Angelo Ferrando & Viviana Mascardi (2017): _Parametric Runtime Verification of Multiagent Systems_.  In Kate Larson, Michael Winikoff, Sanmay Das & Edmund H. Durfee, editors: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2017, São Paulo, Brazil, May 8-12, 2017, ACM, pp. 1457–1459, 10.5555/3091125.3091328.
  * [7] Davide Ancona, Luca Franceschini, Angelo Ferrando & Viviana Mascardi (2021): _RML: Theory and Practice of a Domain Specific Language for Runtime Verification_.  Science of Computer Programming 205, p. 102610, 10.1016/j.scico.2021.102610.
  * [8] Sule Anjomshoae, Amro Najjar, Davide Calvaresi & Kary Främling (2019): _Explainable agents and robots: Results from a systematic literature review_.  In: 18th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2019), Montreal, Canada, May 13–17, 2019, International Foundation for Autonomous Agents and Multiagent Systems, pp. 1078–1088, 10.5555/3306127.3331806.
  * [9] Najwa Abu Bakar & Ali Selamat (2013): _Runtime Verification of Multi-agent Systems Interaction Quality_.  In Ali Selamat, Ngoc Thanh Nguyen & Habibollah Haron, editors: Intelligent Information and Database Systems - 5th Asian Conference, ACIIDS 2013, Kuala Lumpur, Malaysia, March 18-20, 2013, Proceedings, Part I, Lecture Notes in Computer Science 7802, Springer, pp. 435–444, 10.1007/978-3-642-36546-1_45.
  * [10] Ezio Bartocci, Yliès Falcone, Adrian Francalanza & Giles Reger (2018): _Introduction to Runtime Verification_.  In Ezio Bartocci & Yliès Falcone, editors: Lectures on Runtime Verification - Introductory and Advanced Topics, Lecture Notes in Computer Science 10457, Springer, Cham, pp. 1–33, 10.1007/978-3-319-75632-5_1.
  * [11] Ezio Bartocci, Yliès Falcone, Adrian Francalanza & Giles Reger (2018): _Introduction to Runtime Verification_.  In Ezio Bartocci & Yliès Falcone, editors: Lectures on Runtime Verification - Introductory and Advanced Topics, Lecture Notes in Computer Science 10457, Springer, pp. 1–33, 10.1007/978-3-319-75632-5_1.
  * [12] Olivier Boissier, Rafael H Bordini, Jomi F Hübner, Alessandro Ricci & Andrea Santi (2013): _Multi-agent oriented programming with JaCaMo_.  Science of Computer Programming 78(6), pp. 747–761, 10.1016/j.scico.2011.10.004.
  * [13] Rafael H. Bordini, Mehdi Dastani, Jürgen Dix & Amal El Fallah Seghrouchni, editors (2009): _Multi-Agent Programming, Languages, Tools and Applications_.  Springer, 10.1007/978-0-387-89299-3.
  * [14] Rafael H Bordini, Jomi Fred Hübner & Michael Wooldridge (2007): _Programming multi-agent systems in AgentSpeak using Jason_.  John Wiley & Sons, 10.1002/9780470061848.
  * [15] Federico Chesani, Paola Mello, Marco Montali & Paolo Torroni (2009): _Commitment Tracking via the Reactive Event Calculus_.  In Craig Boutilier, editor: IJCAI 2009, Proceedings of the 21st International Joint Conference on Artificial Intelligence, Pasadena, California, USA, July 11-17, 2009, pp. 91–96, 10.5555/1661445.1661461.  Available at http://ijcai.org/Proceedings/09/Papers/026.pdf.
  * [16] Edmund M Clarke (1997): _Model checking_.  In: International Conference on Foundations of Software Technology and Theoretical Computer Science, Springer, pp. 54–56, 10.1007/BFb0058022.
  * [17] Louise A. Dennis, Michael Fisher, Matthew P. Webster & Rafael H. Bordini (2012): _Model checking agent programming languages_.  Autom. Softw. Eng. 19(1), pp. 5–63, 10.1007/s10515-011-0088-x.
  * [18] Ivan Donadello, Mauro Dragoni & Claudio Eccher (2020): _Explaining reasoning algorithms with persuasiveness: a case study for a behavioural change system_.  In: Proceedings of the 35th Annual ACM Symposium on Applied Computing, pp. 646–653, 10.1145/3341105.3373910.
  * [19] Débora C. Engelmann, Lucca Dornelles Cezar, Alison R. Panisson & Rafael H. Bordini (2021): _A Conversational Agent to Support Hospital Bed Allocation_.  In André Britto & Karina Valdivia Delgado, editors: Intelligent Systems - 10th Brazilian Conference, BRACIS 2021, Virtual Event, November 29 - December 3, 2021, Proceedings, Part I, Lecture Notes in Computer Science 13073, Springer, pp. 3–17, 10.1007/978-3-030-91702-9_1.
  * [20] Débora C. Engelmann, Juliana Damasio, Tabajara Krausburg, Olimar Teixeira Borges, Mateus da Silveira Colissi, Alison R. Panisson & Rafael H. Bordini (2021): _Dial4JaCa - A Communication Interface Between Multi-agent Systems and Chatbots_.  In Frank Dignum, Juan Manuel Corchado & Fernando de la Prieta, editors: Advances in Practical Applications of Agents, Multi-Agent Systems, and Social Good. The PAAMS Collection - 19th International Conference, PAAMS 2021, Salamanca, Spain, October 6-8, 2021, Proceedings, Lecture Notes in Computer Science 12946, Springer, pp. 77–88, 10.1007/978-3-030-85739-4_7.
  * [21] Jacques Ferber, Olivier Gutknecht & Fabien Michel (2003): _From Agents to Organizations: An Organizational View of Multi-agent Systems_.  In Paolo Giorgini, Jörg P. Müller & James Odell, editors: Agent-Oriented Software Engineering IV, 4th International Workshop, AOSE 2003, Melbourne, Australia, July 15, 2003, Revised Papers, Lecture Notes in Computer Science 2935, Springer, pp. 214–230, 10.1007/978-3-540-24620-6_15.
  * [22] Angelo Ferrando, Davide Ancona & Viviana Mascardi (2017): _Decentralizing MAS Monitoring with DecAMon_.  In: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2017, São Paulo, Brazil, May 8-12, 2017, ACM, pp. 239–248, 10.5555/3091125.3091164.
  * [23] Michael Fisher, Viviana Mascardi, Kristin Yvonne Rozier, Bernd-Holger Schlingloff, Michael Winikoff & Neil Yorke-Smith (2021): _Towards a framework for certification of reliable autonomous systems_.  Auton. Agents Multi Agent Syst. 35(1), p. 8, 10.1007/s10458-020-09487-2.
  * [24] Bryan Horling & Victor R. Lesser (2004): _A survey of multi-agent organizational paradigms_.  Knowl. Eng. Rev. 19(4), pp. 281–316, 10.1017/S0269888905000317.
  * [25] Jomi F Hubner, Jaime S Sichman & Olivier Boissier (2007): _Developing organised multiagent systems using the MOISE+ model: programming issues at the system and agent levels_.  International Journal of Agent-Oriented Software Engineering 1(3-4), pp. 370–395, 10.1504/IJAOSE.2007.016266.
  * [26] Martin Leucker & Christian Schallhart (2009): _A brief account of runtime verification_.  J. Log. Algebraic Methods Program. 78(5), pp. 293–303, 10.1016/j.jlap.2008.08.004.
  * [27] Yoo Jin Lim, Gwangui Hong, Donghwan Shin, Eunkyoung Jee & Doo-Hwan Bae (2016): _A runtime verification framework for dynamically adaptive multi-agent systems_.  In: 2016 International Conference on Big Data and Smart Computing, BigComp 2016, Hong Kong, China, January 18-20, 2016, IEEE Computer Society, pp. 509–512, 10.1109/BIGCOMP.2016.7425981.
  * [28] Donald W. Loveland (1978): _Automated theorem proving: a logical basis_.  Fundamental studies in computer science 6, North-Holland.  Available at https://www.worldcat.org/oclc/252520243.
  * [29] Alison R. Panisson, Débora C. Engelmann & Rafael H. Bordini (2021): _Engineering Explainable Agents: An Argumentation-Based Approach_.  In Natasha Alechina, Matteo Baldoni & Brian Logan, editors: Engineering Multi-Agent Systems - 9th International Workshop, EMAS 2021, Virtual Event, May 3-4, 2021, Revised Selected Papers, Lecture Notes in Computer Science 13190, Springer, pp. 273–291, 10.1007/978-3-030-97457-2_16.
  * [30] Alison R Panisson, Felipe Meneguzzi, Renata Vieira & Rafael H Bordini (2014): _An approach for argumentation-based reasoning using defeasible logic in multi-agent programming languages_.  In: 11th International Workshop on Argumentation in Multiagent Systems, pp. 1–15.
  * [31] Anand S Rao & Michael P Georgeff (1995): _BDI agents: from theory to practice_.  In: Icmas, 95, pp. 312–319.
  * [32] Alessandro Ricci, Michele Piunti, Mirko Viroli & Andrea Omicini (2009): _Environment programming in CArtAgO_.  In: Multi-agent programming, Springer, pp. 259–288, 10.1007/978-0-387-89299-3_8.
  * [33] Chittra Roungroongsom & Denduang Pradubsuwun (2015): _Formal Verification of Multi-agent System Based on JADE: A Semi-runtime Approach_.  In: Recent Advances in Information and Communication Technology 2015, Springer, pp. 297–306, 10.1007/978-3-319-19024-230.
  * [34] Daniela Schmidt, Alison R. Panisson, Artur Freitas, Rafael H. Bordini, Felipe Meneguzzi & Renata Vieira (2016): _An Ontology-Based Mobile Application for Task Managing in Collaborative Groups_.  In Zdravko Markov & Ingrid Russell, editors: Proceedings of the Twenty-Ninth International Florida Artificial Intelligence Research Society Conference, FLAIRS 2016, Key Largo, Florida, USA, May 16-18, 2016, AAAI Press, pp. 522–526.  Available at http://www.aaai.org/ocs/index.php/FLAIRS/FLAIRS16/paper/view/12956.
  * [35] Paolo Torroni, Pinar Yolum, Munindar P. Singh, Marco Alberti, Federico Chesani, Marco Gavanelli, Evelina Lamma & Paola Mello (2009): _Modelling Interactions via Commitments and Expectations_.  In Virginia Dignum, editor: Handbook of Research on Multi-Agent Systems - Semantics and Dynamics of Organizational Models, IGI Global, pp. 263–284, 10.4018/978-1-60566-256-5.ch011.
  * [36] Michael Winikoff (2017): _BDI agent testability revisited_.  Auton. Agents Multi Agent Syst. 31(5), pp. 1094–1132, 10.1007/s10458-016-9356-2.
  * [37] Michael Winikoff (2017): _Debugging Agent Programs with Why?: Questions_.  In Kate Larson, Michael Winikoff, Sanmay Das & Edmund H. Durfee, editors: Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, AAMAS 2017, São Paulo, Brazil, May 8-12, 2017, ACM, Richland, SC, pp. 251–259, 10.5555/3091125.3091166.
  * [38] Michael Wooldridge (2002): _An Introduction to MultiAgent Systems_.  John wiley & sons Ltd.