* Aljedaani et al. (2020) Wajdi Aljedaani, Yasir Javed, and Mamdouh Alenezi. 2020\.  LDA categorization of security bug reports in Chromium projects. In _Proceedings of the 2020 European Symposium on Software Engineering_. 154–161.
  * Almukaynizi et al. (2017) Mohammed Almukaynizi, Eric Nunes, Krishna Dharaiya, Manoj Senguttuvan, Jana Shakarian, and Paulo Shakarian. 2017.  Proactive identification of exploits in the wild through vulnerability mentions online. In _2017 International Conference on Cyber Conflict (CyCon US)_. IEEE, 82–88.
  * Almukaynizi et al. (2019) Mohammed Almukaynizi, Eric Nunes, Krishna Dharaiya, Manoj Senguttuvan, Jana Shakarian, and Paulo Shakarian. 2019.  Patch before exploited: An approach to identify targeted software vulnerabilities.  In _AI in Cybersecurity_. Springer, 81–113.
  * Anwar et al. (2020) Afsah Anwar, Ahmed Abusnaina, Songqing Chen, Frank Li, and David Mohaisen. 2020.  Cleaning the NVD: Comprehensive quality assessment, improvements, and analyses.  _arXiv preprint arXiv:2006.15074_ (2020).
  * Aota et al. (2020) Masaki Aota, Hideaki Kanehara, Masaki Kubo, Noboru Murata, Bo Sun, and Takeshi Takahashi. 2020\.  Automation of Vulnerability Classification from its Description using Machine Learning. In _2020 IEEE Symposium on Computers and Communications (ISCC)_. IEEE, 1–7.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014\.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ (2014).
  * Barranco et al. (2019) Roberto Camacho Barranco, Arnold P Boedihardjo, and M Shahriar Hossain. 2019.  Analyzing evolving stories in news articles.  _International Journal of Data Science and Analytics_ 8, 3 (2019), 241–256.
  * ben Othmane et al. (2015) Lotfi ben Othmane, Golriz Chehrazi, Eric Bodden, Petar Tsalovski, Achim D Brucker, and Philip Miseldine. 2015\.  Factors impacting the effort required to fix security vulnerabilities. In _International Conference on Information Security_. Springer, 102–119.
  * Bernaschi et al. (2002) Massimo Bernaschi, Emanuele Gabrielli, and Luigi V Mancini. 2002.  REMUS: A security-enhanced operating system.  _ACM Transactions on Information and System Security (TISSEC)_ 5, 1 (2002), 36–61.
  * Bhatt et al. (2021) Navneet Bhatt, Adarsh Anand, and VSS Yadavalli. 2021\.  Exploitability prediction of software vulnerabilities.  _Quality and Reliability Engineering International_ 37, 2 (2021), 648–663.
  * Bhuiyan et al. (2021) Farzana Ahamed Bhuiyan, Md Bulbul Sharif, and Akond Rahman. 2021\.  Security Bug Report Usage for Software Vulnerability Research: A Systematic Mapping Study.  _IEEE Access_ 9 (2021), 28471–28495.
  * Binyamini et al. (2021) Hodaya Binyamini, Ron Bitton, Masaki Inokuchi, Tomohiko Yagyu, Yuval Elovici, and Asaf Shabtai. 2021\.  A Framework for Modeling Cyber Attack Techniques from Security Vulnerability Descriptions. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_. 2574–2583.
  * Blei and McAuliffe (2007) David M. Blei and Jon D. McAuliffe. 2007.  Supervised topic models. In _Proceedings of the 20th International Conference on Neural Information Processing Systems_. 121–128.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003\.  Latent dirichlet allocation.  _Journal of machine learning research_ 3, Jan (2003), 993–1022.
  * Bojanowski et al. (2017) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017.  Enriching word vectors with subword information.  _Transactions of the Association for Computational Linguistics_ 5 (2017), 135–146.
  * Bosu et al. (2014) Amiangshu Bosu, Jeffrey C Carver, Munawar Hafiz, Patrick Hilley, and Derek Janni. 2014.  Identifying the characteristics of vulnerable code changes: An empirical study. In _Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering_. 257–268.
  * Bozorgi et al. (2010) Mehran Bozorgi, Lawrence K Saul, Stefan Savage, and Geoffrey M Voelker. 2010.  Beyond heuristics: learning to classify vulnerabilities and predict exploits. In _Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining_. 105–114.
  * Broadcom ([n. d.]a) Broadcom. [n. d.]a.  Symantec attack signatures.  Retrieved April, 2022 from https://bit.ly/symantec_att_sign
  * Broadcom ([n. d.]b) Broadcom. [n. d.]b.  Symantec threat explorer.  Retrieved April, 2022 from https://bit.ly/symantec_threats
  * Bullough et al. (2017) Benjamin L Bullough, Anna K Yanchenko, Christopher L Smith, and Joseph R Zipkin. 2017.  Predicting exploitation of disclosed software vulnerabilities using open-source data. In _Proceedings of the 3rd ACM on International Workshop on Security And Privacy Analytics_. 45–53.
  * Cabral et al. (2019) George G Cabral, Leandro L Minku, Emad Shihab, and Suhaib Mujahid. 2019.  Class imbalance evolution and verification latency in just-in-time software defect prediction. In _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_. IEEE, 666–676.
  * CERT ([n. d.]) CERT. [n. d.].  Basic fuzzing framework.  Retrieved April, 2022 from https://bit.ly/basic_fuzzing_framework
  * Cha ([n. d.]) Sang Kil Cha. [n. d.].  OFuzz.  Retrieved April, 2022 from https://github.com/sangkilc/ofuzz
  * Cha et al. (2012) Sang Kil Cha, Thanassis Avgerinos, Alexandre Rebert, and David Brumley. 2012.  Unleashing mayhem on binary code. In _2012 IEEE Symposium on Security and Privacy_. IEEE, 380–394.
  * Chandrashekar and Sahin (2014) Girish Chandrashekar and Ferat Sahin. 2014.  A survey on feature selection methods.  _Computers & Electrical Engineering_ 40, 1 (2014), 16–28.
  * Chen et al. (2019a) Haipeng Chen, Jing Liu, Rui Liu, Noseong Park, and VS Subrahmanian. 2019a.  VASE: A Twitter-based vulnerability analysis and score engine. In _2019 IEEE International Conference on Data Mining (ICDM)_. IEEE, 976–981.
  * Chen et al. (2019b) Haipeng Chen, Jing Liu, Rui Liu, Noseong Park, and VS Subrahmanian. 2019b.  VEST: A system for vulnerability exploit scoring & timing. In _IJCAI_. 6503–6505.
  * Chen et al. (2019c) Haipeng Chen, Rui Liu, Noseong Park, and VS Subrahmanian. 2019c.  Using twitter to predict when vulnerabilities will be exploited. In _Proceedings of the 25th International Conference on Knowledge Discovery & Data Mining_. 3143–3152.
  * Chen et al. (2020a) Jinfu Chen, Patrick Kwaku Kudjo, Solomon Mensah, Selasie Aformaley Brown, and George Akorfu. 2020a.  An automatic software vulnerability classification framework using term frequency-inverse gravity moment and feature selection.  _Journal of Systems and Software_ 167 (2020), 110616.
  * Chen et al. (2016) Kewen Chen, Zuping Zhang, Jun Long, and Hao Zhang. 2016\.  Turning from TF-IDF to TF-IGM for term weighting in text classification.  _Expert Systems with Applications_ 66 (2016), 245–260.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  Xgboost: A scalable tree boosting system. In _Proceedings of the 22nd International Conference on Knowledge Discovery and Data Mining_. 785–794.
  * Chen et al. (2020b) Yang Chen, Andrew E Santosa, Ang Ming Yi, Abhishek Sharma, Asankhaya Sharma, and David Lo. 2020b.  A machine learning approach for vulnerability curation. In _Proceedings of the 17th International Conference on Mining Software Repositories_. 32–42.
  * Chen et al. (2010) Zhongqiang Chen, Yuan Zhang, and Zhongrong Chen. 2010\.  A categorization framework for common computer vulnerabilities and exposures.  _Comput. J._ 53, 5 (2010), 551–580.
  * Cortes and Vapnik (1995) Corinna Cortes and Vladimir Vapnik. 1995.  Support-vector networks.  _Machine learning_ 20, 3 (1995), 273–297.
  * Crammer et al. (2006) Koby Crammer, Ofer Dekel, Joseph Keshet, Shai Shalev-Shwartz, and Yoram Singer. 2006.  Online passive aggressive algorithms.  (2006).
  * Cruzes and Dybå (2011) Daniela S Cruzes and Tore Dybå. 2011.  Research synthesis in software engineering: A tertiary study.  _Information and Software Technology_ 53, 5 (2011), 440–455.
  * Das et al. (2021) Siddhartha Shankar Das, Edoardo Serra, Mahantesh Halappanavar, Alex Pothen, and Ehab Al-Shaer. 2021\.  V2W-BERT: A framework for effective hierarchical multiclass classification of software vulnerabilities.  _arXiv preprint arXiv:2102.11498_ (2021).
  * de Oliveira Neto et al. (2019) Francisco Gomes de Oliveira Neto, Richard Torkar, Robert Feldt, Lucas Gren, Carlo A Furia, and Ziwei Huang. 2019.  Evolution of statistical analysis in empirical software engineering research: Current state and steps forward.  _Journal of Systems and Software_ 156 (2019), 246–267.
  * de Sousa et al. (2020) Daniel Alves de Sousa, Elaine Ribeiro de Faria, and Rodrigo Sanches Miani. 2020.  Evaluating the performance of Twitter-based exploit detectors.  _arXiv preprint arXiv:2011.03113_ (2020).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Dissanayake et al. (2020) Nesara Dissanayake, Asangi Jayatilaka, Mansooreh Zahedi, and M Ali Babar. 2020.  Software security patch management–A systematic literature review of challenges, approaches, tools and practices.  _arXiv preprint arXiv:2012.00544_ (2020).
  * Dolan-Gavitt et al. (2016) Brendan Dolan-Gavitt, Patrick Hulin, Engin Kirda, Tim Leek, Andrea Mambretti, Wil Robertson, Frederick Ulrich, and Ryan Whelan. 2016.  Lava: Large-scale automated vulnerability addition. In _2016 IEEE Symposium on Security and Privacy (SP)_. IEEE, 110–121.
  * Dong et al. (2019) Ying Dong, Wenbo Guo, Yueqi Chen, Xinyu Xing, Yuqing Zhang, and Gang Wang. 2019\.  Towards the detection of inconsistencies in public security vulnerability reports. In _28th $\\{$USENIX$\\}$ Security Symposium_. 869–885.
  * Duan et al. (2021) Xuanyu Duan, Mengmeng Ge, Triet Huynh Minh Le, Faheem Ullah, Shang Gao, Xuequan Lu, and M Ali Babar. 2021.  Automated Security Assessment for the Internet of Things. In _2021 IEEE 26th Pacific Rim International Symposium on Dependable Computing (PRDC)_. IEEE, 47–56.
  * Duan et al. (2019) Xu Duan, Jingzheng Wu, Shouling Ji, Zhiqing Rui, Tianyue Luo, Mutian Yang, and Yanjun Wu. 2019.  VulSniper: Focus your attention to shoot fine-grained vulnerabilities. In _IJCAI_. 4665–4671.
  * Dzeroski and Zenko (2002) Saso Dzeroski and Bernard Zenko. 2002.  Is combining classifiers better than selecting the best one?. In _ICML_ , Vol. 2002. Citeseer, 123e30.
  * Edkrantz (2015) MICHEL Edkrantz. 2015\.  _Predicting exploit likelihood for cyber vulnerabilities with machine learning_.  Master’s thesis.
  * Edkrantz et al. (2015) Michel Edkrantz, Staffan Truvé, and Alan Said. 2015\.  Predicting vulnerability exploits in the wild. In _2015 IEEE 2nd International Conference on Cyber Security and Cloud Computing_. IEEE, 513–514.
  * Elbaz et al. (2020) Clément Elbaz, Louis Rilling, and Christine Morin. 2020\.  Fighting N-day vulnerabilities with automated CVSS vector prediction at disclosure. In _the 15th International Conference on Availability, Reliability and Security_. 1–10.
  * ESET ([n. d.]) ESET. [n. d.].  ESET security advisories.  Retrieved April, 2022 from https://bit.ly/eset_virus
  * Evangelista et al. (2020) João Rafael Gonçalves Evangelista, Renato José Sassi, Márcio Romero, and Domingos Napolitano. 2020.  Systematic literature review to investigate the application of open source intelligence (OSINT) with artificial intelligence.  _Journal of Applied Security Research_ (2020), 1–25.
  * Fang et al. (2020) Yong Fang, Yongcheng Liu, Cheng Huang, and Liang Liu. 2020\.  FastEmbed: Predicting vulnerability exploitation possibility based on ensemble machine learning algorithm.  _Plos one_ 15, 2 (2020), e0228439.
  * Finkel et al. (2005) Jenny Rose Finkel, Trond Grenager, and Christopher D Manning. 2005\.  Incorporating non-local information into information extraction systems by gibbs sampling. In _Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05)_. 363–370.
  * FIRST ([n. d.]a) FIRST. [n. d.]a.  Common Vulnerability Scoring System.  Retrieved April, 2022 from https://www.first.org/cvss
  * FIRST ([n. d.]b) FIRST. [n. d.]b.  CVSS version 2.  Retrieved April, 2022 from https://www.first.org/cvss/v2/guide
  * FIRST ([n. d.]c) FIRST. [n. d.]c.  CVSS version 3.  Retrieved April, 2022 from https://www.first.org/cvss/v3.0/specification-document
  * FIRST ([n. d.]d) FIRST. [n. d.]d.  CVSS version 3.1.  Retrieved April, 2022 from https://www.first.org/cvss/v3.1/specification-document
  * Foreman (2019) Park Foreman. 2019\.  _Vulnerability management_.  CRC Press.
  * Foundation ([n. d.]) Wikimedia Foundation. [n. d.].  Wikipedia pages.  Retrieved April, 2022 from https://www.wikipedia.org
  * Future ([n. d.]) Recorded Future. [n. d.].  Recorded Future security advisories.  Retrieved April, 2022 from https://bit.ly/rf_sec
  * Gawron et al. (2017) Marian Gawron, Feng Cheng, and Christoph Meinel. 2017\.  Automatic vulnerability classification using machine learning. In _International Conference on Risks and Security of Internet and Systems_. Springer, 3–17.
  * Gegick et al. (2010) Michael Gegick, Pete Rotella, and Tao Xie. 2010.  Identifying security bug reports via text mining: An industrial case study. In _2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010)_. IEEE, 11–20.
  * Ghaffarian and Shahriari (2017) Seyed Mohammad Ghaffarian and Hamid Reza Shahriari. 2017.  Software vulnerability analysis and discovery using machine-learning and data-mining techniques: A survey.  _ACM Computing Surveys (CSUR)_ 50, 4 (2017), 1–36.
  * Gong et al. (2019) Xi Gong, Zhenchang Xing, Xiaohong Li, Zhiyong Feng, and Zhuobing Han. 2019.  Joint prediction of multiple vulnerability characteristics through multi-task learning. In _2019 24th International Conference on Engineering of Complex Computer Systems (ICECCS)_. IEEE, 31–40.
  * Gonzalez et al. (2019) Danielle Gonzalez, Holly Hastings, and Mehdi Mirakhorli. 2019\.  Automated characterization of software vulnerabilities. In _2019 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 135–139.
  * Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016\.  _Deep learning_.  MIT press.
  * Grieco et al. (2016) Gustavo Grieco, Guillermo Luis Grinblat, Lucas Uzal, Sanjay Rawat, Josselin Feist, and Laurent Mounier. 2016.  Toward large-scale vulnerability discovery using machine learning. In _Proceedings of the Sixth ACM Conference on Data and Application Security and Privacy_. 85–96.
  * Guo et al. (2020) Hao Guo, Zhenchang Xing, and Xiaohong Li. 2020.  Predicting Missing Information of Key Aspects in Vulnerability Reports.  _arXiv preprint arXiv:2008.02456_ (2020).
  * Han et al. (2011) Jiawei Han, Micheline Kamber, and Jian Pei. 2011.  Data mining concepts and techniques third edition.  _The Morgan Kaufmann Series in Data Management Systems_ 5, 4 (2011), 83–124.
  * Han et al. (2000) Jiawei Han, Jian Pei, and Yiwen Yin. 2000.  Mining frequent patterns without candidate generation.  _ACM sigmod record_ 29, 2 (2000), 1–12.
  * Han et al. (2018) Zhuobing Han, Xiaohong Li, Hongtao Liu, Zhenchang Xing, and Zhiyong Feng. 2018.  Deepweak: Reasoning common software weaknesses via knowledge graph embedding. In _2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER)_. IEEE, 456–466.
  * Han et al. (2017) Zhuobing Han, Xiaohong Li, Zhenchang Xing, Hongtao Liu, and Zhiyong Feng. 2017.  Learning to predict severity of software vulnerability using only vulnerability description. In _2017 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 125–136.
  * Hastie et al. (2009) Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2009\.  _The elements of statistical learning: data mining, inference, and prediction_.  Springer Science & Business Media.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Hommersom et al. (2021) Daan Hommersom, Antonino Sabetta, Bonaventura Coppola, and Damian A Tamburri. 2021.  Automated mapping of vulnerability advisories onto their fix commits in open source repositories.  _arXiv preprint arXiv:2103.13375_ (2021).
  * Horawalavithana et al. (2019) Sameera Horawalavithana, Abhishek Bhattacharjee, Renhao Liu, Nazim Choudhury, Lawrence O. Hall, and Adriana Iamnitchi. 2019.  Mentions of security vulnerabilities on reddit, twitter and github. In _IEEE/WIC/ACM International Conference on Web Intelligence_. 200–207.
  * Horwitz et al. (1990) Susan Horwitz, Thomas Reps, and David Binkley. 1990\.  Interprocedural slicing using dependence graphs.  _ACM Transactions on Programming Languages and Systems (TOPLAS)_ 12, 1 (1990), 26–60.
  * Huang et al. (2019) Guoyan Huang, Yazhou Li, Qian Wang, Jiadong Ren, Yongqiang Cheng, and Xiaolin Zhao. 2019\.  Automatic classification method for software vulnerability based on deep neural network.  _IEEE Access_ 7 (2019), 28291–28298.
  * Huang and Wu (2020) Shin-Ying Huang and Yiju Wu. 2020.  Dynamic software vulnerabilities threat prediction through social media contextual analysis. In _Proceedings of the 15th Asia Conference on Computer and Communications Security_. 892–894.
  * Huang et al. (2021) Yujin Huang, Han Hu, and Chunyang Chen. 2021.  Robustness of on-device models: Adversarial attack to deep learning models on android apps. In _2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)_. IEEE, 101–110.
  * Inc ([n. d.]) SecurityFocus Inc. [n. d.].  BugTraq vulnerability database.  Retrieved April, 2022 from http://www.securityfocus.com
  * Inc. ([n. d.]) Secunia Inc. [n. d.].  Secunia vulnerability advisories.  Retrieved April, 2022 from http://secunia.com
  * Jacobs et al. (2020) Jay Jacobs, Sasha Romanosky, Idris Adjerid, and Wade Baker. 2020.  Improving vulnerability remediation through better exploit prediction.  _Journal of Cybersecurity_ 6, 1 (2020), tyaa015.
  * Jacobs et al. (2019) Jay Jacobs, Sasha Romanosky, Benjamin Edwards, Michael Roytman, and Idris Adjerid. 2019\.  Exploit prediction scoring system (EPSS).  _arXiv preprint arXiv:1908.04856_ (2019).
  * Jiang and Atif (2020) Yuning Jiang and Yacine Atif. 2020.  An approach to discover and assess vulnerability severity automatically in cyber-physical systems. In _13th International Conference on Security of Information and Networks_. 1–8.
  * Jiao and Du (2016) Yasen Jiao and Pufeng Du. 2016.  Performance measures in evaluating machine learning based bioinformatics predictors for classifications.  _Quantitative Biology_ 4, 4 (2016), 320–330.
  * Kanakogi et al. (2021) Kenta Kanakogi, Hironori Washizaki, Yoshiaki Fukazawa, Shinpei Ogata, Takao Okubo, Takehisa Kato, Hideyuki Kanuka, Atsuo Hazeyama, and Nobukazu Yoshioka. 2021.  Tracing CAPEC attack patterns from CVE vulnerability information using natural language processing technique. In _Proceedings of the 54th Hawaii International Conference on System Sciences_. 6996.
  * Ke et al. (2017) Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017\.  Lightgbm: A highly efficient gradient boosting decision tree.  _Advances in neural information processing systems_ 30 (2017), 3146–3154.
  * Keele et al. (2007) Staffs Keele et al. 2007\.  _Guidelines for performing systematic literature reviews in software engineering_.  Technical Report. Technical report, Ver. 2.3 EBSE Technical Report. EBSE.
  * Kenna Security ([n. d.]) Inc. Kenna Security. [n. d.].  Kenna Security.  Retrieved April, 2022 from http://www.kennasecurity.com
  * Khan and Parkinson (2018) Saad Khan and Simon Parkinson. 2018.  Review into state of the art of vulnerability assessment using artificial intelligence.  In _Guide to Vulnerability Analysis for Computer Networks and Systems_. Springer, 3–32.
  * Khazaei et al. (2016) Atefeh Khazaei, Mohammad Ghasemzadeh, and Vali Derhami. 2016\.  An automatic method for CVSS score prediction using vulnerabilities description.  _Journal of Intelligent & Fuzzy Systems_ 30, 1 (2016), 89–96.
  * Kim (2014) Yoon Kim. 2014.  Convolutional neural networks for sentence classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, 1746–1751.
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_ (2016).
  * Kivelä et al. (2014) Mikko Kivelä, Alex Arenas, Marc Barthelemy, James P Gleeson, Yamir Moreno, and Mason A Porter. 2014.  Multilayer networks.  _Journal of complex networks_ 2, 3 (2014), 203–271.
  * Kohonen (1990) Teuvo Kohonen. 1990\.  The self-organizing map.  _Proc. IEEE_ 78, 9 (1990), 1464–1480.
  * Kritikos et al. (2019) Kyriakos Kritikos, Kostas Magoutis, Manos Papoutsakis, and Sotiris Ioannidis. 2019.  A survey on vulnerability assessment tools and databases for cloud-based web applications.  _Array_ 3 (2019), 100011.
  * Kudjo et al. (2020) Patrick Kwaku Kudjo, Jinfu Chen, Solomon Mensah, Richard Amankwah, and Christopher Kudjo. 2020.  The effect of Bellwether analysis on software vulnerability severity prediction models.  _Software Quality Journal_ (2020), 1–34.
  * Kudjo et al. (2019) Patrick Kwaku Kudjo, Jinfu Chen, Minmin Zhou, Solomon Mensah, and Rubing Huang. 2019.  Improving the accuracy of vulnerability report classification using term frequency-inverse gravity moment. In _2019 IEEE 19th International Conference on Software Quality, Reliability and Security (QRS)_. IEEE, 248–259.
  * Kumar et al. (2020) Ram Shankar Siva Kumar, Jonathon Penney, Bruce Schneier, and Kendra Albert. 2020.  Legal risks of adversarial machine learning research.  _arXiv preprint arXiv:2006.16179_ (2020).
  * Kursa et al. (2010) Miron B Kursa, Aleksander Jankowski, and Witold R Rudnicki. 2010\.  Boruta–a system for feature selection.  _Fundamenta Informaticae_ 101, 4 (2010), 271–285.
  * Lai et al. (2015) Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015\.  Recurrent convolutional neural networks for text classification. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 29.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed representations of sentences and documents. In _International conference on machine learning_. PMLR, 1188–1196.
  * Le and Babar (2022) Triet HM Le and M Ali Babar. 2022.  On the Use of Fine-grained Vulnerable Code Statements for Software Vulnerability Assessment Models.  _arXiv preprint arXiv:2203.08417_ (2022).
  * Le et al. ([n. d.]) Triet HM Le, Huaming Chen, and M Ali Babar. [n. d.].  Supplementary materials.  Retrieved April, 2022 from https://figshare.com/s/da4d238ecdf9123dc0b8
  * Le et al. (2020a) Triet HM Le, Hao Chen, and M Ali Babar. 2020a.  Deep learning for source code modeling and generation: Models, applications, and challenges.  _ACM Computing Surveys (CSUR)_ 53, 3 (2020), 1–38.
  * Le et al. (2021a) Triet HM Le, Roland Croft, David Hin, and M Ali Babar. 2021a.  A large-scale study of security vulnerability support on developer Q&A websites. In _Evaluation and Assessment in Software Engineering_. 109–118.
  * Le et al. (2020b) Triet HM Le, David Hin, Roland Croft, and M Ali Babar. 2020b.  PUMiner: Mining security posts from developer question and answer websites with PU learning. In _Proceedings of the 17th International Conference on Mining Software Repositories_. 350–361.
  * Le et al. (2021b) Triet HM Le, David Hin, Roland Croft, and M Ali Babar. 2021b.  Deepcva: Automated commit-level vulnerability assessment with deep multi-task learning. In _2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)_. IEEE, 717–729.
  * Le et al. (2019) Triet HM Le, Bushra Sabir, and Muhammad Ali Babar. 2019\.  Automated software vulnerability assessment with concept drift. In _Proceedings of the 16th International Conference on Mining Software Repositories (MSR)_. IEEE, 371–382.
  * Li et al. (2021) Yi Li, Shaohua Wang, and Tien N Nguyen. 2021.  Vulnerability detection with fine-grained interpretations. In _Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering_.
  * Lin et al. (2020) Guanjun Lin, Sheng Wen, Qing-Long Han, Jun Zhang, and Yang Xiang. 2020.  Software vulnerability detection using deep neural networks: a survey.  _Proc. IEEE_ 108, 10 (2020), 1825–1848.
  * Lin et al. (2017) Zhechao Lin, Xiang Li, and Xiaohui Kuang. 2017.  Machine learning in vulnerability databases. In _2017 10th International Symposium on Computational Intelligence and Design (ISCID)_ , Vol. 1. IEEE, 108–113.
  * Liu and Li (2019) Hailong Liu and Bo Li. 2019\.  Automated classification of attacker privileges based on deep neural network. In _International Conference on Smart Computing and Communication_. Springer, 180–189.
  * Liu et al. (2020) Jiakun Liu, Qiao Huang, Xin Xia, Emad Shihab, David Lo, and Shanping Li. 2020\.  Is using deep learning frameworks free? characterizing technical debt in deep learning frameworks. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in Society_. 1–10.
  * Liu et al. (2019) Kai Liu, Yun Zhou, Qingyong Wang, and Xianqiang Zhu. 2019\.  Vulnerability severity prediction with deep neural network. In _2019 5th International Conference on Big Data and Information Analytics (BigDIA)_. IEEE, 114–119.
  * Malhotra et al. (2021) Ruchika Malhotra et al. 2021\.  Severity Prediction of Software Vulnerabilities Using Textual Data. In _Proceedings of International Conference on Recent Trends in Machine Learning, IoT, Smart Cities and Applications_. Springer, 453–464.
  * Manadhata and Wing (2010) Pratyusa K Manadhata and Jeannette M Wing. 2010.  An attack surface metric.  _IEEE Transactions on Software Engineering_ 37, 3 (2010), 371–386.
  * Marcus et al. (1993) Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993\.  Building a large annotated corpus of English: The Penn Treebank.  (1993).
  * Mazuera-Rozo et al. (2021) A. Mazuera-Rozo, A. Mojica-Hanke, M. Linares-Vasquez, and G. Bavota. 2021.  Shallow or deep? An empirical study on detecting vulnerabilities using deep learning. In _Proceedings of the 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC)_. 276–287.
  * Meneely et al. (2013) Andrew Meneely, Harshavardhan Srinivasan, Ayemi Musa, Alberto Rodriguez Tejeda, Matthew Mokary, and Brian Spates. 2013.  When a patch goes bad: Exploring the properties of vulnerability-contributing commits. In _2013 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement_. IEEE, 65–74.
  * Menzies et al. (2018) Tim Menzies, Suvodeep Majumder, Nikhila Balaji, Katie Brey, and Wei Fu. 2018.  500+ times faster than deep learning:(a case study exploring faster methods for text mining stackoverflow). In _2018 IEEE/ACM 15th International Conference on Mining Software Repositories (MSR)_. IEEE, 554–563.
  * Micro ([n. d.]a) Trend Micro. [n. d.]a.  Trend Micro security advisories.  Retrieved April, 2022 from https://bit.ly/trend_micro_sec
  * Micro ([n. d.]b) Trend Micro. [n. d.]b.  ZeroDay Initiative security advisories.  Retrieved April, 2022 from https://bit.ly/zeroday_sec
  * Microsoft ([n. d.]) Microsoft. [n. d.].  Microsoft security advisories.  Retrieved April, 2022 from https://bit.ly/ms_sec_advisories
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Distributed representations of words and phrases and their compositionality.  _arXiv preprint arXiv:1310.4546_ (2013).
  * MITRE ([n. d.]a) MITRE. [n. d.]a.  Common Attack Pattern Enumeration and Classification.  Retrieved April, 2022 from https://capec.mitre.org
  * MITRE ([n. d.]b) MITRE. [n. d.]b.  Common Platform Enumeration.  Retrieved April, 2022 from https://cpe.mitre.org
  * MITRE ([n. d.]c) MITRE. [n. d.]c.  Common Vulnerabilities and Exposures.  Retrieved April, 2022 from https://cve.mitre.org/
  * MITRE ([n. d.]d) MITRE. [n. d.]d.  Common Weakness Enumeration.  Retrieved April, 2022 from https://cwe.mitre.org
  * Mounika et al. (2019) Vanamala Mounika, Xiaohong Yuan, and Kanishka Bandaru. 2019\.  Analyzing CVE database using unsupervised topic modelling. In _2019 International Conference on Computational Science and Computational Intelligence_. 72–77.
  * Murtaza et al. (2016) Syed Shariyar Murtaza, Wael Khreich, Abdelwahab Hamou-Lhadj, and Ayse Basar Bener. 2016.  Mining trends and patterns of software vulnerabilities.  _Journal of Systems and Software_ 117 (2016), 218–228.
  * Na et al. (2016) Sarang Na, Taeeun Kim, and Hwankuk Kim. 2016.  A study on the classification of common vulnerabilities and exposures using naïve bayes. In _International Conference on Broadband and Wireless Computing, Communication and Applications_. Springer, 657–662.
  * Naim et al. (2017) Sheikh Motahar Naim, Arnold P Boedihardjo, and M Shahriar Hossain. 2017.  A scalable model for tracking topical evolution in large document collections. In _2017 IEEE International Conference on Big Data (Big Data)_. IEEE, 726–735.
  * Nakagawa et al. (2019) Shunta Nakagawa, Tatsuya Nagai, Hideaki Kanehara, Keisuke Furumoto, Makoto Takita, Yoshiaki Shiraishi, Takeshi Takahashi, Masami Mohri, Yasuhiro Takano, and Masakatu Morii. 2019.  Character-level convolutional neural network for predicting severity of software vulnerability from vulnerability description.  _IEICE Transactions on Information and Systems_ 102, 9 (2019), 1679–1682.
  * Neuhaus and Zimmermann (2010) Stephan Neuhaus and Thomas Zimmermann. 2010.  Security trend analysis with cve topic models. In _2010 IEEE 21st International Symposium on Software Reliability Engineering_. IEEE, 111–120.
  * NIST ([n. d.]a) NIST. [n. d.]a.  National Vulnerability Database.  Retrieved April, 2022 from https://nvd.nist.gov
  * NIST ([n. d.]b) NIST. [n. d.]b.  Software Assurance Reference Dataset (SARD).  Retrieved April, 2022 from https://samate.nist.gov/SRD
  * NIST ([n. d.]c) NIST. [n. d.]c.  Vulnerability description ontology.  Retrieved April, 2022 from https://bit.ly/nist_vdo
  * Nunes et al. (2016) Eric Nunes, Ahmad Diab, Andrew Gunn, Ericsson Marin, Vineet Mishra, Vivin Paliath, John Robertson, Jana Shakarian, Amanda Thart, and Paulo Shakarian. 2016\.  Darknet and deepnet mining for proactive cybersecurity threat intelligence. In _2016 IEEE Conference on Intelligence and Security Informatics (ISI)_. IEEE, 7–12.
  * Ognawala et al. (2018) Saahil Ognawala, Ricardo Nales Amato, Alexander Pretschner, and Pooja Kulkarni. 2018\.  Automatically assessing vulnerabilities discovered by compositional analysis. In _Proceedings of the 1st International Workshop on Machine Learning and Software Engineering in Symbiosis_. 16–25.
  * Ognawala et al. (2016) Saahil Ognawala, Martín Ochoa, Alexander Pretschner, and Tobias Limmer. 2016.  MACKE: Compositional analysis of low-level vulnerabilities with symbolic execution. In _Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering_. 780–785.
  * Ohira et al. (2015) Masao Ohira, Yutaro Kashiwa, Yosuke Yamatani, Hayato Yoshiyuki, Yoshiya Maeda, Nachai Limsettho, Keisuke Fujino, Hideaki Hata, Akinori Ihara, and Kenichi Matsumoto. 2015\.  A dataset of high impact bugs: Manually-classified issue reports. In _2015 IEEE/ACM 12th Working Conference on Mining Software Repositories_. IEEE, 518–521.
  * Othmane et al. (2017) Lotfi Ben Othmane, Golriz Chehrazi, Eric Bodden, Petar Tsalovski, and Achim D Brucker. 2017.  Time for addressing software security issues: Prediction models and impacting factors.  _Data Science and Engineering_ 2, 2 (2017), 107–124.
  * Ou et al. (2005) Xinming Ou, Sudhakar Govindavajhala, and Andrew W Appel. 2005\.  MulVAL: A logic-based network security analyzer. In _USENIX security symposium_ , Vol. 8. Baltimore, MD, 113–128.
  * OWASP ([n. d.]) OWASP. [n. d.].  Open Web Application Security Project.  Retrieved April, 2022 from https://bit.ly/owasp_main
  * Palacio-Niño and Berzal (2019) Julio-Omar Palacio-Niño and Fernando Berzal. 2019.  Evaluation metrics for unsupervised learning algorithms.  _arXiv preprint arXiv:1905.05667_ (2019).
  * Pan and Yang (2009) Sinno Jialin Pan and Qiang Yang. 2009.  A survey on transfer learning.  _IEEE Transactions on knowledge and data engineering_ 22, 10 (2009), 1345–1359.
  * Pastor-Galindo et al. (2020) Javier Pastor-Galindo, Pantaleone Nespoli, Félix Gómez Mármol, and Gregorio Martínez Pérez. 2020\.  The not yet exploited goldmine of OSINT: Opportunities, open challenges and future trends.  _IEEE Access_ 8 (2020), 10282–10304.
  * Peters et al. (2017) Fayola Peters, Thein Than Tun, Yijun Yu, and Bashar Nuseibeh. 2017\.  Text filtering and ranking for security bug report prediction.  _IEEE Transactions on Software Engineering_ 45, 6 (2017), 615–631.
  * Peters et al. (2018) Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  _arXiv preprint arXiv:1802.05365_ (2018).
  * Project ([n. d.]) Openwall Project. [n. d.].  Openwall security advisories.  Retrieved April, 2022 from https://bit.ly/sec_openwall
  * Rapid7 ([n. d.]) Rapid7. [n. d.].  Metasploit security advisories.  Retrieved April, 2022 from https://www.rapid7.com/db/modules
  * Raschka (2018) Sebastian Raschka. 2018\.  Model evaluation, model selection, and algorithm selection in machine learning.  _arXiv preprint arXiv:1811.12808_ (2018).
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ (2019).
  * Rosenberg et al. (2021) Ishai Rosenberg, Asaf Shabtai, Yuval Elovici, and Lior Rokach. 2021.  Adversarial machine learning attacks and defense methods in the cyber security domain.  _ACM Computing Surveys (CSUR)_ 54, 5 (2021), 1–36.
  * Ruohonen (2017) Jukka Ruohonen. 2017\.  Classifying web exploits with topic modeling. In _2017 28th International Workshop on Database and Expert Systems Applications (DEXA)_. IEEE, 93–97.
  * Ruohonen and Leppänen (2018) Jukka Ruohonen and Ville Leppänen. 2018.  Toward validation of textual information retrieval techniques for software weaknesses. In _International Conference on Database and Expert Systems Applications_. Springer, 265–277.
  * Russell et al. (2018) Rebecca Russell, Louis Kim, Lei Hamilton, Tomo Lazovich, Jacob Harer, Onur Ozdemir, Paul Ellingwood, and Marc McConley. 2018\.  Automated vulnerability detection in source code using deep representation learning. In _2018 17th IEEE international conference on machine learning and applications (ICMLA)_. IEEE, 757–762.
  * Russo et al. (2019) Ernesto R Russo, Andrea D Sorbo, Corrado A Visaggio, and Gerardo Canfora. 2019.  Summarizing vulnerabilities’ descriptions to support experts during vulnerability assessment activities.  _Journal of Systems and Software_ 156 (2019), 84–99.
  * Sabir et al. (2021) Bushra Sabir, Faheem Ullah, M Ali Babar, and Raj Gaire. 2021\.  Machine Learning for Detecting Data Exfiltration: A Review.  _ACM Computing Surveys (CSUR)_ 54, 3 (2021), 1–47.
  * Sabottke et al. (2015) Carl Sabottke, Octavian Suciu, and Tudor Dumitraș. 2015\.  Vulnerability disclosure in the age of social media: Exploiting twitter for predicting real-world exploits. In _24th $\\{$USENIX$\\}$ Security Symposium_. 1041–1056.
  * Sahin and Tosun (2019) Sefa Eren Sahin and Ayse Tosun. 2019.  A conceptual replication on predicting the severity of software vulnerabilities.  In _Proceedings of the Evaluation and Assessment on Software Engineering_. 244–250.
  * Sawadogo et al. (2020) Arthur D Sawadogo, Tegawendé F Bissyandé, Naouel Moha, Kevin Allix, Jacques Klein, Li Li, and Yves Le Traon. 2020\.  Learning to Catch Security Patches.  _arXiv preprint arXiv:2001.09148_ (2020).
  * Security ([n. d.]) Offensive Security. [n. d.].  Exploit Database.  Retrieved April, 2022 from https://www.exploit-db.com
  * SecurityTracker ([n. d.]) SecurityTracker. [n. d.].  SecurityTracker vulnerability database.  Retrieved April, 2022 from https://securitytracker.com
  * Semasaba et al. (2020) Abubakar Omari Abdallah Semasaba, Wei Zheng, Xiaoxue Wu, and Samuel Akwasi Agyemang. 2020.  Literature survey of deep learning-based vulnerability analysis on source code.  _IET Software_ (2020).
  * Services ([n. d.]) Internet Security Services. [n. d.].  Online database X-Force.  Retrieved April, 2022 from http://www.iss.net/xforce
  * Sharma et al. (2021) Ruchi Sharma, Ritu Sibal, and Sangeeta Sabharwal. 2021.  Software vulnerability prioritization using vulnerability description.  _International Journal of System Assurance Engineering and Management_ 12, 1 (2021), 58–64.
  * Shrestha et al. (2020) Prasha Shrestha, Arun Sathanur, Suraj Maharjan, Emily Saldanha, Dustin Arendt, and Svitlana Volkova. 2020.  Multiple social platforms reveal actionable signals for software vulnerability awareness: A study of GitHub, Twitter and Reddit.  _Plos one_ 15, 3 (2020), e0230250.
  * Shuai et al. (2013) Bo Shuai, Haifeng Li, Mengjun Li, Quan Zhang, and Chaojing Tang. 2013.  Automatic classification for vulnerability based on machine learning. In _2013 IEEE International Conference on Information and Automation (ICIA)_. IEEE, 312–318.
  * Singh and Chaturvedi (2020) Shashank Kumar Singh and Amrita Chaturvedi. 2020.  Applying deep learning for discovery and analysis of software vulnerabilities: A brief survey.  _Soft Computing: Theories and Applications_ (2020), 649–658.
  * Smyth (2017) Vincent Smyth. 2017\.  Software vulnerability management: how intelligence helps reduce the risk.  _Network Security_ 2017, 3 (2017), 10–12.
  * Spanos and Angelis (2018) Georgios Spanos and Lefteris Angelis. 2018.  A multi-target approach to estimate software vulnerability characteristics and severity scores.  _Journal of Systems and Software_ 146 (2018), 152–166.
  * Spanos et al. (2017) Georgios Spanos, Lefteris Angelis, and Dimitrios Toloudis. 2017\.  Assessment of vulnerability severity using text mining. In _Proceedings of the 21st Pan-Hellenic Conference on Informatics_. 1–6.
  * Spanos et al. (2013) Georgios Spanos, Angeliki Sioziou, and Lefteris Angelis. 2013\.  WIVSS: a new methodology for scoring information systems vulnerabilities. In _Proceedings of the 17th panhellenic conference on informatics_. 83–90.
  * Spring et al. (2021) Jonathan Spring, Eric Hatleback, Allen Householder, Art Manion, and Deana Shick. 2021\.  Time to change the CVSS?  _IEEE Security & Privacy_ 19, 2 (2021), 74–78.
  * Stanley and Miikkulainen (2002) Kenneth O Stanley and Risto Miikkulainen. 2002.  Evolving neural networks through augmenting topologies.  _Evolutionary computation_ 10, 2 (2002), 99–127.
  * Suciu et al. (2021) Octavian Suciu, Connor Nelson, Zhuoer Lyu, Tiffany Bao, and Tudor Dumitras. 2021.  Expected exploitability: Predicting the development of functional vulnerability exploits.  _arXiv preprint arXiv:2102.07869_ (2021).
  * Sun et al. (2021) Jiamou Sun, Zhenchang Xing, Hao Guo, Deheng Ye, Xiaohong Li, Xiwei Xu, and Liming Zhu. 2021.  Generating informative CVE description from ExploitDB posts by extractive summarization.  _arXiv preprint arXiv:2101.01431_ (2021).
  * Sun et al. (2018) Nan Sun, Jun Zhang, Paul Rimba, Shang Gao, Leo Yu Zhang, and Yang Xiang. 2018\.  Data-driven cybersecurity incident prediction: A survey.  _IEEE Communications Surveys & Tutorials_ 21, 2 (2018), 1744–1772.
  * Tavabi et al. (2018) Nazgol Tavabi, Palash Goyal, Mohammed Almukaynizi, Paulo Shakarian, and Kristina Lerman. 2018\.  Darkembed: Exploit prediction with neural language models. In _the AAAI Conference on Artificial Intelligence_ , Vol. 32.
  * Toloudis et al. (2016) Dimitrios Toloudis, Georgios Spanos, and Lefteris Angelis. 2016\.  Associating the Severity of Vulnerabilities with their Description. In _International Conference on Advanced Information Systems Engineering_. Springer, 231–242.
  * Tripathi et al. (2017) Shubham Tripathi, Gustavo Grieco, and Sanjay Rawat. 2017\.  Exniffer: Learning to prioritize crashes by assessing the exploitability from memory dump. In _2017 24th Asia-Pacific Software Engineering Conference (APSEC)_. IEEE, 239–248.
  * Van Engelen and Hoos (2020) Jesper E Van Engelen and Holger H Hoos. 2020.  A survey on semi-supervised learning.  _Machine Learning_ 109, 2 (2020), 373–440.
  * Vanamala et al. (2020) Mounika Vanamala, Xiaohong Yuan, and Kaushik Roy. 2020.  Topic modeling and classification of Common Vulnerabilities And Exposures database. In _2020 International Conference on Artificial Intelligence, Big Data, Computing and Data Communication Systems (icABCD)_. IEEE, 1–5.
  * Venter et al. (2008) Hein S Venter, Jan HP Eloff, and YL Li. 2008.  Standardising vulnerability categories.  _Computers & Security_ 27, 3-4 (2008), 71–83.
  * Walden (2020) James Walden. 2020\.  The impact of a major security event on an open source project: The case of OpenSSL. In _Proceedings of the 17th International Conference on Mining Software Repositories_. 409–419.
  * Wan et al. (2019) Zhiyuan Wan, Xin Xia, David Lo, and Gail C Murphy. 2019\.  How does machine learning change software development practices?  _IEEE Transactions on Software Engineering_ (2019).
  * Wang and Guo (2010) Ju An Wang and Minzhe Guo. 2010.  Vulnerability categorization using Bayesian networks. In _Proceedings of the sixth annual workshop on cyber security and information intelligence research_. 1–4.
  * Wang et al. (2019) Peichao Wang, Yun Zhou, Baodan Sun, and Weiming Zhang. 2019\.  Intelligent prediction of vulnerability severity level based on text mining and XGBboost. In _2019 Eleventh International Conference on Advanced Computational Intelligence (ICACI)_. IEEE, 72–77.
  * Wang et al. (2020) Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. 2020\.  Generalizing from a few examples: A survey on few-shot learning.  _ACM Computing Surveys (CSUR)_ 53, 3 (2020), 1–34.
  * Wåreus and Hell (2020) Emil Wåreus and Martin Hell. 2020.  Automated CPE labeling of CVE summaries with machine learning. In _International Conference on Detection of Intrusions and Malware, and Vulnerability Assessment_. Springer, 3–22.
  * Warnecke et al. (2020) Alexander Warnecke, Daniel Arp, Christian Wressnegger, and Konrad Rieck. 2020.  Evaluating explanation methods for deep learning in security. In _2020 IEEE European Symposium on Security and Privacy (EuroS &P)_. IEEE, 158–174.
  * Weerawardhana et al. (2014) Sachini Weerawardhana, Subhojeet Mukherjee, Indrajit Ray, and Adele Howe. 2014.  Automated extraction of vulnerability information for home computer security. In _International Symposium on Foundations and Practice of Security_. Springer, 356–366.
  * Wen et al. (2015) Tao Wen, Yuqing Zhang, Ying Dong, and Gang Yang. 2015\.  A novel automatic severity vulnerability assessment framework.  _Journal of Communications_ 10, 5 (2015), 320–329.
  * Williams et al. (2020) Mark A Williams, Roberto Camacho Barranco, Sheikh Motahar Naim, Sumi Dey, M Shahriar Hossain, and Monika Akbar. 2020.  A vulnerability analysis and prediction framework.  _Computers & Security_ 92 (2020), 101751.
  * Williams et al. (2018) Mark A Williams, Sumi Dey, Roberto Camacho Barranco, Sheikh Motahar Naim, M Shahriar Hossain, and Monika Akbar. 2018.  Analyzing evolving trends of vulnerabilities in national vulnerability database. In _2018 IEEE International Conference on Big Data (Big Data)_. IEEE, 3011–3020.
  * Wold et al. (1987) Svante Wold, Kim Esbensen, and Paul Geladi. 1987\.  Principal component analysis.  _Chemometrics and intelligent laboratory systems_ 2, 1-3 (1987), 37–52.
  * Wu et al. (2021) Xiaoxue Wu, Wei Zheng, Xin Xia, and David Lo. 2021\.  Data quality matters: A case study on data label correctness for security bug report prediction.  _IEEE Transactions on Software Engineering_ (2021).
  * Xiao et al. (2018) Chaowei Xiao, Armin Sarabi, Yang Liu, Bo Li, Mingyan Liu, and Tudor Dumitras. 2018\.  From patching delays to infection symptoms: Using risk profiles for an early discovery of vulnerabilities exploited in the wild. In _27th $\\{$USENIX$\\}$ Security Symposium ($\\{$USENIX$\\}$ Security 18)_. 903–918.
  * Xiao et al. (2019) Hongbo Xiao, Zhenchang Xing, Xiaohong Li, and Hao Guo. 2019\.  Embedding and predicting software security entity relationships: A knowledge graph based approach. In _International Conference on Neural Information Processing_. Springer, 50–63.
  * Yamamoto et al. (2015) Yasuhiro Yamamoto, Daisuke Miyamoto, and Masaya Nakayama. 2015\.  Text-mining approach for estimating vulnerability score. In _2015 4th International Workshop on Building Analysis Datasets and Gathering Experience Returns for Security (BADGERS)_. IEEE, 67–73.
  * Yan et al. (2017) Guanhua Yan, Junchen Lu, Zhan Shu, and Yunus Kucuk. 2017\.  Exploitmeter: Combining fuzzing with machine learning for automated evaluation of software exploitability. In _2017 IEEE Symposium on Privacy-Aware Computing (PAC)_. IEEE, 164–175.
  * Yin et al. (2020) Jiao Yin, MingJian Tang, Jinli Cao, and Hua Wang. 2020\.  Apply transfer learning to cybersecurity: Predicting exploitability of vulnerabilities by description.  _Knowledge-Based Systems_ 210 (2020), 106529.
  * Yitagesu et al. (2021) Sofonias Yitagesu, Xiaowang Zhang, Zhiyong Feng, Xiaohong Li, and Zhenchang Xing. 2021\.  Automatic part-of-speech tagging for security vulnerability descriptions. In _2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)_. IEEE, 29–40.
  * Younis and Malaiya (2014) Awad A Younis and Yashwant K Malaiya. 2014.  Using software structure to predict vulnerability exploitation potential. In _2014 IEEE Eighth International Conference on Software Security and Reliability-Companion_. IEEE, 13–18.
  * Zeng et al. (2020) Peng Zeng, Guanjun Lin, Lei Pan, Yonghang Tai, and Jun Zhang. 2020.  Software vulnerability analysis and discovery using deep learning techniques: A survey.  _IEEE Access_ (2020).
  * Zhang et al. (2013) Hongyu Zhang, Liang Gong, and Steve Versteeg. 2013.  Predicting bug-fixing time: an empirical study of commercial software projects. In _2013 35th International Conference on Software Engineering (ICSE)_. IEEE, 1042–1051.
  * Zhang and Thing (2018) Li Zhang and Vrizlynn LL Thing. 2018.  Assisting vulnerability detection by prioritizing crashes with incremental learning. In _TENCON 2018-2018 IEEE Region 10 Conference_. IEEE, 2080–2085.
  * Zhang et al. (2020b) Xiong Zhang, Haoran Xie, Hao Yang, Hongkai Shao, and Minghao Zhu. 2020b.  A general framework to understand vulnerabilities in information systems.  _IEEE Access_ 8 (2020), 121858–121873.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  _arXiv preprint arXiv:1509.01626_ (2015).
  * Zhang et al. (2020a) Yu Zhang, Peter Tiňo, Aleš Leonardis, and Ke Tang. 2020a.  A survey on neural network interpretability.  _arXiv preprint arXiv:2012.14261_ (2020).
  * Zhang and Yang (2021) Yu Zhang and Qiang Yang. 2021.  A survey on multi-task learning.  _IEEE Transactions on Knowledge and Data Engineering_ (2021).
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In _Proceedings of the IEEE international conference on computer vision_. 19–27.
  * Zimmermann et al. (2010) Thomas Zimmermann, Rahul Premraj, Nicolas Bettenburg, Sascha Just, Adrian Schroter, and Cathrin Weiss. 2010.  What makes a good bug report?  _IEEE Transactions on Software Engineering_ 36, 5 (2010), 618–643.
  * Zou et al. (2019) Deqing Zou, Sujuan Wang, Shouhuai Xu, Zhen Li, and Hai Jin. 2019.  $\mu$VulDeePecker: A deep learning-based system for multiclass vulnerability detection.  _IEEE Transactions on Dependable and Secure Computing_ (2019).
  * Zou et al. (2021) Deqing Zou, Yawei Zhu, Shouhuai Xu, Zhen Li, Hai Jin, and Hengkai Ye. 2021\.  Interpreting deep learning-based vulnerability detector predictions based on heuristic searching.  _ACM Transactions on Software Engineering and Methodology (TOSEM)_ 30, 2 (2021), 1–31.
  * Özkan ([n. d.]) Serkan Özkan. [n. d.].  CVE Details.  Retrieved April, 2022 from https://www.cvedetails.com


# Universal notion of classicality based on ontological framework

Shubhayan Sarkar  Center for Theoretical Physics, Polish Academy of Sciences,
Aleja Lotnikow 32/46, 02-668 Warsaw, Poland<EMAIL_ADDRESS>
###### Abstract

Existence of physical reality in the classical world is a well-established
fact from day-to-day observations. However within quantum theory, it is not
straightforward to reach such a conclusion. A framework to analyse how
observations can be described using some physical states of reality in a
theory independent way was recently developed, known as ontological framework.
Different principles when imposed on the ontological level give rise to
different observations in physical experiments. Using the ontological
framework, we formulate a novel notion of classicality termed "universal
classicality" which is based upon the physical principles that in classical
theories pure states are physical states of reality and every projective
measurement just observes the state of the system. We construct a
communication task in which the success probability is bounded from above for
ontological models satisfying the notion of universal classicality. Contrary
to previous notions of classicality which either required systems of dimension
strictly greater than two or atleast three preparations, a violation of
"universal classicality" can be observed using just a pair of qubits and a
pair of incompatible measurements. We further show that violations of
previously known notions of classicality such as preparation non-contexuality
and Bell’s local causality is a violation of universal classicality.

##  I Introduction.

The operational picture of physical theories aim at understanding the physical
world in terms of statistics generated by experiments performed on physical
systems. The novelty of such picture relies on the idea that observable facts
are enough to predict outcomes of some physical experiments without much
emphasis on the underlying physical states of reality, commonly referred to as
ontological states. However, there are limitations of the operational picture
not just towards predictive power but also that such a picture lacks any
deeper understanding of nature. As a consequence from a realist perspective,
any operational prediction needs to be supported by some ontological states of
reality even if they might be unobservable or hidden. Interestingly,
observations from classical world exhibit that observable facts corresponds to
physical realities. However, in any general theory it might also be possible
that observable facts are just a reflection of some underlying physical
reality even when the physical reality can not be directly observed. As a
consequence in classical physics, any laws of physics applicable on the
observable level is valid at the ontological level too. Building on this
viewpoint, Bell in 1964 Bell  could prove that quantum theory is inherently
different from classical physics.

One of the interesting avenues in foundations of quantum theory is to
understand how quantum theory diverges from classical theories. To facilitate
such understanding, different notions of classicality have been suggested like
Bell’s local causality Bell , Kochen-Specker non-contexuality KS ; Cabello ,
and preparation non-contexuality PNC  to name a few. All such notions of
classicality differ based on the assumptions imposed on the ontological level.
The strength of such notions lies in the amount of quantumness which can be
captured by these notions. If quantum theory violates any notion of
classicality, then such a notion of classicality can also be understood as no-
go theorem for quantum theory, i.e., quantum theory can not be compatible with
all the assumptions which define the notion of classicality. For example,
Bell’s theorem shows that any ontological description of quantum theory must
violate the principle of local causality. Apart from their foundational
importance all such notions of classicality have led to various tasks in
cryptography, communication and computation which show quantum advantage over
classical strategies. For example, the security of the well-known
cryptographic scheme $E91$ protocol is based on Bell’s theorem E91 . Infact,
every device independent scheme is based upon violations of some notion of
classicality di1 ; Yang ; Bamps ; sarkar ; Ivan .

In this work, we construct a novel notion of classicality which is named as
"universal classicality" based on the idea that in classical theories, any
pure state is a physical state of reality and any projective measurement just
observes the state of the system. Then, we construct an operational task which
can be violated using only two pure qubit states and two incompatible
measurements. We further show that the notion of universal classicality
implies all the other known notions of classicality like preparation non-
contexuality, Kochen-Specker non-contexuality and Bell’s local causality.

##  II Preliminaries

Before moving on to results, we would like to introduce the notations and
relevant concepts required for this work.

###  II.1 Operational picture

Any experiment performed on some physical system can be understood as prepare,
transform and measure experiment. In each run of the experiment, a preparation
(the prepared system) undergoes a transformation (some dynamical process)
after which the system is measured. The experiment is repeated for large
number of times to gather enough statistics. In any such experiment, it is
always assumed that each run of the experiment is statistically independent of
other runs and the preparation, transformation and measurement remain
consistent throughout the runs of the experiment.

Any preparation in the operational picture is denoted by $P$, transformation
by $T$ and measurements are denoted by $M$ with outcomes denoted by $k$. The
statistics are denoted by $p(k|M,P,T)$ which specifies the probability of
obtaining the outcome $k$ of some measurement $M$ when the preparation $P$
undergoes a transformation $T$. For different physical theories, the
probabilities are obtained in different ways. For example, in quantum theory
the preparations are given by density matrices $\rho$ belonging to some
$d-$dimensional Hilbert space $\mathcal{H}_{d}$, the transformations are
unitary matrices $U$ and the measurements are given by positive operator
valued measure (POVM) $\\{M_{k}\\}$. The probabilities are obtained using the
Born rule, $p(k|M,P,T)=\mathrm{Tr}(U\rho U^{\dagger}M_{k})$. For the rest of
this manuscript, we would not consider systems which evolve. As a consequence,
the scenario can be simplified to only prepare and measure experiments where
the probabilities are denoted by $p(k|M,P)$.

###  II.2 Ontological framework

The idea of ontic states or hidden variables was put forward by Bell in his
seminal work in 1964 Bell . The idea was put in a more rigorous way by
Spekkens and Harrigan Spekkens2  and subsequently by Leifer Leifer . The
ontological framework, provides a basis for realist extensions of any physical
theory and the ontic states can be understood as some "real physical states"
which in general might not be observed directly. The ontic state is denoted by
$\lambda$ which belongs to the ontological state space denoted by $\Lambda$.
In general, any preparation procedure $P$ prepares a distribution of such
ontic states denoted by $\mu(\lambda|P)$ which can be understood as
probability of preparing the ontic state $\lambda$ using a preparation
procedure $P$. It is required that $\Lambda$ is a measurable space along with
the condition that $\int_{\Lambda}\mu(\lambda|P)d\lambda=1$ for all
preparations $P$ which signifies that every preparation must always prepare
some state $\lambda\in\Lambda$ along with the constraint $\mu(\lambda|P)\geq
0$. The set $\Omega_{P}\in\Lambda$ represents the set of $\lambda^{\prime}s$
for some preparation $P$ such that the probability density $\mu(\lambda|P)\neq
0$. Thus, we also have $\int_{\Omega_{P}}\mu(\lambda|P)d\lambda=1$.

Any measurement in the ontological framework just measures the ontic state.
The measurements are represented as response functions denoted by,
$\xi(k|M,\lambda)$ which specifies the probability of obtaining the outcome
$k$ of some measurement $M$ given the ontic state $\lambda$. It is also
required that the response function is measurable along with the condition
$\xi(k|M,\lambda)\geq 0$ and $\sum_{k}\xi(k|M,\lambda)=1$ for all
$\lambda\in\Lambda$.

Using the above prescription, any probabilities obtained in the prepare and
measure experiments can be represented as,

$\displaystyle p(k|M,P)=\int_{\Lambda}\xi(k|M,\lambda)\mu(\lambda|P)d\lambda$
(1)

Note that we assume that the preparation and measurement are related via the
ontic state $\lambda$. This is known as $\lambda-$mediation.

###  II.3 Ontic-Epistemic distinction

One of the most intriguing questions in foundations of quantum theory is
whether quantum state is a physical state of reality or epistemological
knowledge of some underlying reality. To answer this question, the first
requirement is to have a realist perspective and an ontological framework
which can describe quantum theory. If every pure quantum state corresponds to
some unique set of ontological states, then quantum state is ontic or
corresponds to real physical state of reality. On the other hand, if two pure
quantum state share some ontological states, quantum state is epistemic or
just a representation of underlying physical states of reality. It was proven
that for infinite dimensional systems, quantum states are necessarily ontic
Renner . With some additional assumptions, it could be proven that quantum
state is ontic PBR ; Hardy  even for finite dimensional systems. However,
without such additional assumptions the question still remains open whether
quantum states are ontic or epistemic for finite dimensional systems. Further,
there has been a large number of ontological models, in some of which the
quantum states are ontic like Bohmian mechanics Bohm1 ; Bohm2 ,
Beltrametti–Bugajski model BG  and Bell model BM  to name a few. Some other
ontological models of quantum theory in which the quantum state not a physical
reality but just an epistemic knowledge are Kochen-Specker model KS , Spekkens
toy theory Spekkens3 , LJBR model LJBR  and ABCL models ABCL  to name a few.

###  II.4 Pure states and projective measurements

Most of the ontological models aim to characterise pure states and projective
measurements. The reason being every mixed state is just a classical mixture
of pure states and within classical and quantum theory can be realised as a
pure state in some higher dimensional system. Similarly, non-projective
measurements within classical and quantum theory can be realised as a
projective measurement acting on some higher dimensional system NielsenChuang
.

Any preparation $\overline{P}$ which generate pure states can not be described
as a convex mixture of two other preparation $P^{\prime}$ and
$P^{\prime\prime}$. Mathematically, if

$\displaystyle\overline{P}=zP^{\prime}+(1-z)P^{\prime\prime}$ (2)

then, $z=1$ and $P^{\prime}=\overline{P}$. Similarly, any projective
measurement $\overline{M}$ can not be realised as some convex combination of
two different measurements $M^{\prime}$ and $M^{\prime\prime}$.
Mathematically, if

$\displaystyle\overline{M}=zM^{\prime}+(1-z)M^{\prime\prime}$ (3)

then, $z=1$ and $M^{\prime}=\overline{M}$. Now, we proceed towards the main
result of this manuscript.

##  III Results

We present here the novel measure of classicality termed as "universal
classicality" which is based upon two physical assumptions well-supported by
experiments till date. In this section, first we introduce the assumptions
defining "universal classicality" which are imposed on the ontological state
space. Next, we construct a simple operational task and show that the success
probability of the task using classical strategies is bounded from above. We
find states and measurements within quantum theory which gives a higher
success probability than achievable using classical strategies. We begin by
stating the assumptions.

###  III.1 Assumptions

The first assumption defining "universal classicality" is given as,

######  Definition 1 (No overlap).

For two preparations, $\overline{P}_{1}$ and $\overline{P}_{2}$ which generate
pure states, the corresponding ontic distributions
$\mu(\lambda|\overline{P}_{1})$ and $\mu(\lambda|\overline{P}_{2})$ do not
overlap. Mathematically,

$\displaystyle\frac{1}{2}\int_{\Lambda}\left|\mu(\lambda|\overline{P}_{1})-\mu(\lambda|\overline{P}_{2})\right|d\lambda=1$
(4)

This assumption stems from the fact in classical theories, the knowledge about
the preparation producing a pure state imposes that it is a physical state of
reality and as discussed before physical states of reality do not overlap. For
simplicity, let us consider classical Hamiltonian dynamics. In classical
Hamiltonian dynamics, any state that lies within the phase space is described
by the position and momentum of the system $(x,p)$. Now, position and momentum
are physically observable quantities and can be attributed as physical states
of reality or the ontic states. Going back to the ontological framework,
consider a preparation which prepares a state with some fixed energy $E$ which
is a distribution over the phase space states. It is known that energy is a
physically measurable quantity. As a consequence, if we are able to know the
position and momentum of the system, we can predict the energy of the system.
However, if the same phase space state belonged to two different energy
states, then it would not have been a physically measurable quantity. Using a
similar argument, it can concluded that for any preparation which is a
distribution over ontic states must have some unique set of ontic states to be
a physical state of reality upto classical mixture. Note that classical
mixture represent lack of knowledge or ignorance which can be regarded as
epistemic.

The second assumption involves measurements,

######  Definition 2 (Strong duality).

For all pure state preparations $\overline{P}_{i}$, such that
$p(k|\overline{M},\overline{P}_{i})=1$ for some outcome $k$ of projective
measurement $\overline{M}$ imposes that the corresponding response function
$\xi(k|\overline{M},\lambda)=1$ for every
$\lambda\in\displaystyle\bigcap\limits_{i}\Omega_{P_{i}}$ and $0$ for all
other $\lambda\notin\displaystyle\bigcap\limits_{i}\Omega_{P_{i}}$.

The above assumptions stems from the fact that any projective measurement in
the classical world is just observing the state of the system. Likewise,
considering classical Hamiltonian dynamics, projective measurements on the
phase space just reads the state of the system $(x,p)$. In general, the
measurements might not exactly suggest the state of the system but specify
some region of phase space where the system resides. Such scenarios can be
understood as measurements whose different outcomes are classically coarse
grained to a single outcome. Any non-projective measurement within classical
theories are just probabilistic mixture of projective measurements. As a
consequence within classical theories, without loss of generality it can
always be assumed that measurements are projective on some larger dimension
system. Now, we proceed towards constructing the operational task whose
success probability is upper bounded for any classical strategy.

###  III.2 Operational Task

The task simply consists of two players Alice and Bob who are not allowed to
communicate with each other during the run of the experiment. Alice has access
to a preparation box which consists of two inputs $y=\\{0,1\\}$ which generate
two different preparations $P_{y}$. Bob has access to a measurement box with
two inputs $x=\\{0,1\\}$ specifying two different measurements $M_{x}$ with
any number of outputs strictly greater than 1, labelled as
$a=\\{0,1,2,\ldots\\}$. Now, Alice and Bob choose their inputs independent of
each other. As a consequence, a system is prepared by the preparation box with
Alice which is then sent to Bob who measures the system as specified by the
measurement box. The task is repeated for a large number of times to collect
statistics.

Now, we move onto describing the success probability $P_{S}$ which can be
understood as a quantifier of the probability of winning in the above
described task.

$\displaystyle P_{S}=\sum_{a,x,y}c_{a,x,y}p(a|x,y)$ (5)

where $c_{a,x,y}=\frac{1}{4}$ if $a=x.y$ and $0$ otherwise.

Recalling from Bell’s theorem, to observe a quantum advantage it is required
that the correlations shared between Alice and Bob are no-signalling Bellnon .
Similarly, to observe an advantage using quantum theory and constrain
classical theories in the above described operational task (5), we are
required to impose the condition that the measurements are of rank-one, which
can be operationally defined as

Figure 1: The operational task to observe quantum advantage. The preparation
box consists of two inputs labelled by $y=0,1$ which generate two different
preparations. The measurement box consists of two inputs labelled by $x=0,1$
which corresponds to two rank-one measurements of arbitrary number of outcomes
labelled by $a=0,1,2,\ldots$

######  Definition 3 (Rank-one measurements).

For any measurement $M$, if two preparations $P$ and $P^{\prime}$ give the
same outcome with certainty, then $P$ and $P^{\prime}$ can not be
distinguished.

This means for any measurement $M$ and preparation $P$ and $P^{\prime}$, if

$\displaystyle p(k|M,P)=p(k|M,P^{\prime})=1\quad\text{for some $k$}$ (6)

then $p(k|M,P)=p(k|M,P^{\prime})$ for all outcomes $k$ of all measurements
$M$. Note that, for projective measurements if $p(k|M,P)=1$, then $P$ is a
pure state preparation.

Constraining the strategies using the condition of measurements being rank-one
def-3, we establish the following theorem on the success probability as
defined in (5). Note that along with the above assumptions defining "universal
classicality", in any operational task the implicit assumption of free will is
required. The assumption of free will ensures that Alice and Bob can freely
choose respective inputs $(y,x)$ independent of each other’s choices.

######  Theorem.

For any theory which satisfies the assumptions of no overlap def-1 and strong
duality def-2, the success probability (5) is bounded from above as
$P_{S}\leq\frac{3}{4}$.

###### Proof.

Expanding the success probability (5), we have

$\displaystyle
P_{S}=\frac{1}{4}\left(p(0|0,0)+p(0|1,0)+p(0|0,1)+p(1|1,1)\right)$ (7)

Since, the measurements consists of more than one outcome, we have
$\sum_{k}p(k|1,1)=1$, which imposes that $p(0|1,1)+p(1|1,1)\leq 1$. Using
this, we have

$\displaystyle
P_{S}\leq\frac{1}{4}\left(p(0|0,0)+p(0|1,0)+p(0|0,1)-p(0|1,1)\right)+\frac{1}{4}$

Since there is no restriction on dimension, as discussed above any measurement
can be realized as a projective measurement on some larger system. Using this
fact and the ontological framework, we expand the above expression as

$\displaystyle p(0|0,0)+p(0|1,0)+p(0|0,1)-p(0|1,1)$
$\displaystyle=\int_{\Lambda}\xi(0|\overline{M}_{0},\lambda)\left(\mu(\lambda|P_{0})+\mu(\lambda|P_{1})\right)d\lambda$
$\displaystyle+\int_{\Lambda}\xi(0|\overline{M}_{1},\lambda)\left(\mu(\lambda|P_{0})-\mu(\lambda|P_{1})\right)d\lambda$
(8)

Since the measurements are projective, the assumption of strong duality def-2
imposes that for all rank-one projective measurements, the response function
$\xi(0|\overline{M}_{0},\lambda)=1$ for $\lambda\in\Omega_{\overline{P_{0}}}$
and $0$ for $\lambda\notin\Omega_{\overline{P_{0}}}$ and
$\xi(0|\overline{M}_{1},\lambda)=1$ for $\lambda\in\Omega_{\overline{P_{1}}}$
and $0$ for $\lambda\notin\Omega_{\overline{P_{1}}}$ for some preparations
$\overline{P}_{0}$ and $\overline{P}_{1}$ generating pure states. This imposes
that

$\displaystyle p(0|0,0)+p(0|1,0)+p(0|0,1)-p(0|1,1)$
$\displaystyle=\int_{\Omega_{\overline{P_{0}}}}(\mu(\lambda|P_{0})+\mu(\lambda|P_{1}))d\lambda$
$\displaystyle+\int_{\Omega_{\overline{P_{1}}}}(\mu(\lambda|P_{0})-\mu(\lambda|P_{1}))d\lambda$
(9)

Now, the assumption of no-overlap def-1 imposes that either
${\Omega_{\overline{P_{0}}}}$ and ${\Omega_{\overline{P_{1}}}}$ are disjoint
for $\overline{P}_{0}$ and $\overline{P}_{1}$ being distinct or
${\Omega_{\overline{P_{0}}}}$ and ${\Omega_{\overline{P_{1}}}}$ are equivalent
for $\overline{P}_{0}$ and $\overline{P}_{1}$ being same. Let’s first consider
the case when $\overline{P}_{0}$ and $\overline{P}_{1}$ are distinct.

$\displaystyle p(0|0,0)+p(0|1,0)+p(0|0,1)-p(0|1,1)$
$\displaystyle=p_{0,0}+p_{0,1}+p_{1,0}-p_{1,1}\leq 2$ (10)

where $p_{i,j}=\int_{\Omega_{\overline{P_{i}}}}\mu(\lambda|P_{j})d\lambda$ and
we used the fact that the ontological states generated from pure state
preparations do not overlap, which imposes

$\displaystyle p_{0,0}+p_{1,0}$ $\displaystyle\leq$ $\displaystyle
1\quad\text{and},$ $\displaystyle p_{0,1}+p_{1,1}$ $\displaystyle\leq$
$\displaystyle 1$ (11)

For the case when $\overline{P}_{0}$ and $\overline{P}_{1}$ being same, we
have

$\displaystyle p(0|0,0)+p(0|1,0)+p(0|0,1)-p(0|1,1)$
$\displaystyle=2p_{0,0}\leq 2$ (12)

Thus, from (7), (III.2) and (III.2) we can conclude that

$\displaystyle P_{S}\leq\frac{3}{4}.$ (13)

This completes the proof. ∎

Since any classical theory satisfies the assumptions of no-overlap def-1 and
strong duality def-2, the above theorem shows that using classical strategies,
the maximum attainable value of the success probability is bounded from above
by the value $P_{s}=\frac{3}{4}$. For a better understanding of the above
bound, let’s consider a possible classical strategy. Let’s say Alice sends her
inputs $y=\\{0,1\\}$ to Bob. This can be realised by just sending a classical
bit in which case, the constraint 3 imposes that $p(0|0,0)+p(0|0,1)\leq 1$
which in turn imposes from (5) that $P_{s}\leq\frac{3}{4}$. Now, we show that
there exist strategies in quantum theory which can attain a success
probability $P_{s}=\frac{1}{2}+\frac{1}{2\sqrt{2}}$ which is strictly greater
than $\frac{3}{4}$.

###  III.3 Quantum advantage

To witness quantum advantage, the preparations are chosen by Alice as, $P_{0}$
generates the state
$|\psi_{0}\rangle=|+\rangle=\frac{1}{\sqrt{2}}\left(|0\rangle+|1\rangle\right)$
and $P_{1}$ generates the state $|\psi_{1}\rangle=|0\rangle$. The measurements
are chosen by Bob represented as observables are given by,
$\overline{M}_{0}=\frac{1}{\sqrt{2}}\left(\sigma_{z}+\sigma_{x}\right)$ and
$\overline{M}_{1}=\frac{1}{\sqrt{2}}\left(\sigma_{z}-\sigma_{x}\right)$ where
$\sigma_{z}$ and $\sigma_{x}$ corresponds to Pauli-$z$ and Pauli-$x$ matrices
respectively. Calculating the necessary probabilities to evaluate the success
probability (5) using the above chosen quantum states and measurements, we
have

$\displaystyle p(0|0,0)=p(0|1,0)=p(0|0,1)=p(1|1,1)$
$\displaystyle=\cos^{2}\left(\frac{\pi}{8}\right)$ (14)

Thus, the success probability is given by

$\displaystyle
P_{s}=\cos^{2}\left(\frac{\pi}{8}\right)=\frac{1}{2}+\frac{1}{2\sqrt{2}}.$
(15)

The existence of such strategies in quantum theory shows that in the above
presented operational task, quantum theory outperforms classical strategies.
Also, such a violation shows that the assumptions of no-overlap def-1 and
strong duality def-2 are not compatible with quantum theory. Thus, it is
impossible to construct any such ontological model of quantum theory which
have the features of of no-overlap def-1 and strong duality def-2 together.
However, there can exist some ontological model of quantum theory which
violate either of the two assumptions. Some notable ontological models of
quantum theory which violate the assumption of strong duality def-2 but
satisfy the assumption of no-overlap def-1 are Beltrametti–Bugajski model BG
and Bell model BM . Another class of ontological models like the Kochen-
Specker model KS  and Spekkens toy theory Spekkens3  violate the assumption of
no-overlap def-1 but satisfy the assumption of strong duality def-2. In the
next section, we show that how the notion of "universal classicality" implies
all the other known notions of classicality.

##  IV Relations to other notions of classicality

Comparisons between different notions of classicality is based on the amount
of non-classicality which can be captured by such notions. Any newly proposed
notion of classicality can be considered significant if it can capture higher
amount of non-classicality than the previously proposed notions. To show this,
one of the ways is to prove that a certain notion of classicality implies the
previously known notions of classicality.

To show that "universal classicality" imposes other known notions of
classicality, we need an additional auxiliary assumption concerning
preparations which generate mixed states.

######  Definition 4 (Convexity).

For any preparation $P$ that can be realised as a classical mixture of some
preparations $P_{i}^{\prime}s$ such that $P=\sum_{i}z_{i}P_{i}$, then the
corresponding ontological distribution is given by
$\mu(\lambda|P)=\sum_{i}z_{i}\mu(\lambda|P_{i})$ where $z_{i}\geq 0$ and
$\sum_{i}z_{i}=1$.

The above definition holds true for any preparations $P_{i}$. However, for
this work we only consider $P_{i}^{\prime}s$ which are pure states as any
preparation which generate mixed state, can be understood as some classical
mixture of pure states. As pure states represent physical states of reality in
classical theories, any mixture of such pure states would be equivalently
reinstated at the ontological level.

First, we prove that universal classicality and convexity implies preparation
non-contexuality PNC . For this, let us first recall the definition of
preparation non-contexuality. Preparation non-contexuality imposes that if two
preparations are indistinguishable, then their respective ontological
distributions are same. Mathematically, if

$\displaystyle p(k|M,P_{1})=p(k|M,P_{2})\quad\forall k,M$ (16)

then, $\mu(\lambda|P_{1})=\mu(\lambda|P_{2})$ for all $\lambda\in\Lambda$.

Now, we would show that the assumptions of no overlap def-1 and strong duality
def-2 along with the assumption of convexity def-4 when imposed on the
ontological state space imposes preparation non-contexuality. For this, let us
note that as discussed before any measurement can be realized as a projective
measurement in some higher dimensional space. Expressing (16) in the
ontological framework,

$\displaystyle\int_{\Lambda}\xi(k|\overline{M},\lambda)\mu(\lambda|P_{1})d\lambda=\int_{\Lambda}\xi(k|\overline{M},\lambda)\mu(\lambda|P_{2})d\lambda$
(17)

for all outcomes $k$ of all projective measurements $\overline{M}$. To show
$\mu(\lambda|P_{1})=\mu(\lambda|P_{2})$, it is enough to consider rank-one
measurements def-3. As discussed before, the assumption of strong duality
def-2 imposes that $\xi(k|\overline{M},\lambda)=1$ for
$\forall\lambda\in\Omega_{\overline{P}}$ and $0$ for all other
$\lambda\notin\Omega_{\overline{P}}$ for some pure state preparation
$\overline{P}$,

$\displaystyle\int_{\Omega_{\overline{P}}}\mu(\lambda|P_{1})d\lambda=\int_{\Omega_{\overline{P}}}\mu(\lambda|P_{2})d\lambda$
(18)

Now, assuming convexity def-4 we have

$\displaystyle\sum_{i}z_{i,1}\int_{\Omega_{\overline{P}}}\mu(\lambda|\overline{P}_{i,1})d\lambda=\sum_{i}z_{i,2}\int_{\Omega_{\overline{P}}}\mu(\lambda|\overline{P}_{i,2})d\lambda$
(19)

Assuming no overlap def-1, it can be concluded that the above equality (19)
holds iff $\overline{P}_{i,1}=\overline{P}_{i,2}=\overline{P}$ and
$z_{i,1}=z_{i,2}$ for some $i$. Using a similar argument for all outcomes $k$
of all rank-one projective measurements $\overline{M}$ we can conclude that
$\mu(\lambda|P_{1})=\mu(\lambda|P_{2})$ for all $\lambda\in\Lambda$.

Now, we show that universal classicality and convexity also gives rise to a
recently proposed notion of classicality termed bounded ontological
distinctness BOD . For this, let us first recall the definition of bounded
ontological distinctness $(BOD_{P})$. Bounded ontological distinctness imposes
that for distinguishability of preparations at the ontological level is same
as distinguishability at the operational level. Mathematically, for
n-preparations $P_{x}$ when $x\in{0,1,\ldots,n-1}$ if

$\displaystyle\max_{M}\frac{1}{n}\sum_{x}p(k=x|M,P_{x})=p$ (20)

then,

$\displaystyle\frac{1}{n}\int_{\Lambda}\max_{x}\left\\{\mu(\lambda|P_{x})\right\\}d\lambda=p$
(21)

Now, we would show that the assumptions of no overlap def-1 and strong duality
def-2 along with the assumption of convexity def-4 when imposed on the
ontological state space imposes bounded ontological distinctness. Expressing
(20) in the ontological framework,

$\displaystyle\max_{\overline{M}}\frac{1}{n}\sum_{x}\int_{\Lambda}\xi(k=x|\overline{M},\lambda)\mu(\lambda|P_{x})=p$
(22)

Assuming convexity def-4, any ontological distribution can be written as
$\mu(\lambda|P_{x})=\sum_{i}z_{i,x}\mu(\lambda|\overline{P}_{i})$, we have

$\displaystyle\max_{\overline{M}}\frac{1}{n}\sum_{x}\sum_{i}z_{i,x}\int_{\Lambda}\xi(k=x|\overline{M},\lambda)\mu(\lambda|\overline{P}_{i})=p$
(23)

Assuming strong duality def-2 and no overlap def-1 any measurement
$\overline{M}$ that maximises the distinguishing probability must have the
response functions $\xi(k=x|\overline{M},\lambda)$ such that they pick up the
probabilities $z_{i,x}$ which is maximised for each $x$. Thus, we have

$\displaystyle\frac{1}{n}\sum_{i}\max_{x}z_{i,x}=p$ (24)

Now, evaluating (21) assuming convexity def-4 we have,

$\displaystyle\frac{1}{n}\int_{\Lambda}\max_{x}\left\\{\sum_{i}z_{i,x}\mu(\lambda|\overline{P}_{i})\right\\}d\lambda$
$\displaystyle=\frac{1}{n}\sum_{i}\max_{x}z_{i,x}=p$ (25)

where we arrived at the above expression by assuming that pure states do not
overlap def-1.

Using some well established results among different notions of classicality we
have the following hierarchy,

Universal classicality $\displaystyle\implies$ $\displaystyle BOD_{P}$ (26)
$\displaystyle\implies$ Prep non-contexuality
$\displaystyle\begin{subarray}{c}QT\\\ \implies\end{subarray}$ K-S non-
contexuality $\displaystyle\begin{subarray}{c}NS\\\ \implies\end{subarray}$
Bell local-causality

Thus, any violation of Bell’s local causality, Kochen-Specker non-
contexuality, preparation non-contexulaity or bounded ontological distinctness
imposes violation of universal classicality. For a note, the above conclusion
could be reached assuming that ontological models do not violate convexity
def-4 which is based on the fact that there is no successful ontological model
of quantum theory which violates convexity def-4. However, for any general
theory such ontological models can always exist which do not assume convexity
def-4. However, any violation of the above notions using only preparations
which generate pure states would directly impose the violation of universal
classicality.

##  V Conclusions

In the presented work, from a realist perspective we first constructed a novel
notion of classicality termed "universal classicality" based on just two
assumptions imposed on the ontological state space. We justified both the
assumptions based on physical arguments which are satisfied for any known
physical theories of the classical world. Then, we constructed an operational
task which is bounded for any theories which satisfy the assumptions of
"universal classicality". Kochen-Specker non-contexuality and Bell’s local
causality requires systems of atleast dimension three and four respectively to
show a quantum advantage. Bounded ontological distinctness and preparation
non-contexuality can be violated using only qubits, however they require
atleast three and four qubit preparations respectively to show a quantum
advantage. Contrary to this, "universal classicality" can be violated using
just a pair of qubits which makes the above presented operational task much
more applicable in practical scenarios. Then, we showed that the notion of
"universal classicality" implies all the other known notions of classicality
like Bell’s local causality.

Several follow up questions arises from this work. From quantum foundations
perspective, universal classicality might give better lower bounds to the
overlap of the ontological states of quantum theory than previously known
bounds QO1 ; QO2 . Also, the above theorem serves as a no-go theorem for
quantum theory. As a consequence the notion presented in this work might help
construction of newer ontological models. Further, the notion of universal
classicality would give a better understanding of the set of quantum
correlations. As was shown above, the violation of known notions of
classicality would also imply violation of universal classicality. As a
consequence, the set of quantum correlations is enlarged and this might give
interesting results towards boundedness of quantum correlations Cola . From
quantum information perspective, the above presented operational task might
give rise to cryptographic and communication tasks which need less resource to
execute as entanglement is not needed to show quantum advantage. Also, some
device independent schemes might be possible to construct employing the idea
of the presented operational task.

Acknowledgements

We would like to thank Remigiusz Augusiak and Debashish Saha for fruitful
discussions. This work is supported by Foundation for Polish Science through
the First Team project (No First TEAM/2017-4/31).

## References

  * (1) J. S. Bell, Physics (Long Island City, N.Y.) 1, 195 (1964).
  * (2) S. Kochen, and E. P. Specker, Journal of Mathematics and Mechanics 17, 1 (1967).
  * (3) A. Cabello, Phys. Rev. Lett. 101, 210401 (2008)
  * (4) R. W. Spekkens, Phys. Rev. A 71, 052108 (2005)
  * (5) A. K. Ekert  Phys. Rev. Lett. 67, 661 (1991)
  * (6) D. Mayers, and A. Yao, Proc. 39th Ann. Symp. on Foundations of Computer Science (FOCS), 503 (1998).
  * (7) T. H. Yang and M. Navascués, Phys. Rev. A 87, 050102(R) (2013).
  * (8) C. Bamps, and S. Pironio, Phys. Rev. A 91, 052111 (2015).
  * (9) S. Sarkar, D. Saha, J. Kaniewski, R. Augusiak, Self-testing quantum systems of arbitrary local dimension with minimal number of measurements, arXiv:1909.12722
  * (10) I. Šupić, J. Bowles, Quantum 4, 335 (2020).
  * (11) N. Harrigan, R. W. Spekkens,  Foundations of Physics 40, 125 (2010)
  * (12) M. S. Leifer, Quanta 3, 1 (2014)
  * (13) R. Colbeck and R. Renner, New J. Phys. 19 013016 (2017)
  * (14) M. F. Pusey, J. Barrett and T. Rudolph, Nature Physics 8, 475 (2012)
  * (15) L. Hardy, International Journal of Modern Physics 27, 1345012 (2013)
  * (16) L. De Broglie, J. Phys. Radium 8, 225-241 (1927).
  * (17) Bohm, Phys. Rev. 85, 166 (1952).
  * (18) E. G. Beltrametti and S. Bugajski, J. Phys. A: Math. Gen. 28 3329 (1995)
  * (19) J. S. Bell, Rev. Mod. Phys. 38, 447 (1966).
  * (20) R. W. Spekkens, Phys. Rev. A 75, 032110 (2007)
  * (21) P. G. Lewis, D. Jennings, J. Barrett, and T. Rudolph., Phys. Rev. Lett. 109, 150404 (2012)
  * (22) S. Aaronson, A. Bouland, L. Chua, and G. Lowther, Physical Review A 88, 032111 (2013)
  * (23) M. Nielsen and I. Chuang, Quantum Computation and Quantum Information, Cambridge University Press (Cambridge, 2000).
  * (24) N. Brunner, D. Cavalcanti, S. Pironio, V. Scarani and S. Wehner, Rev. Mod. Phys. 86, 419 (2014)
  * (25) A. Chaturvedi and D. Saha, Quantum 4, 345 (2020)
  * (26) M. S. Leifer, Phys. Rev. Lett. 112, 160404 (2014)
  * (27) J. Barrett, E. G. Cavalcanti, R. Lal, and O. J. E. Maroney, Phys. Rev. Lett. 112, 250403 (2014)
  * (28) A. Coladangelo, Quantum 4, 282 (2020).