* Janowitz (1964) Janowitz, M.  The military in the political development of new nations.  Bulletin of the Atomic Scientists 1964, 20, 6–10.
  * DeGrasse (2016) DeGrasse, R.W.  Military expansion, economic decline: Impact of military spending on United States economic performance; Routledge, 2016.
  * Roland (1991) Roland, A.  Technology, Ground Warfare, and Strategy: The Paradox of American Experience.  The Journal of Military History 1991, 55, 447–468.
  * Paxson et al. (1979) Paxson, E.W.; Weiner, M.G.; Wise, R.A.  Interactions Between Tactics and Technology in Ground Warfare.  Technical report, RAND CORP SANTA MONICA CA, 1979.
  * Edmonds (1992) Edmonds, M.  Land Warfare. In International Security in the Modern World; Springer, 1992; pp. 179–206.
  * Harding (2002) Harding, R.  Seapower and naval warfare, 1650-1830; Routledge, 2002.
  * Speller (2018) Speller, I.  Understanding naval warfare; Routledge, 2018.
  * Paris (1992) Paris, M.  Winged warfare: the literature and theory of aerial warfare in Britain, 1859-1917; Manchester University Press, 1992.
  * Colby (1925) Colby, E.  Laws of Aerial Warfare.  Minn. L. Rev. 1925, 10, 207.
  * Garner (1924) Garner, J.W.  Proposed rules for the regulation of aerial warfare.  American Journal of International Law 1924, 18, 56–81.
  * Gingras and Ruby (2000) Gingras, J.L.; Ruby, T.Z.  Morality in Modern Aerial Warfare.  Technical report, AIR UNIV MAXWELL AFB AL, 2000.
  * Dunn (2013) Dunn, D.H.  Drones: disembodied aerial warfare and the unarticulated threat.  International Affairs 2013, 89, 1237–1246.
  * Stoyen (2008) Stoyen, A.D.  Method and system for intelligent agent decision making for tactical aerial warfare, 2008.  US Patent 7,330,844.
  * Klein (2012) Klein, J.J.  Space warfare: Strategy, principles and policy; Routledge, 2012\.
  * Wortzel (2008) Wortzel, L.M.  The Chinese People’s Liberation Army and Space Warfare.  Astropolitics 2008, 6, 112–137.
  * Johnson-Freese (2016) Johnson-Freese, J.  Space warfare in the 21st century: Arming the heavens; Routledge, 2016.
  * Blackman (1986) Blackman, S.S.  Multiple-target tracking with radar applications.  Dedham, MA, Artech House, Inc., 1986, 463 p. 1986.
  * Lipton et al. (1998) Lipton, A.J.; Fujiyoshi, H.; Patil, R.S.  Moving target classification and tracking from real-time video.  Proceedings Fourth IEEE Workshop on Applications of Computer Vision. WACV’98 (Cat. No. 98EX201). IEEE, 1998, pp. 8–14.
  * Li and Jilkov (2003) Li, X.R.; Jilkov, V.P.  Survey of maneuvering target tracking. Part I. Dynamic models.  IEEE Transactions on aerospace and electronic systems 2003, 39, 1333–1364.
  * Kasthurirathna et al. (2014) Kasthurirathna, D.; Piraveenan, M.; Harré, M.  Influence of topology in the evolution of coordination in complex networks under information diffusion constraints.  The European Physical Journal B 2014, 87, 3.
  * Bar-Shalom and Li (1996) Bar-Shalom, Y.; Li, X.R.  Estimation and tracking: Principles, techniques, and software [reviews and abstracts].  IEEE Antennas and Propagation Magazine 1996, 38, 62.
  * Gandhi et al. (2011) Gandhi, R.; Sharma, A.; Mahoney, W.; Sousan, W.; Zhu, Q.; Laplante, P.  Dimensions of cyber-attacks: Cultural, social, economic, and political.  IEEE Technology and Society Magazine 2011, 30, 28–38.
  * Harris (1992) Harris, J.  War and social history: Britain and the home front during the Second World War.  Contemporary European History 1992, 1, 17–35.
  * Maltby and Keeble (2007) Maltby, S.; Keeble, R.  Communicating war: Memory, military and media; Arima Publishing, 2007.
  * Barough et al. (2012) Barough, A.S.; Shoubi, M.V.; Skardi, M.J.E.  Application of Game Theory Approach in Solving the Construction Project Conflicts.  Procedia - Social and Behavioral Sciences 2012, 58, 1586 – 1593.  8th International Strategic Management Conference, doi:blackhttp://dx.doi.org/10.1016/j.sbspro.2012.09.1145.
  * Kasthurirathna and Piraveenan (2014) Kasthurirathna, D.; Piraveenan, M.  Topological stability of evolutionarily unstable strategies.  2014 IEEE Symposium on Evolving and Autonomous Learning Systems (EALS). IEEE, 2014, pp. 35–42.
  * Myerson (1991) Myerson, R.  Game Theory: Analysis of Conflict Harvard Univ.  Press, Cambridge 1991.
  * Von Neumann (1928) Von Neumann, J.  On the theory of games of strategy; Princeton University Press Princeton, NJ, 1928.
  * Von Neumann and Morgenstern (1944) Von Neumann, J.; Morgenstern, O.  Theory of Games and Economic Behavior.(1967 edition) John Wiley and Sons.  Inc., New York, New York 1944, 641.
  * Rasmusen and Blackwell (1994) Rasmusen, E.; Blackwell, B.  Games and information.  Cambridge, MA 1994, 15.
  * Kasthurirathna and Piraveenan (2015) Kasthurirathna, D.; Piraveenan, M.  Emergence of scale-free characteristics in socio-ecological systems with bounded rationality.  Scientific reports 2015, 5.
  * Kasthurirathna et al. (2013) Kasthurirathna, D.; Piraveenan, M.; Harre, M.  Evolution of coordination in scale-free and small world networks under information diffusion constraints.  Advances in Social Networks Analysis and Mining (ASONAM), 2013 IEEE/ACM International Conference on. IEEE, 2013, pp. 183–189.
  * Thedchanamoorthy et al. (2014) Thedchanamoorthy, G.; Piraveenan, M.; Uddin, S.; Senanayake, U.  Influence of vaccination strategies and topology on the herd immunity of complex networks.  Social Network Analysis and Mining 2014, 4, 1–16.
  * Thakor (1991) Thakor, A.V.  Game theory in finance.  Financial Management 1991, pp. 71–94.
  * Bshary and Oliveira (2015) Bshary, R.; Oliveira, R.F.  Cooperation in animals: toward a game theory within the framework of social competence.  Current Opinion in Behavioral Sciences 2015, 3, 31–37.
  * Hadjichrysanthou and Broom (2012) Hadjichrysanthou, C.; Broom, M.  When should animals share food? Game theory applied to kleptoparasitic populations with food sharing.  Behavioral ecology 2012, 23, 977–991.
  * Lundberg and Pollak (1994) Lundberg, S.; Pollak, R.A.  Noncooperative bargaining models of marriage.  The American Economic Review 1994, 84, 132–137.
  * Manser and Brown (1980) Manser, M.; Brown, M.  Marriage and household decision-making: A bargaining analysis.  International economic review 1980, pp. 31–44.
  * Grossbard-Shechtman (1995) Grossbard-Shechtman, S.  Marriage market models.  The new economics of human behavior 1995, pp. 92–112.
  * Murray (2014) Murray, A.G.  A game theory based framework for assessing incentives for local area collaboration with an application to Scottish salmon farming.  Preventive veterinary medicine 2014, 115, 255–262.
  * Gibbons (1992) Gibbons, R.S.  Game theory for applied economists; Princeton University Press, 1992\.
  * Shubik (1970) Shubik, M.  Game theory, behavior, and the paradox of the prisoner’s dilemma: Three solutions.  Journal of Conflict Resolution 1970, 14, 181–193.
  * Polak (2007) Polak, B.  Econ 159: Game theory.  Open Yale courses, Department of Economics, http://oyc. yale. edu/economics 2007.
  * Kasthurirathna et al. (2014) Kasthurirathna, D.; Nguyen, H.; Piraveenan, M.; Uddin, S.; Senanayake, U.  Optimisation of strategy placements for public good in complex networks.  Proceedings of the 2014 International Conference on Social Computing. ACM, 2014, p. 1.
  * Kasthurirathna et al. (2015) Kasthurirathna, D.; Piraveenan, M.; Uddin, S.  Evolutionary stable strategies in networked games: the influence of topology.  Journal of Artificial Intelligence and Soft Computing Research 2015, 5, 83–95.
  * Binmore (2007) Binmore, K.  Playing for real: a text on game theory; Oxford university press, 2007.
  * Hart (1992) Hart, S.  Games in extensive and strategic forms.  Handbook of game theory with economic applications 1992, 1, 19–40.
  * Ritzberger et al. (2002) Ritzberger, K.; others.  Foundations of non-cooperative game theory.  OUP Catalogue 2002.
  * Peleg and Sudhölter (2007) Peleg, B.; Sudhölter, P.  Introduction to the theory of cooperative games; Vol. 34, Springer Science & Business Media, 2007.
  * Bell et al. (2017) Bell, M.; Perera, S.; Piraveenan, M.; Bliemer, M.; Latty, T.; Reid, C.  Network growth models: A behavioural basis for attachment proportional to fitness.  Scientific reports 2017, 7, 42431.
  * Kuhn et al. (1953) Kuhn, H.; Arrow, K.; Tucker, A.  Contributions to the Theory of Games; Number v. 2 in Annals of Mathematics Studies, Princeton University Press, 1953.
  * Kamien and Schwartz (2012) Kamien, M.I.; Schwartz, N.L.  Dynamic optimization: the calculus of variations and optimal control in economics and management; Courier Corporation, 2012.
  * Von Stackelberg (2010) Von Stackelberg, H.  Market structure and equilibrium; Springer Science & Business Media, 2010.
  * Calcagno et al. (2014) Calcagno, R.; Kamada, Y.; Lovo, S.; Sugaya, T.  Asynchronicity and coordination in common and opposing interest games.  Theoretical Economics 2014, 9, 409–434.
  * Schelling (1958) Schelling, T.C.  The strategy of conflict. Prospectus for a reorientation of game theory.  Journal of Conflict Resolution 1958, 2, 203–264.
  * Schelling (1980) Schelling, T.C.  The strategy of conflict; Harvard university press, 1980.
  * Lewis (2008) Lewis, D.  Convention: A philosophical study; John Wiley & Sons, 2008.
  * Nash Jr (1950) Nash Jr, J.F.  The bargaining problem.  Econometrica: Journal of the Econometric Society 1950, pp. 155–162.
  * Myerson (1999) Myerson, R.B.  Nash equilibrium and the history of economic theory.  Journal of Economic Literature 1999, 37, 1067–1082.
  * Rubinstein (1982) Rubinstein, A.  Perfect equilibrium in a bargaining model.  Econometrica: Journal of the Econometric Society 1982, pp. 97–109.
  * Camerer (2011) Camerer, C.F.  Behavioral game theory: Experiments in strategic interaction; Princeton University Press, 2011.
  * Aumann and Hart (1992) Aumann, R.J.; Hart, S.  Handbook of game theory with economic applications; Vol. 1, North-Holland Amsterdam, 1992.
  * Aumann (2008) Aumann, R.  Game theory, The New Palgrave Dictionary of Economics (New York)(SN Durlauf and LE Blume, eds.), 2008.
  * Smith (1993) Smith, J.M.  Evolution and the Theory of Games; Springer, 1993.
  * Le and Boyd (2007) Le, S.; Boyd, R.  Evolutionary dynamics of the continuous iterated Prisoner’s dilemma.  Journal of theoretical biology 2007, 245, 258–267.
  * Cho and Kreps (1987) Cho, I.K.; Kreps, D.M.  Signaling games and stable equilibria.  The Quarterly Journal of Economics 1987, 102, 179–221.
  * Bier et al. (2007-08) Bier, V.; Oliveros, S.; Samuelson, L.  Choosing What to Protect: Strategic Defensive Allocation against an Unknown Attacker.  Journal of Public Economic Theory 2007-08, 9, 563,587.
  * Cotton and Liu (2011) Cotton, C.; Liu, C.  100 Horsemen and the empty city: A game theoretic examination of deception in Chinese military legend.  Journal of peace research 2011, 48, 217–223.
  * Gries et al. (2016) Gries, T.; Haake, C.J.; others.  An Economic Theory of’Destabilization War’.  Technical report, Paderborn University, CIE Center for International Economics, 2016.
  * Krishnamurthy et al. (2007) Krishnamurthy, V.; Maskery, M.; Ngo, M.H.  Game theoretic activation and transmission scheduling in unattended ground sensor networks: A correlated equilibrium approach. In Wireless Sensor Networks: Signal Processing and Communications Perspectives; Wiley, 2007; pp. 349–388.
  * Levine (1995) Levine, A.J.  The pacific war: Japan versus the allies; Vol. 130, Greenwood Publishing Group, 1995.
  * Maskery and Krishnamurthy (2007) Maskery, M.; Krishnamurthy, V.  Network-enabled missile deflection: Games and correlation equilibrium.  IEEE Transactions on Aerospace and Electronic Systems 2007, 43, 843–863.
  * Maskery et al. (2007) Maskery, M.; Krishnamurthy, V.; O’Regan, C.  Decentralized algorithms for netcentric force protection against antiship missiles.  IEEE Transactions on Aerospace and Electronic Systems 2007, 43, 1351–1372.
  * Bachmann et al. (2011) Bachmann, D.J.; Evans, R.J.; Moran, B.  Game theoretic analysis of adaptive radar jamming.  IEEE Transactions on Aerospace and Electronic Systems 2011, 47, 1081–1100.
  * Hamilton and Mesic (2004) Hamilton, T.; Mesic, R.  A simple game-theoretic approach to suppression of enemy defenses and other time critical target analyses.  Technical report, RAND CORP SANTA MONICA CA, 2004.
  * Deligiannis et al. (2016) Deligiannis, A.; Rossetti, G.; Panoui, A.; Lambotharan, S.; Chambers, J.  Power allocation game between a radar network and multiple jammers.  2016 IEEE Radar Conference (RadarConf). IEEE, 2016, pp. 1–5.
  * Garcia et al. (2019) Garcia, E.; Von Moll, A.; Casbeer, D.W.; Pachter, M.  Strategies for defending a coastline against multiple attackers.  2019 IEEE 58th Conference on Decision and Control (CDC). IEEE, 2019, pp. 7319–7324.
  * He and Su (2019) He, B.; Su, H.  Joint Power Allocation and Beamforming between a Multistatic Radar and Jammer Based on Game Theory.  2019 IEEE 11th International Conference on Communication Software and Networks (ICCSN). IEEE, 2019, pp. 337–341.
  * McEneaney et al. (2004) McEneaney, W.M.; Fitzpatrick, B.G.; Lauko, I.G.  Stochastic game approach to air operations.  IEEE Transactions on Aerospace and Electronic Systems 2004, 40, 1191–1216.
  * Wei et al. (2009) Wei, X.; Duan, H.; Wang, Y.  Hybrid game theory and DS evidence approach to multiple UCAVs cooperative air combat decision.  International Symposium on Neural Networks. Springer, 2009, pp. 236–246.
  * Ma et al. (2019) Ma, Y.; Wang, G.; Hu, X.; Luo, H.; Lei, X.  Cooperative occupancy decision making of Multi-UAV in Beyond-Visual-Range air combat: A game theory approach.  IEEE Access 2019, 8, 11624–11634.
  * Evers et al. (2013) Evers, L.; Barros, A.I.; Monsuur, H.  The cooperative ballistic missile defence game.  International Conference on Decision and Game Theory for Security. Springer, 2013, pp. 85–98.
  * Garcia et al. (2017) Garcia, E.; Casbeer, D.W.; Fuchs, Z.E.; Pachter, M.  Cooperative missile guidance for active defense of air vehicles.  IEEE Transactions on Aerospace and Electronic Systems 2017, 54, 706–721.
  * Han et al. (2016) Han, C.Y.; Lunday, B.J.; Robbins, M.J.  A game theoretic model for the optimal location of integrated air defense system missile batteries.  INFORMS Journal on Computing 2016, 28, 405–416.
  * Başpınar and Koyuncu (2020) Başpınar, B.; Koyuncu, E.  Assessment of aerial combat game via optimization-based receding horizon control.  IEEE Access 2020, 8, 35853–35863.
  * Casbeer et al. (2018) Casbeer, D.W.; Garcia, E.; Pachter, M.  The target differential game with two defenders.  Journal of Intelligent & Robotic Systems 2018, 89, 87–106.
  * Keith and Ahner (2021) Keith, A.; Ahner, D.  Counterfactual regret minimization for integrated cyber and air defense resource allocation.  European Journal of Operational Research 2021, 292, 95–107.
  * Zhong et al. (2018) Zhong, X.; Yin, H.; He, Y.; Zhu, H.  Joint Transmit Power and Bandwidth Allocation for Cognitive Satellite Network Based on Bargaining Game Theory.  IEEE Access 2018, 7, 6435–6449.
  * Qiao and Zhao (2013-12) Qiao, R.; Zhao, X.  The Research and Simulation of the Satellite Network Routing Algorithm Based on Game Theory.  2013 International Conference on Information Science and Cloud Computing. IEEE, 2013-12, pp. 124,130.
  * Gu (2010) Gu, D.  A game theory approach to target tracking in sensor networks.  IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 2010, 41, 2–13.
  * Sun et al. (2018) Sun, Q.; Qi, N.; Huo, M.  Optimal strategy for target protection with a defender in the pursuit-evasion scenario.  The Journal of Defense Modeling and Simulation: Applications, Methodology, Technology 2018, 15, 154851291875622.
  * Faruqi (2017) Faruqi, F.A.  Differential game theory with applications to missiles and autonomous systems guidance; John Wiley & Sons, 2017.
  * Evers et al. (2013) Evers, L.; Barros, A.I.; Monsuur, H.  The Cooperative Ballistic Missile Defence Game.  Decision and Game Theory for Security; Das, S.K.; Nita-Rotaru, C.; Kantarcioglu, M., Eds.; Springer International Publishing: Cham, 2013; pp. 85–98.
  * Shinar and Shima (1996) Shinar, J.; Shima, T.  A game theoretical interceptor guidance law for ballistic missile defence.  Proceedings of 35th IEEE Conference on Decision and Control. IEEE, 1996, Vol. 3, pp. 2780–2785.
  * Bogdanović et al. (2018) Bogdanović, N.; Driessen, H.; Yarovoy, A.G.  Target selection for tracking in multifunction radar networks: Nash and correlated equilibria.  IEEE Transactions on Aerospace and Electronic Systems 2018, 54, 2448–2462.
  * Parras et al. (2017) Parras, J.; Zazo, S.; Del Val, J.; Zazo, J.; Macua, S.V.  Pursuit-evasion games: a tractable framework for antijamming games in aerial attacks.  EURASIP Journal on Wireless Communications and Networking 2017, 2017, 69.
  * Halpern (2008) Halpern, J.Y.  Computer science and game theory.  The New Palgrave Dictionary of Economics: Volume 1–8 2008, pp. 984–994.
  * Apt and Grädel (2011) Apt, K.R.; Grädel, E.  Lectures in game theory for computer scientists; Cambridge University Press, 2011.
  * Shoham (2008) Shoham, Y.  Computer science and game theory.  Commun. ACM 2008, 51, 74–79.
  * Tom (2015) Tom, L.  Game-theoretic approach towards network security: A review.  2015 International Conference on Circuits, Power and Computing Technologies [ICCPCT-2015]. IEEE, 2015, pp. 1–4.
  * Jhawar et al. (2016) Jhawar, R.; Mauw, S.; Zakiuddin, I.  Automating cyber defence responses using attack-defence trees and game theory.  European Conference on Cyber Warfare and Security. Academic Conferences International Limited, 2016, p. 163.
  * Gonzalez (2013) Gonzalez, C.  From individual decisions from experience to behavioral game theory: Lessons for cybersecurity. In Moving target defense II; Springer, 2013; pp. 73–86.
  * Hausken et al. (2009) Hausken, K.; Bier, V.M.; Zhuang, J., Defending Against Terrorism, Natural Disaster, and All Hazards.  In Game Theoretic Risk Analysis of Security Threats; Bier, V.M.M.; Azaiez, M.N., Eds.; Springer US: Boston, MA, 2009; pp. 65–97.
  * Kanturska et al. (2009) Kanturska, U.; Schmöcker, J.D.; Fonzone, A.; Bell, M.G.H., Improving Reliability through Multi-Path Routing and Link Defence: An Application of Game Theory to Transport.  In Game Theoretic Risk Analysis of Security Threats; Bier, V.M.M.; Azaiez, M.N., Eds.; Springer US: Boston, MA, 2009; pp. 199–227.
  * Bier (2006) Bier, V., Game-Theoretic and Reliability Methods in Counterterrorism and Security.  In Statistical Methods in Counterterrorism: Game Theory, Modeling, Syndromic Surveillance, and Biometric Authentication; Wilson, A.G.; Wilson, G.D.; Olwell, D.H., Eds.; Springer New York: New York, NY, 2006; pp. 23–40.
  * Cioacă et al. (2013) Cioacă, C.; others.  Considerations on optimum resource allocation in aviation security.  Journal of Defense Resources Management (JoDRM) 2013, 4, 187–192.
  * Gardener and Moffat (2008) Gardener, T.; Moffat, J.  Changing behaviours in defence acquisition: a game theory approach.  Journal of the Operational Research Society 2008, 59, 225–230.
  * Zhang and Mehrjerdi (2013) Zhang, Y.; Mehrjerdi, H.  A survey on multiple unmanned vehicles formation control and coordination: Normal and fault situations.  2013 International conference on unmanned aircraft systems (ICUAS). IEEE, 2013, pp. 1087–1096.
  * Stone (1981) Stone, L.D.  Search and screening: General principles with historical applications (BO Koopman).  SIAM Review 1981, 23, 533–539.
  * Stone (1980) Stone, L.D.  Optimal search for moving targets.  Advances in Applied Probability 1980, 12, 279–280.
  * Washburn et al. (2014) Washburn, A.R.; others.  Two-person zero-sum games 2014.
  * Washburn and Karataş (2015) Washburn, A.; Karataş, M.  Multistatic search theory.  Military Operations Research 2015, 20, 21–38.
  * Jing et al. (2021) Jing, T.; Meng, Q.H.; Ishida, H.  Recent Progress and Trend of Robot Odor Source Localization.  IEEJ Transactions on Electrical and Electronic Engineering 2021.
  * Ristic and Skvortsov (2020) Ristic, B.; Skvortsov, A.  Intermittent Information-Driven Multi-Agent Area-Restricted Search.  Entropy 2020, 22, 635.
  * Ristic et al. (2016) Ristic, B.; Skvortsov, A.; Gunatilaka, A.  A study of cognitive strategies for an autonomous search.  Information Fusion 2016, 28, 1–9.
  * Ristic et al. (2020) Ristic, B.; Gilliam, C.; Moran, W.; Palmer, J.L.  Decentralised multi-platform search for a hazardous source in a turbulent flow.  Information Fusion 2020, 58, 13–23.
  * Hutchinson et al. (2018) Hutchinson, M.; Liu, C.; Chen, W.H.  Information-based search for an atmospheric release using a mobile robot: Algorithm and experiments.  IEEE Transactions on Control Systems Technology 2018, 27, 2388–2402.
  * Park and Oh (2020) Park, M.; Oh, H.  Cooperative information-driven source search and estimation for multiple agents.  Information Fusion 2020, 54, 72–84.
  * Durrant-Whyte and Bailey (2006) Durrant-Whyte, H.; Bailey, T.  Simultaneous localization and mapping: part I.  IEEE robotics & automation magazine 2006, 13, 99–110.
  * Vergassola et al. (2007) Vergassola, M.; Villermaux, E.; Shraiman, B.I.  ?Infotaxis? as a strategy for searching without gradients.  Nature 2007, 445, 406–409.
  * Bénichou et al. (2011) Bénichou, O.; Loverdo, C.; Moreau, M.; Voituriez, R.  Intermittent search strategies.  Reviews of Modern Physics 2011, 83, 81.
  * (139) DARPA.  DARPA Tiles Together a Vision of Mosaic Warfare.  https://www.darpa.mil/work-with-us/darpa-tiles-together-a-vision-of-mosiac-warfare.  Accessed: 2019-09-15.
  * Prokopenko et al. (2005) Prokopenko, M.; Piraveenan, M.; Wang, P.  On convergence of dynamic cluster formation in multi-agent networks.  European Conference on Artificial Life. Springer, 2005, pp. 884–894.
  * Piraveenan et al. (2005) Piraveenan, M.; Prokopenko, M.; Wang, P.; Price, D.  Towards adaptive clustering in self-monitoring multi-agent networks.  International Conference on Knowledge-Based and Intelligent Information and Engineering Systems. Springer, 2005, pp. 796–805.
  * Cliff et al. (2018) Cliff, O.M.; Harding, N.; Piraveenan, M.; Erten, E.Y.; Gambhir, M.; Prokopenko, M.  Investigating spatiotemporal dynamics and synchrony of influenza epidemics in Australia: an agent-based modelling approach.  Simulation Modelling Practice and Theory 2018, 87, 412–431.
  * Chang et al. (2020) Chang, S.L.; Piraveenan, M.; Pattison, P.; Prokopenko, M.  Game theoretic modelling of infectious disease dynamics and intervention methods: a review.  Journal of biological dynamics 2020, 14, 57–89.
  * Piraveenan et al. (2021) Piraveenan, M.; Sawleshwarkar, S.; Walsh, M.; Zablotska, I.; Bhattacharyya, S.; Farooqui, H.H.; Bhatnagar, T.; Karan, A.; Murhekar, M.; Zodpey, S.; others.  Optimal governance and implementation of vaccination programmes to contain the COVID-19 pandemic.  Royal Society open science 2021, 8, 210429.
  * Vaitekunas (2010) Vaitekunas, D.A.  IR susceptibility of naval ships using ShipIR/NTCS.  Infrared Imaging Systems: Design, Analysis, Modeling, and Testing XXI; Holst, G.C.; Krapels, K.A., Eds. International Society for Optics and Photonics, SPIE, 2010, Vol. 7662, pp. 277 – 284.
  * Zalud (2010) Zalud, B.  Unseen But There: A Place for Covert Security 2010.
  * Kim et al. (2019) Kim, S.K.; Cheon, S.P.; Eom, J.H.  A leading cyber warfare strategy according to the evolution of cyber technology after the fourth industrial revolution.  International Journal of Advanced Computer Research 2019, 9, 72–80.


ʻ ‘

# Performance Comparison of Binary Machine Learning Classifiers in Identifying
Code Comment Types: An Exploratory Study

Amila Indika  University of Hawaiʻi at Mānoa
Honolulu, Hawaiʻi, USA
<EMAIL_ADDRESS>Peter Y. Washington  University of Hawaiʻi at Mānoa
Honolulu, Hawaiʻi, USA
<EMAIL_ADDRESS>Anthony Peruma  University of Hawaiʻi at Mānoa
Honolulu, Hawaiʻi, USA
<EMAIL_ADDRESS>
###### Abstract

Code comments are vital to source code as they help developers with program
comprehension tasks. Written in natural language (usually English), code
comments convey a variety of different information, which are grouped into
specific categories. In this study, we construct 19 binary machine learning
classifiers for code comment categories that belong to three different
programming languages. We present a comparison of performance scores for
different types of machine learning classifiers and show that the Linear SVC
classifier has the highest average F1 score of 0.5474.

###### Index Terms:

Code Comments, Natural Language Processing, Machine Learning, Binary
Classification

##  I Introduction

Program comprehension is an essential activity in any software maintenance
task [1], which involves developers reading the code to understand its
purpose, behavior, and shortcomings (if any) [2]. However, while source code
is helpful, it is constrained by the semantics and syntax of programming
language in the amount of information it can provide. In contrast, code
comments are written in natural language (such as English) and are not
restricted to any specific syntax or grammar; they can provide information
that might not be immediately obvious from the source code alone. Developers
use code comments to document various aspects of their code, from explaining
behavior to documenting workarounds to including metadata (such as a license),
all expressed in natural language [3, 4]. While such information in comments
is useful, it would be more productive for developers to group such comments
into related categories automatically. This would help developers locate
pertinent information faster, such as improving code quality by
removing/replacing deprecated code.

The goal of this study is to understand the extent to which different types of
binary machine learning classifiers can identify code comment types. Our
findings show that while no single classifier single-handedly achieves the
highest F1 score for each individual comment type, the Linear SVC has the
highest average F1 score of 0.5474. The outcome of this study will help the
research community understand the strengths and shortcomings of using such
models for this particular task and discover avenues for future research in
this area.

Figure 1: Overview of our methodology for constructing the comment classification models. TABLE I: Comparison of F1 scores associated with each machine learning classifier for each category. The score in green represents the highest F1 score for the specific category. The score in blue is the classifier that has the highest average F1 score.  Category |  | Decision
---
Tree
| K-Nearest
---
Neighbors
| Bernoulli
---
Naive Bayes
| Multi-Layer
---
Perceptron
| Multinomial
---
Naive Bayes
| Random
---
Forest
| Logistic
---
Regression
| Linear
---
SVC
JAVA
Deprecation | 0.5797 | 0.8163 | 0.5195 | 0.7600 | 0.6032 | 0.8085 | 0.7600 | 0.7170
Pointer | 0.6069 | 0.5890 | 0.5697 | 0.5556 | 0.6301 | 0.6418 | 0.6265 | 0.5850
Summary | 0.5787 | 0.5030 | 0.6327 | 0.5532 | 0.5905 | 0.5455 | 0.5842 | 0.5684
Expand | 0.4857 | 0.5833 | 0.5704 | 0.5970 | 0.6097 | 0.6160 | 0.6264 | 0.6100
Ownership | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.0000 | 1.0000
Rational | 0.5455 | 0.5517 | 0.5854 | 0.6122 | 0.5669 | 0.6667 | 0.5833 | 0.5920
Usage | 0.5931 | 0.7255 | 0.7612 | 0.7493 | 0.7740 | 0.7435 | 0.7507 | 0.7413
PHARO
Class References | 0.1778 | 0.2963 | 0.2353 | 0.1600 | 0.2105 | 0.3000 | 0.2069 | 0.2857
Example | 0.7132 | 0.7320 | 0.8063 | 0.7525 | 0.7778 | 0.7438 | 0.7509 | 0.7603
Key Implementation Points | 0.2642 | 0.3125 | 0.3529 | 0.3571 | 0.3902 | 0.2424 | 0.3191 | 0.4124
Collaborators | 0.3810 | 0.2979 | 0.2712 | 0.2609 | 0.2308 | 0.1714 | 0.4138 | 0.4231
Intent | 0.4854 | 0.3488 | 0.3208 | 0.2759 | 0.3390 | 0.5479 | 0.4554 | 0.4301
Key Messages | 0.4628 | 0.4407 | 0.5733 | 0.6055 | 0.5797 | 0.5149 | 0.5 | 0.5254
Responsibilities | 0.3407 | 0.3193 | 0.3867 | 0.4192 | 0.3617 | 0.3585 | 0.4485 | 0.4218
PYTHON
Development Notes | 0.3472 | 0.2314 | 0.2995 | 0.3066 | 0.2769 | 0.3333 | 0.2838 | 0.3145
Parameters | 0.6523 | 0.5831 | 0.6359 | 0.6203 | 0.6387 | 0.6605 | 0.6565 | 0.6337
Summary | 0.3169 | 0.3444 | 0.4017 | 0.3579 | 0.4050 | 0.3432 | 0.4138 | 0.3774
Expand | 0.3514 | 0.4599 | 0.4571 | 0.4643 | 0.4534 | 0.4293 | 0.4519 | 0.4636
Usage | 0.4859 | 0.4266 | 0.5481 | 0.5233 | 0.5342 | 0.5275 | 0.5519 | 0.5387
AVERGE F1 SCORE | 0.4931 | 0.5033 | 0.5225 | 0.5227 | 0.5249 | 0.5366 | 0.5465 | 0.5474

##  II Experiment Design

In this section, we discuss the methodology of our study. Figure 1 depicts an
outline of the activities in our experiment methodology, which we describe
below. Furthermore, our dataset and code are available for
replication/extension at [5].

###  II-A Source Dataset

We utilize an existing dataset of 6,738 class comment sentences produced by
Rani et al. [4] and made available as part of the NLBSE 2023 tool competition
[6]. These comment sentences are obtained from 20 open-source projects
implemented using Java, Python, and Pharo. The authors of the dataset also
perform text preprocessing and splitting of the comments. Furthermore, the
dataset also indicates the category (i.e., class) associated with the comment
sentence. Below is a breakdown of comments categories:

  * •

Java - 2418 comment sentences in 7 categories: summary, pointer, deprecation,
rational, ownership, usage, and expand

  * •

Pharo - 1765 comment sentences in 7 categories: key messages, intent, class
references, example, key implementation points, responsibilities, and
collaborators

  * •

Python - 2555 comment sentences in 5 categories: summary, parameters, usage,
development notes, and expand

For each programming language in the dataset, we execute the activities in the
subsequent subsections.

###  II-B Text Preprocessing

Since code comments are crafted using natural language, there exists the
possibility of developers composing these comments using a variety of tokens
that include numeric digits and non-alphanumeric characters (such as brackets,
parentheses, periods, etc.). Therefore, to improve model classification
performance, it is essential to transform the comment sentences to a standard
and convenient format (i.e., text normalization) [7]. We achieve text
normalization through a series of processing activities:

  * •

Removal of whitespaces from the start and end of the text

  * •

Expansion of contractions (e.g., ‘I’m’ $\rightarrow$ ‘I am’)

  * •

Removal of non-alphanumeric characters

  * •

Removal of single-character words

  * •

Convert text to lowercase

  * •

Removal of stopwords (using the NLTK stopword list [8])

  * •

Reduce words to their base form or root using stemming

  * •

Replace numeric digits with a NUM token

  * •

Replace empty string with an EMT token

###  II-C Filter Category

The purpose of this study is the construction of binary classifiers for each
of the 19 categories mentioned above. Therefore, for each programming language
dataset, we filter the comments for each category. For example, when
processing the Java comments, we filter on each of the 7 categories. For the
activities in the subsequent subsections, we utilize this subset of comments
(i.e., each programming language category) to build and evaluate the
classification model.

###  II-D Train/Test Split

As part of measuring the model’s effectiveness in correctly classifying
comment sentences, it must be tested against a held-out test set containing
comment sentences with known classes. We split the filtered dataset into a
training and test dataset. Contained within the source dataset is a column
that indicates if a specific comment sentence belongs to the test or training
set; we use this column to create the train/test datasets.

###  II-E Feature Representation

The raw comment text must be converted into a set of numerical features that
can be used as input to the model. Most machine learning algorithms can only
process numerical data instead of raw text. Furthermore, converting text to
numerical data reduces the complexity of the data. To this extent, we utilize
the Term Frequency-Inverse Document Frequency (TF-IDF) approach of
representing the raw comment sentence text as numerical values. TF-IDF
calculates the importance of each word in a document based on its frequency
and rarity, which we then use as features for the machine learning model [9].
In this study, we only utilize the comment sentence text as input to the
classification model. In our process, we first fit the vectorizer on the
training data and transform the training data into a matrix of TF-IDF
features. Finally, we utilize this generated vocabulary and TF-IDF
representation scheme to transform the test data into a matrix of TF-IDF
features. This approach ensures that the resulting test data TF-IDF matrix has
the same number of columns (i.e., the same vocabulary) as the training data
and avoids model performance issues.

###  II-F Oversampling

We applied an oversampling technique to the training datasets to mitigate the
issue of having imbalanced classes. Having an imbalanced dataset will lead to
a bias in the model towards the majority class due to the disproportionate
amount of data from that class. For this study, we utilized random
oversampling to generate new samples for the under-represented classes in the
training dataset [10]. We do not oversample the test dataset, as we evaluate
the model using metrics that inherently account for lack of balance
(precision, recall, and F1 score).

###  II-G Model Training & Tuning

Given the complexity of various machine learning classification algorithms,
selecting the most optimal algorithm for a given problem using only
theoretical arguments can be challenging. Hence, in this study, we evaluate 8
common machine learning classification algorithms frequently utilized in
software engineering research [11, 12, 13]. These classification algorithms
fall into different categories:

  * •

Naive Bayes (NB): Multinomial NB and Bernoulli NB

  * •

Support Vector Machines: Linear Support Vector Classifier

  * •

Trees: Decision Tree and Random Forest

  * •

Nearest Neighbors: K-Nearest Neighbors

  * •

Linear Model: Logistic Regression

  * •

Neural Network: Multi-Layer Perceptron

For each classification algorithm listed above, we use grid search in
conjunction with a 10-fold cross-validation to perform a search over specified
hyperparameter values to evaluate the model’s performance. Grid search
utilizes a brute force technique to evaluate all combinations of
hyperparameters to obtain the best performance [14]. We tuned at least one
hyperparameter associated with each classifier in our list. The hyperparameter
values were either a range of values or a predefined set of values. For
example, we evaluated the performance of the Decision Tree classifier by
changing the maximum depth hyperparameter from 5 to 100 in increments of 5.
The complete set of evaluated hyperparameters and their range of possible
values are available in the source code contained within the replication
package (available at [5]).

###  II-H Best/Optimized Model

After hyperparameter tuning, we obtain a model for each classification
algorithm that performs the best when built on the training data. For each of
these optimum models, we capture the values of the hyperparameters and save
the model.

###  II-I Test Predictions

We utilize the optimum model for each classification algorithm to predict the
test data values (i.e., unseen data). We save the predictions as a CSV file.

###  II-J Model Performance Scoring

Next, we utilize standard balanced binary classification metrics (precision,
recall, and F1 score) to evaluate the optimum model’s performance in
predicting the test data values. To calculate these scores, we utilize the
count of true positive (TP), false positive (FP), and false negative (FN)
instances in the predicted test data values. Below we list the formulas for
the three performance measures:

$Precision=\frac{TP}{TP+FP}$

$Recall=\frac{TP}{TP+FN}$

$F1=2*\frac{Precision*Recall}{Precision+Recall}$

We calculate the precision, recall, and F1 scores for each classification
model for each programming language category. Since we are evaluating eight
classifiers across 19 categories, we calculate 190 instances of precision,
recall, and F1 scores. While there are different variants of the F1-score,
such as ‘micro’, ‘macro’, etc., we utilize the ‘binary’ option since our
problem is a binary classification task [15, 16]. Using another variant will
give different (and possibly higher) scores.

##  III Experiment Results

TABLE II: Breakdown of the performance scores of the linear svc classifiers for each programming language category. Category | Precision | Recall | F1
---|---|---|---
JAVA
Deprecation | 0.7308 | 0.7037 | 0.7170
Pointer | 0.5972 | 0.5733 | 0.5850
Summary | 0.5243 | 0.6207 | 0.5684
Expand | 0.5985 | 0.6220 | 0.6100
Ownership | 1.0000 | 1.0000 | 1.0000
Rational | 0.5441 | 0.6491 | 0.5920
Usage | 0.7277 | 0.7554 | 0.7413
PHARO
Class References | 0.3636 | 0.2353 | 0.2857
Example | 0.7929 | 0.7303 | 0.7603
Key Implementation Points | 0.4082 | 0.4167 | 0.4124
Collaborators | 0.4583 | 0.3929 | 0.4231
Intent | 0.4167 | 0.4444 | 0.4301
Key Messages | 0.5636 | 0.4921 | 0.5254
Responsibilities | 0.3974 | 0.4493 | 0.4218
PYTHON
Development Notes | 0.2660 | 0.3846 | 0.3145
Parameters | 0.5956 | 0.6770 | 0.6337
Summary | 0.3361 | 0.4301 | 0.3774
Expand | 0.4322 | 0.5000 | 0.4636
Usage | 0.5438 | 0.5337 | 0.5387
AVERAGE SCORE | 0.5419 | 0.5585 | 0.5474

In this section, we report on the results of our experiment. Table I shows the
F1 scores we obtain for each classification model for each programming
language category. From this table, we observe that no single classification
model achieves the highest F1 score for each programming language category.
Looking at the average F1 scores, we observe that Linear SVC has the highest
average F1 score of 0.5474. It is interesting to note that all models were
able to successfully classify all test data instances of the Java Ownership
category, 489 instances. We also observe that the models seem to fare well
with Java comments than with Pharo and Python comments. In Table II, we
provide the precision and recall score of the optimal Linear Support Vector
Classifier for each programming language category.

It is interesting to note that while several machine learning methods with
varying complexities were explored, the two best-performing models were
constrained to learning linearly separating boundaries: logistic regression
and a support vector classifier with a linear kernel. These methods are less
prone to overfitting due to the simplicity of the decision boundary.

##  IV Threats To Validity

Even though our study includes a variety of classification algorithms, other
types of classifiers can be utilized on the same dataset. Likewise, there is a
possibility that alternative preprocessing and oversampling techniques,
features (for instance, using the NEON tool [17]), and hyperparameters would
yield improved performance scores.

##  V Conclusion & Future Work

In this study, we explored the performance of different types of machine
learning classifiers in performing binary classification of code comment
sentences. Our results show that even though no single classifier can achieve
high F1 scores for all code comment categories, the simplest model we
developed (i.e., a Linear SVC), which is constrained to linear decision
boundaries, yielded the highest average F1 score.

We did not observe a noticeable consistent pattern which might provide
potential explanations as to why the best classifier performs better for some
categories than for others. Understanding these discrepancies is a salient
opportunity for further evaluation.

Future work includes evaluating attention-based deep learning models, such as
BERT [18] and GPT-3 [19]. We hypothesize that such large pre-trained language
models can outperform the models presented here, although the extent to which
is to be determined due to the heterogeneous nature of the prediction task. It
would also be interesting to explore the feature vector representation space
beyond TF-IDF, such as automatically learned representations (notably,
Word2Vec [20, 21]).

Our source code (along with instructions) and generated optimal models are
available for download at [5].

##  VI Acknowledgements

The technical support and advanced computing resources from University of
Hawaiʻi Information Technology Services – Cyberinfrastructure, funded in part
by the National Science Foundation CC* awards #2201428 and #2232862 are
gratefully acknowledged.

## References

  * [1] V. Rajlich and N. Wilde, “The role of concepts in program comprehension,” in Proceedings 10th International Workshop on Program Comprehension, pp. 271–278, IEEE, 2002.
  * [2] W. Maalej, R. Tiarks, T. Roehm, and R. Koschke, “On the comprehension of program comprehension,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 23, no. 4, pp. 1–37, 2014.
  * [3] L. Pascarella and A. Bacchelli, “Classifying code comments in java open-source software systems,” in 2017 IEEE/ACM 14th International Conference on Mining Software Repositories (MSR), pp. 227–237, 2017.
  * [4] P. Rani, S. Panichella, M. Leuenberger, A. Di Sorbo, and O. Nierstrasz, “How to identify class comment types? a multi-language approach for class comment classification,” Journal of systems and software, 2021.
  * [5] https://doi.org/10.5281/zenodo.7686108.
  * [6] R. Kallis, M. Izadi, L. Pascarella, O. Chaparro, and P. Rani, “The nlbse’23 tool competition,” in Proceedings of The 2nd International Workshop on Natural Language-based Software Engineering (NLBSE’23), 2023.
  * [7] D. Jurafsky and J. H. Martin, “Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition,” Prentic e Hall, 2019.
  * [8] S. Bird and E. Loper, “Nltk: the natural language toolkit,” in Proceedings of the ACL 2004 on Interactive poster and demonstration sessions, p. 31, Association for Computational Linguistics, 2004.
  * [9] C. Manning, P. Raghavan, and H. Schütze, Introduction to Information Retrieval.  Cambridge University Press, 2008.
  * [10] T. Amr, Hands-On Machine Learning with scikit-learn and Scientific Python Toolkits.  Packt Publishing, 2020.
  * [11] M. I. Azeem, F. Palomba, L. Shi, and Q. Wang, “Machine learning techniques for code smell detection: A systematic literature review and meta-analysis,” Information and Software Technology, 2019.
  * [12] T. Sharma, M. Kechagia, S. Georgiou, R. Tiwari, and F. Sarro, “A survey on machine learning techniques for source code analysis,” arXiv preprint arXiv:2110.09610, 2021.
  * [13] E. A. AlOmar, A. Peruma, M. W. Mkaouer, C. Newman, A. Ouni, and M. Kessentini, “How we refactor and how we document it? on the use of supervised machine learning algorithms to classify refactoring documentation,” Expert Systems with Applications, 2021.
  * [14] P. Dangeti, Statistics for machine learning.  Packt Publishing Ltd, 2017.
  * [15] G. Ciaburro and P. Joshi, Python Machine Learning Cookbook: Over 100 recipes to progress from smart data analytics to deep learning using real-world datasets, 2nd Edition.  Packt Publishing, 2019.
  * [16] “sklearn.metrics.f1_score — scikit-learn 1.2.1 documentation.” https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html.
  * [17] A. Di Sorbo, C. A. Visaggio, M. Di Penta, G. Canfora, and S. Panichella, “An nlp-based tool for software artifacts analysis,” in IEEE International Conference on Software Maintenance and Evolution, ICSME 2021, Luxembourg, September 27 - October 1, 2021, IEEE, 2021.
  * [18] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), (Minneapolis, Minnesota), pp. 4171–4186, Association for Computational Linguistics, June 2019.
  * [19] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  * [20] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” arXiv preprint arXiv:1301.3781, 2013\.
  * [21] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” Advances in neural information processing systems, vol. 26, 2013.