* Barsi and Kassemi [2013a] Barsi S, Kassemi M.  Investigation of tank pressurization and pressure control—part i: Experimental study.  Journal of Thermal Science and Engineering Applications 2013a;5(4).  doi:10.1115/1.4023891.
  * Barsi and Kassemi [2013b] Barsi S, Kassemi M.  Investigation of tank pressurization and pressure control—part II: Numerical modeling.  Journal of Thermal Science and Engineering Applications 2013b;5(4).  doi:10.1115/1.4023892.
  * van Beek et al. [2023] van Beek A, Karkaria VN, Chen W.  Digital twins for the designs of systems: a perspective.  Structural and Multidisciplinary Optimization 2023;66(3).  doi:10.1007/s00158-023-03488-x.
  * Bertsekas [2019] Bertsekas D.  Reinforcement Learning and Optimal Control.  Athena Scientific, 2019.
  * Bhatnagar et al. [2009] Bhatnagar S, Sutton RS, Ghavamzadeh M, Lee M.  Natural actor–critic algorithms.  Automatica 2009;45(11):2471--82.  URL: https://linkinghub.elsevier.com/retrieve/pii/S0005109809003549. doi:10.1016/j.automatica.2009.07.008.
  * Bhowmik and Spee [1999] Bhowmik S, Spee R.  Performance Optimization for Doubly Fed Wind Power Generation Systems.  IEEE TRANSACTIONS ON INDUSTRY APPLICATIONS 1999;35(4).
  * Bianchi et al. [2007] Bianchi FD, De Battista H, Mantz RJ.  Wind turbine control systems: principles, modelling and gain scheduling design.  volume 19.  Springer, 2007.
  * Bocquet [2011] Bocquet M.  Ensemble kalman filtering without the intrinsic need for inflation.  Nonlinear Processes in Geophysics 2011;18(5):735--50.  doi:10.5194/npg-18-735-2011.
  * Bocquet et al. [2019] Bocquet M, Brajard J, Carrassi A, Bertino L.  Data assimilation as a deep learning tool to infer ode representations of dynamical models.  2019\. .
  * Bocquet and Farchi [2023] Bocquet M, Farchi A.  Introduction to the principles and methods of data assimilation in the geosciences.  Technical Report; École des Ponts ParisTech; 2023\.
  * Boedecker et al. [2014a] Boedecker J, Springenberg JT, Wülfing J, Riedmiller M.  Approximate real-time optimal control based on sparse gaussian process models.  In: 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL). 2014a. p. 1--8.  doi:10.1109/ADPRL.2014.7010608.
  * Boedecker et al. [2014b] Boedecker J, Springenberg JT, Wülfing J, Riedmiller M.  Approximate real-time optimal control based on sparse gaussian process models.  In: 2014 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL). 2014b. p. 1--8.  doi:10.1109/ADPRL.2014.7010608.
  * Bossanyi [2000] Bossanyi EA.  The Design of closed loop controllers for wind turbines.  Wind Energ 2000;3(3):149--63.  URL: https://onlinelibrary.wiley.com/doi/10.1002/we.34. doi:10.1002/we.34.
  * Bradley [2019] Bradley AM.  Pde-constrained optimization and the adjoint method 2019;.
  * Brajard et al. [2020] Brajard J, Carassi A, Bocquet M, Bertino L.  Combining data assimilation and machine learning to emulate a dynamical model from sparse and noisy observations: a case study with the Lorenz 96 model.  Journal of Computational Science 2020;44:101171.  URL: http://arxiv.org/abs/2001.01520. doi:10.1016/j.jocs.2020.101171; arXiv:2001.01520 [physics, stat].
  * Branlard et al. [2023] Branlard E, Jonkman J, Brown C, Zang J.  A digital-twin solution for floating offshore wind turbines validated using a full-scale prototype.  Wind Energy Science Discussions 2023;2023:1--34.  URL: https://wes.copernicus.org/preprints/wes-2023-50/. doi:10.5194/wes-2023-50.
  * Buizza et al. [2022] Buizza C, Casas CQ, Nadler P, Mack J, Marrone S, Titus Z, Cornec CL, Heylen E, Dur T, Ruiz LB, Heaney C, Lopez JAD, Kumar KS, Arcucci R.  Data learning: Integrating data assimilation and machine learning.  Journal of Computational Science 2022;58:101525.  doi:10.1016/j.jocs.2021.101525.
  * Cai et al. [2021] Cai X, Kolomenskiy D, Nakata T, Liu H.  A cfd data-driven aerodynamic model for fast and precise prediction of flapping aerodynamics in various flight velocities.  Journal of Fluid Mechanics 2021;915:A114.
  * Canaday et al. [2020] Canaday D, Pomerance A, Gauthier DJ.  Model-free control of dynamical systems with deep reservoir computing 2020;arXiv:2010.02285.
  * Cao et al. [2003] Cao Y, Li S, Petzold L, Serban R.  Adjoint sensitivity analysis for differential-algebraic equations: The adjoint DAE system and its numerical solution.  SIAM Journal on Scientific Computing 2003;24(3):1076--89.  doi:10.1137/s1064827501380630.
  * Carrassi et al. [2017] Carrassi A, Bocquet M, Bertino L, Evensen G.  Data assimilation in the geosciences - an overview on methods, issues and perspectives 2017;arXiv:1709.02798.
  * Cengel and Ghajar [2019] Cengel Y, Ghajar A.  Heat and Mass Transfer: Fundamentals and Applications.  6th ed.  McGraw Hill, 2019.
  * Chai and Wilhite [2014] Chai PR, Wilhite AW.  Cryogenic thermal system analysis for orbital propellant depot 2014;102:35--46.  URL: https://linkinghub.elsevier.com/retrieve/pii/S0094576514001738. doi:10.1016/j.actaastro.2014.05.013.
  * Chang and Lin [2011] Chang CC, Lin CJ.  LIBSVM.  ACM Transactions on Intelligent Systems and Technology 2011;2(3):1--27.  doi:10.1145/1961189.1961199.
  * Chatzilygeroudis et al. [2019] Chatzilygeroudis K, Vassiliades V, Stulp F, Calinon S, Mouret JB.  A survey on policy search algorithms for learning robot controllers in a handful of trials.  2019\.  arXiv:1807.02303.
  * Chen et al. [2018] Chen RTQ, Rubanova Y, Bettencourt J, Duvenaud D.  Neural ordinary differential equations.  In: Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook, NY, USA: Curran Associates Inc.; NIPS’18; 2018\. p. 6572–6583.
  * Chen et al. [1990] Chen S, Billings S, Grant P.  Non-linear system identification using neural networks.  International Journal of Control 1990;51(6):1191--214.  doi:10.1080/00207179008934126.
  * Cheng et al. [2016] Cheng B, Tobalske BW, Powers DR, Hedrick TL, Wethington SM, Chiu GT, Deng X.  Flight mechanics and control of escape manoeuvres in hummingbirds. i. flight kinematics.  Journal of Experimental Biology 2016;219(22):3518--31.
  * Cheng et al. [2023] Cheng S, Quilodran-Casas C, Ouala S, Farchi A, Liu C, Tandeo P, Fablet R, Lucor D, Iooss B, Brajard J, Xiao D, Janjic T, Ding W, Guo Y, Carrassi A, Bocquet M, Arcucci R.  Machine learning with data assimilation and uncertainty quantification for dynamical systems: a review.  2023\.  URL: http://arxiv.org/abs/2303.10462; arXiv:2303.10462 [cs].
  * Chinesta et al. [2020] Chinesta F, Cueto E, Abisset-Chavanne E, Duval JL, Khaldi FE.  Virtual, Digital and Hybrid Twins: A New Paradigm in Data-Based Engineering and Engineered Data.  Arch Computat Methods Eng 2020;27(1):105--34.  URL: http://link.springer.com/10.1007/s11831-018-9301-4. doi:10.1007/s11831-018-9301-4.
  * Deisenroth and Rasmussen [2011] Deisenroth M, Rasmussen C.  Pilco: A model-based and data-efficient approach to policy search.  2011\. p. 465--72.
  * Dickinson et al. [1999] Dickinson MH, Lehmann FO, Sane SP.  Wing rotation and the aerodynamic basis of insect flight.  Science 1999;284(5422):1954--60.
  * Dimet et al. [2016] Dimet FXL, Navon IM, Ştefănescu R.  Variational data assimilation: Optimization and optimal control.  In: Data Assimilation for Atmospheric, Oceanic and Hydrologic Applications (Vol. III). Springer International Publishing; 2016. p. 1--53.  doi:10.1007/978-3-319-43415-5_1.
  * Errico [1997] Errico RM.  What is an adjoint model?  Bulletin of the American Meteorological Society 1997;78(11).
  * Evensen [2009] Evensen G.  Data Assimilation: The Ensemble Kalman Filter.  2nd ed.  Spinger, 2009.
  * Fahim et al. [2022] Fahim M, Sharma V, Cao TV, Canberk B, Duong TQ.  Machine learning-based digital twin for predictive modeling in wind turbines.  IEEE Access 2022;10:14184--94.  URL: https://api.semanticscholar.org/CorpusID:246420062.
  * Fei et al. [2019] Fei F, Tu Z, Yang Y, Zhang J, Deng X.  Flappy hummingbird: An open source dynamic simulation of flapping wing robots and animals.  In: 2019 International Conference on Robotics and Automation (ICRA). IEEE; 2019. p. 9223--9.
  * Geer [2021] Geer AJ.  Learning earth system models from observations: machine learning or data assimilation?  Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 2021;379(2194).  doi:10.1098/rsta.2020.0089.
  * Gonzalez and Yu [2018] Gonzalez J, Yu W.  Non-linear system modeling using LSTM neural networks.  IFAC-PapersOnLine 2018;51(13):485--9.  doi:10.1016/j.ifacol.2018.07.326.
  * Goodfellow et al. [2016] Goodfellow I, Bengio Y, Courville A, Bengio Y.  Deep learning.  volume 1.  MIT press Cambridge, 2016.
  * Haarnoja et al. [2018] Haarnoja T, Zhou A, Abbeel P, Levine S.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  2018\.  arXiv:1801.01290.
  * Haghshenas et al. [2023] Haghshenas A, Hasan A, Osen O, Mikalsen ET.  Predictive digital twin for offshore wind farms.  Energy Informatics 2023;6(1):1--26.
  * Haider et al. [2020] Haider N, Shahzad A, Qadri MNM, Shah SIA.  Recent progress in flapping wings for micro aerial vehicle applications.  Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science 2020;235(2):245--64.  doi:10.1177/0954406220917426.
  * Hastings et al. [2005] Hastings L, Tucker S, Flachbart R, Hedayat A, Nelson S.  Marshall space flight center in-space cryogenic fluid management program overview.  In: 41st AIAA/ASME/SAE/ASEE Joint Propulsion Conference and Exhibit. American Institute of Aeronautics and Astronautics; 2005. doi:10.2514/6.2005-3561.
  * Hastings et al. [2003] Hastings LJ, Flachbart R, Martin J, Hedayat A, Fazah M, Lak T, Nguyen H, Bailey J.  Spray Bar Zero-Gravity Vent System for On-Orbit Liquid Hydrogen Storage.  Technical Memorandum (TM) NASA/TM-2003-212926; National Aeronautics and Space Administration, Marshall Space Flight Center; Alabama 35812; 2003.  URL: https://ntrs.nasa.gov/api/citations/20040000092/downloads/20040000092.pdf.
  * Hedengren et al. [2014] Hedengren JD, Shishavan RA, Powell KM, Edgar TF.  Nonlinear modeling, estimation and predictive control in APMonitor.  Computers & Chemical Engineering 2014;70:133--48.  URL: https://linkinghub.elsevier.com/retrieve/pii/S0098135414001306. doi:10.1016/j.compchemeng.2014.04.013.
  * Hochreiter and Schmidhuber [1997] Hochreiter S, Schmidhuber J.  Long short-term memory.  Neural Computation 1997;9(8):1735--80.  doi:10.1162/neco.1997.9.8.1735.
  * Howlader et al. [2010] Howlader AM, Urasaki N, Uchida K, Yona A, Senjyu T, Kim CH, Saber AY.  Parameter Identification of Wind Turbine for Maximum Power-point Tracking Control.  Electric Power Components and Systems 2010;38(5):603--14.  URL: http://www.tandfonline.com/doi/abs/10.1080/15325000903376974. doi:10.1080/15325000903376974.
  * Hunt et al. [1992] Hunt K, Sbarbaro D, Żbikowski R, Gawthrop P.  Neural networks for control systems—a survey.  Automatica 1992;28(6):1083--112.  URL: https://www.sciencedirect.com/science/article/pii/000510989290053I. doi:https://doi.org/10.1016/0005-1098(92)90053-I.
  * Imai et al. [2020] Imai R, Nishida K, Kawanami O, Umemura Y, Himeno T.  Ground based experiment and numerical calculation on thermodynamic vent system in propellant tank for future cryogenic propulsion system.  Cryogenics 2020;109:103095.  doi:10.1016/j.cryogenics.2020.103095.
  * Jaeger and Haas [2004] Jaeger H, Haas H.  Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication.  Science 2004;304(5667):78--80.  doi:10.1126/science.1091277.
  * Jiang et al. [2021] Jiang W, Sun P, Li P, Zuo Z, Huang Y.  Transient thermal behavior of multi-layer insulation coupled with vapor cooled shield used for liquid hydrogen storage tank.  Energy 2021;231:120859.  doi:10.1016/j.energy.2021.120859.
  * Johnson [2004] Johnson KE.  Adaptive torque control of variable speed wind turbines.  University of Colorado at Boulder, 2004\.
  * Johnson [2006] Johnson KE.  Control of variable-speed wind turbines: standard and adaptive techniques for maximizing energy capture.  IEEE Control Systems Magazine 2006;26(3):70--81.
  * Jonkman [2006] Jonkman BJ.  TurbSim user’s guide.  Technical Report; National Renewable Energy Lab.(NREL), Golden, CO (United States); 2006.
  * Jonkman et al. [2009] Jonkman J, Butterfield S, Musial W, Scott G.  Definition of a 5-MW reference wind turbine for offshore system development.  Technical Report; National Renewable Energy Lab.(NREL), Golden, CO (United States); 2009.
  * Kalnay et al. [2007] Kalnay E, Li H, Miyoshi T, Yang SC, Ballabrera-Poy J.  4-d-var or ensemble kalman filter?  Tellus A: Dynamic Meteorology and Oceanography 2007;59(5):758.  doi:10.1111/j.1600-0870.2007.00261.x.
  * Khandelwal et al. [2021] Khandelwal P, Nadler P, Arcucci R, Knottenbelt W, Guo YK.  A scalable inference method for large dynamic economic systems 2021;arXiv:2110.14346.
  * Kruyt et al. [2014] Kruyt JW, Quicazán-Rubio EM, Van Heijst GF, Altshuler DL, Lentink D.  Hummingbird wing efficacy depends on aspect ratio and compares with helicopter rotors.  Journal of the royal society interface 2014;11(99):20140585.
  * Kurutach et al. [2018] Kurutach T, Clavera I, Duan Y, Tamar A, Abbeel P.  Model-ensemble trust-region policy optimization.  arXiv preprint arXiv:180210592 2018;.
  * Lahoz et al. [2010] Lahoz W, Khattatov B, Menard R, editors.  Data Assimilation.  Springer Berlin Heidelberg, 2010\.  doi:10.1007/978-3-540-74703-1.
  * Laks et al. [2009] Laks JH, Pao LY, Wright AD.  Control of wind turbines: Past, present, and future.  In: 2009 American Control Conference. IEEE; 2009. doi:10.1109/acc.2009.5160590.
  * LeCun et al. [1989] LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD.  Backpropagation applied to handwritten zip code recognition.  Neural Computation 1989;1(4):541--51.  doi:10.1162/neco.1989.1.4.541.
  * Lee et al. [2016] Lee Y, Lua KB, Lim T, Yeo K.  A quasi-steady aerodynamic model for flapping flight with improved adaptability.  Bioinspiration & biomimetics 2016;11(3):036005.
  * Leishman [2002] Leishman JG.  Challenges in modelling the unsteady aerodynamics of wind turbines.  Wind Energy: An International Journal for Progress and Applications in Wind Power Conversion Technology 2002;5(2-3):85--132.
  * Lemmon et al. [2018] Lemmon EW, Bell IH, Huber ML, McLinden MO.  NIST Standard Reference Database 23: Reference Fluid Thermodynamic and Transport Properties-REFPROP, Version 10.0, National Institute of Standards and Technology.  2018\.  URL: https://www.nist.gov/srd/refprop. doi:https://doi.org/10.18434/T4/1502528.
  * Lillicrap et al. [2019] Lillicrap TP, Hunt JJ, Pritzel A, Heess N, Erez T, Tassa Y, Silver D, Wierstra D.  Continuous control with deep reinforcement learning.  2019\.  arXiv:1509.02971.
  * Lin et al. [1991] Lin C, Dresar N, Hasan M.  Pressure control analysis of cryogenic storage systems.  Journal of Propulsion and Power 1991;20.  doi:10.2514/1.10387.
  * Liu and MacArt [2023] Liu X, MacArt JF.  Adjoint-based machine learning for active flow control.  2023\.  arXiv:2307.09980.
  * Liu and Wang [2021] Liu XY, Wang JX.  Physics-informed dyna-style model-based deep reinforcement learning for dynamic control 2021;doi:10.1098/rspa.2021.0618. arXiv:2108.00128.
  * Ljung [2008] Ljung L.  Perspectives on system identification.  IFAC Proceedings Volumes 2008;41(2):7172--84.  doi:10.3182/20080706-5-kr-1001.01215.
  * Ljung et al. [2020] Ljung L, Andersson CR, Tiels K, Schön TB.  Deep learning and system identification.  IFAC-PapersOnLine 2020;URL: https://api.semanticscholar.org/CorpusID:226118683.
  * Lorenc [1986] Lorenc AC.  Analysis methods for numerical weather prediction.  Quarterly Journal of the Royal Meteorological Society 1986;112(474):1177--94.  doi:10.1002/qj.49711247414.
  * Lorenc et al. [2015] Lorenc AC, Bowler NE, Clayton AM, Pring SR, Fairbairn D.  Comparison of hybrid-4denvar and hybrid-4dvar data assimilation methods for global NWP.  Monthly Weather Review 2015;143(1):212--29.  doi:10.1175/mwr-d-14-00195.1.
  * Luo et al. [2022a] Luo FM, Xu T, Lai H, Chen XH, Zhang W, Yu Y.  A survey on model-based reinforcement learning 2022a;(arXiv:2206.09328).  URL: http://arxiv.org/abs/2206.09328. doi:10.48550/arXiv.2206.09328. arXiv:2206.09328 [cs].
  * Luo et al. [2022b] Luo FM, Xu T, Lai H, Chen XH, Zhang W, Yu Y.  A survey on model-based reinforcement learning.  2022b.  arXiv:2206.09328.
  * Lutter et al. [2019] Lutter M, Ritter C, Peters J.  Deep lagrangian networks: Using physics as model prior for deep learning 2019;arXiv:1907.04490.
  * Lutter et al. [2020] Lutter M, Silberbauer J, Watson J, Peters J.  Differentiable physics models for real-world offline model-based reinforcement learning.  2020\.  arXiv:2011.01734.
  * Madhavan [1993] Madhavan P.  Recurrent neural network for time series prediction.  In: Proceedings of the 15th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. IEEE; 1993. doi:10.1109/iembs.1993.978527.
  * Marques et al. [2023] Marques P, Ahizi S, Mendez MA.  Real Time Data Assimilation for the Thermodynamic Modeling of a Cryogenic Fuel Tank.  In: 36th International Conference on Efficiency, Cost, Optimization, Simulation and Environmental Impact of Energy Systems (ECOS 2023). Las Palmas De Gran Canaria, Spain: ECOS 2023; 2023\. p. 1041--52.  URL: http://www.proceedings.com/069564-0095.html. doi:10.52202/069564-0095.
  * Mer et al. [2016a] Mer S, Fernandez D, Thibault JP, Corre C.  Optimal design of a thermodynamic vent system for cryogenic propellant storage.  Cryogenics 2016a;80:127--37.  doi:10.1016/j.cryogenics.2016.09.012.
  * Mer et al. [2016b] Mer S, Thibault JP, Corre C.  Active insulation technique applied to the experimental analysis of a thermodynamic control system for cryogenic propellant storage.  Journal of Thermal Science and Engineering Applications 2016b;8(2).  doi:10.1115/1.4032761.
  * Mnih et al. [2013a] Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M.  Playing atari with deep reinforcement learning 2013a;arXiv:1312.5602.
  * Mnih et al. [2013b] Mnih V, Kavukcuoglu K, Silver D, Graves A, Antonoglou I, Wierstra D, Riedmiller M.  Playing atari with deep reinforcement learning 2013b;arXiv:http://arxiv.org/abs/1312.5602v1.
  * Mnih et al. [2015] Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, Graves A, Riedmiller M, Fidjeland AK, Ostrovski G, Petersen S, Beattie C, Sadik A, Antonoglou I, King H, Kumaran D, Wierstra D, Legg S, Hassabis D.  Human-level control through deep reinforcement learning.  Nature 2015;518(7540):529--33.  doi:10.1038/nature14236.
  * Moerland et al. [2022a] Moerland TM, Broekens J, Plaat A, Jonker CM.  Model-based reinforcement learning: A survey 2022a;(arXiv:2006.16712).  URL: http://arxiv.org/abs/2006.16712. doi:10.48550/arXiv.2006.16712. arXiv:2006.16712 [cs, stat].
  * Moerland et al. [2022b] Moerland TM, Broekens J, Plaat A, Jonker CM.  Model-based reinforcement learning: A survey.  2022b.  arXiv:2006.16712.
  * Moriarty and Hansen [2005] Moriarty PJ, Hansen AC.  AeroDyn theory manual.  Technical Report; National Renewable Energy Lab., Golden, CO (US); 2005.
  * Motil et al. [2007] Motil S, Meyer M, Tucker S.  Cryogenic fluid management technologies for advanced green propulsion systems.  In: 45th AIAA Aerospace Sciences Meeting and Exhibit. American Institute of Aeronautics and Astronautics; 2007. doi:10.2514/6.2007-343.
  * Nadler et al. [2020] Nadler P, Arcucci R, Guo Y.  A neural sir model for global forecasting.  In: Alsentzer E, McDermott MBA, Falck F, Sarkar SK, Roy S, Hyland SL, editors. Proceedings of the Machine Learning for Health NeurIPS Workshop. PMLR; volume 136 of Proceedings of Machine Learning Research; 2020\. p. 254--66.  URL: https://proceedings.mlr.press/v136/nadler20a.html.
  * Nagabandi et al. [2017] Nagabandi A, Kahn G, Fearing RS, Levine S.  Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning 2017;arXiv:1708.02596.
  * Nelles [2001] Nelles O.  Nonlinear System Identification.  Springer Berlin Heidelberg, 2001\.  doi:10.1007/978-3-662-04323-3.
  * Nicolao [2003] Nicolao GD.  System identification : Problems and perspectives.  In: 12th Workshop on Qualitative Reasoning. 2003\. URL: https://api.semanticscholar.org/CorpusID:26963940.
  * Norgaard et al. [2000] Norgaard M, Ravn O, Poulsen N, Hansen L.  Neural Networks for Modelling and Control of Dynamic Systems.  Advanced Textbooks in Control and Signal Processing. Springer London, 2000.
  * Olatunji et al. [2021] Olatunji OO, Adedeji PA, Madushele N, Jen TC.  Overview of digital twin technology in wind turbine fault diagnosis and condition monitoring.  2021 IEEE 12th International Conference on Mechanical and Intelligent Manufacturing Technologies (ICMIMT) 2021;:201--7URL: https://api.semanticscholar.org/CorpusID:236190314.
  * Ortega-Jiménez and Dudley [2018] Ortega-Jiménez VM, Dudley R.  Ascending flight and decelerating vertical glides in anna’s hummingbirds.  Journal of Experimental Biology 2018;221(24):jeb191171.
  * Panzarella et al. [2004] Panzarella C, Plachta D, Kassemi M.  Pressure control of large cryogenic tanks in microgravity 2004;44(6):475--83.  URL: https://linkinghub.elsevier.com/retrieve/pii/S0011227504000633. doi:10.1016/j.cryogenics.2004.03.009.
  * Panzarella and Kassemi [2003] Panzarella CH, Kassemi M.  On the validity of purely thermodynamic descriptions of two-phase cryogenic fluid storage 2003;484:41--68.  URL: http://www.journals.cambridge.org/abstract_S0022112003004002. doi:10.1017/S0022112003004002.
  * Pao and Johnson [2009] Pao LY, Johnson KE.  A tutorial on the dynamics and control of wind turbines and wind farms.  In: 2009 American Control Conference. IEEE; 2009. doi:10.1109/acc.2009.5160195.
  * Pillonetto et al. [2023] Pillonetto G, Aravkin A, Gedon D, Ljung L, Ribeiro AH, Schön TB.  Deep networks for system identification: a survey 2023;arXiv:2301.12832.
  * Pimenta et al. [2020] Pimenta F, Pacheco J, Branco CM, Teixeira CM, Magalhães F.  Development of a digital twin of an onshore wind turbine using monitoring data.  Journal of Physics: Conference Series 2020;1618(2):022065.  URL: https://dx.doi.org/10.1088/1742-6596/1618/2/022065. doi:10.1088/1742-6596/1618/2/022065.
  * Pino et al. [2023] Pino F, Schena L, Rabault J, Mendez MA.  Comparative analysis of machine learning methods for active flow control.  Journal of Fluid Mechanics 2023;958.  doi:10.1017/jfm.2023.76.
  * Pu and Kalnay [2018] Pu Z, Kalnay E.  Numerical weather prediction basics: Models, numerical methods, and data assimilation.  In: Handbook of Hydrometeorological Ensemble Forecasting. Springer Berlin Heidelberg; 2018\. p. 1--31.  doi:10.1007/978-3-642-40457-3_11-1.
  * Puterman [1994] Puterman ML.  Markov Decision Processes.  Wiley, 1994.  doi:10.1002/9780470316887.
  * Qin et al. [2021] Qin X, Li P, Sun P, Huang Y.  Testing and comparison of a thermodynamic vent system operating in different modes in a liquid nitrogen tank.  Applied Thermal Engineering 2021;197:117393.  doi:10.1016/j.applthermaleng.2021.117393.
  * Rahman et al. [2022] Rahman A, Drgoňa J, Tuor A, Strube J.  Neural ordinary differential equations for nonlinear system identification 2022;arXiv:2203.00120.
  * Raissi et al. [2019] Raissi M, Perdikaris P, Karniadakis GE.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  Journal of Computational physics 2019;378:686--707.
  * Ramesh and Ravindran [2023] Ramesh A, Ravindran B.  Physics-informed model-based reinforcement learning.  2023\.  arXiv:2212.02179.
  * Rasheed et al. [2019] Rasheed A, San O, Kvamsdal T.  Digital Twin: Values, Challenges and Enablers.  2019\.  URL: http://arxiv.org/abs/1910.01719; arXiv:1910.01719 [eess].
  * Rasmussen and Williams [2005] Rasmussen CE, Williams CKI.  Gaussian Processes for Machine Learning.  MIT Press Ltd, 2005.
  * Routray et al. [2016] Routray A, Osuri KK, Pattanayak S, Mohanty UC.  Introduction to data assimilation techniques and ensemble kalman filter.  In: Advanced Numerical Modeling and Data Assimilation Techniques for Tropical Cyclone Prediction. Springer Netherlands; 2016. p. 307--30.  doi:10.5822/978-94-024-0896-6_11.
  * Saint-Drenan et al. [2020] Saint-Drenan YM, Besseau R, Jansen M, Staffell I, Troccoli A, Dubus L, Schmidt J, Gruber K, Simões SG, Heier S.  A parametric model for wind turbine power curves incorporating environmental conditions.  Renewable Energy 2020;157:754--68.  URL: https://www.sciencedirect.com/science/article/pii/S0960148120306613. doi:https://doi.org/10.1016/j.renene.2020.04.123.
  * Salzman [1996] Salzman JA.  Fluid management in space-based systems.  In: Engineering, Construction, and Operations in Space V. American Society of Civil Engineers; 1996\. doi:10.1061/40177(207)71.
  * Sane [2003] Sane SP.  The aerodynamics of insect flight.  Journal of experimental biology 2003;206(23):4191--208.
  * Sane and Dickinson [2001] Sane SP, Dickinson MH.  The control of flight force by a flapping wing: lift and drag production.  Journal of experimental biology 2001;204(15):2607--26.
  * Sastry and Isidori [1989] Sastry SS, Isidori A.  Adaptive control of linearizable systems.  IEEE Transactions on Automatic Control 1989;34(11):1123--31.
  * Schaul et al. [2015] Schaul T, Quan J, Antonoglou I, Silver D.  Prioritized experience replay.  2015\.  doi:10.48550/ARXIV.1511.05952.
  * Schaul et al. [2016] Schaul T, Quan J, Antonoglou I, Silver D.  Prioritized experience replay 2016;arXiv:1511.05952.
  * Schoukens and Ljung [2019] Schoukens J, Ljung L.  Nonlinear system identification: A user-oriented road map.  IEEE Control Systems 2019;39(6):28--99.  doi:10.1109/mcs.2019.2938121.
  * Schwenzer et al. [2021] Schwenzer M, Ay M, Bergs T, Abel D.  Review on model predictive control: an engineering perspective.  Int J Adv Manuf Technol 2021;117(5-6):1327--49.  URL: https://link.springer.com/10.1007/s00170-021-07682-3. doi:10.1007/s00170-021-07682-3.
  * Silver et al. [2016] Silver D, Huang A, Maddison CJ, Guez A, Sifre L, van den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, Dieleman S, Grewe D, Nham J, Kalchbrenner N, Sutskever I, Lillicrap T, Leach M, Kavukcuoglu K, Graepel T, Hassabis D.  Mastering the game of go with deep neural networks and tree search.  Nature 2016;529(7587):484--9.
  * Silver et al. [2018] Silver D, Hubert T, Schrittwieser J, Antonoglou I, Lai M, Guez A, Lanctot M, Sifre L, Kumaran D, Graepel T, Lillicrap T, Simonyan K, Hassabis D.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  Science 2018;362(6419):1140--4.
  * [136] Silver D, Lever G, Heess N, Degris T, Wierstra D, Riedmiller M.  Deterministic Policy Gradient Algorithms ;.
  * Sjöberg et al. [1994] Sjöberg J, Hjalmarsson H, Ljung L.  Neural networks in system identification.  IFAC Proceedings Volumes 1994;27(8):359--82.  doi:10.1016/s1474-6670(17)47737-8.
  * Smola and Schölkopf [2004] Smola AJ, Schölkopf B.  A tutorial on support vector regression.  Statistics and Computing 2004;14(3):199--222.  doi:10.1023/b:stco.0000035301.49549.88.
  * Staffell and Green [2014] Staffell I, Green R.  How does wind farm performance decline with age?  Renewable Energy 2014;66:775--86.  URL: https://www.sciencedirect.com/science/article/pii/S0960148113005727. doi:https://doi.org/10.1016/j.renene.2013.10.041.
  * Stengel [1994] Stengel RF.  Optimal control and estimation.  Courier Corporation, 1994.
  * Sutton [1991] Sutton RS.  Dyna, an integrated architecture for learning, planning, and reacting.  SIGART Bull 1991;2(4):160–163.  URL: https://doi.org/10.1145/122344.122377. doi:10.1145/122344.122377.
  * Sutton and Barto [2018] Sutton RS, Barto AG.  Reinforcement learning: An introduction.  MIT press, 2018.
  * Suykens et al. [1996] Suykens JAK, Vandewalle JPL, Moor BLRD.  Artificial Neural Networks for Modelling and Control of Non-Linear Systems.  Springer US, 1996.  doi:10.1007/978-1-4757-2493-6.
  * Szita [2012] Szita I.  Reinforcement learning in games.  In: Adaptation, Learning, and Optimization. Springer Berlin Heidelberg; 2012. p. 539--77.
  * Taha et al. [2012] Taha HE, Hajj MR, Nayfeh AH.  Flight dynamics and control of flapping-wing mavs: a review.  Nonlinear Dynamics 2012;70:907--39.
  * Talagrand and Courtier [1987] Talagrand O, Courtier P.  Variational assimilation of meteorological observations with the adjoint vorticity equation. i: Theory.  Quarterly Journal of the Royal Meteorological Society 1987;113(478):1311--28.  doi:10.1002/qj.49711347812.
  * Tang and Hsieh [2001] Tang Y, Hsieh WW.  Coupling neural networks to incomplete dynamical systems via variational data assimilation.  Monthly Weather Review 2001;129(4):818--34.  doi:10.1175/1520-0493(2001)129<0818:cnntid>2.0.co;2.
  * Tekinerdogan [2022] Tekinerdogan B.  On the notion of digital twins: A modeling perspective 2022;11(1):15.  URL: https://www.mdpi.com/2079-8954/11/1/15. doi:10.3390/systems11010015.
  * Uc-Cetina et al. [2022] Uc-Cetina V, Navarro-Guerrero N, Martin-Gonzalez A, Weber C, Wermter S.  Survey on reinforcement learning for language processing.  Artificial Intelligence Review 2022;56(2):1543--75.  doi:10.1007/s10462-022-10205-5.
  * Wagner et al. [2019] Wagner R, Schleich B, Haefner B, Kuhnle A, Wartzack S, Lanza G.  Challenges and potentials of digital twins and industry 4.0 in product design and production for high performance products.  Procedia CIRP 2019;84:88--93.  doi:10.1016/j.procir.2019.04.219.
  * Wang et al. [2017] Wang B, Huang Y, Wu J, Wang T, Lei G.  Experimental study on pressure control of liquid nitrogen tank by thermodynamic vent system.  Applied Thermal Engineering 2017;125:1037--46.  URL: https://www.sciencedirect.com/science/article/pii/S1359431117315491. doi:https://doi.org/10.1016/j.applthermaleng.2017.07.067.
  * Wang et al. [2000] Wang B, Zou X, Zhu J.  Data assimilation and its applications.  Proceedings of the National Academy of Sciences 2000;97(21):11143--4.  doi:10.1073/pnas.97.21.11143.
  * Weber et al. [2017] Weber T, Racanière S, Reichert DP, Buesing L, Guez A, Rezende DJ, Badia AP, Vinyals O, Heess N, Li Y, Pascanu R, Battaglia P, Hassabis D, Silver D, Wierstra D.  Imagination-augmented agents for deep reinforcement learning 2017;arXiv:1707.06203.
  * Werner and Peitz [2023] Werner S, Peitz S.  Learning a model is paramount for sample efficiency in reinforcement learning control of pdes 2023;arXiv:2302.07160.
  * Whitney and Wood [2010] Whitney JP, Wood RJ.  Aeromechanics of passive rotation in flapping flight.  Journal of fluid mechanics 2010;660:197--220.
  * Willard et al. [2020] Willard J, Jia X, Xu S, Steinbach M, Kumar V.  Integrating scientific knowledge with machine learning for engineering and environmental systems 2020;doi:10.48550/ARXIV.2003.04919.
  * Wright and Davidson [2020] Wright L, Davidson S.  How to tell the difference between a model and a digital twin.  Advanced Modeling and Simulation in Engineering Sciences 2020;7(1).  doi:10.1186/s40323-020-00147-4.
  * Xue et al. [2023] Xue Y, Cai X, Xu R, Liu H.  Wing kinematics-based flight control strategy in insect-inspired flight systems: Deep reinforcement learning gives solutions and inspires controller design in flapping mavs.  Biomimetics 2023;8(3):295.
  * Zhang and Moore [1991] Zhang C, Moore KL.  System identification using neural networks.  [1991] Proceedings of the 30th IEEE Conference on Decision and Control 1991;:873--874 vol.1URL: https://api.semanticscholar.org/CorpusID:60798054.
  * Zhang and Constantinescu [2023] Zhang H, Constantinescu EM.  Optimal checkpointing for adjoint multistage time-stepping schemes.  Journal of Computational Science 2023;66:101913.  doi:10.1016/j.jocs.2022.101913.
  * Zheng and Jin [2022] Zheng X, Jin T.  A reliable method of wind power fluctuation smoothing strategy based on multidimensional non-linear exponential smoothing short-term forecasting.  IET Renewable Power Generation 2022;16(16):3573--86.  doi:10.1049/rpg2.12395.


11institutetext: Ludwig Maximilian University of Munich, Germany
22institutetext: Siemens AG, Germany 33institutetext: Technical University of
Munich, Germany
$\,{}^{*}$ equal contribution

# Graphhopper: Multi-Hop Scene Graph Reasoning for Visual Question Answering

Rajat Koner *1*1 Hang Li *1*122 Marcel Hildebrandt *1*122 Deepan Das 33 Volker
Tresp 1122 Stephan Günnemann 33

###### Abstract

Visual Question Answering (VQA) is concerned with answering free-form
questions about an image. Since it requires a deep semantic and linguistic
understanding of the question and the ability to associate it with various
objects that are present in the image, it is an ambitious task and requires
multi-modal reasoning from both computer vision and natural language
processing. We propose Graphhopper, a novel method that approaches the task by
integrating knowledge graph reasoning, computer vision, and natural language
processing techniques. Concretely, our method is based on performing context-
driven, sequential reasoning based on the scene entities and their semantic
and spatial relationships. As a first step, we derive a scene graph that
describes the objects in the image, as well as their attributes and their
mutual relationships. Subsequently, a reinforcement learning agent is trained
to autonomously navigate in a multi-hop manner over the extracted scene graph
to generate reasoning paths, which are the basis for deriving answers. We
conduct an experimental study on the challenging dataset GQA, based on both
manually curated and automatically generated scene graphs. Our results show
that we keep up with human performance on manually curated scene graphs.
Moreover, we find that Graphhopper outperforms another state-of-the-art scene
graph reasoning model on both manually curated and automatically generated
scene graphs by a significant margin.

###### Keywords:

Visual Question Answering (VQA) Knowledge Graph Reasoning Scene Graph
Reasoning Multi-Modal Reasoning Reinforcement Learning

##  1 Introduction

Figure 1: Example of an image and the corresponding scene graph. Since the
scene graph is a directed graph with typed edges, it resembles a knowledge
graph and permits the application of knowledge-base completion techniques.

Visual Question Answering (VQA) is a challenging task that involves
understanding and reasoning over two data modalities, i.e., images and natural
language. Given an image and a free-form question which formulates a query
about the presented scene — the issue is for the algorithm to find the correct
answer.

VQA has been studied from the perspective of scene and knowledge graphs [33,
6], as well as vision-language reasoning [10, 1]. To study VQA, various real-
world data sets, such as the VQA data set [4, 24], have been generated. It has
been argued that, in the VQA data set, many of the apparently challenging
reasoning tasks can be solved by an algorithm through exploiting trivial prior
knowledge, and thus by shortcuts to proper reasoning (e.g., clouds are white
or doors are made of wood). To address these shortcomings, the GQA dataset
[17] has been developed. Compared to other real-world datasets, GQA is more
suitable for evaluating reasoning abilities since the images and questions are
carefully filtered to make the data less prone to biases.

Plenty of VQA approaches are agnostic towards the explicit relational
structure of the objects in the presented scene and rely on monolithic neural
network architectures that process regional features of the image separately
[2, 39]. While these methods led to promising results on previous datasets,
they lack explicit compositional reasoning abilities, which results in weaker
performance on more challenging datasets such as GQA. Other works [34, 31, 15]
perform reasoning on explicitly detected objects and interactive semantic and
spatial relationships among them. These approaches are closely related to the
scene graph representations [19] of an image, where detected objects are
labeled as nodes and relationships between the objects are labeled as edges.
In this work, we aim to combine VQA techniques with recent research advances
in the area of statistical relation learning on knowledge graphs (KGs). KGs
provide human-understandable, structured representations of knowledge about
the real world via collections of factual statements. Inspired by multi-hop
reasoning methods on KGs such as [8, 38, 12], we propose Graphhopper, a novel
method that models the VQA task as a path-finding problem on scene graphs. The
underlying idea can be summarized with the phrase: Learn to walk to the
correct answer. More specifically, given an image, we consider a scene graph
and train a reinforcement learning agent to conduct a policy-guided random
walk on the scene graph until a conclusive inference path is obtained. In
contrast to purely embedding-based approaches, our method provides explicit
reasoning chains that lead to the derived answers. To sum up, our major
contributions are as follows.

  * •

Graphhopper is the first VQA method that employs reinforcement learning for
multi-hop reasoning on scene graphs.

  * •

We conduct a thorough experimental study on the challenging VQA dataset named
QGA to show the compositional and interpretable nature of our model.

  * •

To analyze the reasoning capabilities of our method, we consider manually
curated (ground truth) scene graphs. This setting isolates the noise
associated with the visual perception task and focuses solely on the language
understanding and reasoning task. Thereby, we can show that our method
achieves human-like performance.

  * •

Based on both the manually curated scene graphs and our own automatically
generated scene graphs, we show that Graphhopper outperforms the Neural State
Machine (NMS), a state-of-the-art scene graph reasoning model that operates in
a setting, similar to Graphhopper.

Moreover, we are the first group to conduct experiments and publish the code
on generated scene graphs for the GQA dataset111Code is available at
:https://github.com/rajatkoner08/Graphhopper.The remainder of this work is
organized as follows. We review related literature in the next section.
Section 3 introduces the notation and describes the methodology of
Graphhopper. Section 4 and Section 5 detail an experimental study on the
benchmark dataset GQA. Furthermore, through a rigorous study using both
manually-curated ground-truth and generated scene graphs, we examine the
reasoning capabilities of Graphhopper. We conclude in Section 6.

##  2 Related Work

#### Visual Question Answering:

Various models have been proposed that perform VQA on both real-world [4, 17]
and artificial datasets [18]. Currently, leading VQA approaches can be
categorized into two different branches: First, monolithic neural networks,
which perform implicit reasoning on latent representations obtained from
fusing the two data modalities. Second, multi-hop methods that form explicit
symbolic reasoning chains on a structured representation of the data.
Monolithic network architectures obtain visual features from the image either
in the form of individual detected objects or by processing the whole image
directly via convolutional neural networks (CNNs). The derived embeddings are
usually scored against a fixed answer set along with the embedding of the
question obtained from a sequence model. Moreover, co-attention mechanisms are
frequently employed to couple the vision and the language models allowing for
interactions between objects from both modalities [20, 2, 5, 40, 41].
Monolithic networks are among the dominant methods on previous real-world VQA
datasets such as [4]. However, they suffer from the black-box problem and
possess limited reasoning capabilities with respect to complex questions that
require long reasoning chains (see [7] for a detailed discussion).

Explicit reasoning methods combine the sub-symbolic representation learning
paradigm with symbolic reasoning approaches over structured representations of
the image. Most of the popular explicit reasoning approaches follow the idea
of neural module networks (NMNs) [3] which perform a sequence of reasoning
steps realized by forward passes through specialized neural networks that each
correspond to predefined reasoning subtasks. Thereby, NMNs construct
functional programs by dynamically assembling the modules resulting in a
question-specific neural network architecture. In contrast to the monolithic
neural network architectures described above, these methods contain a natural
transparency mechanism via functional programs. However, while NMN-related
methods (e.g., [14, 26]) exhibit good performance on synthetic datasets such
as CLEVR [18], they require functional module layouts as additional
supervision signals to obtain good results. Closely related to our method is
the Neural State Machine (NSM) proposed by [16]. NSM’s underlying idea
consists of first constructing a scene graph from an image and treating it as
a state machine. Concretely, the nodes correspond to states and edges to
transitions. Then, conditioned on the question, a sequence of instructions is
derived that indicates how to traverse the scene graph and arrive at the
answer. In contrast to NSM, we treat path-finding as a decision problem in a
reinforcement learning setting. Concretely, we outline in the next section how
extracting predictive paths from scene graphs can be naturally formulated in
terms of a goal-oriented random walk induced by a stochastic policy that
allows the approach to balance between exploration and exploitation. Moreover,
our framework integrates state-of-the-art techniques from graph representation
learning and NLP. This paper only considers basic policy gradient methods, but
more sophisticated reinforcement learning techniques will be employed in
future works.

#### Statistical Relational Learning:

Machine learning methods for KG reasoning aim at exploiting statistical
regularities in observed connectivity patterns. These methods are studied
under the umbrella of statistical relational learning (SRL) [27]. In recent
years, KG embeddings have become the dominant approach in SRL. The underlying
idea is that graph features that explain the connectivity pattern of KGs can
be encoded in low-dimensional vector spaces. In the embedding spaces, the
interactions among the embeddings for entities and relations can be
efficiently modeled to produce scores that predict the validity of a triple.
Despite achieving good results in KG reasoning tasks, most embedding-based
methods have problems capturing the compositionality expressed by long
reasoning chains. This often limits their applicability in complex reasoning
tasks. Recently, multi-hop reasoning methods such as MINERVA [8] and DeepPath
[38] were proposed. Both methods are based on the idea that a reinforcement
learning agent is trained to perform a policy-guided random walk until the
answer entity to a query is reached. Thereby, the path finding problem of the
agent can be modeled in terms of a sequential decision making task framed as a
Markov decision process (MDP). The method that we propose in this work follows
a similar philosophy, in the sense that we train an RL agent to navigate on a
scene graph to the correct answer node. However, a conceptual difference is
that the agents in MINERVA and DeepPath perform walks on large-scale knowledge
graphs exploiting repeating statistical patterns. Thereby, the policies
implicitly incorporate approximate rules. In addition, instead of free-form
processing questions, the query in the KG reasoning setting is structured as a
pair of symbolic entities. That is why we propose a wide range of
modifications to adjust our method to the challenging VQA setting.

##  3 Method

The task of VQA is framed as a scene graph traversal problem. Starting from a
hub node that is connected to all other nodes, an agent sequentially samples
transitions to neighboring nodes on the scene graph until the node
corresponding to the answer is reached. In this way, by adding transitions to
the current path, the reasoning chain is successively extended. Before
describing the decision problem of the agent, we introduce the notation that
we use throughout this work.

#### Notation:

A scene graph is a directed multigraph where each node corresponds to a scene
entity which is either an object associated with a bounding box or an
attribute of an object. Each scene entity comes with a type that corresponds
to the predicted object or attribute label. Typed edges specify how scene
entities are related to each other. More formally, let $\mathcal{E}$ denote
the set of scene entities and consider the set of binary relations
$\mathcal{R}$. Then a scene graph
$\mathcal{SG}\subset\mathcal{E}\times\mathcal{R}\times\mathcal{E}$ is a
collection of ordered triples $(s,p,o)$ – subject, predicate, and object. For
example, as shown in Figure 1, the triple (motorcycle-1, has_part, tire-1)
indicates that both a motorcycle (subject) and a tire (object) are detected in
the image. The predicate has_part indicates the relation between the entities.
Moreover, we denote with $p^{-1}$ the inverse relation corresponding to the
predicate $p$. For the remainder of this work, we impose completeness with
respect to inverse relations in the sense that for every
$(s,p,o)\in\mathcal{SG}$ it is implied that $(o,p^{-1},s)\in\mathcal{SG}$.

Figure 2: The architecture of our scene graph reasoning module.

#### Environment

The state space of the agent $\mathcal{S}$ is given by
$\mathcal{E}\times\mathcal{Q}$ where $\mathcal{E}$ are the nodes of a scene
graph $\mathcal{SG}$ and $\mathcal{Q}$ denotes the set of all questions. The
state at time $t$ is the entity $e_{t}$ at which the agent is currently
located and the question $Q$. Thus, a state $S_{t}\in\mathcal{S}$ for time
$t\in\mathbb{N}$ is represented by $S_{t}=\left(e_{t},Q\right)$. The set of
available actions from a state $S_{t}$ is denoted by $\mathcal{A}_{S_{t}}$. It
contains all outgoing edges from the node $e_{t}$ together with their
corresponding object nodes. More formally,
$\mathcal{A}_{S_{t}}=\left\\{(r,e)\in\mathcal{R}\times\mathcal{E}:S_{t}=\left(e_{t},Q\right)\land\left(e_{t},r,e\right)\in\mathcal{SG}\right\\}\,.$
Moreover, we denote with $A_{t}\in\mathcal{A}_{S_{t}}$ the action that the
agent performed at time $t$. We include self-loops for each node in
$\mathcal{SG}$ that produce a NO_OP-label. These self-loops allow the agent to
remain at the current location if it reaches the answer node. Furthermore, the
introduction of inverse relations allows agent to transit freely in any
direction between two nodes.

The environments evolve deterministically by updating the state according to
previous action. Formally, the transition function at time $t$ is given by
$\delta_{t}({S_{t}},A_{t}):=\left(e_{t+1},Q\right)$ with
$S_{t}=\left(e_{t},Q\right)$ and $A_{t}=\left(r,e_{t+1}\right)$.

Auxiliary Nodes :  In addition to standard entity relation nodes present in a
scene graph, we introduce a few auxiliary nodes (e.g. hub node). The
underlying rationale for the inclusion of auxiliary nodes is that they
facilitate the walk for the agent or help to frame the QA-task as a goal-
oriented walk on the scene graph. These additional nodes are included during
run-time graph traversal, but they are ignored during the compile time such as
when computing node embedding. For example, we add a hub node (hub) to every
scene graph which is connected to all other nodes. The agent then starts the
scene graph traversal from a hub with global connectivity. Furthermore for a
binary question, we add YES and NO nodes to the scene entities that correspond
to the final location of the agent. The agent can then transition to either
the YES or the NO node.

#### Question and Scene Graph Processing

We initialize words in $Q$ with GloVe embeddings [29] with dimension $d=300$.
Similarly we initialize entities and relations in $\mathcal{SG}$ with the
embeddings of their type labels. In the scene graph, the node embeddings are
passed through a multi-layered graph attention network (GAT) [36]. Extending
the idea from graph convolutional networks [22] with a self-attention
mechanism, GATs mimic the convolution operator on regular grids where an
entity embedding is formed by aggregating node features from its neighbors.
Relations and inverse relations between nodes allows context to flow in both
ways through GAT. Thus, the resulting embeddings are context-aware, which
makes nodes with the same type, but different graph neighborhoods,
distinguishable. To produce an embedding for the question $Q$, we first apply
a Transformer [35], followed by a mean pooling operation.

Finally, since we added auxiliary YES and NO nodes to the scene graph for
binary questions, we train a feedforward neural network to classify query-type
(i.e., questions that query for an object in the depicted scene) and binary
questions. This network consists of two fully connected layers with ReLU
activation on the intermediate output. We find that it is easy to distingquish
between query and binary questions (e.g., query questions usually begin with
What, Which, How, etc., whereas binary questions usually begin with Do, Is,
etc.). Since our classifier achieves 99.99% accuracy we will ignore the error
in question classification in the following discussions.

#### Policy

We denote the agent’s history until time $t$ with the tuple
$H_{t}=\left(H_{t-1},A_{t-1}\right)$ for $t\geq 1$ and $H_{0}=hub$ along with
$A_{0}=\emptyset$ for $t=0$. The history is encoded via a multilayered LSTM
[13]

$\mathbf{h}_{t}=\textrm{LSTM}\left(\mathbf{a}_{t-1}\right)\,,$ (1)

where
$\mathbf{a}_{t-1}=\left[\mathbf{r}_{t-1},\mathbf{e}_{t}\right]\in\mathbb{R}^{2d}$
corresponds to the embedding of the previous action with $\mathbf{r}_{t-1}$
and $\mathbf{e}_{t}$ denoting the embeddings of the edge and the target node
into $\mathbb{R}^{d}$, respectively. The history-dependent action distribution
is given by

$\mathbf{d}_{t}=\textrm{softmax}\left(\mathbf{A}_{t}\left(\mathbf{W}_{2}\textrm{ReLU}\left(\mathbf{W}_{1}\left[\mathbf{h}_{t},\mathbf{Q}\right]\right)\right)\right)\,,$
(2)

where the rows of $\mathbf{A}_{t}\in\mathbb{R}^{|\mathcal{A}_{S_{t}}|\times
d}$ contain latent representations of all admissible actions. Moreover,
$\mathbf{Q}\in\mathbb{R}^{d}$ encodes the question $Q$. The action
$A_{t}=(r,e)\in\mathcal{A}_{S_{t}}$ is drawn according to
$\textrm{categorical}\left(\mathbf{d}_{t}\right)$. Equations (1) and (2)
induce a stochastic policy $\pi_{\theta}$, where $\theta$ denotes the set of
trainable parameters.

#### Rewards and Optimization

After sampling $T$ transitions, a terminal reward is assigned according to

$R=\begin{cases}1&\text{if $e_{T}$ is the answer to $Q$,}\\\
0&\text{otherwise.}\end{cases}$ (3)

We employ REINFORCE [37] to maximize the expected rewards. Thus, the agent’s
maximization problem is given by

$\operatorname*{arg\,max}_{\theta}\mathbb{E}_{Q\sim\mathcal{T}}\mathbb{E}_{A_{1},A_{2},\dots,A_{N}\sim\pi_{\theta}}\left[R\left|\vphantom{\frac{1}{1}}\right.e_{c}\right]\,,$
(4)

where $\mathcal{T}$ denote the set of training questions. During training the
first expectation in Equation (4) is substituted with the empirical average
over the training set. The second expectation is approximated by the empirical
average over multiple rollouts. We also employ a moving average baseline to
reduce the variance. Further, we use entropy regularization with parameter
$\lambda\in\mathbb{R}_{\geq 0}$ to enforce exploration. During inference, we
do not sample paths but perform a beam search with width 20 based on the
transition probabilities given by Equation (2).

Additional details on the model, the training and the inference procedure
along with sketches of the algorithms, and a complexity analysis can be found
in the supplementary material.

##  4 Dataset and Experimental Setup

In this section we introduce the dataset and detail the experimental protocol.

###  4.1 Dataset

The GQA dataset [17] has been introduced with the goal of addressing key
shortcomings of previous VQA datasets, such as CLEVR [18] or the VQA dataset
[4]. GQA is more suitable for evaluating the reasoning and compositional
abilities of a model in a realistic setting. It contains 113K images, and
around 1.2M questions split into roughly $80\%/10\%/10\%$ for the training,
validation, and testing. The overall vocabulary size consists of 3097 words,
including 1702 object classes, 310 relationships, and 610 object attributes.
Due to the large number of objects and relationships present in GQA, we used a
pruned version of the dataset (see Section 5) for our generated scene graph.
In this work, we have conducted two primary experiments. First, we report the
results on manually curated scene graphs provided in the GQA dataset. In this
setting, the true reasoning and language understanding capabilities of our
model can be analyzed. Afterward, we evaluate the performance of our model
with the generated scene graphs on pruned GQA dataset. It shows the
performance of our model on noisy generated data. We have used state of the
art Relation Transformer Network (RTN) [23] for the scene graph generation and
DetectoRS [30] for object detection. We have conducted all the experiments on
“test-dev” split of the GQA.

#### Question Types:

The questions are designed to evaluate the reasoning abilities such as visual
verification, relational reasoning, spatial reasoning, comparison, and logical
reasoning. These questions can be categorized either according to structural
or semantic criteria. An overview of the different question types is given in
supplementary (see Table 4).

###  4.2 Experimental Setup

#### Scene Graph Reasoning:

Regarding the model parameters, we apply 300 dimensional GloVe embeddings to
both the questions and the graphs (i.e., edges and nodes). Moreover, we employ
a two-layer GAT [36] model. The dropout [32] probability of each layer is set
to 0.1. The first layer has eight attention heads. Each head has eight latent
features which are concatenated to form the output features of that layer. The
output layer has eight attention heads with mean aggregation, so that the
output also has 300-dimensional features. We apply dropout with $p=0.1$ to the
attention coefficients at each layer. This essentially means that each node is
exposed to a stochastically sampled neighborhood during training. Moreover, we
employ a two-layer Transformer [35] decoder model. The model dimension is set
to 300, and the key and query dimensions are both set to 64 with dropout
$p=0.1$ . The LSTM of the policy networks consists of a uni-directional layer
with hidden size 300. Finally, the agent performs a fixed number of
transitions. In question answering, most questions concern one subject to be
explored within one reasoning path originated from the start node. Hence, we
set the maximum number of steps to 4, without resetting. By contrast, the
binary questions have 8 steps and a reset frequency of 4. In other words, the
agent is prompted to the hub node after the fourth step.

#### Training the Graphhopper:

In terms of the training procedure, the GAT, the Transformer, and the policy
networks are initialized with Glorot [11] initialization. We train our model
with data from the val_balanced_questions tier. We use a batch size of 64 and
sample a batch of questions along with their associated graphs. We collect 20
stochastic rollouts for each question performed in a vectorized form to
utilize parallel computation. For each batch, we collect the rewards when a
complete forward pass is done. Then the gradients are approximated from the
rewards and applied to update the weights. We employ the Adam optimizer [21]
with a learning rate of $10^{-4}$ for all trainable weights. The coefficient
for the action entropy, which balances exploration and exploitation, starts
from 0.2 and decreases exponentially at each step with a factor 0.99.

Next to other standard Python libraries, we mainly employed PyTorch [28]. All
experiments were conducted on a machine with one NVIDIA RTX 2080 Ti GPU and 64
GB RAM. Training the scene graph reasoner of Graphhopper for 40 epochs on GQA
takes around 10 hours, testing about 1 hour.

###  4.3 Performance Metrics

Along with the accuracy (i.e., Hits@1) on open questions (“Open”), binary
questions (yes/no) (“Binary”), and the overall accuracy (“Accuracy”), we also
report the additional metric “Consistency” (answers should not contradict
themselves), “Validity” (answers are in the range of a question; e.g., red is
a valid answer when asked for the color of an object), “Plausibility” (answers
should be reasonable; e.g., red is a reasonable color of an apple reasonable,
blue is not), as proposed in [17].

##  5 Results and Discussion

As outlined before, VQA is a challenging task, and there is still a
significant performance gap between state-of-the-art VQA methods and human
performance on challenging, real-world datasets such as GQA (see [17]).
Similar to other existing methods, our architecture involves multiple
components, and it important to be able to analyse the performance of the
different modules and processing steps in isolation. Therefore we first
present the results of our experiments on manually curated, ground-truth scene
graphs provided in the GQA dataset and compare the performance of Graphhopper
against NSM and humans. This setting allows us to isolate the noise from the
visual perception component and quantify our methods’ reasoning capabilities.
Subsequently, we present the results with our own generated scene graphs.

In addition, we also observed that the inclusion of auxiliary nodes helps the
agent to achieve efficient performance. Hub node performs better compare to
starting from any random nodes, as its facilitate easier forward and
backtracking from a node. For binary question instead of YES or NO node, we
experimented where the path of the agent was processed by another classifier
(e.g., a logistic regression) and the classification logits were assigned as
rewards. However, this led to inferior results; most likely due to the absence
of a weight-sharing mechanism and due to the noisy reward signal produced by
the classifier. These observations supports our assumption on the role of
auxiliary nodes we have used in scene graph.

#### Reproducing NSM:

[15] proposed the state of the art method named NSM for VQA. NSM is the
conceptually most similar method, as it also exploits the scene graph
reasoning for VQA. We consider NSM to be our baseline method for comparison.
However, their approach to reasoning is different from ours. To compare the
reasoning ability of our method with the same generated scene graph, we tried
to reproduce NSM, as the code for NSM is not open-sourced. We have used the
the available parameters from [15] and the implementation from [9].

Table 1: A comparison of Graphhopper with human performance and NSM based on manually curated scene graphs.  Method | Binary | Open | Consistency | Validity | Plausibility | Accuracy
---|---|---|---|---|---|---
Human [17] | 91.2 | 87.4 | 98.4 | 98.9 | 97.2 | 89.3
NSM [15] | 51.03 | 18.79 | 81.36 | 83.69 | 79.12 | 34.5
Graphhopper | 92.18 | 92.40 | 91.92 | 93.68 | 93.13 | 92.30

###  5.1 Results on Manually Curated Scene Graphs

In this section, we report on an experimental study with Graphhopper on the
manually curated scene graphs provided along with the GQA dataset. Table 1
shows the performance of Graphhopper and compares it with the human
performance reported in [17] and with the performance of NSM on the same
underlying manually curated scene graphs. We find that Graphhopper strictly
outperforms NSM with respect to all performance measures. In particular, on
the open questions, the performance gap is significant. Moreover, Graphhopper
also slightly outperforms humans with respect to the accuracy on both types of
questions. On the other hand, concerning the supplementary performance
measures consistency, validity, and plausibility, Graphhopper is outperformed
by humans but nevertheless consistently reaches high values. Overall, these
results can be seen as a testament of the reasoning capabilities and establish
an upper bound to the performance of Graphhopper.

###  5.2 Results on automatically generated graph

The process of generating a graph representation for visual data is a costly
and complex procedure. Although the scene graph generation is not the main
focus of this work, it constituted one of the major challenges to create good
scene graph for GQA due to the following facts:

  * •

There is no open source code for GQA scene graph generation or object
detection.

  * •

A large number of instances and an uneven class distribution in GQA leads to a
significant drop in the accuracy compared to existing scene graph datasets
(see [24]).

  * •

There is a lack of attribute prediction models in modern object detection
frameworks.

In this work, we adress all of these challenges as our model’s performance is
directly dependent on the quality of the scene graph. We will also open-source
our code base for transparency and accelerate the development scene graph-
based reasoning for VQA.

#### Generation of Scene Graph:

To address these problems, first, we choose two state-of-the-art network, RTN
[23] for scene graph generation, and DetectoRS [30] for object detection. The
transformer [35] based architecture of RTN and its contextual scene graph
embedding is most closely related to our architecture and for our future
expansion. To make Graphhopper generic to any scene graph generator, we
haven’t use contextualized embedding from RTN, instead we rely on GAT for
contextualization.

#### Pruning of GQA:

GQA has more than 6 times the number of relationships compared to Visual
Genome [24], which is the most used scene graph generation dataset, and
contains more than 18 times the number of objects compared to the most common
object detection dataset COCO [25]. Also, the class distribution is highly
skewed which causes a significant drop in the accuracy for both the object
detection and the scene graph generation task. To efficiently prune the number
of instances, we take the first 800 classes, 170 relationships, and 200
attributes based on their frequency of occurrence in the training questions
and answers. This pruning allows us to reduce more than $60\%$ of the words
while covering more than $96\%$ of the combined answers in the training set.

#### Attribute prediction:

One of the shortcomings of existing scene graph generation and object
detection networks is that they do not predict the attributes (e.g., the color
or size of an object) of a detected object. Therefore, we have incorporated
the attribute prediction for answering the question on GQA. Contextualized
object embedding from RTN [23] is used for attribute prediction as

$P_{attribute}=\sigma(W(Obj_{context},P_{obj}))\,,$ (5)

where $W$ ,$Obj_{context}$, $P_{obj}$ , $P_{attribute}$ are the weight
matrices of a linear layer, the contextual embedding of an object, the
probability distribution over all objects and the probability distribution
over the attributes. $\sigma$ denotes the sigmoid function.
We have trained both the object detector and the scene graph generator on a
pruned version of GQA with their respective default parameters after the
prepossessing. This helps to increase the coverage of all the instances (e.g.,
objects, attributes, relationships ) on training questions from $52\%$ to
$77\%$ implying that our generated scene graph now covers $77\%$ of all
instances that represent answers to the training questions.

Table 2: A comparison of our method with NSM, based on generated scene graphs. Graphhopper (pr) indicates that we employed predicted relations from RTN [23]. Method | Binary | Open | Consistency | Validity | Plausibility | Accuracy
---|---|---|---|---|---|---
NSM [15] | 51.88 | 19.83 | 82.01 | 86.28 | 81.75 | 35.34
Graphhopper | 69.48 | 44.69 | 83.64 | 89.42 | 85.13 | 56.69
Graphhopper (pr) | 85.84 | 77.27 | 92.98 | 92.26 | 89.50 | 81.41

(a) Question: Is the color of the number the same as that of the wristband?
Answer: No.

(b) Question: What is the name of the appliance that is not small?
Answer: Refrigerator.

(c) Do both the pepper and the vegetable to the right of the ice cube have
green color?
Answer: Yes.

Figure 3: Three examples question and the corresponding images and paths.

(a)

(b)

(c)

Figure 4: Comparison of the performance of our model on various Scene Graph
generation settings, (left) accuracy across various semantic instances
(“Attribute”,“Global”,“Relation” etc) required to answer a question (middle)
accuracy on multiple types of question category (“Choose”,“Logical”,“Verify”
etc) and (right) accuracy on minimum number of steps needed to reach the
answer node.

Table 2, shows the performance of Graphhopper in two settings: First, with a
generated graph where we predict the classes, the attributes, and
relationships using our own pipeline. Second, where we only use the predicted
relationships from RTN [23] (with ground truth objects and attributes). We
find that Graphhopper consistently outperforms NSM [15] based on the generated
graph. Moreover, in the “pr” or predicted relations setting, it achieves an
even higher score as the graphs do not contain any misprediction from the
object detector. These encouraging results show superior reasoning abilities
both on the generated graph and generated relationships between objects.

###  5.3 Discussion on the Reasoning Ability

To further analyze the reasoning abilities of Graphhopper, Figure 4
disentangles the results according to different types of questions: 5 semantic
types (left) and 5 structural types (middle). Moreover, we report the
performance of Graphhopper according to the length of the reasoning path
(right) (see the supplementary material for additional information). Moreover,
we show the performance of Graphhopper separately for each of the three scene
graph settings that we considered in this work. Figure 4(a) shows performance
on a manually curated scene graph that depicts the actual performance in an
ideal environment. Figure 4(b) illustrates the performance based on only the
predicted relationships between objects. This setting shows the performance of
Graphhopper along with a scene graph generator. Finally, Figure 4(c) depicts
the performance based on the object detector, the scene graph generator, and
Graphhopper. First and foremost, we find that Graphhopper consistently
achieves high accuracy on all types of questions in every setting. Moreover,
we find that the performance of Graphhopper does not suffer if answering the
questions requires many reasoning steps. We conjecture that this is because
high-complexity questions are harder to answer, but due to proper
contextualization of the embeddings (e.g., via the GAT and the Transformer),
the agent can extract the specific information that identifies the correct
target node. The good performance on these high-complexity questions can be
seen as evidence that Graphhopper can efficiently translate the question into
a transition on the scene graph hopping until the correct answer is reached.

#### Examples of Reasoning Path:

Figure 3 shows three examples of scene graph traversals of Graphhopper that
lead to the correct answer. One can see in these examples that the sequential
reasoning process over explicit scene graph entities makes the reasoning
process more comprehensible. In the case of wrong predictions, the extracted
path may offer insights into the mechanics of Graphhopper and facilitate
debugging.

##  6 Conclusion

We have proposed Graphhopper, a novel method for visual question answering
that integrates existing KG reasoning, computer vision, and natural language
processing techniques. Concretely, an agent is trained to extract conclusive
reasoning paths from scene graphs. To analyze the reasoning abilities of our
method, we conducted a rigorous experimental study on both manually curated
and generated scene graphs. Based on the manually curated scene graphs we
showed that Graphhopper reaches human performance. Moreover, we find that, on
our own automatically generated scene graph, Graphhopper outperform another
state-of-the-art scene graph reasoning model with respect to all considered
performance metrics. In future works, we plan to combine scene graphs with
common sense knowledge graphs to further enhance the reasoning abilities of
Graphhopper.

## References

  * [1] Abbasnejad, E., Teney, D., Parvaneh, A., Shi, J., Hengel, A.v.d.: Counterfactual vision and language learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10044–10054 (2020)
  * [2] Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6077–6086 (2018)
  * [3] Andreas, J., Rohrbach, M., Darrell, T., Klein, D.: Neural module networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 39–48 (2016)
  * [4] Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Lawrence Zitnick, C., Parikh, D.: Vqa: Visual question answering. In: Proceedings of the IEEE international conference on computer vision. pp. 2425–2433 (2015)
  * [5] Cadene, R., Ben-Younes, H., Cord, M., Thome, N.: Murel: Multimodal relational reasoning for visual question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1989–1998 (2019)
  * [6] Chen, L., Zhang, H., Xiao, J., He, X., Pu, S., Chang, S.F.: Counterfactual critic multi-agent training for scene graph generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 4613–4623 (2019)
  * [7] Chen, W., Gan, Z., Li, L., Cheng, Y., Wang, W., Liu, J.: Meta module network for compositional visual reasoning. arXiv preprint arXiv:1910.03230 (2019)
  * [8] Das, R., Dhuliawala, S., Zaheer, M., Vilnis, L., Durugkar, I., Krishnamurthy, A., Smola, A., McCallum, A.: Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning. In: ICLR (2018)
  * [9] Eyzaguirre, C.: Nsm. https://github.com/charlespwd/project-title (2019)
  * [10] Gan, Z., Chen, Y.C., Li, L., Zhu, C., Cheng, Y., Liu, J.: Large-scale adversarial training for vision-and-language representation learning. arXiv preprint arXiv:2006.06195 (2020)
  * [11] Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics. pp. 249–256 (2010)
  * [12] Hildebrandt, M., Serna, J.A.Q., Ma, Y., Ringsquandl, M., Joblin, M., Tresp, V.: Reasoning on knowledge graphs with debate dynamics. arXiv preprint arXiv:2001.00461 (2020)
  * [13] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation 9(8), 1735–1780 (1997)
  * [14] Hu, R., Andreas, J., Rohrbach, M., Darrell, T., Saenko, K.: Learning to reason: End-to-end module networks for visual question answering. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 804–813 (2017)
  * [15] Hudson, D., Manning, C.D.: Learning by abstraction: The neural state machine. In: Advances in Neural Information Processing Systems. pp. 5901–5914 (2019)
  * [16] Hudson, D.A., Manning, C.D.: Compositional attention networks for machine reasoning. arXiv preprint arXiv:1803.03067 (2018)
  * [17] Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. arXiv preprint arXiv:1902.09506 (2019)
  * [18] Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., Girshick, R.: Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 2901–2910 (2017)
  * [19] Johnson, J., Krishna, R., Stark, M., Li, L.J., Shamma, D., Bernstein, M., Fei-Fei, L.: Image retrieval using scene graphs. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3668–3678 (2015)
  * [20] Kim, J.H., Jun, J., Zhang, B.T.: Bilinear attention networks. In: Advances in Neural Information Processing Systems. pp. 1564–1574 (2018)
  * [21] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [22] Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
  * [23] Koner, R., Sinhamahapatra, P., Tresp, V.: Relation transformer network. arXiv preprint arXiv:2004.06193 (2020)
  * [24] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision 123(1), 32–73 (2017)
  * [25] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740–755. Springer (2014)
  * [26] Mao, J., Gan, C., Kohli, P., Tenenbaum, J.B., Wu, J.: The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv:1904.12584 (2019)
  * [27] Nickel, M., Murphy, K., Tresp, V., Gabrilovich, E.: A review of relational machine learning for knowledge graphs. Proceedings of the IEEE 104(1), 11–33 (2015)
  * [28] Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang, E., DeVito, Z., Lin, Z., Desmaison, A., Antiga, L., Lerer, A.: Automatic differentiation in pytorch (2017)
  * [29] Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP). pp. 1532–1543 (2014), http://www.aclweb.org/anthology/D14-1162
  * [30] Qiao, S., Chen, L.C., Yuille, A.: Detectors: Detecting objects with recursive feature pyramid and switchable atrous convolution. arXiv preprint arXiv:2006.02334 (2020)
  * [31] Shi, J., Zhang, H., Li, J.: Explainable and explicit visual reasoning over scene graphs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 8376–8384 (2019)
  * [32] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15(1), 1929–1958 (2014)
  * [33] Tang, K., Niu, Y., Huang, J., Shi, J., Zhang, H.: Unbiased scene graph generation from biased training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 3716–3725 (2020)
  * [34] Teney, D., Liu, L., van Den Hengel, A.: Graph-structured representations for visual question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1–9 (2017)
  * [35] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in neural information processing systems. pp. 5998–6008 (2017)
  * [36] Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. arXiv preprint arXiv:1710.10903 (2017)
  * [37] Williams, R.J.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8(3-4), 229–256 (1992)
  * [38] Xiong, W., Hoang, T., Wang, W.Y.: Deeppath: A reinforcement learning method for knowledge graph reasoning. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017). ACL, Copenhagen, Denmark (9 2017)
  * [39] Yang, Z., He, X., Gao, J., Deng, L., Smola, A.: Stacked attention networks for image question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 21–29 (2016)
  * [40] Yu, Z., Yu, J., Fan, J., Tao, D.: Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In: Proceedings of the IEEE international conference on computer vision. pp. 1821–1830 (2017)
  * [41] Zhu, C., Zhao, Y., Huang, S., Tu, K., Ma, Y.: Structured attentions for visual question answering. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1291–1300 (2017)

##  Appendix 0.A Details on Model Training and Inference

###  0.A.1 Training Details

In order to optimize the training objective given by Equation (4), we use
REINFORCE [37] to obtain the gradient approximation

$\hat{g}=\sum_{t=0}^{T-1}{\nabla_{\theta}\log\pi_{\theta}(A_{t}|S_{t})(\sum_{t^{{}^{\prime}}=t}^{T-1}{R_{t^{{}^{\prime}}}\gamma^{t^{{}^{\prime}}-t}-B(S_{t})})}\,,$
(6)

where $\gamma$ is the discount factor for the reward. The gradient of the
weights are aggregated over multiple rollouts. To reduce the variance, we
adopt a moving average baseline function $B(S_{t})$. The baseline function is
an approximation of the value of a state $S_{t}$. We could have employed more
sophisticated methods such as advantage network or actor-critic algorithm.
However, we find the current baseline works sufficiently well. Formally, the
baseline function consists of a non-trainable variable $b$ and a
hyperparameter $\lambda$. The baseline is updated by $b_{t+1}=\lambda
b_{t}+(1-\lambda)r_{t}$ at each optimization step. Another technique that
affects the training speed is the reward normalization. Concretely, the
accumulated rewards at each time step for each rollout are collected and
normalized after subtraction of the baseline value.

We introduce a regularization term on the entropy of the resulting probability
distribution from the policy network $\pi_{\theta}(A_{t}|S_{t})$, which
enforces that the agent explores the SG. The regularization is controlled by a
hyperparameter $\beta$. In addition, we apply exponential decay to $\beta$
during training so that $\beta$ converges to zero.

Moreover, we use the chain rule to calculate the gradients of the parameters
of the graph encoder (GAT) $\theta_{GAT}$ and the question encoder
(Transformer) $\theta_{Transformer}$. The weight updates can be performed via
gradient ascent, $\theta\leftarrow\theta+\eta\hat{g}$ or more advanced
optimization methods such as Adam [21].

Input: Question $Q$, Scene Graph
$\mathcal{SG}\subset\mathcal{E}\times\mathcal{R}\times\mathcal{E}$

Model : Policy Network with
$\theta:=\\{\theta_{GAT},\theta_{Transformer},\theta_{Agent}\\}$, Baseline
with $b$

1 for  _$i\leftarrow 0$ to $N$_ do // Loop over epochs

2 Initialize $Q$ and $\mathcal{SG}$ with GloVe embeddings;

$Q\leftarrow\mbox{GAT}(Q)$ // Update the question with the question encoder

$\mathcal{SG}\leftarrow\mbox{Transformer}(\mathcal{SG})$ // Update the SG with
the graph encoder

$C\leftarrow[]$ // Initialize the trajectory buffer

3 for  _$r\leftarrow 0$ to $N$_ do // Loop over samples

$\tau\leftarrow[]$ // Initialize the trajectory

$E_{0}\leftarrow$ hub // Initialize the start position

$A_{0}\leftarrow$ dummy // Initialize the dummy start action

4 for  _$t\leftarrow 0$ to $T$_ do // Loop over time steps

5 if  _$t\%\Delta==0$_ then // Restart and prompt the agent to the hub node

// so that the agent is aware of its own action

$E_{t+1}\leftarrow$ hub // Set next nodes to the hub node

$A_{t+1}\leftarrow$ dummy // Set next actions to the dummy return action

6

7 end if

Sample an action ($A_{t}$, $E_{t+1}$) from $d_{t}$ $\tau$.append($A_{t}$,
$E_{t}$) // Extend the trajectory

$E_{t}\leftarrow E_{t+1}$ // Move the agent to the next entity

8

9 end for

$C$.append($\tau$) // Collect the trajectory

10

11 end for

$r\leftarrow R(C)$ // Gather rewards

$g\leftarrow\sum_{t=0}^{T-1}{\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})(\sum_{t^{{}^{\prime}}=t}^{T-1}{r_{t^{{}^{\prime}}}\gamma^{t^{{}^{\prime}}-t}-b(s_{t})})}$
// Approximate gradients

$\theta\leftarrow\theta+\eta g$ // Update the policy network

$b\leftarrow b+(1-\lambda)r$ // Update the baseline function

12

13 end for

Algorithm 1 Training regime

###  0.A.2 Inference

Beam search is used to infer the answer to a given question. Our inference
approach is based on evaluating how likely specific paths are appearing among
all possible paths with a fixed length. More specifically, given an input
question, the agent’s initial location is given by the hub node. At each time
step, the agent scores the next permissible actions based on the learned
policy. The value of action represents the transition probability from the
current node to a target node. Next, we keep the top $k$ (also known as beam
width) paths among all possible transitions and move the agent to the
corresponding targets. This computation is iteratively performed until the
maximum number of transitions is reached. In the end, we obtain multiple
rollouts ranked by the path probabilities. The target node (i.e. the last
node) of the path is regarded as an answer candidate. Unlike Monte Carlo
sampling which does not consider path probabilities, beam search yields better
answer candidates, as it always chooses the best choice within the search
region. The algorithm for inference is summarized in table 2.

#### Inference Complexity

The inference of our method is computationally efficient. Unlike other methods
that need to iterate through each candidate answer for a final prediction, we
only need to run the inference once so that the score of each answer is
obtained. Let $d$ denote the embedding dimension of the words and entities.
Analytically, the embedding stage has asymptotic complexity
$\mathcal{O}(|\mathcal{E}|+|\mathcal{R}|+|Q|)$. For the GAT, the
implementation of a single attention head and multi-head attention is similar.
In particular, they have the same time complexity
$\mathcal{O}(|\mathcal{E}|dd^{{}^{\prime}}+|\mathcal{R}|d^{{}^{\prime}})$. The
computation of the question encoding is given by
$\mathcal{O}(|\mathcal{Q}|^{2}d)$. It is efficient as it only runs once for
each question and is used for arbitrary times during random walks. Also, the
length of the questions $Q$ is usually short (less than 30 words). Finally,
during the random walk sampling, the agents complxity is given by
$\mathcal{O}(T(D^{2}+|\mathcal{E}|d))$, where $d$ is dominant. The inference
time depends largely on the path length.

Input: Question $Q$, Scene graph
$\mathcal{SG}\subset\mathcal{E}\times\mathcal{R}\times\mathcal{E}$

Output: Answer

1 Initialize $Q$ and $\mathcal{SG}$ with GloVe embeddings;

$Q\leftarrow\mbox{GAT}(Q)$ // Update the question with the question encoder

$\mathcal{SG}\leftarrow\mbox{Transformer}(\mathcal{SG})$ // Update the SG with
the graph encoder.

$P\leftarrow[]$ // Initialize the probability register

$\tau\leftarrow[]$ // Initialize the trajectory

$E_{0}\leftarrow$ hub // Initialize the start position

$A_{0}\leftarrow$ dummy // Initialize the dummy start action

2 for  _$t\leftarrow 0$ to $T$_ do // Loop over time steps

3 for  _$r\leftarrow 0$ to $N$_ do // Loop over rollouts

4 if  _$t\%\Delta==0$_ then // Restart and prompt the agent to the hub node

// so that the agent is aware of its own action

$E_{t+1}\leftarrow$ hub // Set next nodes to the hub node

$A_{t+1}\leftarrow$ dummy // Set next actions to the dummy return action

5

6 end if

Forward pass through the policy network to generate candidate actions
$\\{(A_{t},E_{t})\\}$ along with their probabilities $\\{p_{i}\\}$
$\tau$.append($\\{(A_{t},E_{t})\\}$) // Extend the trajectory

$P$.append($\\{p_{i}\\}$) // Store corresponding probabilities

7

8 end for

$indicies\leftarrow argmax(P,k)$ // Filter indices of top k probabilities from
P

$\tau\leftarrow\tau[indices]$ // Choose top k paths ranked by their
probabilities

$E_{t+1}\leftarrow e\in\tau$ // Conduct corresponding transitions

9

10 end for

Prediction $\leftarrow\tau[0]$ // Predict the end entity of the top path as
the answer

Algorithm 2 Inference with beam search

###  0.A.3 Complexity Analysis

For analyzing the complexity of our method, we provide all the parameters
contained in the building blocks. Moreover, we present the number of
operations of a forward pass - the complete run that derives the answer from a
given $Q$ and $\mathcal{SG}$. They are listed in the table 3.

Group | Name | No. Parameters | No. Operations
---|---|---|---
Word Embeddings* | Entity | $N_{e}\times d$ | $\mathcal{O}(N)$
| Relation | $N_{r}\times D$ | $\mathcal{O}(N)$
GAT | Conv layer weight | $d\times H_{1}$ | $\mathcal{O}(BHN_{e})$
| Conv layer attention | $d\times H_{1}$ | $\mathcal{O}(BHN_{e})$
| Conv layer bias | $H_{1}$ | $\mathcal{O}(BHN_{e})$
Transformer | Positional encoder | $d\times d$ | $\mathcal{O}(N_{e})$
| Layer self attention $(qkv)$ | $H_{t}(512)\times d$ | $3\times H_{t}\times d$
| Self attn norm $(W,b)$ | $d$ | $2\times d$
| Layer enc attn | $H_{t}(512)\times d$ | $3\times H_{t}\times d$
| Enc attn norm $(W,b)$ | $d$ | $2\times d$
| Pos ffn 1 $(W,b)$ | $d\times d+d$ | $d\times d+d$
| Pos ffn 2 $(W,b)$ | $d\times d+d$ | $d\times d+d$
| Pos ffn norm$(W,b)$ | $2d$ | $2\times d$
| Enc attn norm $(W,b)$ | $d$ | $2\times d$
Agent-MLP | Dense 0 | $4d\times 4d+4d$ | $(H\times 4d+4d)\times T$
| Dense 1 | $2d\times 4d+2d$ | $(H\times 2d+2d)\times T$
Agent-LSTM | Lstm_cell $W_{ih}$ | $4d\times 16d$ | $(4d\times 16d)\times T$
| Lstm_cell ih | $16d$ | $(16d)\times T$
| Lstm_cell $W_{hh}$ | $4d\times 16d$ | $(4d\times 16d)\times T$
| Lstm_cell $b_{hh}$ | $16d$ | $(16d)\times T$
Table 3: An overview of the number parameters and the asymptotic number of
operations for the individual modules. The batch size is indicated by $B$. $T$
corresponds to the number of time steps. $D$ and $H$ denote the embedding size
and hidden size, respectively. Blocks are marked with a ”*” if their weights
are not trainable.

##  Appendix 0.B Additional Details on the Dataset GQA

In this section we describe various question category and their type. We list
the question based on semantic and structural categories. We further grouped
them based on their entity type like object, attribute, category etc. Table 4,
describes the detailed list of question category.

Table 4: List of question examples in the GQA dataset. Category | Type | Description |  Example
---|---|---|---
Semantics | Object | Existence of object |  Are there any doors that are not made of metal?
Attribute | Property about an object |  Does the soap dispenser that is to the right of the other soap dispenser have small size and white color?
Category | Identify an object class |  What kind of animal is standing?
Relation | Relationship of object |  What is the food that is to the left of the white object that is to the left of the chocolate called?
Global | Overall scene property |  Which place is it?
Structural | Query | Open-form question |  What type of furniture is to the left of the silver device which is to the left of the helmet?
Choose | Choose from alternatives |  What are the floating people in the ocean doing, riding or swimming?
Verify | Simple yes/no question |  Are there statues above the brass clock that is on the building?
Compare | Comparison of objects |  Are the drawers made of the same material as the cages?
Logical | And/or operators |  Are both the giraffe near the building and the giraffe that is to the left of the tray standing?