* Li et al. [2018a] Li, J., Peng, H., Liu, L., Xiong, G., Du, B., Ma, H., Wang, L., & Bhuiyan, M. Z. A. (2018a).  Graph cnns for urban traffic passenger flows prediction.  In 2018 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI) (pp. 29--36).  IEEE.
  * Li & Zhu [2021] Li, M., & Zhu, Z. (2021).  Spatial-temporal fusion graph neural networks for traffic flow forecasting.  In Proceedings of the AAAI Conference on Artificial Intelligence (pp. 4189--4196).  volume 35.
  * Li et al. [2020b] Li, W., Wang, X., Zhang, Y., & Wu, Q. (2020b).  Traffic flow prediction over muti-sensor data correlation with graph convolution network.  Neurocomputing, .
  * Li et al. [2020c] Li, W., Yang, X., Tang, X., & Xia, S. (2020c).  Sdcn: Sparsity and diversity driven correlation networks for traffic demand forecasting.  In 2020 International Joint Conference on Neural Networks (IJCNN) (pp. 1--8).  IEEE.
  * Li & Moura [2020] Li, Y., & Moura, J. M. (2020).  Forecaster: A graph transformer for forecasting spatial and time-dependent data.  In Proceedings of the Twenty-fourth European Conference on Artificial Intelligence.
  * Li et al. [2018b] Li, Y., Yu, R., Shahabi, C., & Liu, Y. (2018b).  Diffusion convolutional recurrent neural network: Data-driven traffic forecasting.  In International Conference on Learning Representations (ICLR ’18).
  * Li et al. [2020d] Li, Z., Li, L., Peng, Y., & Tao, X. (2020d).  A two-stream graph convolutional neural network for dynamic traffic flow forecasting.  In 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI) (pp. 355--362).  IEEE.
  * Li et al. [2020e] Li, Z., Sergin, N. D., Yan, H., Zhang, C., & Tsung, F. (2020e).  Tensor completion for weakly-dependent data on graph for metro passenger flow prediction.  In Proceedings of the AAAI Conference on Artificial Intelligence.  volume 34.
  * Li et al. [2019] Li, Z., Xiong, G., Chen, Y., Lv, Y., Hu, B., Zhu, F., & Wang, F.-Y. (2019).  A hybrid deep learning approach with gcn and lstm for traffic flow prediction.  In 2019 IEEE Intelligent Transportation Systems Conference (ITSC) (pp. 1929--1933).  IEEE.
  * Li et al. [2020f] Li, Z., Xiong, G., Tian, Y., Lv, Y., Chen, Y., Hui, P., & Su, X. (2020f).  A multi-stream feature fusion approach for traffic prediction.  IEEE Transactions on Intelligent Transportation Systems, .
  * Liao et al. [2018] Liao, B., Zhang, J., Wu, C., McIlwraith, D., Chen, T., Yang, S., Guo, Y., & Wu, F. (2018).  Deep sequence learning with auxiliary information for traffic prediction.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 537--546).
  * Lin et al. [2018] Lin, L., He, Z., & Peeta, S. (2018).  Predicting station-level hourly demand in a large-scale bike-sharing network: A graph convolutional neural network approach.  Transportation Research Part C: Emerging Technologies, 97, 258--276.
  * Liu et al. [2020a] Liu, J., Ong, G. P., & Chen, X. (2020a).  Graphsage-based traffic speed forecasting for segment network with sparse data.  IEEE Transactions on Intelligent Transportation Systems, .
  * Liu et al. [2020b] Liu, L., Chen, J., Wu, H., Zhen, J., Li, G., & Lin, L. (2020b).  Physical-virtual collaboration modeling for intra-and inter-station metro ridership prediction.  IEEE Transactions on Intelligent Transportation Systems, .
  * Liu et al. [2019] Liu, L., Zhou, T., Long, G., Jiang, J., & Zhang, C. (2019).  Learning to propagate for graph meta-learning.  In Advances in Neural Information Processing Systems (pp. 1039--1050).
  * Liu et al. [2020c] Liu, R., Zhao, S., Cheng, B., Yang, H., Tang, H., & Yang, F. (2020c).  St-mfm: A spatiotemporal multi-modal fusion model for urban anomalies prediction.  In Proceedings of the Twenty-fourth European Conference on Artificial Intelligence.
  * Lu et al. [2020a] Lu, B., Gan, X., Jin, H., Fu, L., & Zhang, H. (2020a).  Spatiotemporal adaptive gated graph convolution network for urban traffic flow forecasting.  In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (pp. 1025--1034).
  * Lu et al. [2019a] Lu, M., Zhang, K., Liu, H., & Xiong, N. (2019a).  Graph hierarchical convolutional recurrent neural network (ghcrnn) for vehicle condition prediction.  arXiv preprint arXiv:1903.06261, .
  * Lu et al. [2020b] Lu, Z., Lv, W., Cao, Y., Xie, Z., Peng, H., & Du, B. (2020b).  Lstm variants meet graph neural networks for road speed prediction.  Neurocomputing, .
  * Lu et al. [2019b] Lu, Z., Lv, W., Xie, Z., Du, B., & Huang, R. (2019b).  Leveraging graph neural network with lstm for traffic speed prediction.  In 2019 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI) (pp. 74--81).  IEEE.
  * Luca et al. [2020] Luca, M., Barlacchi, G., Lepri, B., & Pappalardo, L. (2020).  Deep learning for human mobility: a survey on data and models.  arXiv preprint arXiv:2012.02825, .
  * Luo et al. [2020] Luo, M., Du, B., Klemmer, K., Zhu, H., Ferhatosmanoglu, H., & Wen, H. (2020).  D3p: Data-driven demand prediction for fast expanding electric vehicle sharing systems.  Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4, 1--21.
  * Lv et al. [2020] Lv, M., Hong, Z., Chen, L., Chen, T., Zhu, T., & Ji, S. (2020).  Temporal multi-graph convolutional network for traffic flow prediction.  IEEE Transactions on Intelligent Transportation Systems, .
  * Maas & Bloem [2020] Maas, T., & Bloem, P. (2020).  Uncertainty intervals for graph-based spatio-temporal traffic prediction.  arXiv preprint arXiv:2012.05207, .
  * Mallick et al. [2020] Mallick, T., Balaprakash, P., Rask, E., & Macfarlane, J. (2020).  Graph-partitioning-based diffusion convolution recurrent neural network for large-scale traffic forecasting.  Transportation Research Record, (p. 0361198120930010).
  * Mallick et al. [2021] Mallick, T., Balaprakash, P., Rask, E., & Macfarlane, J. (2021).  Transfer learning with graph neural networks for short-term highway traffic forecasting.  In 2020 25th International Conference on Pattern Recognition (ICPR) (pp. 10367--10374).  IEEE.
  * Manibardo et al. [2020] Manibardo, E. L., Laña, I., & Del Ser, J. (2020).  Deep learning for road traffic forecasting: Does it make a difference?  arXiv preprint arXiv:2012.02260, .
  * Mena-Oreja & Gozalvez [2020] Mena-Oreja, J., & Gozalvez, J. (2020).  A comprehensive evaluation of deep learning-based techniques for traffic prediction.  IEEE Access, 8, 91188--91212.
  * Mohanty & Pozdnukhov [2018] Mohanty, S., & Pozdnukhov, A. (2018).  Graph cnn+ lstm framework for dynamic macroscopic traffic congestion prediction.  In International Workshop on Mining and Learning with Graphs.
  * Mohanty et al. [2020] Mohanty, S., Pozdnukhov, A., & Cassidy, M. (2020).  Region-wide congestion prediction and control using deep learning.  Transportation Research Part C: Emerging Technologies, 116, 102624.
  * Opolka et al. [2019] Opolka, F. L., Solomon, A., Cangea, C., Veličković, P., Liò, P., & Hjelm, R. D. (2019).  Spatio-temporal deep graph infomax.  In Representation Learning on Graphs and Manifolds, ICLR 2019 Workshop.
  * Oreshkin et al. [2021] Oreshkin, B. N., Amini, A., Coyle, L., & Coates, M. (2021).  Fc-gaga: Fully connected gated graph architecture for spatio-temporal traffic forecasting.  In Proceedings of the AAAI Conference on Artificial Intelligence (pp. 9233--9241).  volume 35.
  * Ou et al. [2020] Ou, J., Sun, J., Zhu, Y., Jin, H., Liu, Y., Zhang, F., Huang, J., & Wang, X. (2020).  Stp-trellisnets: Spatial-temporal parallel trellisnets for metro station passenger flow prediction.  In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (pp. 1185--1194).
  * Pan et al. [2019] Pan, Z., Liang, Y., Wang, W., Yu, Y., Zheng, Y., & Zhang, J. (2019).  Urban traffic prediction from spatio-temporal data using deep meta learning.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 1720--1730).
  * Pan et al. [2020] Pan, Z., Zhang, W., Liang, Y., Zhang, W., Yu, Y., Zhang, J., & Zheng, Y. (2020).  Spatio-temporal meta learning for urban traffic prediction.  IEEE Transactions on Knowledge and Data Engineering, .
  * Park et al. [2020] Park, C., Lee, C., Bahng, H., Tae, Y., Jin, S., Kim, K., Ko, S., & Choo, J. (2020).  St-grat: A novel spatio-temporal graph attention networks for accurately forecasting dynamically changing road speed.  In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (pp. 1215--1224).
  * Pavlyuk [2019] Pavlyuk, D. (2019).  Feature selection and extraction in spatiotemporal traffic forecasting: a systematic literature review.  European Transport Research Review, 11, 6.
  * Peng et al. [2020] Peng, H., Wang, H., Du, B., Bhuiyan, M. Z. A., Ma, H., Liu, J., Wang, L., Yang, Z., Du, L., Wang, S. et al. (2020).  Spatial temporal incidence dynamic graph neural networks for traffic flow forecasting.  Information Sciences, 521, 277--290.
  * Pian & Wu [2020] Pian, W., & Wu, Y. (2020).  Spatial-temporal dynamic graph attention networks for ride-hailing demand prediction.  arXiv preprint arXiv:2006.05905, .
  * Pope et al. [2019] Pope, P. E., Kolouri, S., Rostami, M., Martin, C. E., & Hoffmann, H. (2019).  Explainability methods for graph convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 10772--10781).
  * Qin et al. [2020a] Qin, K., Xu, Y., Kang, C., & Kwan, M.-P. (2020a).  A graph convolutional network model for evaluating potential congestion spots based on local urban built environments.  Transactions in GIS, .
  * Qin et al. [2020b] Qin, T., Liu, T., Wu, H., Tong, W., & Zhao, S. (2020b).  Resgcn: Residual graph convolutional network based free dock prediction in bike sharing system.  In 2020 21st IEEE International Conference on Mobile Data Management (MDM) (pp. 210--217).  IEEE.
  * Qiu et al. [2020] Qiu, H., Zheng, Q., Msahli, M., Memmi, G., Qiu, M., & Lu, J. (2020).  Topological graph convolutional network-based urban traffic flow and density prediction.  IEEE Transactions on Intelligent Transportation Systems, .
  * Qu et al. [2020] Qu, Y., Zhu, Y., Zang, T., Xu, Y., & Yu, J. (2020).  Modeling local and global flow aggregation for traffic flow forecasting.  In International Conference on Web Information Systems Engineering (pp. 414--429).  Springer.
  * Ramadan et al. [2020] Ramadan, A., Elbery, A., Zorba, N., & Hassanein, H. S. (2020).  Traffic forecasting using temporal line graph convolutional network: Case study.  In ICC 2020-2020 IEEE International Conference on Communications (ICC) (pp. 1--6).  IEEE.
  * Ren & Xie [2019] Ren, Y., & Xie, K. (2019).  Transfer knowledge between sub-regions for traffic prediction using deep learning method.  In International Conference on Intelligent Data Engineering and Automated Learning (pp. 208--219).  Springer.
  * Sánchez et al. [2020] Sánchez, C. S., Wieder, A., Sottovia, P., Bortoli, S., Baumbach, J., & Axenie, C. (2020).  Gannster: Graph-augmented neural network spatio-temporal reasoner for traffic forecasting.  In International Workshop on Advanced Analysis and Learning on Temporal Data (AALTD).  Springer.
  * Scarselli et al. [2008] Scarselli, F., Gori, M., Tsoi, A. C., Hagenbuchner, M., & Monfardini, G. (2008).  The graph neural network model.  IEEE transactions on neural networks, 20, 61--80.
  * Shao et al. [2020] Shao, K., Wang, K., Chen, L., & Zhou, Z. (2020).  Estimation of urban travel time with sparse traffic surveillance data.  In Proceedings of the 2020 4th High Performance Computing and Cluster Technologies Conference & 2020 3rd International Conference on Big Data and Artificial Intelligence (pp. 218--223).
  * Shen et al. [2020] Shen, Y., Jin, C., & Hua, J. (2020).  Ttpnet: A neural network for travel time prediction based on tensor decomposition and graph embedding.  IEEE Transactions on Knowledge and Data Engineering, .
  * Shi et al. [2020] Shi, H., Yao, Q., Guo, Q., Li, Y., Zhang, L., Ye, J., Li, Y., & Liu, Y. (2020).  Predicting origin-destination flow via multi-perspective graph convolutional network.  In 2020 IEEE 36th International Conference on Data Engineering (ICDE) (pp. 1818--1821).  IEEE.
  * Shi & Yeung [2018] Shi, X., & Yeung, D.-Y. (2018).  Machine learning for spatiotemporal sequence forecasting: A survey.  arXiv preprint arXiv:1808.06865, .
  * Shin & Yoon [2020] Shin, Y., & Yoon, Y. (2020).  Incorporating dynamicity of transportation network with multi-weight traffic graph convolutional network for traffic forecasting.  IEEE Transactions on Intelligent Transportation Systems, .
  * Shleifer et al. [2019] Shleifer, S., McCreery, C., & Chitters, V. (2019).  Incrementally improving graph wavenet performance on traffic prediction.  arXiv preprint arXiv:1912.07390, .
  * Song et al. [2020a] Song, C., Lin, Y., Guo, S., & Wan, H. (2020a).  Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting.  In Proceedings of the AAAI Conference on Artificial Intelligence.  volume 34.
  * Song et al. [2020b] Song, Q., Ming, R., Hu, J., Niu, H., & Gao, M. (2020b).  Graph attention convolutional network: Spatiotemporal modeling for urban traffic prediction.  In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC) (pp. 1--6).  IEEE.
  * Sun et al. [2020] Sun, J., Zhang, J., Li, Q., Yi, X., Liang, Y., & Zheng, Y. (2020).  Predicting citywide crowd flows in irregular regions using multi-view graph convolutional networks.  IEEE Transactions on Knowledge and Data Engineering, (pp. 1--1).
  * Sun et al. [2020] Sun, X., Li, J., Lv, Z., & Dong, C. (2020).  Traffic flow prediction model based on spatio-temporal dilated graph convolution.  KSII Transactions on Internet & Information Systems, 14.
  * Sun et al. [2021] Sun, Y., Wang, Y., Fu, K., Wang, Z., Zhang, C., & Ye, J. (2021).  Constructing geographic and long-term temporal graph for traffic forecasting.  In 2020 25th International Conference on Pattern Recognition (ICPR) (pp. 3483--3490).  IEEE.
  * Tang et al. [2020a] Tang, C., Sun, J., & Sun, Y. (2020a).  Dynamic spatial-temporal graph attention graph convolutional network for short-term traffic flow forecasting.  In 2020 IEEE International Symposium on Circuits and Systems (ISCAS) (pp. 1--5).  IEEE.
  * Tang et al. [2020b] Tang, C., Sun, J., Sun, Y., Peng, M., & Gan, N. (2020b).  A general traffic flow prediction approach based on spatial-temporal graph attention.  IEEE Access, 8, 153731--153741.
  * Tedjopurnomo et al. [2020] Tedjopurnomo, D. A., Bao, Z., Zheng, B., Choudhury, F., & Qin, A. (2020).  A survey on modern deep neural network for traffic prediction: Trends, methods and challenges.  IEEE Transactions on Knowledge and Data Engineering, .
  * Tian et al. [2020] Tian, K., Guo, J., Ye, K., & Xu, C.-Z. (2020).  St-mgat: Spatial-temporal multi-head graph attention networks for traffic forecasting.  In 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI) (pp. 714--721).  IEEE.
  * Varghese et al. [2020] Varghese, V., Chikaraishi, M., & Urata, J. (2020).  Deep learning in transport studies: A meta-analysis on the prediction accuracy.  Journal of Big Data Analytics in Transportation, (pp. 1--22).
  * Vaswani et al. [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017).  Attention is all you need.  Advances in neural information processing systems, 30, 5998--6008.
  * Veličković et al. [2018] Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., & Bengio, Y. (2018).  Graph attention networks.  In International Conference on Learning Representations.
  * Wang et al. [2018a] Wang, B., Luo, X., Zhang, F., Yuan, B., Bertozzi, A. L., & Brantingham, P. J. (2018a).  Graph-based deep modeling and real time forecasting of sparse spatio-temporal data.  arXiv preprint arXiv:1804.00684, .
  * Wang et al. [2020a] Wang, C., Zhang, K., Wang, H., & Chen, B. (2020a).  Auto-stgcn: Autonomous spatial-temporal graph convolutional network search based on reinforcement learning and existing research results.  arXiv preprint arXiv:2010.07474, .
  * Wang et al. [2020b] Wang, F., Xu, J., Liu, C., Zhou, R., & Zhao, P. (2020b).  Mtgcn: A multitask deep learning model for traffic flow prediction.  In International Conference on Database Systems for Advanced Applications (pp. 435--451).  Springer.
  * Wang et al. [2020c] Wang, H.-W., Peng, Z.-R., Wang, D., Meng, Y., Wu, T., Sun, W., & Lu, Q.-C. (2020c).  Evaluation and prediction of transportation resilience under extreme weather events: A diffusion graph convolutional approach.  Transportation Research Part C: Emerging Technologies, 115, 102619.
  * Wang et al. [2020d] Wang, Q., Guo, B., Ouyang, Y., Shu, K., Yu, Z., & Liu, H. (2020d).  Spatial community-informed evolving graphs for demand prediction.  In Proceedings of The European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD 2020).
  * Wang et al. [2020e] Wang, S., Miao, H., Chen, H., & Huang, Z. (2020e).  Multi-task adversarial spatial-temporal networks for crowd flow prediction.  In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (pp. 1555--1564).
  * Wang et al. [2018b] Wang, X., Chen, C., Min, Y., He, J., Yang, B., & Zhang, Y. (2018b).  Efficient metropolitan traffic prediction based on graph recurrent neural network.  arXiv preprint arXiv:1811.00740, .
  * Wang et al. [2020f] Wang, X., Guan, X., Cao, J., Zhang, N., & Wu, H. (2020f).  Forecast network-wide traffic states for multiple steps ahead: A deep learning approach considering dynamic non-local spatial correlation and non-stationary temporal dependency.  Transportation Research Part C: Emerging Technologies, 119, 102763. URL: http://www.sciencedirect.com/science/article/pii/S0968090X20306756. doi:https://doi.org/10.1016/j.trc.2020.102763.
  * Wang et al. [2020g] Wang, X., Ma, Y., Wang, Y., Jin, W., Wang, X., Tang, J., Jia, C., & Yu, J. (2020g).  Traffic flow prediction via spatial temporal graph neural network.  In Proceedings of The Web Conference 2020 WWW ’20 (p. 1082–1092).  New York, NY, USA: Association for Computing Machinery.  URL: https://doi.org/10.1145/3366423.3380186. doi:10.1145/3366423.3380186.
  * Wang et al. [2020h] Wang, Y., Xu, D., Peng, P., Xuan, Q., & Zhang, G. (2020h).  An urban commuters’ od hybrid prediction method based on big gps data.  Chaos: An Interdisciplinary Journal of Nonlinear Science, 30, 093128\.
  * Wang et al. [2019] Wang, Y., Yin, H., Chen, H., Wo, T., Xu, J., & Zheng, K. (2019).  Origin-destination matrix prediction via graph convolution: a new perspective of passenger demand modeling.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (pp. 1227--1235).
  * Wei & Sheng [2020] Wei, C., & Sheng, J. (2020).  Spatial-temporal graph attention networks for traffic flow forecasting.  In IOP Conference Series: Earth and Environmental Science (p. 012065).  IOP Publishing volume 587.
  * Wei et al. [2019] Wei, L., Yu, Z., Jin, Z., Xie, L., Huang, J., Cai, D., He, X., & Hua, X.-S. (2019).  Dual graph for traffic forecasting.  IEEE Access, .
  * Wright et al. [2019] Wright, M. A., Ehlers, S. F., & Horowitz, R. (2019).  Neural-attention-based deep learning architectures for modeling traffic dynamics on lane graphs.  In 2019 IEEE Intelligent Transportation Systems Conference (ITSC) (pp. 3898--3905).  IEEE.
  * Wu et al. [2020a] Wu, M., Zhu, C., & Chen, L. (2020a).  Multi-task spatial-temporal graph attention network for taxi demand prediction.  In Proceedings of the 2020 5th International Conference on Mathematics and Artificial Intelligence (pp. 224--228).
  * Wu et al. [2018a] Wu, T., Chen, F., & Wan, Y. (2018a).  Graph attention lstm network: A new model for traffic flow forecasting.  In 2018 5th International Conference on Information Science and Control Engineering (ICISCE) (pp. 241--245).  IEEE.
  * Wu et al. [2018b] Wu, Y., Tan, H., Qin, L., Ran, B., & Jiang, Z. (2018b).  A hybrid deep learning based traffic flow prediction method and its understanding.  Transportation Research Part C: Emerging Technologies, 90, 166--180.
  * Wu et al. [2020b] Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., & Philip, S. Y. (2020b).  A comprehensive survey on graph neural networks.  IEEE Transactions on Neural Networks and Learning Systems, .
  * Wu et al. [2020c] Wu, Z., Pan, S., Long, G., Jiang, J., Chang, X., & Zhang, C. (2020c).  Connecting the dots: Multivariate time series forecasting with graph neural networks.  In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining KDD ’20 (p. 753–763).  New York, NY, USA: Association for Computing Machinery.  URL: https://doi.org/10.1145/3394486.3403118. doi:10.1145/3394486.3403118.
  * Wu et al. [2019] Wu, Z., Pan, S., Long, G., Jiang, J., & Zhang, C. (2019).  Graph wavenet for deep spatial-temporal graph modeling.  In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19 (pp. 1907--1913).  International Joint Conferences on Artificial Intelligence Organization.  URL: https://doi.org/10.24963/ijcai.2019/264. doi:10.24963/ijcai.2019/264.
  * Xiao et al. [2020] Xiao, G., Wang, R., Zhang, C., & Ni, A. (2020).  Demand prediction for a public bike sharing program based on spatio-temporal graph convolutional networks.  Multimedia Tools and Applications, (pp. 1--19).
  * Xie et al. [2020a] Xie, P., Li, T., Liu, J., Du, S., Yang, X., & Zhang, J. (2020a).  Urban flow prediction from spatiotemporal data using machine learning: A survey.  Information Fusion, 59, 1--12.
  * Xie et al. [2020b] Xie, Q., Guo, T., Chen, Y., Xiao, Y., Wang, X., & Zhao, B. Y. (2020b).  Deep graph convolutional networks for incident-driven traffic speed prediction.  In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (pp. 1665--1674).
  * Xie et al. [2020c] Xie, Y., Xiong, Y., & Zhu, Y. (2020c).  Istd-gcn: Iterative spatial-temporal diffusion graph convolutional network for traffic speed forecasting.  arXiv preprint arXiv:2008.03970, .
  * Xie et al. [2020d] Xie, Y., Xiong, Y., & Zhu, Y. (2020d).  Sast-gnn: A self-attention based spatio-temporal graph neural network for traffic prediction.  In International Conference on Database Systems for Advanced Applications (pp. 707--714).  Springer.
  * Xie et al. [2019] Xie, Z., Lv, W., Huang, S., Lu, Z., Du, B., & Huang, R. (2019).  Sequential graph neural network for urban road traffic speed prediction.  IEEE Access, .
  * Xin et al. [2020] Xin, Y., Miao, D., Zhu, M., Jin, C., & Lu, X. (2020).  Internet: Multistep traffic forecasting by interacting spatial and temporal features.  In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (pp. 3477--3480).
  * Xiong et al. [2020] Xiong, X., Ozbay, K., Jin, L., & Feng, C. (2020).  Dynamic origin--destination matrix prediction with line graph neural networks and kalman filter.  Transportation Research Record, (p. 0361198120919399).
  * Xu et al. [2019] Xu, D., Dai, H., Wang, Y., Peng, P., Xuan, Q., & Guo, H. (2019).  Road traffic state prediction based on a graph embedding recurrent neural network under the scats.  Chaos: An Interdisciplinary Journal of Nonlinear Science, 29, 103125\.
  * Xu et al. [2020a] Xu, D., Wei, C., Peng, P., Xuan, Q., & Guo, H. (2020a).  Ge-gan: A novel deep learning framework for road traffic state estimation.  Transportation Research Part C: Emerging Technologies, 117, 102635.
  * Xu et al. [2020b] Xu, M., Dai, W., Liu, C., Gao, X., Lin, W., Qi, G.-J., & Xiong, H. (2020b).  Spatial-temporal transformer networks for traffic flow forecasting.  arXiv preprint arXiv:2001.02908, .
  * Xu et al. [2020c] Xu, X., Zheng, H., Feng, X., & Chen, Y. (2020c).  Traffic flow forecasting with spatial-temporal graph convolutional networks in edge-computing systems.  In 2020 International Conference on Wireless Communications and Signal Processing (WCSP) (pp. 251--256).  IEEE.
  * Xu & Li [2019] Xu, Y., & Li, D. (2019).  Incorporating graph attention and recurrent architectures for city-wide taxi demand prediction.  ISPRS International Journal of Geo-Information, 8, 414.
  * Xu et al. [2020d] Xu, Z., Kang, Y., Cao, Y., & Li, Z. (2020d).  Spatiotemporal graph convolution multifusion network for urban vehicle emission prediction.  IEEE Transactions on Neural Networks and Learning Systems, .
  * Yang et al. [2020] Yang, F., Chen, L., Zhou, F., Gao, Y., & Cao, W. (2020).  Relational state-space model for stochastic multi-object systems.  In International Conference on Learning Representations.
  * Yang et al. [2019] Yang, S., Ma, W., Pi, X., & Qian, S. (2019).  A deep learning approach to real-time parking occupancy prediction in transportation networks incorporating multiple spatio-temporal data sources.  Transportation Research Part C: Emerging Technologies, 107, 248--265.
  * Yao et al. [2020] Yao, X., Gao, Y., Zhu, D., Manley, E., Wang, J., & Liu, Y. (2020).  Spatial origin-destination flow imputation using graph convolutional networks.  IEEE Transactions on Intelligent Transportation Systems, .
  * Ye et al. [2021] Ye, J., Sun, L., Du, B., Fu, Y., & Xiong, H. (2021).  Coupled layer-wise graph convolution for transportation demand prediction.  In Proceedings of the AAAI Conference on Artificial Intelligence (pp. 4617--4625).  volume 35.
  * Ye et al. [2020a] Ye, J., Zhao, J., Ye, K., & Xu, C. (2020a).  How to build a graph-based deep learning architecture in traffic domain: A survey.  IEEE Transactions on Intelligent Transportation Systems, .
  * Ye et al. [2020b] Ye, J., Zhao, J., Ye, K., & Xu, C. (2020b).  Multi-stgcnet: A graph convolution based spatial-temporal framework for subway passenger flow forecasting.  In 2020 International Joint Conference on Neural Networks (IJCNN) (pp. 1--8).  IEEE.
  * Yeghikyan et al. [2020] Yeghikyan, G., Opolka, F. L., Nanni, M., Lepri, B., & Liò, P. (2020).  Learning mobility flows from urban features with spatial interaction models and neural networks.  In 2020 IEEE International Conference on Smart Computing (SMARTCOMP) (pp. 57--64).  IEEE.
  * Yin et al. [2020] Yin, X., Wu, G., Wei, J., Shen, Y., Qi, H., & Yin, B. (2020).  Multi-stage attention spatial-temporal graph networks for traffic prediction.  Neurocomputing, .
  * Yin et al. [2021] Yin, X., Wu, G., Wei, J., Shen, Y., Qi, H., & Yin, B. (2021).  Deep learning on traffic prediction: Methods, analysis and future directions.  IEEE Transactions on Intelligent Transportation Systems, .
  * Ying et al. [2019] Ying, Z., Bourgeois, D., You, J., Zitnik, M., & Leskovec, J. (2019).  Gnnexplainer: Generating explanations for graph neural networks.  In Advances in neural information processing systems (pp. 9244--9255).
  * Yoshida et al. [2019] Yoshida, A., Yatsushiro, Y., Hata, N., Higurashi, T., Tateiwa, N., Wakamatsu, T., Tanaka, A., Nagamatsu, K., & Fujisawa, K. (2019).  Practical end-to-end repositioning algorithm for managing bike-sharing system.  In 2019 IEEE International Conference on Big Data (Big Data) (pp. 1251--1258).  IEEE.
  * Yu et al. [2020a] Yu, B., Lee, Y., & Sohn, K. (2020a).  Forecasting road traffic speeds by considering area-wide spatio-temporal dependencies based on a graph convolutional neural network (gcn).  Transportation Research Part C: Emerging Technologies, 114, 189--204.
  * Yu et al. [2019a] Yu, B., Li, M., Zhang, J., & Zhu, Z. (2019a).  3d graph convolutional networks with temporal graphs: A spatial information free framework for traffic forecasting.  arXiv preprint arXiv:1903.00919, .
  * Yu et al. [2018] Yu, B., Yin, H., & Zhu, Z. (2018).  Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting.  In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18 (pp. 3634--3640).  International Joint Conferences on Artificial Intelligence Organization.  URL: https://doi.org/10.24963/ijcai.2018/505. doi:10.24963/ijcai.2018/505.
  * Yu et al. [2019b] Yu, B., Yin, H., & Zhu, Z. (2019b).  St-unet: A spatio-temporal u-network for graph-structured time series modeling.  arXiv preprint arXiv:1903.05631, .
  * Yu & Gu [2019] Yu, J. J. Q., & Gu, J. (2019).  Real-time traffic speed estimation with graph convolutional generative autoencoder.  IEEE Transactions on Intelligent Transportation Systems, 20, 3940--3951.
  * Yu et al. [2020b] Yu, L., Du, B., Hu, X., Sun, L., Han, L., & Lv, W. (2020b).  Deep spatio-temporal graph convolutional network for traffic accident prediction.  Neurocomputing, .
  * Yuan et al. [2010] Yuan, J., Zheng, Y., Zhang, C., Xie, W., Xie, X., Sun, G., & Huang, Y. (2010).  T-drive: driving directions based on taxi trajectories.  In Proceedings of the 18th SIGSPATIAL International conference on advances in geographic information systems (pp. 99--108).
  * Zhang et al. [2019a] Zhang, C., James, J., & Liu, Y. (2019a).  Spatial-temporal graph attention networks: A deep learning approach for traffic forecasting.  IEEE Access, 7, 166246--166256.
  * Zhang et al. [2020a] Zhang, H., Liu, J., Tang, Y., & Xiong, G. (2020a).  Attention based graph covolution networks for intelligent traffic flow analysis.  In 2020 IEEE 16th International Conference on Automation Science and Engineering (CASE) (pp. 558--563).  IEEE.
  * Zhang et al. [2020b] Zhang, J., Chen, F., Cui, Z., Guo, Y., & Zhu, Y. (2020b).  Deep learning architecture for short-term passenger flow forecasting in urban rail transit.  IEEE Transactions on Intelligent Transportation Systems, .
  * Zhang et al. [2020c] Zhang, J., Chen, F., & Guo, Y. (2020c).  Multi-graph convolutional network for short-term passenger flow forecasting in urban rail transit.  IET Intelligent Transport Systems, .
  * Zhang et al. [2018a] Zhang, J., Shi, X., Xie, J., Ma, H., King, I., & Yeung, D.-Y. (2018a).  Gaan: Gated attention networks for learning on large and spatiotemporal graphs.  arXiv preprint arXiv:1803.07294, .
  * Zhang et al. [2017] Zhang, J., Zheng, Y., & Qi, D. (2017).  Deep spatio-temporal residual networks for citywide crowd flows prediction.  In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (pp. 1655--1661).
  * Zhang et al. [2020d] Zhang, K., He, F., Zhang, Z., Lin, X., & Li, M. (2020d).  Graph attention temporal convolutional network for traffic speed forecasting on road networks.  Transportmetrica B: Transport Dynamics, (pp. 1--19).
  * Zhang et al. [2019b] Zhang, N., Guan, X., Cao, J., Wang, X., & Wu, H. (2019b).  A hybrid traffic speed forecasting approach integrating wavelet transform and motif-based graph convolutional recurrent neural network.  arXiv preprint arXiv:1904.06656, .
  * Zhang et al. [2020e] Zhang, Q., Chang, J., Meng, G., Xiang, S., & Pan, C. (2020e).  Spatio-temporal graph structure learning for traffic forecasting.  In Proceedings of the AAAI Conference on Artificial Intelligence.  volume 34.
  * Zhang et al. [2018b] Zhang, Q., Jin, Q., Chang, J., Xiang, S., & Pan, C. (2018b).  Kernel-weighted graph convolutional network: A deep learning approach for traffic forecasting.  In 2018 24th International Conference on Pattern Recognition (ICPR) (pp. 1018--1023).  IEEE.
  * Zhang & Guo [2020] Zhang, T., & Guo, G. (2020).  Graph attention lstm: A spatio-temperal approach for traffic flow forecasting.  IEEE Intelligent Transportation Systems Magazine, .
  * Zhang et al. [2019c] Zhang, T., Jin, J., Yang, H., Guo, H., & Ma, X. (2019c).  Link speed prediction for signalized urban traffic network using a hybrid deep learning approach.  In 2019 IEEE Intelligent Transportation Systems Conference (ITSC) (pp. 2195--2200).  IEEE.
  * Zhang et al. [2020f] Zhang, W., Liu, H., Liu, Y., Zhou, J., & Xiong, H. (2020f).  Semi-supervised hierarchical recurrent graph neural network for city-wide parking availability prediction.  In Proceedings of the AAAI Conference on Artificial Intelligence.  volume 34.
  * Zhang et al. [2020g] Zhang, W., Liu, H., Liu, Y., Zhou, J., Xu, T., & Xiong, H. (2020g).  Semi-supervised city-wide parking availability prediction via hierarchical recurrent graph neural network.  IEEE Transactions on Knowledge and Data Engineering, .
  * Zhang et al. [2020h] Zhang, X., Huang, C., Xu, Y., & Xia, L. (2020h).  Spatial-temporal convolutional graph attention networks for citywide traffic flow forecasting.  In Proceedings of the 29th ACM International Conference on Information & Knowledge Management (pp. 1853--1862).
  * Zhang et al. [2020i] Zhang, X., Zhang, Z., & Jin, X. (2020i).  Spatial-temporal graph attention model on traffic forecasting.  In 2020 13th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI) (pp. 999--1003).  IEEE.
  * Zhang et al. [2019d] Zhang, Y., Cheng, T., & Ren, Y. (2019d).  A graph deep learning method for short-term traffic forecasting on large road networks.  Computer-Aided Civil and Infrastructure Engineering, 34, 877--896.
  * Zhang et al. [2020j] Zhang, Y., Cheng, T., Ren, Y., & Xie, K. (2020j).  A novel residual graph convolution deep learning model for short-term network-based traffic forecasting.  International Journal of Geographical Information Science, 34, 969--995.
  * Zhang et al. [2020k] Zhang, Y., Dong, X., Shang, L., Zhang, D., & Wang, D. (2020k).  A multi-modal graph neural network approach to traffic risk forecasting in smart urban sensing.  In 2020 17th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON) (pp. 1--9).  IEEE.
  * Zhang et al. [2020l] Zhang, Y., Lu, M., & Li, H. (2020l).  Urban traffic flow forecast based on fastgcrnn.  Journal of Advanced Transportation, 2020.
  * Zhang et al. [2019e] Zhang, Y., Wang, S., Chen, B., & Cao, J. (2019e).  Gcgan: Generative adversarial nets with graph cnn for network-scale traffic prediction.  In 2019 International Joint Conference on Neural Networks (IJCNN) (pp. 1--8).  IEEE.
  * Zhang et al. [2020m] Zhang, Z., Cui, P., & Zhu, W. (2020m).  Deep learning on graphs: A survey.  IEEE Transactions on Knowledge and Data Engineering, .
  * Zhang et al. [2019f] Zhang, Z., Li, M., Lin, X., Wang, Y., & He, F. (2019f).  Multistep speed prediction on traffic networks: A deep learning approach considering spatio-temporal dependencies.  Transportation research part C: emerging technologies, 105, 297--322.
  * Zhao et al. [2020a] Zhao, B., Gao, X., Liu, J., Zhao, J., & Xu, C. (2020a).  Spatiotemporal data fusion in graph convolutional networks for traffic prediction.  IEEE Access, .
  * Zhao et al. [2020b] Zhao, H., Yang, H., Wang, Y., Wang, D., & Su, R. (2020b).  Attention based graph bi-lstm networks for traffic forecasting.  In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC) (pp. 1--6).  IEEE.
  * Zhao et al. [2019] Zhao, L., Song, Y., Zhang, C., Liu, Y., Wang, P., Lin, T., Deng, M., & Li, H. (2019).  T-gcn: A temporal graph convolutional network for traffic prediction.  IEEE Transactions on Intelligent Transportation Systems, .
  * Zhao et al. [2021] Zhao, T., Liu, Y., Neves, L., Woodford, O., Jiang, M., & Shah, N. (2021).  Data augmentation for graph neural networks.  In Proceedings of the 30th International Joint Conference on Artificial Intelligence.  AAAI Press.
  * Zheng et al. [2020a] Zheng, B., Hu, Q., Ming, L., Hu, J., Chen, L., Zheng, K., & Jensen, C. S. (2020a).  Spatial-temporal demand forecasting and competitive supply via graph convolutional networks.  arXiv preprint arXiv:2009.12157, .
  * Zheng et al. [2020b] Zheng, C., Fan, X., Wang, C., & Qi, J. (2020b).  Gman: A graph multi-attention network for traffic prediction.  In Proceedings of the AAAI Conference on Artificial Intelligence.  volume 34.
  * Zhou et al. [2020a] Zhou, F., Yang, Q., Zhang, K., Trajcevski, G., Zhong, T., & Khokhar, A. (2020a).  Reinforced spatio-temporal attentive graph neural networks for traffic forecasting.  IEEE Internet of Things Journal, .
  * Zhou et al. [2020b] Zhou, F., Yang, Q., Zhong, T., Chen, D., & Zhang, N. (2020b).  Variational graph neural networks for road traffic prediction in intelligent transportation systems.  IEEE Transactions on Industrial Informatics, .
  * Zhou et al. [2020c] Zhou, J., Cui, G., Hu, S., Zhang, Z., Yang, C., Liu, Z., Wang, L., Li, C., & Sun, M. (2020c).  Graph neural networks: A review of methods and applications.  AI Open, 1, 57--81.
  * Zhou et al. [2020d] Zhou, Q., Gu, J.-J., Ling, C., Li, W.-B., Zhuang, Y., & Wang, J. (2020d).  Exploiting multiple correlations among urban regions for crowd flow prediction.  Journal of Computer Science and Technology, 35, 338--352.
  * Zhou et al. [2019] Zhou, X., Shen, Y., & Huang, L. (2019).  Revisiting flow information for traffic prediction.  arXiv preprint arXiv:1906.00560, .
  * Zhou et al. [2020e] Zhou, Z., Wang, Y., Xie, X., Chen, L., & Liu, H. (2020e).  Riskoracle: A minute-level citywide traffic accident forecasting framework.  In Proceedings of the AAAI Conference on Artificial Intelligence.  volume 34.
  * Zhou et al. [2020f] Zhou, Z., Wang, Y., Xie, X., Chen, L., & Zhu, C. (2020f).  Foresee urban sparse traffic accidents: A spatiotemporal multi-granularity perspective.  IEEE Transactions on Knowledge and Data Engineering, .
  * Zhu et al. [2019] Zhu, H., Luo, Y., Liu, Q., Fan, H., Song, T., Yu, C. W., & Du, B. (2019).  Multistep flow prediction on car-sharing systems: A multi-graph convolutional neural network with attention mechanism.  International Journal of Software Engineering and Knowledge Engineering, 29, 1727--1740.
  * Zhu et al. [2020a] Zhu, H., Xie, Y., He, W., Sun, C., Zhu, K., Zhou, G., & Ma, N. (2020a).  A novel traffic flow forecasting method based on rnn-gcn and brb.  Journal of Advanced Transportation, 2020.
  * Zhu et al. [2020b] Zhu, J., Han, X., Deng, H., Tao, C., Zhao, L., Tao, L., & Li, H. (2020b).  Kst-gcn: A knowledge-driven spatial-temporal graph convolutional network for traffic forecasting.  arXiv preprint arXiv:2011.14992, .
  * Zhu et al. [2020c] Zhu, J., Tao, C., Deng, H., Zhao, L., Wang, P., Lin, T., & Li, H. (2020c).  Ast-gcn: Attribute-augmented spatiotemporal graph convolutional network for traffic forecasting.  arXiv preprint arXiv:2011.11004, .


11institutetext: University of Pittsburgh, Pittsburgh, PA, USA
11email<EMAIL_ADDRESS>
22institutetext: University of Pittsburgh Medical Center, Pittsburgh, PA, USA

# Anatomy-Guided Weakly-Supervised Abnormality Localization in Chest X-rays

Ke Yu${}^{(\textrm{{\char 0\relax}})}$ 11 Shantanu Ghosh 11 Zhexiong Liu 11
Christopher Deible 22
Kayhan Batmanghelich 11

###### Abstract

Creating a large-scale dataset of abnormality annotation on medical images is
a labor-intensive and costly task. Leveraging _weak supervision_ from readily
available data such as radiology reports can compensate lack of large-scale
data for anomaly detection methods. However, most of the current methods only
use image-level pathological observations, failing to utilize the relevant
_anatomy mentions_ in reports. Furthermore, Natural Language Processing
(NLP)-mined weak labels are noisy due to label sparsity and linguistic
ambiguity. We propose an Anatomy-Guided chest X-ray Network (AGXNet) to
address these issues of weak annotation. Our framework consists of a cascade
of two networks, one responsible for identifying anatomical abnormalities and
the second responsible for pathological observations. The critical component
in our framework is an anatomy-guided attention module that aids the
downstream observation network in focusing on the relevant anatomical regions
generated by the anatomy network. We use Positive Unlabeled (PU) learning to
account for the fact that lack of mention does not necessarily mean a negative
label. Our quantitative and qualitative results on the MIMIC-CXR dataset
demonstrate the effectiveness of AGXNet in disease and anatomical abnormality
localization. Experiments on the NIH Chest X-ray dataset show that the learned
feature representations are transferable and can achieve the state-of-the-art
performances in disease classification and competitive disease localization
results. Our code is available at https://github.com/batmanlab/AGXNet.

###### Keywords:

Weakly-supervised learning PU learning Disease detection Class activation map
Residual attention

##  1 Introduction

There is considerable interest in developing automated abnormality detection
systems for chest X-rays (CXR) in order to improve radiologists’ workflow
efficiency and reduce observational oversights [19, 3, 13, 6]. Typically,
training a high-precision detection model using deep learning requires high-
quality annotations. However, collecting large-scale annotations by clinical
experts is time-consuming and prohibitively expensive. This motivates weakly-
supervised learning (WSL) methods [24, 16, 14, 2] that leverage weak
supervision from paired CXR reports that are readily available on a large
scale. There are, however, two unaddressed challenges: (1) The entire report
is often summarized to a small set of disease labels, which misses the
opportunity of incorporating anatomical context, and (2) Weak labels derived
from radiology reports using Natural Language Processing (NLP) are noisy due
to the sparsity of the labels and the ambiguity of the language. Improper
handling of labeling noise can result in underperforming models. We propose a
novel WSL framework to bridge these two gaps.

Figure 1:  Schematic diagram of the proposed AGXNet. Our architecture
comprises of two classification networks, $g_{a}$ for anatomical abnormalities
and $g_{o}$ for pathological observations. Relations $(a_{i},o_{k})=P$,
$(a_{j},o_{k})=P$ are parsed from the report and represent that the
observation $o_{k}$ is annotated as Present in two anatomical landmarks
$a_{i},a_{j}$, respectively. We obtain the anatomy-guided attention map
$H^{k}$ by aggregating CAMs of $a_{i},a_{j}$ and incorporate the $H^{k}$ as a
residual attention into $g_{o}$. The symbols $f_{a},f_{o}$ denote the
intermediate anatomy and observation features, respectively.

A variety of automated text labelers [9, 24, 18, 11] have been developed to
extract image-level observations from CXR reports. However, they do not
consider _anatomy mentions_ that provide important contexts for associated
observations. Aiming to fill this gap, the recently developed pipelines, Chest
ImaGenome [26] and RadGraph [10] extract fine-grained “observation-located at-
anatomy” relations, (e.g., “opacity in the right lower lung”), from CXR
reports. Yet none of them has been incorporated into a weakly-supervised
disease detection method. We build our WSL framework based on the RadGraph
dataset. Radiologists typically employ a systematic approach [21] when reading
CXR images to ensure that no significant abnormality is missed. This approach
is essentially documented in CXR reports which typically have imaging
observations paired with the related anatomical locations. We design an
architecture that reflects this process.

The second challenge is that weak labels extracted from CXR reports are
inherently noisy. Typically, only a subset of weak labels is mentioned in a
report, and unlabeled data is handled using a basic zero-imputation strategy.
However, lack of mention does not necessarily mean a negative label. For
example, a CXR report that contains mentions of strong visual clues for
pneumonia, such as opacity or consolidation, may not specifically establish
pneumonia as a diagnosis due to human variability in the reporting process. We
also consider uncertainty mentions (e.g., may, possible, can’t exclude) as
unlabeled, where the noise originates from the intrinsic ambiguities in
reports. We formulate this problem as a Positive Unlabeled (PU) learning [1]
where the learner has access to a small set of examples with high-confidence
labels (either positive or negative) and a large amount of unlabeled data
mixed by positive and negative examples with an unknown mixture ratio.

In this paper, we present Anatomy-Guided chest X-ray Networks (AGXNet), a
novel WSL framework that leverages information from pathological observations
and their associated anatomical landmarks mentioned in CXR reports. Our
architecture consists of a cascade of two networks, with the upstream anatomy
network tasked with identifying anatomical abnormalities in CXR images and the
downstream observation network tasked with identifying pathological
observations. The inclusion of an anatomy-guided attention (AGA) module aims
to aid the observation network paying attention to abnormalities in the
context of associated anatomical landmarks. During training, the AGA module
also provides a feedback loop to the upstream anatomy network, thus mutually
improving the quality of both types of features. In addition, we adopt a PU
learning technique to estimate the fraction of positive samples in the
unlabeled data and use a self-training approach to iteratively reduce noise in
labels. Our model is trained end-to-end. We evaluate the proposed framework on
the MIMIC-CXR [12] dataset. Results show that the proposed AGXNet model
outperforms both supervised and weakly-supervised baselines in disease
localization. In-depth ablation studies demonstrate that both AGA module and
PU learning can help to improve localization accuracy. We further evaluate the
pre-trained encoders on the NIH Chest X-ray dataset [24] and show that the
transferred models achieve the state-of-the-art (SOTA) results in disease
classification and competitive performance in disease localization.

##  2 Methods

We parse the report into an adjacency matrix that encodes the relations
between the observations and anatomical landmarks. The AGA module connects two
networks that are used to predict the presence of observations and anatomical
abnormalities in CXR images. We use PU learning to explicitly address noise in
unlabeled observations. The proposed framework is illustrated in Fig. 1.

Report Representation. We use the RadGraph [10] to parse reports to anatomical
landmark tokens $\\{a_{j}\\}$, observation tokens $\\{o_{k}\\}$, and encode
their relations using an adjacency matrix $A$. An entry $A(j,k)$ can take one
of the three variants $P,N,$ or $U$, representing whether the observation
$o_{k}$ is Present, Absent or Unlabeled in the anatomical landmark $a_{j}$.
Note, for $o_{k}$ that is mentioned without a specified location (e.g., “No
evidence of pneumothorax”), we link it to a special anatomical landmark token
named unspecified. Fig. A.1 shows an example adjacent matrix. We summarize the
image-level labels for observation and anatomical landmark using the adjacency
matrix. An observation $o_{k}$ is assigned a positive label ($y(o_{k})=1$) if
column $A(\cdot,k)$ has at least one $P$ entry, a negative label
($y(o_{k})=0$) if column $A(\cdot,k)$ has no $P$ entry and at least one $N$
entry, and assigned as unlabeled ($y(o_{k})=u$) if column $A(\cdot,k)$ only
has $U$ entries. For anatomy, an anatomical landmark is labeled as abnormal
$y(a_{j})=1$ if row $A(j,\cdot)$ has at least one $P$ entry, otherwise it is
labeled as normal $y(a_{j})=0$.

Anatomy Network. The anatomy network $g_{a}$ is responsible for identifying
abnormalities from anatomical landmarks. The weighted binary cross-entropy
(BCE) loss for $a_{j}$ is given by:

$\mathcal{L}_{a_{j}}(\theta_{a},w_{a}^{j})=-b_{j}^{+}\sum_{y(a_{j})=1}\text{ln}(g_{a}(x))-b_{j}^{-}\sum_{y(a_{j})=0}\text{ln}(1-g_{a}(x)),$
(1)

where $\theta_{a}$ is the parameter of anatomy encoder network, $w_{a}^{j}$ is
the classification weight corresponding to $a_{j}$, $x$ is an image,
$g_{a}(x)$ is the predicted probability of being abnormal in $a_{j}$,
$b_{j}^{+}$ and $b_{j}^{-}$ are balancing factors introduced in [20].

AGA. We introduce the AGA module to guide the downstream observation network
$g_{o}$ focusing on the relevant anatomical regions mentioned in the radiology
report. In detail, we construct the AGA map $H^{k}$ for observation $o_{k}$ by
aggregating the class activation maps (CAMs) [27] for locations $\\{a_{j}\\}$
where $o_{k}$ was positively observed, i.e., $A(a_{j},o_{k})=P$. Formally,

$H^{k}(\theta_{a},w_{a}^{j})=\sum_{j}\mathds{1}\\{A(a_{j},o_{k})=P\\}\underbrace{\sum_{c}w_{a}^{j}f_{a}(:,:,c;\theta_{a})}_{\text{CAM}(a_{j})},$
(2)

where $f_{a}(:,:,c)$ represents the activation of channel $c$ in anatomy
feature map $f_{a}$, and $\mathds{1}\\{A(a_{j},o_{k})=P\\}$ is an indicator
function. We then transform the values of $H^{k}$ to $[0,1]$ using the min-max
normalization.

Observation Network. The observation network $g_{o}$ is responsible for
identifying the presence of pathological observations in CXR images. We
incorporate the $H^{k}$ into $g_{o}$ as a residual attention map and modify
the observation feature map $f_{o}$ as follows,
$f^{\prime}_{o_{k}}=(1+\beta\cdot H^{k})\odot f_{o}$, where $\odot$ indicates
element-wise multiplication for spatial positions, and $\beta$ is a scaling
hyperparameter. The weighted BCE loss for $o_{k}$ is given by:

$\mathcal{L}_{o_{k}}(\mathbf{\Theta})=-b_{k}^{+}\sum_{y(o_{k})=1}\text{ln}(g_{o}(x,H^{k}))-b_{k}^{-}\sum_{y(o_{k})\in\\{0,u\\}}\text{ln}(1-g_{o}(x,H^{k})),$
(3)

where $\mathbf{\Theta}=(\theta_{o},w_{o}^{k},\theta_{a},w_{a}^{j})$,
$\theta_{o}$ is the parameter of observation encoder network, $w_{o}^{k}$ is
the classification weight corresponding to $o_{k}$, $g_{o}(x,H^{k})$ is the
predicted probability of $o_{k}$ being present in the image, $b_{k}^{+}$ and
$b_{k}^{+}$ are balancing factors. It is important to note that mapping all
unlabeled examples to negative ($|y(o_{k})\in\\{0,u\\}|=|N|+|U|$) would
overlook the noise in the unlabeled data, which can degrade model performance.
We explicitly handle the randomness present in unlabeled data and formulate
this problem using a PU learning approach.

PU Learning. The distribution of unlabeled data $P_{u}$ can be decomposed as,
$P_{u}=\alpha P_{p}+(1-\alpha)P_{n}$, where $\alpha$ denotes the mixture
proportion of positive examples in the unlabeled data, and $P_{p},P_{n}$
denotes the class-conditional distribution for positive and negative class,
respectively. We adopt the method Best Bin Estimation (BBE) proposed by Garg
et al. [5] to estimate $\alpha$. In short, let $X_{p},X_{n},X_{u}$ denote the
positive, negative and unlabeled samples for $o_{k}$ in the validation data
set, $\hat{F}_{p}(z),\hat{F}_{u}(z)$ denote the empirical cumulative
distributions of the predicted probabilities of the observation network, named
$Z_{p},Z_{u}$. The mixture proportion $\alpha$ is estimated by minimizing the
upper confidence bound of the ratio, namely
$(1-\hat{F}_{u}(z))/(1-\hat{F}_{p}(z))$. We integrate the BBE with an
iterative self-training approach summarized as follows: (1) warm-start
training with treating all unlabeled samples as negative, (2) estimating
$\alpha$ using BBE, (3) removing $\alpha$ fraction of unlabeled training
samples scored as most positive and relabeling the rest $1-\alpha$ unlabeled
samples as negative, (4) updating the model using positive samples ($|P|$) and
provisional negative samples ($|N|+(1-\alpha)|U|$). We repeat steps 2 to 4
until the classifiers reach the best validation performance.

Optimization. The final loss is given by:
$\mathcal{L}=\frac{1}{N_{a}}\sum_{j}^{N_{a}}\mathcal{L}_{a_{j}}+\frac{1}{N_{o}}\sum_{k}^{N_{o}}\mathcal{L}_{o_{k}}$,
where $N_{a}$, $N_{o}$ are the number of anatomy and observation labels used
in training, respectively. The network is trained end-to-end. Importantly,
during training, the gradients $\partial\mathcal{L}_{o_{k}}/\partial\theta$,
$\partial\mathcal{L}_{o_{k}}/\partial w_{a}^{j}$ provide a feedback loop to
the anatomy network through the AGA module, mutually reinforcing the anatomy
and observation features. During inference time, we do not require any text
input and set $H^{k}=0$.

##  3 Experiments

Experiments are carried out to evaluate the performance of AGXNet in
abnormality classification and localization. We conduct ablation studies to
validate the efficiency of the AGA module and PU learning . We also test the
robustness and transferability of the learned anatomy and observation
features.

Experimental Details. We first evaluate the proposed AGXNet on the MIMIC-CXR
dataset [12]. The RadGraph’s inference dataset [10] provides annotations
automatically generated by DYGIE++ [23] model for 220,763 MIMIC-CXR reports.
We select the corresponding 220,763 frontal images from the MIMIC-CXR dataset
and obtain their adjacency matrix representations from the RadGraph
annotations. The adjacency matrix of each sample has 48 rows (46 anatomical
landmarks, 1 unspecified, 1 other anatomies representing anatomy tokens in the
tail distribution) and 64 columns (63 mostly mentioned observations and 1
other observations representing observation tokens in the tail distribution).
Appx. B provides additional details of anatomical labels. For evaluation of
disease localization, we use a held out set [22] of MIMIC-CXR images with 390
bounding boxes (BBox) for pneumonia ($196/390$) and pneumothorax ($194/390$)
annotated by board-certified radiologists. For evaluation of anatomical
abnormality localization, we utilize the anatomy BBox from the Chest ImaGenome
dataset [26], which were extracted by an atlas-based detection pipeline. We
use a 80%-10%-10% train-validation-test split with no patient shared across
splits. For both anatomy and observation encoders in AGXNet, we use
DenseNet-121 [8] with pre-trained weights from ImageNet [4] as the backbone.
We train our framework to predict presence of abnormality in the 46 anatomical
landmarks and the presence of two diseases (i.e., pneumonia and pneumothorax)
that have ground truth BBox in [22]. The $\beta$ is set to be $0.1$ based on
validation results. We optimize the networks using SGD with
$\text{momentum}=0.9$, $\text{weight decay}=10e^{-4}$ and stop training once
the validation error reaches minimum. The learning rate is set to be 0.01 and
divided by 10 every 6 epochs. We resize all images to $512\times 512$ without
any data augmentation and set the batch size as 16. The model is implemented
in PyTorch and trained on a single NVIDIA GPU with 32G of memory.

Evaluation Metric. We produce disease-specific CAMs for pneumonia and
pneumothorax, and anatomy-specific CAMs for the anatomical landmarks. We apply
a thresholding-based bounding box generation method and extract isolated
regions in which pixel values are greater than the 95% quantile of the CAM’s
pixel value distribution. We evaluate the generated boxes against the ground
truth BBox using intersection over union ratio (IoU). A generated box is
considered as a true positive when $\text{IoU}>T(\text{IoU})$, where $T(\ast)$
is a threshold.

Table 1: Disease localization on MIMIC-CXR. Results are reported as the
average over 5 independent runs with standard deviations. The highest values
are highlighted in bold, and the best results without BBox annotation are
underlined.

IoU @ 0.1 IoU @ 0.25 IoU @ 0.5 Disease Model Supervised Recall Precision
Recall Precision Recall Precision RetinaNet [15] ✓ $0.52_{\pm 0.04}$
$0.19_{\pm 0.00}$ $0.41_{\pm 0.04}$ $0.15_{\pm 0.00}$ $\bm{0.26_{\pm 0.00}}$
$0.09_{\pm 0.00}$ CheXpert [9] ✗ $0.68_{\pm 0.01}$ $0.40_{\pm 0.01}$
$0.47_{\pm 0.01}$ $0.28_{\pm 0.00}$ $0.14_{\pm 0.01}$ $0.08_{\pm 0.01}$ AGXNet
w/o AGA ✗ $0.67_{\pm 0.04}$ $0.41_{\pm 0.03}$ $0.54_{\pm 0.04}$ $0.33_{\pm
0.03}$ $0.16_{\pm 0.02}$ $\bm{0.10_{\pm 0.01}}$ AGXNet w/ AGA ✗ $\bm{0.75_{\pm
0.01}}$ $\bm{0.44_{\pm 0.01}}$ $\bm{0.60_{\pm 0.02}}$ $\bm{0.35_{\pm 0.01}}$
$0.16_{\pm 0.00}$ $0.09_{\pm 0.00}$ Pneumothorax AGXNet w/ AGA + PU ✗
$0.74_{\pm 0.03}$ $0.43_{\pm 0.02}$ $0.58_{\pm 0.03}$ $0.34_{\pm 0.02}$
$0.13_{\pm 0.01}$ $0.07_{\pm 0.01}$ RetinaNet [15] ✓ $0.66_{\pm 0.04}$
$0.27_{\pm 0.04}$ $0.62_{\pm 0.04}$ $0.26_{\pm 0.09}$ $\bm{0.42_{\pm 0.04}}$
$\bm{0.17_{\pm 0.03}}$ CheXpert [9] ✗ $0.73_{\pm 0.01}$ $0.40_{\pm 0.01}$
$0.51_{\pm 0.01}$ $0.28_{\pm 0.01}$ $0.19_{\pm 0.01}$ $0.10_{\pm 0.01}$ AGXNet
w/o AGA ✗ $0.69_{\pm 0.01}$ $0.44_{\pm 0.01}$ $0.59_{\pm 0.02}$ $0.37_{\pm
0.02}$ $0.18_{\pm 0.02}$ $0.11_{\pm 0.01}$ AGXNet w/ AGA ✗ $0.67_{\pm 0.02}$
$0.46_{\pm 0.01}$ $0.58_{\pm 0.02}$ $0.39_{\pm 0.02}$ $0.18_{\pm 0.03}$
$0.12_{\pm 0.02}$ Pneumonia AGXNet w/ AGA + PU ✗ $\bm{0.73_{\pm 0.01}}$
$\bm{0.47_{\pm 0.01}}$ $\bm{0.62_{\pm 0.01}}$ $\bm{0.40_{\pm 0.01}}$
$0.20_{\pm 0.01}$ $0.13_{\pm 0.01}$

Evaluating Disease Localization. We trained a RetinaNet [15] using the
annotated BBox from [22] as the supervised baseline and a DenseNet-121 using
CheXpert [9] disease labels for CAM-based localization as the WSL baseline. We
investigated three variants of AGXNet to understand the effects of AGA module
and PU learning. Tab. 1 shows that AGXNet trained with PU learning achieved
the best localization results for pneumonia at $T(\text{IoU})=0.1$ and $0.25$,
while AGXNet trained without PU learning performed best in pneumothorax
localization at the same IoU thresholds. Note that the label noise in
pneumothorax is intrinsically low, thus adding PU learning may not
significantly improve its localization accuracy. RetinaNet achieved the best
localization results at $T(\text{IoU})=0.5$, probably due to its direct
predictions for the coordinates of BBox.

Table 2: Ablation studies on AGA and PU Learning. Results are averaged over 5
runs. $\alpha$: mixture proportion of positive samples. AA: anatomic
abnormality. LAL: left apical lung. LL: left lung. RAL: right apical lung.
RLL: right lower lung. RL: right lung.

Model Classification AA Localization Pneumonia Pneumothorax Accuracy @ IoU =
0.25 $\alpha$ AUPRC $\alpha$ AUPRC LAL LL RAL RLL RL AGXNet w/o AGA - 0.57 -
0.55 0.02 0.10 0.08 0.43 0.12 AGXNet w/ AGA - 0.57 - 0.57 0.28 0.22 0.49 0.63
0.22 AGXNet w/ AGA + PU 0.14 0.62 0.03 0.57 0.38 0.23 0.50 0.69 0.22

Ablation Studies. Tab. 2 shows additional results of ablation studies for
AGXNet, including (1) disease classification performance on the positive and
negative samples in the test set using the area under the precision-recall
curve (AUPRC), and (2) anatomical abnormality localization accuracy at
$T(\text{IoU})=0.25$ for $\\{a_{j}\\}$, where $A(a_{j},o_{k})=P$ and
$o_{k}\in\\{\text{pneumonia, pneumothorax}\\}$. Effect of AGA: Results in Tab.
1 and Tab. 2 show that the AGA module significantly improved the results of
pneumothorax detection and anatomical abnormality detection, suggesting that
the attention module mutually enhanced both types of features. Fig. 2 shows
the qualitative comparison of two models with (M2) and without (M1) the AGA
module. M2 correctly detects pneumothorax and the relevant abnormal anatomical
landmarks in both examples, while M1 fails to do so. Furthermore, in the
second example, a pigtail catheter is applied to treat pneumothorax and acts
as the only discriminative feature used by M1, while M2 is more robust against
this shortcut and detects disease in the correct anatomical location. Effect
of PU Learning: Tab. 2 shows that the estimated mixture proportion of positive
examples ($\alpha$) is significantly higher in pneumonia ($14\%$) than in
pneumothorax ($3\%$), reflecting the fact that there is considerable
variability in pneumonia diagnoses [17]. Accordingly, results in Tab. 1 and
Tab. 2 show that using the PU learning technique improves the performance of
pneumonia classification and localization, while it is less effective for
pneumothorax whose label noise is intrinsically low.

Figure 2: AGXNet with (M2) or without (M1) the AGA module for pneumothorax
detection. Heatmaps indicate the CAMs. The blue and green boxes stand for
ground truth anatomy and disease annotations. The red boxes are generated from
the CAMs.  Table 3: The test AUROCs for the NIH Chest X-ray disease
classification task.

Method Atelectasis Cardiomegaly Effusion Infiltration Mass Nodule Pneumonia
Pneumothorax Mean Scratch 0.79 0.91 0.88 0.69 0.81 0.70 0.70 0.82 0.79 Wang et
al. [24] 0.72 0.81 0.78 0.61 0.71 0.67 0.63 0.81 0.72 Wang et al. [25] 0.73
0.84 0.79 0.67 0.73 0.69 0.72 0.85 0.75 Liu et al. [16] 0.79 0.91 0.88 0.69
0.81 0.73 0.75 0.89 0.80 Rajpurkar et al. [20] 0.82 0.91 0.88 0.72 0.86 0.78
0.76 0.89 0.83 Han et al. [7] 0.84 0.93 0.88 0.72 0.87 0.79 0.77 0.90 0.84
AGXNet - Ana. 0.84 0.91 0.90 0.71 0.86 0.80 0.74 0.86 0.83 AGXNet - Obs. 0.85
0.92 0.90 0.72 0.86 0.80 0.76 0.87 0.84 AGXNet - Both 0.85 0.92 0.90 0.72 0.87
0.80 0.76 0.88 0.84

Table 4: The localization accuracy for the NIH Chest X-ray disease
localization task. The highest values are highlighted in bold, and the best
results without BBox annotation are underlined.

T(IoU) Method Supervised Atelectasis Cardiomegaly Effusion Infiltration Mass
Nodule Pneumonia Pneumothorax Mean 0.1 Scratch ✗ 0.46 0.85 0.61 0.40 0.48 0.11
0.56 0.25 0.47 Han et al. [7] ✓ 0.72 0.96 0.88 0.93 0.74 0.45 0.65 0.64 0.75
Wang et al. [24] ✗ 0.69 0.94 0.66 0.71 0.40 0.14 0.63 0.38 0.57 AGXNet - Ana.
✗ 0.64 0.99 0.66 0.69 0.69 0.29 0.73 0.29 0.62 AGXNet - Obs. ✗ 0.61 0.99 0.70
0.70 0.69 0.23 0.72 0.55 0.65 AGXNet - Both ✗ 0.71 1.00 0.69 0.68 0.68 0.22
0.71 0.62 0.66 0.3 Scratch ✗ 0.10 0.45 0.34 0.15 0.14 0.00 0.31 0.06 0.19 Han
et al. [7] ✓ 0.39 0.85 0.60 0.67 0.43 0.21 0.40 0.45 0.50 Wang et al. [24] ✗
0.24 0.46 0.30 0.28 0.15 0.04 0.17 0.13 0.22 AGXNet - Ana. ✗ 0.20 0.48 0.40
0.43 0.28 0.00 0.46 0.13 0.30 AGXNet - Obs. ✗ 0.22 0.62 0.45 0.46 0.27 0.01
0.54 0.31 0.36 AGXNet - Both ✗ 0.27 0.56 0.47 0.45 0.27 0.01 0.50 0.36 0.36

Transfer Learning on NIH Chest X-ray. We pre-trained an
$\text{AGXNet}+\text{PU}$ model on the MIMIC-CXR dataset using the 46 anatomy
labels and 8 observation labels listed in C.1. We then fine-tuned the
encoder(s) and re-trained a classifier using the NIH Chest X-ray dataset [24].
We investigated three variants of fine-tuning regimes: (1) only using anatomy
encoder, (2) only using observation encoder, and (3) using both encoders and
their concatenated embeddings. We compare our transferred models with a
baseline model trained from scratch using the NIH Chest X-ray dataset and a
series of relevant baselines. We evaluate the classification performance using
the area under the ROC curve (AUROC) score and the localization accuracy at
$T(\text{IoU})=0.1$, $0.3$ across the 8 diseases. Tab. 3 shows that all
variants of fine-tuned AGXNet models achieve disease classification
performances comparable to the SOTA method [7], demonstrating that both
learned anatomy and observation features are robust and transferable. Tab. 4
shows that all variants of fine-tuned AGXNet models outperform learning from
scratch and the existing CAM-based baseline method [24] in disease
localization. Note that the model proposed by Han et al. [7] achieved higher
accuracy by utilizing the ground truth BBox during training, therefore it is
not directly comparable and should be viewed as an upper bound method.

##  4 Conclusion

In this work, we propose a novel WSL framework to incorporate anatomical
contexts mentioned in radiology reports to facilitate disease detection on
corresponding CXR images. In addition, we use a PU learning approach to
explicitly handle noise in unlabeled data. Experimental evaluations on the
MIMIC-CXR dataset show that the addition of anatomic knowledge and the use of
PU learning improve abnormality localization. Experiments on the NIH Chest
X-ray datasets demonstrate that the learned anatomical and pathological
features are transferable and encode robust classification and localization
information.

###  4.0.1 Acknowledgements

This work was partially supported by NIH Award Number 1R01HL141813-01, NSF
1839332 Tripod+X, and SAP SE. We are grateful for the computational resources
provided by Pittsburgh SuperComputing grant number TG-ASC170024.

## References

  * [1] Bekker, J., Davis, J.: Learning from positive and unlabeled data: A survey. Machine Learning 109(4), 719–760 (2020), https://doi.org/10.1007/s10994-020-05877-5
  * [2] Bhalodia, R., Hatamizadeh, A., Tam, L., Xu, Z., Wang, X., Turkbey, E., Xu, D.: Improving pneumonia localization via cross-attention on medical images and reports. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 571–581. Springer (2021), https://doi.org/10.1007/978-3-030-87196-3_53
  * [3] Castellino, R.A.: Computer aided detection (cad): an overview. Cancer Imaging 5(1), 17 (2005)
  * [4] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [5] Garg, S., Wu, Y., Smola, A.J., Balakrishnan, S., Lipton, Z.: Mixture proportion estimation and pu learning: A modern approach. Advances in Neural Information Processing Systems 34 (2021)
  * [6] Gromet, M.: Comparison of computer-aided detection to double reading of screening mammograms: review of 231,221 mammograms. American Journal of Roentgenology 190(4), 854–859 (2008)
  * [7] Han, Y., Chen, C., Tewfik, A., Glicksberg, B., Ding, Y., Peng, Y., Wang, Z.: Knowledge-augmented contrastive learning for abnormality classification and localization in chest x-rays with radiomics using a feedback loop. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2465–2474 (2022)
  * [8] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4700–4708 (2017)
  * [9] Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI conference on artificial intelligence. vol. 33, pp. 590–597 (2019)
  * [10] Jain, S., Agrawal, A., Saporta, A., Truong, S.Q., Bui, T., Chambon, P., Zhang, Y., Lungren, M.P., Ng, A.Y., Langlotz, C., et al.: Radgraph: Extracting clinical entities and relations from radiology reports (2021)
  * [11] Jain, S., Smit, A., Truong, S.Q., Nguyen, C.D., Huynh, M.T., Jain, M., Young, V.A., Ng, A.Y., Lungren, M.P., Rajpurkar, P.: Visualchexbert: addressing the discrepancy between radiology report labels and image labels. In: Proceedings of the Conference on Health, Inference, and Learning. pp. 105–115 (2021)
  * [12] Johnson, A.E., Pollard, T.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Peng, Y., Lu, Z., Mark, R.G., Berkowitz, S.J., Horng, S.: Mimic-cxr-jpg - chest radiographs with structured labels (version 2.0.0). PhysioNet (2019), https://doi.org/10.13026/8360-t248
  * [13] Lakhani, P., Sundaram, B.: Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. Radiology 284(2), 574–582 (2017)
  * [14] Li, Z., Wang, C., Han, M., Xue, Y., Wei, W., Li, L.J., Fei-Fei, L.: Thoracic disease identification and localization with limited supervision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8290–8299 (2018)
  * [15] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980–2988 (2017)
  * [16] Liu, J., Zhao, G., Fei, Y., Zhang, M., Wang, Y., Yu, Y.: Align, attend and locate: Chest x-ray diagnosis via contrast induced attention network with limited supervision. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 10632–10641 (2019)
  * [17] Neuman, M.I., Lee, E.Y., Bixby, S., Diperna, S., Hellinger, J., Markowitz, R., Servaes, S., Monuteaux, M.C., Shah, S.S.: Variability in the interpretation of chest radiographs for the diagnosis of pneumonia in children. Journal of hospital medicine 7(4), 294–298 (2012)
  * [18] Peng, Y., Wang, X., Lu, L., Bagheri, M., Summers, R., Lu, Z.: Negbio: a high-performance tool for negation and uncertainty detection in radiology reports. AMIA Summits on Translational Science Proceedings 2018, 188 (2018)
  * [19] Qin, C., Yao, D., Shi, Y., Song, Z.: Computer-aided detection in chest radiography based on artificial intelligence: a survey. Biomedical engineering online 17(1), 1–23 (2018)
  * [20] Rajpurkar, P., Irvin, J., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C., Shpanskaya, K., et al.: Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225 (2017)
  * [21] Sait, S., Tombs, M.: Teaching medical students how to interpret chest x-rays: the design and development of an e-learning resource. Advances in Medical Education and Practice 12, 123 (2021)
  * [22] Tam, L.K., Wang, X., Turkbey, E., Lu, K., Wen, Y., Xu, D.: Weakly supervised one-stage vision and language disease detection using large scale pneumonia and pneumothorax studies. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 45–55. Springer (2020), https://doi.org/10.1007/978-3-030-59719-1_5
  * [23] Wadden, D., Wennberg, U., Luan, Y., Hajishirzi, H.: Entity, relation, and event extraction with contextualized span representations. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). pp. 5784–5789 (Nov 2019). https://doi.org/10.18653/v1/D19-1585, https://aclanthology.org/D19-1585
  * [24] Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2097–2106 (2017)
  * [25] Wang, X., Peng, Y., Lu, L., Lu, Z., Summers, R.M.: Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 9049–9058 (2018)
  * [26] Wu, J.T., Agu, N.N., Lourentzou, I., Sharma, A., Paguio, J.A., Yao, J.S., Dee, E.C., Mitchell, W.G., Kashyap, S., Giovannini, A., Celi, L.A., Moradi, M.: Chest imagenome dataset for clinical reasoning. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) (2021)
  * [27] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2921–2929 (2016)

##  Appendix 0.A Example of Adjacency Matrix

Figure 0.A.1: An example of adjacency matrix derived from RadGraph annotations
for a report. ANAT denotes anatomy mention. OBS-DP, OBS-DA and OBS-U denote
observation mentions that are definitely present, definitely absent and
uncertain.

##  Appendix 0.B Anatomical Abnormality Localization

Figure 0.B.2: Abnormality localization for different anatomical landmarks
within the same image. The blue boxes stand for ground truth anatomy
annotations. The red boxes stand for the detected abnormal regions in the
anatomical landmark. The predicted probabilities of anatomy network in AGXNet
are shown in the titles.

##  Appendix 0.C Transfer Learning on the NIH Chest X-ray Dataset

Table 0.C.1: Summary of observation labels used for pre-training.

Observation P (%) N (%) U (%) Top 3 Associated Anatomical Landmarks (% in P)
Atelectasis 21.9 0.3 77.8 Lung bases (28.4) Unspecified (15.6) Left lower lung
(12.5) Effusion 23.6 49.2 27.2 Pleural (40.5) Right pleural (22.5) Left
pleural (21.2) Pneumonia 3.5 14.1 82.4 Unspecified (68.6) Lower left lobe
(7.5) Lower right lobe (6.7) Pneumothorax 3.8 60.8 35.5 Right apical lung
(22.1) Left apical lung (16.5) Right lung (15.9) Cardiomegaly 14.1 0.2 85.7
Unspecified (99.8) Mediastinal (0.05) Heart (0.02) Enlarge 13.6 0.8 85.6 Heart
(77.1) Mediastinal (4.7) Unspecified (2.9) Nodule 1.6 0.6 97.8 Unspecified
(13) Upper right lobe (5.4) Upper left lobe (4) Mass 1.1 0.4 98.5 Unspecified
(11.4) Right hilar (10.1) Mediastinal (7.5)

Table 0.C.2: The localization accuracy for the NIH Chest X-ray disease
localization task by IoU levels.

T(IoU) Method Atelectasis Cardiomegaly Effusion Infiltration Mass Nodule
Pneumonia Pneumothorax Mean 0.1 AGXNet - Ana. 0.64 0.99 0.66 0.69 0.69 0.29
0.73 0.29 0.62 AGXNet - Obs. 0.61 0.99 0.70 0.70 0.69 0.23 0.72 0.55 0.65
AGXNet - Both 0.71 1.00 0.69 0.68 0.68 0.22 0.71 0.62 0.66 0.3 AGXNet - Ana.
0.20 0.48 0.40 0.43 0.28 0.00 0.46 0.13 0.30 AGXNet - Obs. 0.22 0.62 0.45 0.46
0.27 0.01 0.54 0.31 0.36 AGXNet - Both 0.27 0.56 0.47 0.45 0.27 0.01 0.50 0.36
0.36 0.5 AGXNet - Ana. 0.03 0.03 0.10 0.21 0.09 0.00 0.19 0.02 0.08 AGXNet -
Obs. 0.03 0.01 0.13 0.24 0.14 0.00 0.26 0.08 0.11 AGXNet - Both 0.03 0.01 0.12
0.26 0.11 0.00 0.25 0.14 0.12

##  Appendix 0.D Effect of Scaling Hyperparameter $\beta$ in Residual
Attention

Table 0.D.3: Disease localization on MIMIC-CXR.

IoU @ 0.1 IoU @ 0.25 IoU @ 0.5 Disease AGXNet Recall Precision Recall
Precision Recall Precision $\beta=0$ $0.67_{\pm 0.04}$ $0.41_{\pm 0.03}$
$0.54_{\pm 0.04}$ $0.33_{\pm 0.03}$ $\bm{0.16_{\pm 0.02}}$ $\bm{0.10_{\pm
0.01}}$ $\beta=0.05$ $0.71_{\pm 0.02}$ $0.41_{\pm 0.02}$ $0.58_{\pm 0.02}$
$0.33_{\pm 0.01}$ $0.15_{\pm 0.01}$ $0.09_{\pm 0.00}$ Pneumothorax $\beta=0.1$
$\bm{0.74_{\pm 0.03}}$ $\bm{0.43_{\pm 0.02}}$ $\bm{0.58_{\pm 0.03}}$
$\bm{0.34_{\pm 0.02}}$ $0.13_{\pm 0.01}$ $0.07_{\pm 0.01}$ $\beta=0$
$0.69_{\pm 0.01}$ $0.44_{\pm 0.01}$ $0.59_{\pm 0.02}$ $0.37_{\pm 0.02}$
$0.18_{\pm 0.02}$ $0.11_{\pm 0.01}$ $\beta=0.05$ $0.71_{\pm 0.01}$ $0.45_{\pm
0.02}$ $0.58_{\pm 0.02}$ $0.36_{\pm 0.02}$ $0.17_{\pm 0.01}$ $0.11_{\pm 0.01}$
Pneumonia $\beta=0.1$ $\bm{0.73_{\pm 0.01}}$ $\bm{0.47_{\pm 0.01}}$
$\bm{0.62_{\pm 0.01}}$ $\bm{0.40_{\pm 0.01}}$ $\bm{0.20_{\pm 0.01}}$
$\bm{0.13_{\pm 0.01}}$