* [84] W. Chen, Z. Zheng, J. Cui, E. Ngai, P. Zheng, and Y. Zhou, “Detecting ponzi schemes on ethereum: Towards healthier blockchain technology,” in Proceedings of the 2018 world wide web conference, pp. 1409–1418, 2018.
  * [85] W. Chen, Z. Zheng, E. C.-H. Ngai, P. Zheng, and Y. Zhou, “Exploiting blockchain data to detect smart ponzi schemes on ethereum,” IEEE Access, vol. 7, pp. 37575–37586, 2019.
  * [86] L. Wang, H. Cheng, Z. Zheng, A. Yang, and X. Zhu, “Ponzi scheme detection via oversampling-based long short-term memory for smart contracts,” Knowledge-Based Systems, vol. 228, p. 107312, 2021.
  * [87] Y. Zhang, W. Yu, Z. Li, S. Raza, and H. Cao, “Detecting ethereum ponzi schemes based on improved lightgbm algorithm,” IEEE Transactions on Computational Social Systems, vol. 9, no. 2, pp. 624–637, 2021.
  * [88] E. Jung, M. Le Tilly, A. Gehani, and Y. Ge, “Data mining-based ethereum fraud detection,” in 2019 IEEE International Conference on Blockchain (Blockchain), pp. 266–273, IEEE, 2019.
  * [89] J. Peng and G. Xiao, “Detection of smart ponzi schemes using opcode,” in Blockchain and Trustworthy Systems: Second International Conference, BlockSys 2020, Dali, China, August 6–7, 2020, Revised Selected Papers 2, pp. 192–204, Springer, 2020.
  * [90] S. Fan, S. Fu, H. Xu, and X. Cheng, “Al-spsd: Anti-leakage smart ponzi schemes detection in blockchain,” Information Processing & Management, vol. 58, no. 4, p. 102587, 2021.
  * [91] A. Aljofey, Q. Jiang, and Q. Qu, “A supervised learning model for detecting ponzi contracts in ethereum blockchain,” in International Conference on Big Data and Security, pp. 657–672, Springer, 2021.
  * [92] W. Sun, G. Xu, Z. Yang, and Z. Chen, “Early detection of smart ponzi scheme contracts based on behavior forest similarity,” in 2020 IEEE 20th International Conference on Software Quality, Reliability and Security (QRS), pp. 297–309, IEEE, 2020.
  * [93] B. Fu, X. Yu, and T. Feng, “Ct-gcn: a phishing identification model for blockchain cryptocurrency transactions,” International Journal of Information Security, vol. 21, no. 6, pp. 1223–1232, 2022.
  * [94] L. Chen, J. Peng, Y. Liu, J. Li, F. Xie, and Z. Zheng, “Phishing scams detection in ethereum transaction network,” ACM Transactions on Internet Technology (TOIT), vol. 21, no. 1, pp. 1–16, 2020.
  * [95] Q. Yuan, B. Huang, J. Zhang, J. Wu, H. Zhang, and X. Zhang, “Detecting phishing scams on ethereum based on transaction records,” in 2020 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1–5, IEEE, 2020.
  * [96] Z. Yuan, Q. Yuan, and J. Wu, “Phishing detection on ethereum via learning representation of transaction subgraphs,” in Blockchain and Trustworthy Systems: Second International Conference, BlockSys 2020, Dali, China, August 6–7, 2020, Revised Selected Papers 2, pp. 178–191, Springer, 2020.
  * [97] J. Wu, Q. Yuan, D. Lin, W. You, W. Chen, C. Chen, and Z. Zheng, “Who are the phishers? phishing scam detection on ethereum via network embedding,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 52, no. 2, pp. 1156–1166, 2020.
  * [98] Y. Xia, J. Liu, and J. Wu, “Phishing detection on ethereum via attributed ego-graph embedding,” IEEE Transactions on Circuits and Systems II: Express Briefs, vol. 69, no. 5, pp. 2538–2542, 2022.
  * [99] H. Wen, J. Fang, J. Wu, and Z. Zheng, “Transaction-based hidden strategies against general phishing detection framework on ethereum,” in 2021 IEEE International Symposium on Circuits and Systems (ISCAS), pp. 1–5, IEEE, 2021.
  * [100] T. Wen, Y. Xiao, A. Wang, and H. Wang, “A novel hybrid feature fusion model for detecting phishing scam on ethereum using deep neural network,” Expert Systems with Applications, vol. 211, p. 118463, 2023.
  * [101] Y. Wang, Z. Liu, J. Xu, and W. Yan, “Heterogeneous network representation learning approach for ethereum identity identification,” IEEE Transactions on Computational Social Systems, 2022.
  * [102] X. Zhou, W. Yang, and X. Tian, “Detecting phishing accounts on ethereum based on transaction records and egat,” Electronics, vol. 12, no. 4, p. 993, 2023.
  * [103] J. Kim, S. Lee, Y. Kim, S. Ahn, and S. Cho, “Graph learning-based blockchain phishing account detection with a heterogeneous transaction graph,” Sensors, vol. 23, no. 1, p. 463, 2023.
  * [104] A. H. H. Kabla, M. Anbar, S. Manickam, and S. Karupayah, “Eth-psd: A machine learning-based phishing scam detection approach in ethereum,” IEEE Access, vol. 10, pp. 118043–118057, 2022.
  * [105] W. Chen, J. Cui, X. Guo, Z. Chen, and Y. Lu, “Misbehavior detection on blockchain data,” in Blockchain Intelligence, pp. 95–133, Springer, 2021.
  * [106] W. Chen, Y. Xu, Z. Zheng, Y. Zhou, J. E. Yang, and J. Bian, “Detecting” pump & dump schemes” on cryptocurrency market using an improved apriori algorithm,” in 2019 IEEE International Conference on Service-Oriented System Engineering (SOSE), pp. 293–2935, IEEE, 2019.
  * [107] P. Monamo, V. Marivate, and B. Twala, “Unsupervised learning for robust bitcoin fraud detection,” in 2016 Information Security for South Africa (ISSA), pp. 129–134, IEEE, 2016.
  * [108] P. M. Monamo, V. Marivate, and B. Twala, “A multifaceted approach to bitcoin fraud detection: Global and local outliers,” in 2016 15th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 188–194, IEEE, 2016.
  * [109] K. Toyoda, T. Ohtsuki, and P. T. Mathiopoulos, “Identification of high yielding investment programs in bitcoin via transactions pattern analysis,” in GLOBECOM 2017-2017 IEEE Global Communications Conference, pp. 1–6, IEEE, 2017.
  * [110] J. Lorenz, M. I. Silva, D. Aparício, J. T. Ascensão, and P. Bizarro, “Machine learning methods to detect money laundering in the bitcoin blockchain in the presence of label scarcity,” in Proceedings of the First ACM International Conference on AI in Finance, pp. 1–8, 2020.
  * [111] A. Wahrstätter, J. Gomes, S. Khan, and D. Svetinovic, “Improving cryptocurrency crime detection: Coinjoin community detection approach,” IEEE Transactions on Dependable and Secure Computing, 2023.
  * [112] A. Aljofey, A. Rasool, Q. Jiang, and Q. Qu, “A feature-based robust method for abnormal contracts detection in ethereum blockchain,” Electronics, vol. 11, no. 18, p. 2937, 2022.
  * [113] T. Ashfaq, R. Khalid, A. S. Yahaya, S. Aslam, A. T. Azar, S. Alsafari, and I. A. Hameed, “A machine learning and blockchain based efficient fraud detection mechanism,” Sensors, vol. 22, no. 19, p. 7162, 2022.
  * [114] H. Hall, P. Baiz, and P. Nadler, “Efficient analysis of transactional data using graph convolutional networks,” in Machine Learning and Principles and Practice of Knowledge Discovery in Databases: International Workshops of ECML PKDD 2021, Virtual Event, September 13-17, 2021, Proceedings, Part II, pp. 210–225, Springer, 2022.
  * [115] Y. Elmougy and O. Manzi, “Anomaly detection on bitcoin, ethereum networks using gpu-accelerated machine learning methods,” in 2021 31st International Conference on Computer Theory and Applications (ICCTA), pp. 166–171, IEEE, 2021.
  * [116] M. Weber, G. Domeniconi, J. Chen, D. K. I. Weidele, C. Bellei, T. Robinson, and C. E. Leiserson, “Anti-money laundering in bitcoin: Experimenting with graph convolutional networks for financial forensics,” arXiv preprint arXiv:1908.02591, 2019.
  * [117] C. Lee, S. Maharjan, K. Ko, and J. W.-K. Hong, “Toward detecting illegal transactions on bitcoin using machine-learning methods,” in Blockchain and Trustworthy Systems: First International Conference, BlockSys 2019, Guangzhou, China, December 7–8, 2019, Proceedings 1, pp. 520–533, Springer, 2020.
  * [118] H. Han, R. Wang, Y. Chen, K. Xie, and K. Zhang, “Research on abnormal transaction detection method for blockchain,” in International Conference on Blockchain and Trustworthy Systems, pp. 223–236, Springer, 2022.
  * [119] T. Min and W. Cai, “Portrait of decentralized application users: an overview based on large-scale ethereum data,” CCF Transactions on Pervasive Computing and Interaction, vol. 4, no. 2, pp. 124–141, 2022.
  * [120] V. Patel, S. Rajasegarar, L. Pan, J. Liu, and L. Zhu, “Evangcn: Evolving graph deep neural network based anomaly detection in blockchain,” in Advanced Data Mining and Applications: 18th International Conference, ADMA 2022, Brisbane, QLD, Australia, November 28–30, 2022, Proceedings, Part I, pp. 444–456, Springer, 2022.
  * [121] V. Patel, L. Pan, and S. Rajasegarar, “Graph deep learning based anomaly detection in ethereum blockchain network,” in International Conference on Network and System Security, pp. 132–148, Springer, 2020.
  * [122] S. Al-E’mari, M. Anbar, Y. Sanjalawe, and S. Manickam, “A labeled transactions-based dataset on the ethereum network,” in Advances in Cyber Security: Second International Conference, ACeS 2020, Penang, Malaysia, December 8-9, 2020, Revised Selected Papers 2, pp. 61–79, Springer, 2021.
  * [123] L. Liu, W.-T. Tsai, M. Z. A. Bhuiyan, H. Peng, and M. Liu, “Blockchain-enabled fraud discovery through abnormal smart contract detection on ethereum,” Future Generation Computer Systems, vol. 128, pp. 158–166, 2022.
  * [124] F. Scicchitano, A. Liguori, M. Guarascio, E. Ritacco, and G. Manco, “A deep learning approach for detecting security attacks on blockchain.,” in ITASEC, pp. 212–222, 2020.
  * [125] T. Hu, X. Liu, T. Chen, X. Zhang, X. Huang, W. Niu, J. Lu, K. Zhou, and Y. Liu, “Transaction-based classification and detection approach for ethereum smart contract,” Information Processing & Management, vol. 58, no. 2, p. 102462, 2021.
  * [126] B. Podgorelec, M. Turkanović, and S. Karakatič, “A machine learning-based method for automated blockchain transaction signing including personalized anomaly detection,” Sensors, vol. 20, no. 1, p. 147, 2019.
  * [127] H. Baek, J. Oh, C. Y. Kim, and K. Lee, “A model for detecting cryptocurrency transactions with discernible purpose,” in 2019 Eleventh International Conference on Ubiquitous and Future Networks (ICUFN), pp. 713–717, IEEE, 2019.
  * [128] I. Alarab and S. Prakoonwit, “Effect of data resampling on feature importance in imbalanced blockchain data: Comparison studies of resampling techniques,” Data Science and Management, 2022.
  * [129] E. V. Feldman, A. N. Ruchay, V. K. Matveeva, and V. D. Samsonova, “Bitcoin abnormal transaction detection based on machine learning,” in Recent Trends in Analysis of Images, Social Networks and Texts: 9th International Conference, AIST 2020, Skolkovo, Moscow, Russia, October 15–16, 2020 Revised Supplementary Proceedings 9, pp. 205–215, Springer, 2021.
  * [130] B. Chen, F. Wei, and C. Gu, “Bitcoin theft detection based on supervised machine learning algorithms,” Security and Communication Networks, vol. 2021, pp. 1–10, 2021.
  * [131] M. Rwibasira and R. Suchithra, “Adobsvm: Anomaly detection on block chain using support vector machine,” Measurement: Sensors, vol. 24, p. 100503, 2022.
  * [132] R. Zhang, G. Zhang, L. Liu, C. Wang, and S. Wan, “Anomaly detection in bitcoin information networks with multi-constrained meta path,” Journal of Systems Architecture, vol. 110, p. 101829, 2020.
  * [133] C. Zhao and Y. Guan, “A graph-based investigation of bitcoin transactions,” in Advances in Digital Forensics XI: 11th IFIP WG 11.9 International Conference, Orlando, FL, USA, January 26-28, 2015, Revised Selected Papers 11, pp. 79–95, Springer, 2015.
  * [134] M. Deepa and D. Akila, “Cost-effective anomaly detection for blockchain transactions using unsupervised learning,” in Intelligent Computing and Innovation on Data Science, pp. 445–453, Springer, 2021.
  * [135] S. Farrugia, J. Ellul, and G. Azzopardi, “Detection of illicit accounts over the ethereum blockchain,” Expert Systems with Applications, vol. 150, p. 113318, 2020.
  * [136] J. Wu, J. Liu, W. Chen, H. Huang, Z. Zheng, and Y. Zhang, “Detecting mixing services via mining bitcoin transaction network with hybrid motifs,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 52, no. 4, pp. 2237–2249, 2021.
  * [137] Y. Li, Y. Cai, H. Tian, G. Xue, and Z. Zheng, “Identifying illicit addresses in bitcoin network,” in Blockchain and Trustworthy Systems: Second International Conference, BlockSys 2020, Dali, China, August 6–7, 2020, Revised Selected Papers 2, pp. 99–111, Springer, 2020.
  * [138] K. Kanemura, K. Toyoda, and T. Ohtsuki, “Identification of darknet markets’ bitcoin addresses by voting per-address classification results,” in 2019 IEEE International Conference on Blockchain and Cryptocurrency (ICBC), pp. 154–158, IEEE, 2019.
  * [139] A. Elbaghdadi, S. Mezroui, and A. El Oualkadi, “Svm: an approach to detect illicit transaction in the bitcoin network,” in Innovations in Smart Cities Applications Volume 4: The Proceedings of the 5th International Conference on Smart City Applications, pp. 1130–1141, Springer, 2021.
  * [140] I. Alarab and S. Prakoonwit, “Graph-based lstm for anti-money laundering: Experimenting temporal graph convolutional network with bitcoin data,” Neural Processing Letters, pp. 1–19, 2022.
  * [141] S. Eloul, S. J. Moran, and J. Mendel, “Improving streaming cryptocurrency transaction classification via biased sampling and graph feedback,” in Annual Computer Security Applications Conference, pp. 761–772, 2021.
  * [142] D. Boughaci and A. A. Alkhawaldeh, “Enhancing the security of financial transactions in blockchain by using machine learning techniques: Towards a sophisticated security tool for banking and finance,” in 2020 First International Conference of Smart Systems and Emerging Technologies (SMARTTECH), pp. 110–115, IEEE, 2020.
  * [143] A. Ahmed, “Anti-money laundering recognition through the gradient boosting classifier,” Academy of Accountingand Financial Studies Journal, vol. 25, no. 5, 2021.
  * [144] I. Alarab, S. Prakoonwit, and M. I. Nacer, “Competence of graph convolutional networks for anti-money laundering in bitcoin blockchain,” in Proceedings of the 2020 5th international conference on machine learning technologies, pp. 23–27, 2020.
  * [145] H. Zheng, B. Wen, and Y. Li, “Recognize illegal transactions in the bitcoin network using graph attention with dikw,” in 2021 IEEE 23rd Int Conf on High Performance Computing & Communications; 7th Int Conf on Data Science & Systems; 19th Int Conf on Smart City; 7th Int Conf on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), pp. 2118–2123, IEEE, 2021.
  * [146] M. Ostapowicz and K. Żbikowski, “Detecting fraudulent accounts on blockchain: a supervised approach,” in International Conference on Web Information Systems Engineering, pp. 18–31, Springer, 2020.
  * [147] H. Sun, N. Ruan, and H. Liu, “Ethereum analysis via node clustering,” in Network and System Security: 13th International Conference, NSS 2019, Sapporo, Japan, December 15–18, 2019, Proceedings 13, pp. 114–129, Springer, 2019.
  * [148] D. Vassallo, V. Vella, and J. Ellul, “Application of gradient boosting algorithms for anti-money laundering in cryptocurrencies,” SN Computer Science, vol. 2, pp. 1–15, 2021.
  * [149] B. Kılıç, A. Sen, and C. Özturan, “Fraud detection in blockchains using machine learning,” in 2022 Fourth International Conference on Blockchain Computing and Applications (BCCA), pp. 214–218, IEEE, 2022.
  * [150] Z. Yuan, T. Yang, and J. Cao, “Eth-tt: A novel approach for detecting ethereum malicious accounts,” in Artificial Intelligence: Second CAAI International Conference, CICAI 2022, Beijing, China, August 27–28, 2022, Revised Selected Papers, Part II, pp. 84–94, Springer, 2023.
  * [151] F. Poursafaei, G. B. Hamad, and Z. Zilic, “Detecting malicious ethereum entities via application of machine learning classification,” in 2020 2nd Conference on Blockchain Research & Applications for Innovative Networks and Services (BRAINS), pp. 120–127, IEEE, 2020.
  * [152] R. F. Ibrahim, A. M. Elian, and M. Ababneh, “Illicit account detection in the ethereum blockchain using machine learning,” in 2021 International Conference on Information Technology (ICIT), pp. 488–493, IEEE, 2021.
  * [153] R. M. Aziz, M. F. Baluch, S. Patel, and A. H. Ganie, “Lgbm: a machine learning approach for ethereum fraud detection,” International Journal of Information Technology, pp. 1–11, 2022.
  * [154] J. Zhou, S. Yan, and J. Zhang, “Prediction and analysis of illegal accounts on ethereum based on catboost algorithm,” in 2022 International Conference on Big Data, Information and Computer Network (BDICN), pp. 63–67, IEEE, 2022.
  * [155] A. Bella Baci, K. Brousmiche, I. Amal, F. Abdelhédi, and L. Rigaud, “Detecting illicit ethereum accounts based on their transaction history and properties and using machine learning,” in The International Conference on Deep Learning, Big Data and Blockchain (DBB 2022), pp. 97–108, Springer, 2022.
  * [156] F. Poursafaei, R. Rabbany, and Z. Zilic, “Sigtran: Signature vectors for detecting illicit activities in blockchain transaction networks,” in Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp. 27–39, Springer, 2021.
  * [157] T. Chen, Z. Li, Y. Zhu, J. Chen, X. Luo, J. C.-S. Lui, X. Lin, and X. Zhang, “Understanding ethereum via graph analysis,” ACM Transactions on Internet Technology (TOIT), vol. 20, no. 2, pp. 1–32, 2020.
  * [158] D. Saveetha and G. Maragatham, “Design of blockchain enabled intrusion detection model for detecting security attacks using deep learning,” Pattern Recognition Letters, vol. 153, pp. 24–28, 2022.
  * [159] O. Sanda, M. Pavlidis, S. Seraj, and N. Polatidis, “Long-range attack detection on permissionless blockchains using deep learning,” Expert Systems with Applications, vol. 218, p. 119606, 2023.
  * [160] M. Caprolu, S. Raponi, G. Oligeri, and R. Di Pietro, “Cryptomining makes noise: Detecting cryptojacking via machine learning,” Computer Communications, vol. 171, pp. 126–139, 2021.
  * [161] S. Boudko, H. Abie, M. Boscolo, and D. Ferrario, “Predictive analytics service for security of blockchain and peer-to-peer payment solutions,” in Information Science and Applications, pp. 71–81, Springer, 2021.
  * [162] B. B. Zarpelão, R. S. Miani, and M. Rajarajan, “Detection of bitcoin-based botnets using a one-class classifier,” in Information Security Theory and Practice: 12th IFIP WG 11.2 International Conference, WISTP 2018, Brussels, Belgium, December 10–11, 2018, Revised Selected Papers 12, pp. 174–189, Springer, 2019.
  * [163] W. Chen, H. Xu, L. Jia, and Y. Gao, “Machine learning model for bitcoin exchange rate prediction using economic and technology determinants,” International Journal of Forecasting, vol. 37, no. 1, pp. 28–43, 2021.
  * [164] G. Yan, S. Wang, S. Li, and B. Lu, “Multi-player dynamic game model for bitcoin transaction bidding prediction,” The North American Journal of Economics and Finance, vol. 60, p. 101631, 2022.
  * [165] H. Jang and J. Lee, “An empirical study on modeling and prediction of bitcoin prices with bayesian neural networks based on blockchain information,” Ieee Access, vol. 6, pp. 5427–5437, 2017.
  * [166] C. G. Akcora, A. K. Dey, Y. R. Gel, and M. Kantarcioglu, “Forecasting bitcoin price with graph chainlets,” in Advances in Knowledge Discovery and Data Mining: 22nd Pacific-Asia Conference, PAKDD 2018, Melbourne, VIC, Australia, June 3-6, 2018, Proceedings, Part III 22, pp. 765–776, Springer, 2018.
  * [167] S. Velankar, S. Valecha, and S. Maji, “Bitcoin price prediction using machine learning,” in 2018 20th International Conference on Advanced Communication Technology (ICACT), pp. 144–147, IEEE, 2018.
  * [168] A. Dutta, S. Kumar, and M. Basu, “A gated recurrent unit approach to bitcoin price prediction,” Journal of Risk and Financial Management, vol. 13, no. 2, p. 23, 2020.
  * [169] C. Cai, W. Li, H. Han, and M. Liu, “Risk scenario-based value estimation of bitcoin,” Procedia Computer Science, vol. 199, pp. 1198–1204, 2022.
  * [170] O. Poyser, “Exploring the dynamics of bitcoin’s price: a bayesian structural time series approach,” Eurasian Economic Review, vol. 9, no. 1, pp. 29–60, 2019.
  * [171] L. Kristoufek, “What are the main drivers of the bitcoin price? evidence from wavelet coherence analysis,” PloS one, vol. 10, no. 4, p. e0123923, 2015.
  * [172] I. Georgoula, D. Pournarakis, C. Bilanakos, D. Sotiropoulos, and G. M. Giaglis, “Using time-series and sentiment analysis to detect the determinants of bitcoin prices,” Available at SSRN 2607167, 2015.
  * [173] S. Y. Yang and J. Kim, “Bitcoin market return and volatility forecasting using transaction network flow properties,” in 2015 IEEE Symposium Series on Computational Intelligence, pp. 1778–1785, IEEE, 2015.
  * [174] X. Li and C. A. Wang, “The technology and economic determinants of cryptocurrency exchange rates: The case of bitcoin,” Decision support systems, vol. 95, pp. 49–60, 2017.
  * [175] Z. Chen, C. Li, and W. Sun, “Bitcoin price prediction using machine learning: An approach to sample dimension engineering,” Journal of Computational and Applied Mathematics, vol. 365, p. 112395, 2020.
  * [176] P. Jaquart, D. Dann, and C. Weinhardt, “Short-term bitcoin market prediction via machine learning,” The journal of finance and data science, vol. 7, pp. 45–66, 2021.
  * [177] N. Antulov-Fantulin, D. Tolic, M. Piskorec, Z. Ce, and I. Vodenska, “Inferring short-term volatility indicators from the bitcoin blockchain,” in International Conference on Complex Networks and their Applications, pp. 508–520, Springer, 2018.
  * [178] S. Ranjan, P. Kayal, and M. Saraf, “Bitcoin price prediction: A machine learning sample dimension approach,” Computational Economics, pp. 1–20, 2022.
  * [179] A. M. Kanji, I. Chaudhary, R. L. Shankar, and G. Srinivasa, “Predicting the price direction of bitcoin using twitter data and machine learning,” in 2022 IEEE 2nd International Conference on Data Science and Computer Application (ICDSCA), pp. 46–52, IEEE, 2022.
  * [180] M. Liu, G. Li, J. Li, X. Zhu, and Y. Yao, “Forecasting the price of bitcoin using deep learning,” Finance research letters, vol. 40, p. 101755, 2021.
  * [181] S. Ji, J. Kim, and H. Im, “A comparative study of bitcoin price prediction using deep learning,” Mathematics, vol. 7, no. 10, p. 898, 2019.
  * [182] P. Lamothe-Fernández, D. Alaminos, P. Lamothe-López, and M. A. Fernández-Gámez, “Deep learning methods for modeling bitcoin price,” Mathematics, vol. 8, no. 8, p. 1245, 2020.
  * [183] S. Yogeshwaran, M. J. Kaur, and P. Maheshwari, “Project based learning: predicting bitcoin prices using deep learning,” in 2019 IEEE Global Engineering Education Conference (EDUCON), pp. 1449–1454, IEEE, 2019.
  * [184] D. C. Mallqui and R. A. Fernandes, “Predicting the direction, maximum, minimum and closing prices of daily bitcoin exchange rate using machine learning techniques,” Applied Soft Computing, vol. 75, pp. 596–606, 2019.
  * [185] S. McNally, J. Roche, and S. Caton, “Predicting the price of bitcoin using machine learning,” in 2018 26th euromicro international conference on parallel, distributed and network-based processing (PDP), pp. 339–343, IEEE, 2018.
  * [186] M. Mudassir, S. Bennbaia, D. Unal, and M. Hammoudeh, “Time-series forecasting of bitcoin prices using high-dimensional features: a machine learning approach,” Neural computing and applications, pp. 1–15, 2020.
  * [187] P. K. Nagula and C. Alexakis, “A new hybrid machine learning model for predicting the bitcoin (btc-usd) price,” Journal of Behavioral and Experimental Finance, vol. 36, p. 100741, 2022.
  * [188] I. Chalkiadakis, A. Zaremba, G. W. Peters, and M. J. Chantler, “On-chain analytics for sentiment-driven statistical causality in cryptocurrencies,” Blockchain: Research and Applications, vol. 3, no. 2, p. 100063, 2022.
  * [189] X. Li and L. Du, “Bitcoin daily price prediction through understanding blockchain transaction pattern with machine learning methods,” Journal of Combinatorial Optimization, vol. 45, no. 1, p. 4, 2023.
  * [190] H.-M. Kim, G.-W. Bock, and G. Lee, “Predicting ethereum prices with machine learning based on blockchain information,” Expert Systems with Applications, vol. 184, p. 115480, 2021.
  * [191] M. Saad, J. Choi, D. Nyang, J. Kim, and A. Mohaisen, “Toward characterizing blockchain-based cryptocurrencies for highly accurate predictions,” IEEE Systems Journal, vol. 14, no. 1, pp. 321–332, 2019.
  * [192] H. J. Singh and A. S. Hafid, “Prediction of transaction confirmation time in ethereum blockchain using machine learning,” in International Congress on Blockchain and Applications, pp. 126–133, Springer, 2019.
  * [193] A. M. Fajge, S. Goswami, A. Srivastava, and R. Halder, “Wait or reset gas price?: A machine learning-based prediction model for ethereum transactions’ waiting time,” in 2021 IEEE 20th International Conference on Trust, Security and Privacy in Computing and Communications (TrustCom), pp. 1153–1160, IEEE, 2021.
  * [194] V. C. Oliveira, J. Almeida Valadares, J. E. A. Sousa, A. Borges Vieira, H. S. Bernardino, S. Moraes Villela, and G. Dias Goncalves, “Analyzing transaction confirmation in ethereum using machine learning techniques,” ACM SIGMETRICS Performance Evaluation Review, vol. 48, no. 4, pp. 12–15, 2021.
  * [195] D. Lan, H. Wang, C. Yin, L. Zhou, C. Ge, and X. Lu, “Gas price prediction based on machine learning combined with ethereum mempool,” in 2022 IEEE 19th International Conference on Mobile Ad Hoc and Smart Systems (MASS), pp. 346–354, IEEE, 2022.
  * [196] R. Mars, A. Abid, S. Cheikhrouhou, and S. Kallel, “A machine learning approach for gas price prediction in ethereum blockchain,” in 2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC), pp. 156–165, IEEE, 2021.
  * [197] G. A. Pierro and H. Rocha, “The influence factors on ethereum transaction fees,” in 2019 IEEE/ACM 2nd International Workshop on Emerging Trends in Software Engineering for Blockchain (WETSEB), pp. 24–31, IEEE, 2019.
  * [198] L. Zhang, J. Wang, W. Wang, Z. Jin, Y. Su, and H. Chen, “Smart contract vulnerability detection combined with multi-objective detection,” Computer Networks, vol. 217, p. 109289, 2022.
  * [199] J.-W. Liao, T.-T. Tsai, C.-K. He, and C.-W. Tien, “Soliaudit: Smart contract vulnerability assessment based on machine learning and fuzz testing,” in 2019 Sixth International Conference on Internet of Things: Systems, Management and Security (IOTSMS), pp. 458–465, IEEE, 2019.
  * [200] P. Momeni, Y. Wang, and R. Samavi, “Machine learning model for smart contracts security analysis,” in 2019 17th International Conference on Privacy, Security and Trust (PST), pp. 1–6, IEEE, 2019.
  * [201] P. Qian, Z. Liu, Q. He, R. Zimmermann, and X. Wang, “Towards automated reentrancy detection for smart contracts based on sequential models,” IEEE Access, vol. 8, pp. 19685–19695, 2020.
  * [202] J. Song, H. He, Z. Lv, C. Su, G. Xu, and W. Wang, “An efficient vulnerability detection model for ethereum smart contracts,” in Network and System Security: 13th International Conference, NSS 2019, Sapporo, Japan, December 15–18, 2019, Proceedings 13, pp. 433–442, Springer, 2019.
  * [203] W. Wang, J. Song, G. Xu, Y. Li, H. Wang, and C. Su, “Contractward: Automated vulnerability detection models for ethereum smart contracts,” IEEE Transactions on Network Science and Engineering, vol. 8, no. 2, pp. 1133–1144, 2020.
  * [204] A. K. Gogineni, S. Swayamjyoti, D. Sahoo, K. K. Sahu, and R. Kishore, “Multi-class classification of vulnerabilities in smart contracts using awd-lstm, with pre-trained encoder inspired from natural language processing,” IOP SciNotes, vol. 1, no. 3, p. 035002, 2020.
  * [205] J. Huang, S. Han, W. You, W. Shi, B. Liang, J. Wu, and Y. Wu, “Hunting vulnerable smart contracts via graph embedding based bytecode matching,” IEEE Transactions on Information Forensics and Security, vol. 16, pp. 2144–2156, 2021.
  * [206] H. Wu, Z. Zhang, S. Wang, Y. Lei, B. Lin, Y. Qin, H. Zhang, and X. Mao, “Peculiar: Smart contract vulnerability detection based on crucial data flow graph and pre-training techniques,” in 2021 IEEE 32nd International Symposium on Software Reliability Engineering (ISSRE), pp. 378–389, IEEE, 2021.
  * [207] X. Yu, H. Zhao, B. Hou, Z. Ying, and B. Wu, “Deescvhunter: A deep learning-based framework for smart contract vulnerability detection,” in 2021 International Joint Conference on Neural Networks (IJCNN), pp. 1–8, IEEE, 2021.
  * [208] Z. Gao, “When deep learning meets smart contracts,” in Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, pp. 1400–1402, 2020.
  * [209] N. Ashizawa, N. Yanai, J. P. Cruz, and S. Okamura, “Eth2vec: Learning contract-wide code representations for vulnerability detection on ethereum smart contracts,” Blockchain: Research and Applications, vol. 3, no. 4, p. 100101, 2022.
  * [210] G. Palaiokrassas, S. Scherrers, I. Ofeidis, and L. Tassiulas, “Leveraging machine learning for multichain defi fraud detection,” arXiv preprint arXiv:2306.07972, 2023.
  * [211] C. Tenopir, S. Allard, K. Douglass, A. U. Aydinoglu, L. Wu, E. Read, M. Manoff, and M. Frame, “Data sharing by scientists: practices and perceptions,” PloS one, vol. 6, no. 6, p. e21101, 2011.
  * [212] Y. Kim and P. Zhang, “Understanding data sharing behaviors of stem researchers: The roles of attitudes, norms, and data repositories,” Library & Information Science Research, vol. 37, no. 3, pp. 189–200, 2015.
  * [213] W. Zenk-Möltgen, E. Akdeniz, A. Katsanidou, V. Naßhoven, and E. Balaban, “Factors influencing the data sharing behavior of researchers in sociology and political science,” Journal of documentation, vol. 74, no. 5, pp. 1053–1073, 2018.
  * [214] L. Perrier, E. Blondal, and H. MacDonald, “The views, perspectives, and experiences of academic researchers with data sharing and reuse: A meta-synthesis,” PloS one, vol. 15, no. 2, p. e0229182, 2020.


# Learning-on-the-Drive:
Self-supervised Adaptation of Visual Offroad Traversability Models

Eric Chen∗1, Cherie Ho∗2, Mukhtar Maulimov2 and Chen Wang3, and Sebastian
Scherer2 *Both authors contributed equally.1Eric Chen is with the Department
of Computer Science and Mathematics, Harvey Mudd College, USA<EMAIL_ADDRESS>Cherie Ho, Mukhtar Maulimov, and Sebastian Scherer are with the Robotics
Institute, Carnegie Mellon University, USA {cherieh, mmaulimo,
<EMAIL_ADDRESS>Wang is with the Department of Computer Science and
Engineering, State University of New York at Buffalo, USA<EMAIL_ADDRESS>
###### Abstract

Autonomous off-road driving requires understanding traversability, which
refers to the suitability of a given terrain to drive over. When offroad
vehicles travel at high speed ($>10/$), they need to reason at long-range
($50$-$100$) for safe and deliberate navigation. Moreover, vehicles often
operate in new environments and under different weather conditions. LiDAR
provides accurate estimates robust to visual appearances, however, it is often
too noisy beyond 30m for fine-grained estimates due to sparse measurements.
Conversely, visual-based models give dense predictions at further distances
but perform poorly at all ranges when out of training distribution. To address
these challenges, we present ALTER, an offroad perception module that adapts-
on-the-drive to combine the best of both sensors. Our visual model
continuously learns from new near-range LiDAR measurements. This self-
supervised approach enables accurate long-range traversability prediction in
novel environments without hand-labeling. Results on two distinct real-world
offroad environments show up to 52.5% improvement in traversability estimation
over LiDAR-only estimates and 38.1% improvement over non-adaptive visual
baseline.

##  I INTRODUCTION

Consider a robot driving fast in a forest, as in Figure 1. Autonomous off-road
driving requires the ability to estimate traversability accurately to
determine the suitability of a given terrain for a vehicle to drive over. The
robot must differentiate between bushes that cannot be traversed, tall grass
that can be traversed but poses a risk, and trails that are ideal for high-
speed traversal in order to make intelligent navigation decisions. While
offroad driving has demonstrated many successes in the past decade [1, 2, 3,
4], most efforts are limited to slow speeds lower than 5m/s and are carefully
developed for particular scenarios.

Challenges arise as we increase the speed ($>10/$) and the variety of terrains
an autonomous system must navigate. To tackle these challenges, an effective
offroad perception system should possess three key properties:

  1. 1.

Accurate at long-range When operating at a high speed, the vehicle requires
accurate traversability estimates beyond reactive range ($>30$) for more
deliberate and safe navigation.

  2. 2.

Robust in unseen environments Off-road vehicles often operate in new
environments, e.g., desert and forest, or with different weather conditions,
e.g., sunny and cloudy.

  3. 3.

Fine-grained prediction The complexity of offroad environments requires
differentiating terrain beyond binary classifications of traversable or non-
traversable. For example, differentiating between various traversable areas:
medium-risk tall grass, and low-risk trail.

Figure 1: We present ALTER, an offroad perception framework that adapts on-
the-drive to provide accurate long-range traversability prediction. We
validate our adaptive method on two real-world offroad scenarios where it is
able to differentiate between key terrains (trail, grass, obstacles) at a far
distance with less than one minute of online learning.

Laser range sensors such as LiDAR are typically used to build a geometric
understanding of the environment. Fine-grained traversability estimates can be
generated by combining extracted LiDAR features, such as object height and
smoothness [2, 5]. LiDAR provides accurate estimates robust to visual
appearances, but its noise increases with range due to sparsity of points. On
the contrary, camera-based methods output dense predictions at further
distances [6, 7]. However, these models require a large human-annotated
dataset and perform poorly when out of training distribution.

Online self-supervised techniques are used to adapt a visual model with
appearance-invariant LIDAR measurements to leverage the benefits of both LiDAR
and camera while remaining robust in unseen environments. However, prior
LiDAR-based works either only produce binary traversable or non-traversable
classification [8, 9, 10], or learn in colored point cloud space [11] which
limits estimation to within reliable LiDAR range, limiting its long-range
capability.

To simultaneously address the 3 needed properties, we present ALTER, Adaptive
Long-range Traversibility EstimatoR. Our perception framework adapts its
visual traversability model online using current LiDAR measurements to deliver
accurate traversability estimates in unseen environments. By inferring in
image space, our method produces pixel-wise traversability estimates that
provide dense measurement at long-range. ALTER produces continuous-value
traversability labels to provide fine-grained estimates needed for navigation.
We present an example of offroad driving in Figure 1, where the vehicle can
learn to differentiate between key terrains at a far distance with less than
one minute of online learning.

Conceptually, we bridge the benefits of appearance-invariant LiDAR and dense
camera measurements via online self-supervised learning. To label without
manual annotation, we first accumulate near-range LiDAR information to build a
large accurate long-range map, and then extract features from the accumulated
map to label fine-grained traversability in 3D space. Lastly, we project the
3D labels to a previous image to produce dense long-range pixel-wise labels.
During online learning, we continuously finetune a model with the latest self-
labeled data to remain robust in unseen environments.

Our contributions are threefold:

  1. 1.

To the best of our knowledge, we propose the first online learning framework
to produce fine-grained, image-space traversability estimation supervised by
LiDAR measurements. While the fine-grained or image-space estimation was
independently explored, ours is the first to combine all elements to produce
accurate long-range estimates that adapt to new environments.

  2. 2.

We present a novel method to generate fine-grained and long-range pixel-wise
traversability labels with LiDAR measurements. Previous works provide a binary
traversability label, while we provide a continuous traversability score [8,
9]. In addition, the previous works use instantaneous LiDAR measurement to
label which restricts the labeling range and accuracy. Our method uses
accumulated LiDAR scans, extending the labeling range from 15 to 100.

  3. 3.

We show, on 2 visually distinct environments, within 1 minute of combined data
collection and training, our adaptive visual method produces up to 52.5%
improvement in traversability estimation over LiDAR-only estimates and 34.3%
improvement over visual models trained in another environment.

##  II RELATED WORKS

Traversability estimation: Traversability estimation involves understanding
high-dimensional information to determine how suitable a specific terrain is
for the vehicle to drive over. The two most common sensors for offroad
perception are cameras and LiDARs. We find works that use LiDAR sensors to
achieve this where a map is built with each grid cell containing some
geometric statistics to help quantify traversability [12, 1, 5]. However,
LiDAR measurements are sparse, and therefore noisy, at the distance. Many
works therefore incorporate visual predictions to increase the range [4, 7].
For example, Maturana et al. [4] classifies an image into semantic classes,
such as trail, grass, etc. However, these visual models are trained with a
large amount of hand-annotated data and fail when deployed in environments
different from the training distribution.

Cross-modal Self-supervised Learning: To alleviate the need for hard-to-
acquire human annotation, self-supervised learning is often employed to
generate labeled data automatically, with applications in natural language
processing[13], healthcare[14], and robotics[15]. This paradigm lends itself
well to combining complementary information across multiple sensors. Most
related, we find successes in learning visual models with another sensor, for
example, IMU[16, 17], force-torque signals [18], acoustic waveforms [19] and
odometry [20, 21]. However, the above works rely on a static training dataset
and may still fail when out of distribution.

Figure 2: ALTER framework: Our adaptive approach estimates traversability from
current images using models trained online with new LiDAR data. Online
Labeling: We accumulate LiDAR scans and label in 3D their traversability costs
given LIDAR features. The 3D labels are then projected onto the image plane to
form training labels. Online Learning: ALTER continuously trains the model
with the latest training buffer to adapt to new appearances. Online Inference:
We predict pixel-wise fine-grained traversability scores for new images with
the latest model.

Online Learning: Online learning investigates applications where the testing
distribution is constantly changing by rapidly adapting to new data
distributions. Common techniques include repeatedly training with the latest
training buffer [22] or discarding outdated clusters of data [8]. More
recently, with the rise of larger image datasets such as ImageNet [23],
finetuning from pretrained model emerges as a more sample-efficient method for
online learning [24]. However, fine-tuning a model on new tasks can lead to
"forgetting" of previously learned information, known as catastrophic
forgetting [25]. Continual learning approaches prevent forgetting by memory-
based methods[26] and regularization-based methods [27].

Online Self-Supervised Learning for Traversability Estimation: For high-speed
off-road driving, it is important to have accurate sensing at range even in
new environments. We find multiple works that leverage cross-modal self-
supervised learning and online learning to provide dense visual traversability
estimation in unseen environments. Several works learn visual cues for
traversability prediction by using stereo measurements as supervision[22, 28].
Hadsell et al. [22] segments camera images into multiple classes based on
stereo-estimated object height. However, the authors also cite the
unreliability of stereo range sensing to appearance and lighting changes as a
key limitation.

Our work builds on successful approaches that adopt online self-supervised
learning to combine LiDAR and cameras[8, 3, 11]. Sofman et al. [11] learns a
mapping between LiDAR-based traversability to a colored point cloud space.
However, the estimation is limited to within reliable LiDAR range, while our
method can provide estimates beyond LiDAR sensing. We also find works that
infer in image space [8, 9]. However, in both works, the prediction is limited
to binary classes: traversable or non-traversable, which is overly
conservative for the complex off-road environment. In contrast, our work
provides a fine-grained traversability estimation. To address the limitations
of past work, we seek to build an adaptive visual model that can learn from
LiDAR measurements to provide fine-grained, pixel-wise predictions.

##  III APPROACH

###  III-A Overview

We aim to estimate fine-grained traversability at long-range while being
robust to previously unseen environments. We achieve this by inferring in
image space while adapting our visual model online with new LiDAR
measurements.

As seen in Figure 2, ALTER is composed of three key steps:

  1. 1.

Extracting image labels from accumulated LiDAR scans;

  2. 2.

Training a visual model with the self-labeled data;

  3. 3.

Infer on incoming images with online-learned models.

We describe each step in detail in the following subsections.

###  III-B Online Self-Supervised Labeling

We first generate pixel-wise traversability labels from current LiDAR
measurements that will be used to adapt the visual models. Figure 3 describes
the overall labeling process. Crucially, we generate the labels using
accumulated near-range measurements to produce accurate long-range labels.

####  III-B1 Labeling in LiDAR Space

##### Accumulating LiDAR scans into voxel map

LiDAR scans (Figure 3a) are first registered with an IMU-based SLAM algorithm
[29]. The resulting registered scans are then accumulated in a $120\times 120$
robot-centered 3D voxel map at 0.25 resolution.

##### Extracting LiDAR Features

We extract object height and planarity features from the accumulated map to
inform traversability, with object height helping find large obstacles and
planarity helping to differentiate between free space areas such as grass and
trail. For object height $f_{\text{h}}$, we estimate the ground plane with a
Markov Random Field-based Filter that interpolates between the heights of the
neighboring lowest dense cells. For each voxel pillar, we label object height
of each of its voxel by the difference between the height of tallest voxel and
ground, as seen in Figure 3 (c).

On the other hand, for the planarity $f_{\text{p}}$, we calculate the
eigenvalues for each top-down cell, by applying singular value decomposition
on the aggregated LiDAR points in the cell. Using the eigenvalues, we
calculate planarity by $\frac{(\lambda_{2}-\lambda_{3})\times
2}{\lambda_{1}}$. As seen in Figure 3 (c), planarity allows us to
differentiate between trail/smooth ground (more planar) and grass/low
vegetation (less planar) and is commonly used in offroad literatures[5, 30].
However, calculating planarity requires a high density of accurately
registered points, which is significantly less likely at a distance, resulting
in noisy far-range measurements. ALTER estimates in image space, therefore,
produces accurate, long-range measurements.

##### Combining Features to Traversability Costs

We next combine the two LiDAR features into a continuous traversability score.
To this end, we formulate a cost function that generally captures three
terrain types of increasing severity: a trail that is low in height and high
in planarity will be low cost, grass that is low in height and low in
planarity will be medium cost, and untraversable obstacles that are above a
height threshold are marked as high cost. We use the following cost function
$J$, where $h_{\text{thresh}}=1$ is a threshold:

$J(f_{\text{h}},f_{\text{p}})=\begin{cases}10&\text{if }f_{\text{h}}\geq
h_{\text{thresh}}\\\ 6-f_{\text{p}}\in[0,6]&\text{if
}f_{\text{h}}<h_{\text{thresh}}\end{cases}$ (1)

An example labeled 3D voxel map is shown in Figure 3 (d).

Figure 3: Self-Supervised Traversability Labeling Pipeline: Our self-
supervised approach generates pixel-wise labels without manual labeling.
First, LiDAR scans are registered and accumulated into a voxel map. Then,
LiDAR features (object height, planarity) are extracted from the accumulated
near-range map. Given a voxel’s LiDAR features, its traversability cost is
calculated. Finally, the 3D labels are projected into image space to generate
pixel-wise image labels. Our labels are able to distinguish between regions of
different traversability, such as, trail, grass and tall obstacles, for
visually distinct environments.

####  III-B2 Projecting LiDAR Labels to Image

Given labeled voxels in 3D space, we then project the information to image
space to self-generate a pixel-wise label mask for visual traversability
prediction. Figure 3(e) shows an example image, label pair.

##### Near-to-far projection

As described earlier, LiDAR measurements are often noisy at long ranges. To
address the insufficient information at range, we perform a near-to-far
learning similar to [31], which associates accumulated near-range measurements
to supervise previous measurements. Specifically, we associate an image taken
at time $t$ with a voxel map after a delay at $t+d$. The resulting labels are
more accurate at farther range and cover a larger distance.

To capture accurate labels, we trim the admissible voxels to be within 15 of
past and future robot odometries, within which LiDAR provides good estimates,
and within 100m of the odometry at the image frame.

##### Raycasting from camera view

While projecting a future accumulated map improves the labeling accuracy, a
naive projection of LiDAR labels will also include labels that would have been
occluded from the initial viewpoint. To eliminate these labels, we perform
Digital Differential Analysis (DDA)-style raycasting [32] starting from the
image viewpoint at time $t$ into the voxel map at time $t+d$ to select
suitable voxels to project. Finally, the four corners of each raycasted voxel
surface closest to the camera $\mathbf{p}_{w}$ in world coordinates are
projected to pixel positions in image coordinates $\mathbf{p}_{c}$ using a
pinhole camera model, as follows: $\mathbf{p}_{c}=M_{\text{int}}\cdot
M_{\text{ext}}\cdot\mathbf{p}_{w},$ where $M_{\text{int}}$ is the intrinsic
matrix, $M_{\text{ext}}$ is the extrinsic matrix given by the odometry of the
LiDAR sensor at time $t$. For each raycasted voxel surface, the enveloped
space by its projected corners is filled in by the associated 3D
traversability cost.

By labeling accumulated LiDAR measurement in a near-to-far manner, we can
accurately label pixels up to 100m along the driven-over path, instead of just
15m. This results in more accurate and filled-out pixel-wise labels in image
space.

###  III-C Online Learning

As new image/label pairs are generated from self-supervision, we continuously
train models to adapt-on-the-drive to predict fine-grained pixel-wise
traversability scores.

####  III-C1 Model Training

We use a U-Net model [33] as the base architecture with a MobileNet backbone
[34] pretrained on ImageNet [23] to quickly adapt to changing environments. We
train our model using a modified Mean Squared Error, that ignores unknown
pixels, as the loss function between predicted values and self-supervised
values. Due to dataset imbalance between different costs, we adjust the
training loss to weigh pixels of different traversability costs using median
frequency balancing[35]. To keep training time low, we train with a rolling
buffer of 10 seconds with a frame per second, for 10 seconds. We use the top-
half image to encourage the model to learn traversability at a distance.

Figure 4: Qualitative results after 1 minute of adaptation:  Our online-
learned method is able to differentiate nuanced terrain (trail, grass,
obstacle) at long-range within one minute of combined data collect and
training. Compared to LiDAR-only baseline, our method is able to better
differentiate grass and trail at the distance. Compared to a model pretrained
on another environment, our adaptive method provides more robust predictions
when deployed in a new environment.

####  III-C2 Continuous Model Selection

For more generalizable performance, we choose which model weights over a
training cycle to deploy based on its performance in a wider validation buffer
with older data. In this work, we choose 8 evenly-spaced frames, that does not
overlap with training frames, from the latest 20 seconds of data as our
validation buffer. During training, the best-performing model out of all
epochs with respect to the validation buffer, is then used for inference and
for continuous finetuning in the next training cycle.

###  III-D Online Traversability Inference

We infer on incoming images with our online-learned models using the top half
of the image to focus on long-range predictions. For cleaner visualizations,
we use an offroad segmentation model [36] to remove the sky.

##  IV RESULTS

We evaluate the robustness and adaptivity of ALTER on two real-world offroad
datasets collected from distinct environments. Through our experiments, we aim
to answer the following research questions:
Q1: Does adaptation enable more robust performance in new environments than
pretrained visual models?
Q2: Does our visual model provide more accurate long-range perception than
LiDAR-only estimates?
Q3: How do online learning parameters affect performance?
Q4: How do key design choices affect performance?
Q5: How does ALTER perform in previously-learned environment?

###  IV-A Data Collection

We use a Polaris RZR platform equipped with RGB cameras (Multisense S27) and
LiDARs (Velodyne VLP-32C) as our data collection vehicle. We collect data in
two visually distinct environments: forest and hill. The forest sequence was
collected in a forest hill in Pittsburgh, PA with generally dark-green
vegetation, green grass and some dark-brown colored trails. The hill sequence
was collected in a hilly region in San Luis Obispo, CA featuring dry yellow
grass with sparse trees. The images are acquired at 1024 $\times$ 750 pixels
and downsampled to 512 $\times$ 384 pixels.

###  IV-B Baselines and Metrics

We compare our method with two baselines to compare its ability to capture
traversability at range. The baselines represent the two paradigms that our
method hopes to bridge:

  1. 1.

LiDAR-only: the geometric traversability maps estimated with accumulated LiDAR
points. For comparison, the estimated score are raycasted into image space.

  2. 2.

Non-adaptive: the visual model trained in an environment other than the one we
are testing on. We use a model trained with 50 epochs.

To compare the two methods, we calculate the mean square error (MSE) of the
known pixels between the predicted image and the self-supervised labels. As
the focus of this paper is long-range traversability estimates, we exclude
points that are within 10 meters of the robot for comparison.

###  IV-C Comparison to baselines after 1 minute of adaptation

| Forest Env. |  | Hill Env. |
---|---|---|---|---
Methods | MSE $\downarrow$ |  | Difference
---
from LiDAR $\downarrow$
MSE $\downarrow$ |  | Difference
---
from LiDAR $\downarrow$
ALTER (ours) | 2.81$\pm$1.41 | -52.5 % | 3.81$\pm$2.98 | -34.3 %
Non-adaptive | 3.66$\pm$1.96 | -38.1% | 6.16$\pm$4.03 | +6.1 %
LiDAR-only | 5.92$\pm$1.38 | - | 5.80$\pm$1.35 | -

TABLE I: Quantitative comparison to baselines after 1 minute of adaptation: We
compare our online method to a LiDAR-only baseline and a visual model trained
in another environment. Our adaptive approach provides more robustness to new
environments than the baseline visual model. With less than a minute of data
collection and training time, our model produces up to $52\%$ improvement over
LIDAR.

Experimental Setup: We conduct simulated online experiments in the two
environments to evaluate the performance of our adaptive method. In each
environment, we use a 85 seconds sequence for training and inference, and we
initialize the starting visual model with ImageNet[23] weights. Each of our
training buffer contains 10 seconds of labeled image data. The labels are
generated using a delayed projection of 5 seconds, meaning each label contains
voxel information from time $t$ to $t+5$ seconds, where t is the image
acquisition time. The model is allowed 10 seconds of training on each buffer
before moving onto the next. This results in finishing training of the first
model 25 seconds into the run and subsequent models trained in 10 seconds
intervals. The self-supervised labels are preprocessed with a real-time
implementation in progress. We train continuously for four models with a
learning rate of $1e^{-3}$, ending the learning process at 55. The last 30
seconds (55-85 from start) are used as our test set. Table I quantitatively
compares our online-learned method with the baselines, with our method
providing more accurate long-range results after less than 1 minute of
adaptation.

Does adaptation enable more robust prediction in new environments? After 55
seconds of data collection and training, our proposed method consistently
outperforms the non-adaptive visual baseline, with up to 38.1% improvement.
The experiment validates that adapting with self-supervised labels enables
more robust performance in an out-of-distribution environment. Figure 4 shows
representative qualitative outputs, where the online-learned module can
clearly distinguish low-cost planar trails, medium-cost grassy regions and
high-cost obstacles in both environments. In contrast, the baseline that is
trained in different environments fails to generalize and often estimates
trail region that is safe to traverse as medium-cost.

Figure 5: Model performance over time:  We show the prediction on a held-out
frame in Forest environment from the continuously-learning model. As expected,
as more data is collected, model performance increases resulting in more fine-
grained estimates with clearer boundaries at further terrain.

Figure 5 compares the performance of the continuously-learned models on the
test set. As expected, error decreases with more training data and training
time. While the first model (data collected and model trained within 35s) can
discern the trail at close-range, it mislabels grass regions as having high
cost. In general, we observe that the later models provide more fine-grained
estimates and better boundary prediction between terrains.

Does the adapted visual model provide longer-range estimates? Additionally, we
observe our adaptive visual prediction can provide more accurate
traversability estimates than LiDAR at range, with up to a 52.5% reduction in
error. In the first row of Figure 4, we show a scenario where our online-
learned prediction differentiates the trail from the grass region at far-
range, matching the self-supervised labels. In contrast, the map generated
from LiDAR does not distinguish the two areas well as the planarity feature of
the trail requires a high density of points to be accurate. Moreover, in the
third row of Figure 4, the LiDAR-only baseline is unable to provide estimates
on the trail at a distance as those cells do not yet have the required number
of points. In contrast, our online-learned model is able to predict the trail
and grass much further than even the general voxel map size.

###  IV-D Online Performance Analysis

We examine the online performance of our adaptive method and the impacts of
online learning hyperparameters.

Figure 6: Online performance with our adaptive approach: Top:  We compare our
online method to baselines. For clearer visualization, we apply a moving
average of 3 seconds. By 55 seconds from start, our adaptive visual model
consistently provides better or comparable estimates compared to both
baselines in both environments. Bottom:  Over the test sequence, 6 models were
trained starting from 25 and at 10 intervals. We show the first frame when a
new model is deployed. As time increases, the performance improves. By 55
seconds, our model is able to differentiate between grass and trail at the
distance, which a LiDAR-only method is not.

We leave our method to be fully adaptive throughout the sequence, with a
starting model initialized with ImageNet weights. Figure 6 shows the online
adaptive performance on the forest sequence, starting at 25 seconds when the
first model is trained. By 55 seconds from the start (third model deployed, 30
seconds of data used), our adaptive visual model consistently provides better
or comparable estimates compared to both baselines. In addition, we show the
online prediction of the initial frame when a new model is deployed. By 55
seconds, we see our system has sufficiently adapted to the environment, able
to provide good long-range fine-grained estimates.

We next quantify the impact of buffer length and learning rate on online
performance.

Figure 7: Average MSE given buffer length and learning rate: We find a buffer
length of 10 seconds and a learning rate of $1e^{-3}$ to be optimal, providing
a good tradeoff of reactivity and robustness.

Figure 7 shows the average error with various combinations of buffer length
and learning rate. In both environments, we find a buffer length of 10 seconds
and a learning rate of $1e^{-3}$ to be optimal for our application. We find
that the highest errors occur when buffer length is 1 second. This is expected
as the model may have overfitted to the very limited data. Alternatively, a
buffer length of 25s is suboptimal as the latest model is effectively 30
seconds behind (25 training time, 5 accumulation time) the last data in the
training buffer. Such a long delay does not allow for the reactivity needed
for our application.

###  IV-E Ablation Studies

|  | Forest Env.
---
MSE $\downarrow$
| Hill Env.
---
MSE $\downarrow$
Ours | 2.81 | 4.83
w/o Accum | 6.85 | 7.93
w/o Pretrain-ext | 3.36 | 6.51
TABLE II: Ablation Studies of Key Modules: Average MSE loss in test set. We
find learning with accurate labels (labeling with accumulated LiDAR scans) and
learning with pretrained extractors to be critical for online performance.

Table II studies the relative impact of key design decisions to online
performance. We investigate the impact of our accumulation of LiDAR scans for
labeling (w/o Accum) and our use of a model with a pretrained ImageNet
extractor (w/o Pretrain-ext). Similar to previous section, we apply our
adaptive method throughout the entire sequence and report MSE error of the
last 30 seconds of each sequence. As expected, learning from delayed
accumulated LiDAR labels significantly improves performance as the training
signal is denser and closer to ground truth. We then compare performance
between initializing with a pretrained ImageNet extractor versus a random
initialization (train from scratch). We find learning with the pretrained
extractor also significantly improves adaptive performance.

Method |  | Training
---
Sequence
| Hill Env.
---
MSE $\downarrow$
| Forest Env.
---
MSE$\downarrow$
| Average
---
MSE$\downarrow$
Ours | forest | 18.9 | 2.81 | 10.85
Ours | hill | 3.81 | 5.87 | 4.84
Ours | forest$\rightarrow$hill | 4.02 | 4.50 | 4.26
Ours + EWC[27] | forest$\rightarrow$hill | 4.12 | 3.52 | 3.82
TABLE III: Performance in Previously-Learned Environments: We continually
adapt on forest and then hill sequence. Both variants of our method achieve
better average performance when learned in both environments than with just
one. Our method with EWC loss is able to better retain previous knowledge
after learning in a new environment.

###  IV-F Performance in Previously-Learned Environment

We investigate how ALTER performs in previously-learned environments after
learning in new environments, i.e., avoids catastrophic forgetting [25]. In
our experiment, we first adapt a model to the forest sequence, then adapt to
the hill sequence and evaluate the resulting model on the test sequences of
both environments. We introduce an extension of our method to include Elastic
Weight Consolidation (EWC) loss [27] (ours + EWC), which penalizes learning on
weights important to previous tasks to limit forgetting. We set the EWC loss
weight to $5e^{-3}$. Table III shows that after learning in the new hill
environment, our methods (1 and 2 row) are still able to retain previous
knowledge and achieve better performance in the forest environment compared to
if they had not learned in it. By introducing EWC, we can better retain
previous information, achieving the best average performance. Finally, we find
that if it is possible to give our system one minute (an acceptable period for
offroad driving) to adapt, resetting our model and learning solely in the test
environment (3 and 4 row) performs best. Note that our model is also able to
work with other lifelong/continual learning techniques such as [37], but it is
out of the scope of this work.

##  V CONCLUSIONS

In this paper, we present ALTER, an offroad perception framework that adapts
on-the-drive to provide accurate traversability prediction in novel
environments. Central to this approach, is a self-supervised framework that
adapts a visual model online with near-range LiDAR measurements. We validate
with two offroad datasets that our adaptive visual method significantly
increases robustness when operating in new environments. Within one minute of
combined data collect and training, the model is able to differentiate between
key terrains to provide fine-grained traversability estimates. We also show
that the visual prediction is able to provide more accurate long-range
measurements compared to LiDAR, especially on terrains that require dense
points to differentiate.

We are interested in extending our adaptive perception framework in multiple
directions. Instead of inferring only from images, we would like to learn a
visual-lidar model online to leverage two sensors at inference time. We would
also like to incorporate more LiDAR features such as, slope, density. Finally,
we would like to incorporate more intelligent data sampling [38, 26].

## ACKNOWLEDGMENT

Distribution Statement ‘A’ (Approved for Public Release, Distribution
Unlimited). This research was sponsored by DARPA (#HR001121C0189). The views,
opinions, and/or findings expressed are those of the author(s) and should not
be interpreted as representing the official views or policies of the
Department of Defense or the U.S. Government. We thank JSV and EF for their
assistance.

## References

  * [1] D. Langer, J. Rosenblatt, and M. Hebert, “A behavior-based system for off-road navigation,” _IEEE Transactions on Robotics and Automation_ , vol. 10, no. 6, pp. 776 – 783, December 1994.
  * [2] A. Kelly, O. Amidi, M. Bode, M. Happold, H. Herman, T. Pilarski, P. Rander, A. Stentz, N. Vallidis, and R. Warner, “Toward reliable off road autonomous vehicles operating in challenging environments,” in _Proceedings of 9th International Symposium on Experimental Robotics (ISER ’04)_ , June 2004, pp. 599 – 608.
  * [3] J. A. Bagnell, D. Bradley, D. Silver, B. Sofman, and A. Stentz, “Learning for autonomous navigation,” _IEEE Robotics and Automation Magazine_ , vol. 17, no. 2, pp. 74–84, 2010.
  * [4] D. Maturana, P.-W. Chou, M. Uenoyama, and S. Scherer, “Real-time semantic mapping for autonomous off-road navigation,” in _Proceedings of 11th International Conference on Field and Service Robotics (FSR ’17)_ , September 2017, pp. 335 – 350.
  * [5] S. Triest, M. G. Castro, P. Maheshwari, M. Sivaprakasam, W. Wang, and S. Scherer, “Learning risk-aware costmaps via inverse reinforcement learning for off-road navigation,” 2023.
  * [6] D. Maturana, P.-W. Chou, M. Uenoyama, and S. Scherer, “Real-time semantic mapping for autonomous off-road navigation,” in _Field and Service Robotics_. Springer, 2018, pp. 335–350.
  * [7] T. Guan, D. Kothandaraman, R. Chandra, A. J. Sathyamoorthy, K. Weerakoon, and D. Manocha, “Ga-nav: Efficient terrain segmentation for robot navigation in unstructured outdoor environments,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 3, pp. 8138–8145, 2022.
  * [8] H. Dahlkamp, A. Kaehler, D. Stavens, S. Thrun, and G. R. Bradski, “Self-supervised monocular road detection in desert terrain.” in _Robotics: science and systems_ , vol. 38. Philadelphia, 2006.
  * [9] D. Maier, M. Bennewitz, and C. Stachniss, “Self-supervised obstacle detection for humanoid navigation using monocular vision and sparse laser data,” in _2011 IEEE International Conference on Robotics and Automation_. IEEE, 2011, pp. 1263–1269.
  * [10] S. Zhou, J. Xi, M. W. McDaniel, T. Nishihata, P. Salesses, and K. Iagnemma, “Self-supervised learning to visually detect terrain surfaces for autonomous robots operating in forested terrain,” _Journal of Field Robotics_ , vol. 29, no. 2, pp. 277–297, 2012.
  * [11] B. Sofman, E. Lin, J. A. Bagnell, N. Vandapel, and A. Stentz, “Improving robot navigation through self-supervised online learning,” in _Robotics: Science and Systems_ , 2006.
  * [12] T. Overbye and S. Saripalli, “G-vom: A gpu accelerated voxel off-road mapping system,” in _2022 IEEE Intelligent Vehicles Symposium (IV)_ , 2022, pp. 1480–1486.
  * [13] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, June 2019, pp. 4171–4186.
  * [14] E. Tiu, E. Talius, P. Patel, C. Langlotz, A. Ng, and P. Rajpurkar, “Expert-level detection of pathologies from unannotated chest x-ray images via self-supervised learning,” _Nature Biomedical Engineering_ , vol. 6, pp. 1–8, 09 2022.
  * [15] M. Sivaprakasam, S. Triest, W. Wang, P. Yin, and S. Scherer, “Improving off-road planning techniques with learned costs from physical interactions,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE Press, 2021, p. 4844–4850.
  * [16] M. Guaman Castro, S. Triest, W. Wang, J. M. Gregory, F. Sanchez, J. G. Rogers III, and S. Scherer, “How does it feel? self-supervised costmap learning for off-road vehicle traversability,” IEEE, 2023.
  * [17] O. Mayuku, B. W. Surgenor, and J. A. Marshall, “A self-supervised near-to-far approach for terrain-adaptive off-road autonomous driving,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 14 054–14 060.
  * [18] L. Wellhausen, A. Dosovitskiy, R. Ranftl, K. Walas, C. Cadena, and M. Hutter, “Where should i walk? predicting terrain properties from images via self-supervised learning,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 1509–1516, 2019.
  * [19] J. Zürn, W. Burgard, and A. Valada, “Self-supervised visual terrain classification from unsupervised acoustic feature learning,” _IEEE Transactions on Robotics_ , vol. 37, no. 2, pp. 466–481, 2021.
  * [20] R. Schmid, D. Atha, F. Schöller, S. Dey, S. Fakoorian, K. Otsu, B. Ridge, M. Bjelonic, L. Wellhausen, M. Hutter, and A.-a. Agha-mohammadi, “Self-supervised traversability prediction by learning to reconstruct safe terrain,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2022, pp. 12 419–12 425.
  * [21] X. Yao, J. Zhang, and J. Oh, “Rca: Ride comfort-aware visual navigation via self-supervised learning,” 2022.
  * [22] R. Hadsell, A. Erkan, P. Sermanet, M. Scoffier, U. Muller, and Y. LeCun, “Deep belief net learning in a long-range vision system for autonomous off-road driving,” in _2008 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2008, pp. 628–633.
  * [23] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [24] K.-Y. Lee, Y. Zhong, and Y.-X. Wang, “Do pre-trained models benefit equally in continual learning?” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , January 2023, pp. 6485–6493.
  * [25] R. M. French, “Catastrophic forgetting in connectionist networks,” _Trends in Cognitive Sciences_ , vol. 3, no. 4, pp. 128–135, 1999.
  * [26] C. de Masson d’Autume, S. Ruder, L. Kong, and D. Yogatama, “Episodic memory in lifelong language learning,” _CoRR_ , vol. abs/1906.01076, 2019.
  * [27] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, “Overcoming catastrophic forgetting in neural networks,” _Proceedings of the National Academy of Sciences_ , vol. 114, no. 13, pp. 3521–3526, 2017.
  * [28] M. Happold, M. Ollis, and N. Johnson, “Enhancing supervised terrain classification with predictive unsupervised learning,” in _Proceedings of Robotics: Science and Systems_ , Philadelphia, USA, August 2006.
  * [29] S. Zhao, H. Zhang, P. Wang, L. Nogueira, and S. Scherer, “Super odometry: Imu-centric lidar-visual-inertial estimator for challenging environments,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2021, pp. 8729–8736.
  * [30] J.-F. Lalonde, N. Vandapel, D. F. Huber, and M. Hebert, “Natural terrain classification using three-dimensional ladar data for ground robot mobility,” _Journal of Field Robotics_ , vol. 23, no. 10, pp. 839–861, 2006\.
  * [31] C. Wellington, A. Courville, and A. T. Stentz, “A generative model of terrain for autonomous navigation in vegetation,” _The International Journal of Robotics Research_ , vol. 25, no. 12, pp. 1287–1304, 2006.
  * [32] J. Amanatides and A. Woo, “A fast voxel traversal algorithm for ray tracing,” _Proceedings of EuroGraphics_ , vol. 87, 08 1987.
  * [33] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” 2015.
  * [34] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” 2017, cite arxiv:1704.04861.
  * [35] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoder-decoder architecture for image segmentation,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 39, no. 12, pp. 2481–2495, 2017.
  * [36] Y. Yang, X. Meng, W. Yu, T. Zhang, J. Tan, and B. Boots, “Learning semantics-aware locomotion skills from human demonstration,” in _Conference on Robot Learning_. PMLR, 2023, pp. 2205–2214.
  * [37] C. Wang, Y. Qiu, D. Gao, and S. Scherer, “Lifelong graph learning,” in _IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022\.
  * [38] C. Wang, Y. Qiu, W. Wang, Y. Hu, S. Kim, and S. Scherer, “Unsupervised online learning for robotic interestingness with visual memory,” _IEEE Transactions on Robotics (T-RO)_ , 2021.