figurec 5cm(1cm,10cm)  Publication doi

5cm(20.5cm,10cm)  Preprint doi

# Validation of a Zero-Shot Learning Natural Language Processing Tool for Data
Abstraction from Unstructured Healthcare Data

Basil Kaufmann ID Milton and Carroll Petrie Department of Urology, Icahn
School of Medicine at Mount Sinai, New York, USA  Dallin Busby ID Milton and
Carroll Petrie Department of Urology, Icahn School of Medicine at Mount Sinai,
New York, USA  Chandan Krushna Das ID Milton and Carroll Petrie Department of
Urology, Icahn School of Medicine at Mount Sinai, New York, USA  Neeraja Tillu
ID Milton and Carroll Petrie Department of Urology, Icahn School of Medicine
at Mount Sinai, New York, USA  Mani Menon  Milton and Carroll Petrie
Department of Urology, Icahn School of Medicine at Mount Sinai, New York, USA
Ashutosh K. Tewari ID Milton and Carroll Petrie Department of Urology, Icahn
School of Medicine at Mount Sinai, New York, USA  Michael A. Gorin ID Milton
and Carroll Petrie Department of Urology, Icahn School of Medicine at Mount
Sinai, New York, USA

###### Abstract

Objectives: To describe the development and validation of a zero-shot learning
natural language processing (NLP) tool for abstracting data from unstructured
text contained within PDF documents, such as those frequently found within
electronic health record systems. Materials and Methods: A data abstraction
tool based on the GPT-3.5 model from OpenAI was developed and compared to
three physician human abstractors in terms of time to task completion and
accuracy for abstracting data on 14 unique variables from a set of 199 de-
identified radical prostatectomy pathology reports. The reports were processed
by the software tool in vectorized and scanned formats to establish the impact
of optical character recognition on data abstraction. The tool was assessed
for superiority for data abstraction speed and non-inferiority for accuracy.
Results: The human abstractors required a mean of 101 seconds (95% CI, 97 to
104 seconds) per report for data abstraction, with times varying from 15 to
284 seconds. In comparison, the software tool required a mean of 12.8 second
(95% CI, 12.3 to 13.3 seconds) to process the vectorized reports and a mean of
15.8 seconds (95% CI 15.1 to 16.5 seconds) to process the scanned reports (P <
0.001 for all paired comparisons). The overall accuracies of the three human
abstractors were 94.7% (95% CI, 93.8 to 95.5%), 97.8% (95% CI, 97.2 to 98.3%),
and 96.4% (95% CI, 95.6 to 97%) for the combined set of 2786 datapoints. The
software tool had an overall accuracy of 94.2% (95% CI, 93.3 to 94.9%) for the
vectorized reports, proving to be non-inferior to the human abstractors at a
margin of -10% ($\alpha$=0.025). The tool had a slightly lower accuracy of
88.7% (95% CI 87.5 to 89.9%) using the scanned reports, proving to be non-
inferiority to 2 out of 3 human abstractors. Conclusion: The developed zero-
shot learning NLP tool affords researchers comparable levels of accuracy to
that of human abstractors, with significant time savings benefits. Because of
the lack of need for task-specific model training, the developed tool is
highly generalizable and can be used for a wide variety of data abstraction
tasks, even outside the field of medicine.

_K_ eywords data abstraction $\cdot$ natural language processing $\cdot$ large
language models

##  1 Introduction

The advent of electronic health records (EHRs) has transformed clinical
research by offering access to vast amounts of patient data. An essential part
of EHRs are free-text fields. Unlike their structured counterparts, free-text
fields offer a nuanced and comprehensive perspective on individual patient
cases, capturing a depth of clinical information that is often not found in
structured data. With that said, the use of unstructured data in clinical
research is fraught with challenges[1, 2, 3]. Chief among them is the labor-
intensive task of data extraction. Additionally, the lack of standardization
in free-text entries, due to their subjective and individualistic nature,
complicates data aggregation and comparison across different records or
healthcare providers. Furthermore, inconsistencies in the quality and
completeness of data further heighten this challenge for researchers. This has
prompted the need for sophisticated natural language processing (NLP)
techniques for data abstraction, but this has the downside of introducing the
potential for misinterpretation or omission of vital information[4, 5, 6, 7].

Recent advancements in artificial intelligence technologies, particularly the
introduction of large language models (LLMs) with zero-shot learning
capabilities, offer a promising solution to the abstraction of unstructured
healthcare data[8, 9, 10, 11, 12]. Zero-shot learning is a concept in machine
learning where a model is able to accurately classify data into categories
that it has not previously encountered during its training phase. Thus, zero-
shot learning makes it possible to forego the highly technical and time-
consuming work of first training a model for a given data abstraction task.

Herein, we describe the development of an LLM-based tool that utilizes zero-
shot learning to abstract unstructured healthcare data contained within
Portable Document Format (PDF) files. Following successful development of the
tool, we benchmarked its performance in terms of time to task completion and
accuracy for abstracting data from a set of radical prostatectomy pathology
reports. We compare the performance of this tool to three physician data
abstractors.

##  2 Material and Methods

Software Development. A zero-shot learning tool for data abstraction from PDF
documents was developed using Python programming language version 3.10. The
completed tool incorporates functions from several third-party libraries
including PyMuPDF for text extraction [13], TensorFlow Hub (Google DeepMind,
London, UK) for loading a universal sentence encoder for creating semantic
embedding from the extracted data[14], Scikit-learn for performing nearest
neighbor searches[15], and an advanced programming interface (API) from OpenAI
(San Francisco, CA) for answer generation[16]. The software relies on the use
of an API key provided by OpenAI that serves as an authentication token for
accessing the “text-davinci-003” LLM. This model is based on GPT-3.5 and has
been optimized for instruction-following tasks such as data abstraction[17].

Figure 1 provides a graphic representation of the steps followed by the
software to perform the data abstraction task. First, the program extracts
text from one or more PDF files specified by the user and removes any unwanted
characters or white spaces. The extracted text is next divided into smaller
chunks and processed by the sentence encoder model [14]. This step generates
numerical embeddings of the text and creates an index of the data using a
nearest neighbor algorithm. The software then performs a semantic search of
the embedded data and finds relevant chunks of text based on the user’s input
question. Next, a prompt is formulated for the text-davinci-003 model,
incorporating the relevant text segments and the user’s question of interest.
The API returns the model’s response, and this text is cleaned and placed in
an Excel spreadsheet (Microsoft Corporation, Redwood, WA) for downstream data
cleaning, standardization, and analysis.

Figure 1: Workflow of the developed zero-shot learning NLP tool for data
abstraction from unstructured text contained with PDF documents. Multiple PDF
files (such as scanned pathology reports), are processed from a designated
input folder and converted into text. Based on the user’s question, the tool
applies an AI algorithm to isolate specific information. Finally, it outputs
an Excel file containing the cleaned and extracted data into a predetermined
output folder.

Because the developed tool uses zero-shot learning for data abstraction, no
training steps are required for its implementation. The user is only required
to provide a prompt written in plain English to query against a provided set
of PDF documents. For example, a researcher may have a series of articles from
which they are interested in abstracting the name of each article’s first
author. The user can prompt the tool to ‘Query the input files and return the
name of the first author of each article.’ The result of this will be an Excel
file with a column containing author names and an identifier linking the row
to its respective input PDF file. The code underlying our tool is publicly
available on GitHub (https://github.com/kaufmannb/PDF-Extractor). In addition,
we have made available an executable file that researchers without computer
coding experience can use immediately for data abstraction
(https://rebrand.ly/5sxu4rn).

Datasets. The performance of the developed NLP tool was evaluated using a
publicly available set of 199 de-identified radical prostatectomy pathology
reports from The Cancer Genome Atlas (TCGA) program [18]. Although provided in
PDF format, these pathology reports were previously printed and then scanned
by the submitting sites. To mitigate potential errors introduced by optical
character recognition (OCR), the reports were first converted back to a
“vectorized” format as they originally appeared in the EHR. This was performed
by exporting each document into .docx format using Adobe Acrobat Pro (Adobe,
Inc., San Jose, CA) and then manually correcting any introduced errors in
Microsoft Word. The corrected reports were saved back into PDF format,
maintaining the integrity of the original formatting and layout (Supplementary
Figure 1). These vectorized reports served as the input for our main round of
data abstraction by the NLP tool. Additionally, a secondary analysis was
performed using the original scanned reports to quantify the impact of OCR
errors on the tool’s performance.

Assessment of the Data Abstraction Tool. Following completed development of
the data abstraction tool and generation of the testing datasets, we benchmark
its performance against three human data abstractors (all of whom are
physicians) in terms of time to task completion and accuracy of data
abstraction. The human abstractors and software tool were asked to perform the
identical task of abstracting 14 variables from the pathology reports (Table
1). These variables were selected based on their clinical relevance and
alignment with the protocol for examination of radical prostatectomy specimens
by the College of American Pathologists [19].

Table 1: List of abstracted variables.

Responses from the three human abstractors were collected using a custom-built
website that displayed each pathology report in a PDF viewer, accompanied by
an input form designed to capture the abstracted data. Each variable featured
a specific set of dropdown answer choices, ensuring that only predefined
answers could be selected, thereby precluding the possibility of random or
extraneous responses. Responses, along with processing times, were logged for
each abstracted report.

To extract data using the NLP tool, the following prompt was utilized:
“Complete the following list of variables with the corresponding values
extracted from the given pathology report: pT-Stage, Primary Gleason Grade,
Secondary Gleason Grade, Gleason Sum Score, Tertiary Gleason Pattern or Grade,
Extraprostatic Extension, Seminal Vesical Invasion, Lymphovascular Invasion,
Perineural Invasion, Surgical Margin Status, pN-Stage, Number of Lymph Nodes
Removed, Number of Lymph Nodes Involved by Cancer, Specific Prostate Weight in
g.” Additionally, the program was instructed to “Compose a comprehensive reply
to the query using the search results given. If the search results mention
multiple subjects with the same name, create separate answers for each. Only
include information found in the results and don’t add any additional
information. Make sure the answer is correct and don’t output false content.
If the text does not relate to the query, simply state ‘Found Nothing’. Ignore
outlier search results which has nothing to do with the question. Only answer
what is asked. The answer should be short and concise.”

The mean time required for data abstraction was calculated for the three human
abstractors as well as the NLP tool using the two sets of PDF reports.
Similarly, the accuracy of data abstraction was calculated for these groups in
terms of overall accuracy for the combined set of datapoints (14 variables x
199 reports = 2,786 datapoints) and at the individual variable level. For this
analysis, the ground truth (i.e. the correct answer for each data point) was
established from the individual responses provided by the human abstractors.
The ground truth response was defined when there was agreement between at
least two of the data abstractors. In cases with any degree of disagreement
among the abstractors, the report was re-examined and if a clear error was
identified on part of the dissenting abstractor, the majority consensus was
upheld as the ground truth. However, if the re-examination did not reveal an
obvious error, further discussions between the abstractors were held until
consensus was reached. In instances where agreement between the three
abstractors could not be achieved, a fourth reviewer was involved.

Statistical Analysis. Differences in data abstraction times were assessed for
statistical significance using the paired Student’s t-test. Comparisons were
separately made between the NLP tool and each human abstractor for the two
sets of PDF reports. Additionally, the performance of the NLP was compared to
itself in a paired fashion using the two datasets. For comparisons of
superiority, a P-value of <0.05 was considered statistically significant. A
similar paired analysis was performed to assess the accuracy of data
abstraction using the McNemar’s test of paired proportions. The accuracy of
the NLP tool was assessed for non-inferiority to the human abstractors using a
non-inferiority margin of -10% ($\alpha$ = 0.025). Data analysis and
visualization was performed using R version 4.0.2 (R Foundation for
Statistical Computing).

##  3 Results

As a group, the human abstractors required a mean of 101 seconds (95% CI, 97
to 104 seconds) per report for data abstraction, with times varying from 15 to
284 seconds. Assessed individually, the three human abstractors had data
collection times of 104 seconds (95% CI 99 to 109 seconds), 94 seconds (95% CI
89 to 100 seconds), and 103 seconds (95% CI 96 to 110), respectively. In
comparison, the software tool required a mean of 12.8 second (95% CI, 12.3 to
13.3 seconds) to process the vectorized reports and a mean of 15.8 seconds
(95% CI 15.1 to 16.5 seconds) to process the scanned reports (P < 0.001 for
all paired comparisons). The NLP tool required significantly less time to
process the vectorized reports than the scanned documents (P < 0.001). Our
observations with respect to data abstraction times are highlighted in Figure
2.

Figure 2: Violin plot showing the distribution of times required for data
abstraction.

Compared to the ground truth, the overall accuracies of the three human
abstractors were 94.7% (95% CI, 93.8 to 95.5%), 97.8% (95% CI, 97.2 to 98.3%),
and 96.4% (95% CI, 95.6 to 97%) for the combined set of 2786 datapoints. In
contrast, the software tool achieved an overall accuracy of 94.2% (95% CI,
93.3 to 94.9%) for the vectorized reports and 88.7% (95% CI 87.5 to 89.9%) for
the scanned reports. The overall accuracy of the tool for abstracting data
from the vectorized reports proved to be non-inferior to each of the three
human abstractors (Supplementary Figure 2). For the scanned reports, the tool
was non-inferior to 2 out of 3 human abstractors. A significantly higher
degree of accuracy was achieved by the tool for the task of overall data
abstraction when using the vectorized reports (Supplementary Figure 3).

Figure 3: Accuracy of data abstraction by the three humans and NLP tool.

Two of the three human abstractors achieved an accuracy of >90% for each of
the 14 individual variables. The third abstractor achieved this level of
accuracy for 12 out of 14 variables. In contrast, the software achieved >90%
accuracy for 11 variables using the vectorized reports whereas this level of
accuracy was achieved for only 8 parameters when using the scanned reports.

Figure 3 provides a summary of the accuracies achieved by the three human
abstractors as well as the NLP tool using both datasets.

##  4 Discussion

The present study explored the time efficiency and accuracy of a zero-shot
learning NLP tool for abstracting data from PDF pathology reports of radical
prostatectomy specimens. Our data show that the tool is capable of achieving a
7-fold increase in processing speed compared to human abstractors while
maintaining an overall abstraction accuracy of 94%. Notably, the tool was non-
inferior to the three humans at a -10% margin for the overall task of
abstracting 2,786 datapoints from vectorized PDF documents. Although the tool
performed slightly worse using the dataset of scanned reports, non-inferiority
was achieved relative to two out of the three human abstractors. The
performance of the software tool varied across the 14 variables examined,
reaching an accuracy of over 90% accuracy in 11 out of 14 variables. However,
the tool struggled with certain variables such as "Tertiary Gleason Pattern"
or "Number of Lymph Nodes Removed,” achieving accuracies in the range of 83%.

Previous research has employed various machine learning algorithms to parse
non-structured data from EHR systems [20, 21, 22, 9, 23, 24, 25, 26, 6, 27,
28, 29]. However, these tools have uniformly required complex supervised
training methods. This requirement is highly labor intensive and limits the
number of variables one can practically extract. Furthermore, the reliance on
supervised and semi-supervised learning methods limits generalizability of
tools to unfamiliar sources of data. These limitations were evident in a study
by McCowan et al. [23] who used a rule-based and support vector machine
classifiers to extract information on tumor (T) and nodal (N) stages from lung
cancer pathology reports. This study required a training set of 710 reports
and yet only achieved accuracy levels of 74% and 86% for T and N stage
classification, respectively. Abedian et al. [20] looked the broaden the
capability of NLP tools for abstracting cancer staging data from a wider range
of cancer types, but this required and enormous training data from 555,681
pathology reports and was still only limited to four outputs (ICD10 code and
T, N, and metastasis stage). While other more advanced tools, such as DeepPhe,
have been specifically engineered to widen the breath of data abstraction
capabilities, such tools still require supervised training methods and/or have
seen only limited validated in a handful of cancer types [26].

Compared to the tools mentioned above, our method utilizes a zero-shot
learning strategy, which requires no task-specific training. This gives our
tool the capacity to abstract data on any topic from a body of PDF documents.
As urologists, the application described in this report was based on clinical
data abstraction from radical prostatectomy reports, but the developed tool
could be employed for any number of applications within the field of medicine.
Additionally, our tool has the potential be used across a broad range of
industries that relay on insights from data contained in free-text format.

For the development of the software, we specifically utilized the publicly
available text-davinci-003 model from OpenAI’s GPT-3 family. Compared to other
iterations of GPT-3, the model used in our study was previously shown to excel
in tasks like Named Entity Recognition (NER) to identify and categorize key
information from unstructured text [17]. This study also showed that the text-
davinci-003 model was particularly well suited for interpreting complex
sentences and phrases, as shown in tasks like Part-of-speech Tagging [30] and
Semantic Matching [31].

The prompt crafted by the user and sent to text-davinci-003 model is crucial
for achieving the desired output. GPT is highly sensitive to the form and
choice of wording, and there are even specialists, known as "prompt
engineers", who excel in this task [32, 33]. An example of this can be found
in a recent study that evaluated the specific manner in which multiple choice
questions are presented to an LLM can significantly impact the accuracy of
response [34]. These authors found that simply asking multiple choice
questions of the model in the usual manner resulted in an accuracy of 55.8%
when indeterminate responses were censored and 36.1% when indeterminate
responses were included on the United States Medical Licensing Examination
Step 1. When the users next asked the model to provide a rationale for each
answer choice, the accuracy increased to 64.5% when indeterminate responses
were censored and 41.2% when indeterminate responses were included.

While our model was highly accurate for the abstraction of 11 of the selected
variables, more modest results were achieved for the remaining 3. As described
by Zheng et al. [35], the majority of incorrect responses from LLMs can be
grouped into four categories of error: (1) comprehension, (2) factualness, (3)
specificity, and (4) inference. In errors of comprehension, the model does not
fully understand the context or the intent of the question or the information
in the report, whereas for errors of factualness the model recalls incorrect
or outdated information. In terms of specificity, these errors occur because
the model does not provide enough information that is relevant to the question
to derive a useful answer. For example if it should extract the Gleason score
8, it answers with “high” instead of providing the specific number. Finally,
for inference errors, the model might have all the necessary information to
answer a question, but it fails to “reason” with the facts effectively to
arrive at the correct answer.

The tool’s performance is not solely dependent on the GPT component or prompt
generation. In the application selected for our tool, parsing PDFs is a
critical aspect of the process. Various Python libraries are available for PDF
parsing, but their efficacy can vary, potentially leading to data loss or
errors during the conversion from PDF to text. Particularly, data in complex
layouts or tabular formats might be misinterpreted. We found in our sub-
analysis of the scanned reports, that OCR made a difference in overall
accuracy of 5.5%, leading to a decrease in performance of the NLP tool to
88.7%.

Despite the challenges mentioned, we feel the developed tool represents a
significant advancement in the ability for researchers to abstract data
without the need for individual NLP model creation or training. We feel that
if vectorized PDFs or improved OCR and text extraction methods are utilized
with carefully crafted prompts specific to individual variables, the error
rate achieved would balance well against the gain of 7-fold improvement in
abstraction efficiency, freeing human abstractors to perform more critical
research tasks. We should note, however, that the tool developed for this
research report is not quite ready for immediate use in clinical practice, as
it does not include end-to-end encryption which is required to meet the
standards of the United States Health Insurance Portability and Accountability
Act (HIPAA) or the General Data Protection Regulation (GDPR) of the European
Union.

##  5 Conclusion

The presented zero-shot learning NLP tool affords researchers comparable
levels of accuracy to that of human abstractors, with significant time savings
benefits. Because our tool does not require task-specific training, it can be
adopted for a limitless number of data abstraction tasks, including
applications outside the field of medicine. Continued refinement, particularly
in terms of optical character recognition (OCR) capabilities, may enhance its
practical use, speed, and degree of accuracy.

##  6 Acknowledgements

The research was supported by a grant from the Swiss Cancer League (Krebsliga
Schweiz), Swiss Urological Society, and the Menon Family Foundation.

## References

  * Garza et al. [2022] Maryam Y Garza, Tremaine Williams, Sahiti Myneni, Susan H Fenton, Songthip Ounpraseuth, Zhuopei Hu, Jeannette Lee, Jessica Snowden, Meredith N Zozus, and Anita C Walden.  Measuring and controlling medical record abstraction (mra) error rates in an observational study.  _BMC Medical Research Methodology_ , 22(1):227, 2022.  ISSN 1471-2288.
  * Kong [2019] Hyoun-Joong Kong.  Managing unstructured big data in healthcare system.  _Healthcare informatics research_ , 25(1):1–2, 2019.
  * Polnaszek et al. [2016] Brock Polnaszek, Andrea Gilmore-Bykovskyi, Melissa Hovanes, Rachel Roiland, Patrick Ferguson, Roger Brown, and Amy JH Kind.  Overcoming the challenges of unstructured data in multi-site, electronic medical record-based abstraction.  _Medical care_ , 54(10):e65, 2016.
  * Alzoubi et al. [2019] Hadeel Alzoubi, Raid Alzubi, Naeem Ramzan, Daune West, Tawfik Al-Hadhrami, and Mamoun Alazab.  A review of automatic phenotyping approaches using electronic health records.  _Electronics_ , 8(11):1235, 2019.  ISSN 2079-9292.
  * Miller et al. [2022] Benjamin T Miller, Aldo Fafaj, Luciano Tastaldi, Hemasat Alkhatib, Samuel Zolin, Raha AlMarzooqi, Chao Tu, Diya Alaedeen, Ajita S Prabhu, and David M Krpata.  Capturing surgical data: Comparing a quality improvement registry to natural language processing and manual chart review.  _Journal of Gastrointestinal Surgery_ , 26(7):1490–1494, 2022.  ISSN 1091-255X.
  * Savova et al. [2019] Guergana K Savova, Ioana Danciu, Folami Alamudun, Timothy Miller, Chen Lin, Danielle S Bitterman, Georgia Tourassi, and Jeremy L Warner.  Use of natural language processing to extract clinical cancer phenotypes from electronic medical records.  _Cancer research_ , 79(21):5463–5470, 2019.  ISSN 0008-5472.
  * Sheikhalishahi et al. [2019] Seyedmostafa Sheikhalishahi, Riccardo Miotto, Joel T Dudley, Alberto Lavelli, Fabio Rinaldi, and Venet Osmani.  Natural language processing of clinical notes on chronic diseases: systematic review.  _JMIR medical informatics_ , 7(2):e12239, 2019\.
  * Dunn et al. [2022] Alexander Dunn, John Dagdelen, Nicholas Walker, Sanghoon Lee, Andrew S Rosen, Gerbrand Ceder, Kristin Persson, and Anubhav Jain.  Structured information extraction from complex scientific text with fine-tuned large language models.  _arXiv preprint arXiv:2212.05238_ , 2022.
  * Leyh-Bannurah et al. [2018] Sami-Ramzi Leyh-Bannurah, Zhe Tian, Pierre I Karakiewicz, Ulrich Wolffgang, Guido Sauter, Margit Fisch, Dirk Pehrke, Hartwig Huland, Markus Graefen, and Lars Budäus.  Deep learning for natural language processing in urology: State-of-the-art automated extraction of detailed pathologic prostate cancer data from narratively written electronic health records.  _JCO clinical cancer informatics_ , 2:1–9, 2018.  ISSN 2473-4276.
  * [10] Bernardino Romera-Paredes and Philip Torr.  An embarrassingly simple approach to zero-shot learning.  In _International conference on machine learning_ , pages 2152–2161. PMLR.
  * Sun et al. [2021] Xiaohong Sun, Jinan Gu, and Hongying Sun.  Research progress of zero-shot learning.  _Applied Intelligence_ , 51:3600–3614, 2021.  ISSN 0924-669X.
  * Yim et al. [2016] Wen-wai Yim, Meliha Yetisgen, William P Harris, and Sharon W Kwan.  Natural language processing in oncology: a review.  _JAMA oncology_ , 2(6):797–804, 2016.  ISSN 2374-2437.
  * RN1 [a] Pymupdf. version 1.22.3. retrieved from https://github.com/pymupdf/pymupdf, a.
  * Cer et al. [2018] Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, and Chris Tar.  Universal sentence encoder.  _arXiv preprint arXiv:1803.11175_ , 2018.
  * RN1 [b] Scikit-learn. version 1.2.2. retrieved from https://scikit-learn.org/stable/.  b.
  * RN1 [c] Openai python library. version 0.27.7. retrieved from https://pypi.org/project/openai/.  c.
  * Ye et al. [2023] Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, and Yang Shen.  A comprehensive capability analysis of gpt-3 and gpt-3.5 series models.  _arXiv preprint arXiv:2303.10420_ , 2023.
  * [18] National Cancer Institute.  The cancer genome atlas program (tcga).
  * Paner et al. [2021] G Paner, J Srigley, Jason Pettus, Giovanna Angela Giannico, Joseph Sirintrapun, and Lara R Harik.  Protocol for the examination of radical prostatectomy specimens from patients with carcinoma of the prostate gland.  _College of American Pathologists (CAP): Northfield, IL, USA_ , 2021\.
  * Abedian et al. [2021] Sajjad Abedian, Evan T Sholle, Prakash M Adekkanattu, Marika M Cusick, Stephanie E Weiner, Jonathan E Shoag, Jim C Hu, and Thomas R Campion Jr.  Automated extraction of tumor staging and diagnosis information from surgical pathology reports.  _JCO Clinical Cancer Informatics_ , 5:1054–1061, 2021.  ISSN 2473-4276.
  * Glaser et al. [2018] Alexander P Glaser, Brian J Jordan, Jason Cohen, Anuj Desai, Philip Silberman, and Joshua J Meeks.  Automated extraction of grade, stage, and quality information from transurethral resection of bladder tumor pathology reports using natural language processing.  _JCO Clinical Cancer Informatics_ , 2:1–8, 2018.  ISSN 2473-4276.
  * Kim et al. [2014] B. J. Kim, M. Merchant, C. Zheng, A. A. Thomas, R. Contreras, S. J. Jacobsen, and G. W. Chien.  A natural language processing program effectively extracts key pathologic findings from radical prostatectomy reports.  _J Endourol_ , 28(12):1474–8, 2014.  ISSN 0892-7790.  doi: 10.1089/end.2014.0221.
  * McCowan et al. [2007] Iain A McCowan, Darren C Moore, Anthony N Nguyen, Rayleen V Bowman, Belinda E Clarke, Edwina E Duhig, and Mary-Jane Fry.  Collection of cancer stage data by classifying free-text medical reports.  _Journal of the American Medical Informatics Association_ , 14(6):736–745, 2007.  ISSN 1527-974X.
  * [24] Iain McCowan, Darren Moore, and Mary-Jane Fry.  Classification of cancer stage from free-text histology reports.  In _2006 International Conference of the IEEE Engineering in Medicine and Biology Society_ , pages 5153–5156. IEEE.  ISBN 1424400325.
  * Nguyen et al. [2010] Anthony N Nguyen, Michael J Lawley, David P Hansen, Rayleen V Bowman, Belinda E Clarke, Edwina E Duhig, and Shoni Colquist.  Symbolic rule-based classification of lung cancer stages from free-text pathology reports.  _Journal of the American Medical Informatics Association_ , 17(4):440–445, 2010.  ISSN 1527-974X.
  * Savova et al. [2017] G. K. Savova, E. Tseytlin, S. Finan, M. Castine, T. Miller, O. Medvedeva, D. Harris, H. Hochheiser, C. Lin, G. Chavan, and R. S. Jacobson.  Deepphe: A natural language processing system for extracting cancer phenotypes from clinical records.  _Cancer Res_ , 77(21):e115–e118, 2017.  ISSN 0008-5472 (Print) 0008-5472.  doi: 10.1158/0008-5472.Can-17-0615.
  * Warner et al. [2016] Jeremy L Warner, Mia A Levy, Michael N Neuss, Jeremy L Warner, Mia A Levy, and Michael N Neuss.  Recap: feasibility and accuracy of extracting cancer stage information from narrative electronic health record data.  _Journal of oncology practice_ , 12(2):157–158, 2016.  ISSN 1554-7477.
  * Yala et al. [2017] Adam Yala, Regina Barzilay, Laura Salama, Molly Griffin, Grace Sollender, Aditya Bardia, Constance Lehman, Julliette M Buckley, Suzanne B Coopey, and Fernanda Polubriaginof.  Using machine learning to parse breast pathology reports.  _Breast cancer research and treatment_ , 161:203–211, 2017\.  ISSN 0167-6806.
  * Zhou et al. [2022] Sicheng Zhou, Nan Wang, Liwei Wang, Hongfang Liu, and Rui Zhang.  Cancerbert: a cancer domain-specific language model for extracting breast cancer phenotypes from electronic health records.  _Journal of the American Medical Informatics Association_ , 29(7):1208–1216, 2022.  ISSN 1527-974X.
  * Schmid [1994] Helmut Schmid.  Part-of-speech tagging with neural networks.  _arXiv preprint cmp-lg/9410018_ , 1994.
  * [31] Jyun-Yu Jiang, Mingyang Zhang, Cheng Li, Michael Bendersky, Nadav Golbandi, and Marc Najork.  Semantic text matching for long-form documents.  In _The world wide web conference_ , pages 795–806.
  * Lee et al. [2023] Peter Lee, Sebastien Bubeck, and Joseph Petro.  Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine.  _New England Journal of Medicine_ , 388(13):1233–1239, 2023.  ISSN 0028-4793.
  * White et al. [2023] Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt.  A prompt pattern catalog to enhance prompt engineering with chatgpt.  _arXiv preprint arXiv:2302.11382_ , 2023.
  * Kung et al. [2023] Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, and James Maningo.  Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models.  _PLoS digital health_ , 2(2):e0000198, 2023.  ISSN 2767-3170.
  * Zheng et al. [2023] Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang.  Why does chatgpt fall short in answering questions faithfully?  _arXiv preprint arXiv:2304.10513_ , 2023.

##  7 Supplementary Material

List of suppfigures 1 Example of a scanned (A) and vectorized (B) de-
identified radical prostatectomy pathology reports.

List of suppfigures 2 Non-inferiority analysis of accuracy of the NLP tool
versus human abstractors using vectorized (A) and scanned reports (B). Lines
represent 97.5% confidence intervals of differences in proportions of
accuracy, with points marking the mean difference. The blue dashed line
denotes the -10% non-inferiority margin.

List of suppfigures 3 Non-inferiority analysis of accuracy of the NLP tool
using vectorized versus scanned reports. Lines represent 97.5% confidence
intervals of differences in proportions of accuracy, with points marking the
mean difference.


# Local_INN: Implicit Map Representation and Localization with
Invertible Neural Networks
††thanks:

Zirui Zang, Hongrui Zheng, Johannes Betz, Rahul Mangharam All authors are with
the University of Pennsylvania, Department of Electrical and Systems
Engineering, 19104, Philadelphia, PA, USA. Emails: {zzang, hongruiz, joebetz,
<EMAIL_ADDRESS>
###### Abstract

Robot localization is an inverse problem of finding a robot’s pose using a map
and sensor measurements. In recent years, Invertible Neural Networks (INNs)
have successfully solved ambiguous inverse problems in various fields. This
paper proposes a framework that solves the localization problem with INN. We
design an INN that provides implicit map representation in the forward path
and localization in the inverse path. By sampling the latent space in
evaluation, Local_INN outputs robot poses with covariance, which can be used
to estimate the uncertainty. We show that the localization performance of
Local_INN is on par with current methods with much lower latency. We show
detailed 2D and 3D map reconstruction from Local_INN using poses exterior to
the training set. We also provide a global localization algorithm using
Local_INN to tackle the kidnapping problem.

##  I Introduction

Robot localization is a problem of finding a robot’s pose using a map and
sensor measurements, like LiDAR scans. Any moving robot needs to interact with
the physical world correctly. However, finding injective mappings between
measurements and poses is difficult because sensor measurements from multiple
distant poses can be similar.

To solve this ambiguity, the widely adopted method, Monte Carlo Localization
(MCL)[1, 2] uses random hypothesis sampling and sensor measurement updates to
infer the pose. Other common approaches are to use Bayesian filtering[3] or to
find better-distinguishable global descriptors on the map[4, 5]. Recent
developments in localization research usually propose better measurement
models or feature extractors within these frameworks. On contrary, this paper
proposes a new approach to frame the localization problem as an ambiguous
inverse problem and solve it with an invertible neural network (INN). We claim
that INN is naturally suitable for the localization problem with many
benefits, as we will show in this paper.

Robot localization is an inverse problem, which is when we are given a set of
observations and try to find the causal factors. Usually, it’s easier to
calculate the expected observations if given the causal factors. In the
context of LiDAR-based localization, the robot’s pose in the environment
causes the particular scan measurements. In addition, when given a map, we can
easily simulate LiDAR scans from any pose on the map.

Figure 1: Local_INN is a framework of localization with invertible neural
networks. Compared to current localization methods, Local_INN stores map
information within the neural network. Evaluation of Local_INN in forward
direction gives compressed map information, and in the reverse direction gives
accurate localization with fast runtime and uncertainty estimation.

Invertible neural networks such as normalizing flows[6, 7, 8, 9] have been
used to solve inverse problems in various fields[10, 11, 12, 13, 14, 15]. It
learns a bijective mapping between the source and target distributions with a
series of invertible transformations. It uses a latent space to capture the
lost ambiguous information during training. We use pose-scan data pairs to
train such a bijective mapping. The forward path is from pose to scan and the
reverse path is from scan to pose. Because INNs require the same input and
output dimensions, we use a Variational Autoencoder (VAE)[16] to reduce the
dimension of the lidar scans and use Positional Encoding[17] to augment the
dimension of the poses. With the help of conditional inputs, we can reduce the
ambiguity of the inverse problem. In our case, we use zones in the map
calculated from the previous pose of the robot as conditional input into the
INN. During the evaluation, we sample the latent space to find the full
posterior distribution of the pose, given a sensor measurement. We validated
our method in localization experiments with 2D and 3D LiDARs, both in
simulation and with real data. To summarize, this paper has four major
contributions:

  1. 1.

Map Compression: Local_INN provides an implicit map representation and a
localization method within one neural network. Map files are no longer needed
when localizing.

  2. 2.

Uncertainty Estimation: Local_INN outputs not just a pose but a distribution
of inferred poses, the covariance of which can be used as the confidence of
the neural network when fusing with other sensors, enhancing the overall
robustness.

  3. 3.

Fast and Accurate: We demonstrate that the localization performance of
Local_INN is comparable to particle filter at slow speed and better at high
speed with much lower latency with 2D LiDAR experiments.

  4. 4.

Ability to Generalize: We demonstrate that the framework of Local_INN can
learn complex 3D open-world environments and provides accurate localization.
We also provide an algorithm for global localization with Local_INN.

##  II Related Work

Local_INN sits at the intersection of two research fields: Localization and
Normalizing Flows. In this section, we will briefly introduce both fields.

###  II-A Lidar-based Localization

Monte Carlo Localization (MCL)[1], ever since its introduction, has been a
popular localization framework for its reliable performance and the modularity
to swap the motion or measurement model with any desired method. Many
developments in localization seek to improve within the framework of MCL.[18,
19, 20] Outside the framework of MCL, people have used methods such as
Bayesian inference[21], RNNs[22, 23], global descriptors[24, 25], or combining
them[26]. We propose Local_INN as a new framework of solve the problem.

For learning-based localization methods, uncertainty estimations of the neural
networks become a challenge. There are efforts to approximate the
uncertainty[27, 28, 29], but it hasn’t been widely applied. Local_INN comes
naturally with an uncertainty estimation due to the use of normalizing flows.

Large map size is also becoming a burden as pointed out by [30]. After the
advent of NeRF[31], it was clear that neural networks are very capable of
implicitly representing spatial information. There are developments in using
neural networks for implicit map representation in the SLAM pipeline[32, 33].
Local_INN builds on that while providing a method of localization.

###  II-B Normalizing Flows

A normalizing flow is a series of invertible transformations that gradually
transform a source data distribution into a target data distribution. Methods
of achieving such bijective mappings have been developing rapidly in recent
years[9]. Real-valued non-volume preserving (RealNVP) transformations
introduced by Dinh et al.[7] use coupling layers that are efficient to compute
in both forward and reverse processes. Although newer normalizing flows have
better expressiveness[34, 35, 36], we choose to use RealNVP for its
efficiency.

The framework of solving ambiguous inverse problems using normalizing flows
was introduced by Ardizonne et al.[10] and was later extended by [11, 37] to
include a conditional input that is concatenated to the vectors inside the
coupling layers. They proposed to use a latent variable to encode the lost
information in training due to the ambiguity of the problem. During the
evaluation, repeatedly sampling the latent variable can give the full
posterior distribution given the input. In this paper, we added a VAE to the
framework so that we can use high-dimensional input. The use of latent
variables gives us distributions of estimated poses, which we can use to
calculate the covariance.

##  III Methodology

LiDARs are widely used in moving robots and autonomous vehicles. 2D or 3D
LiDARs produce one or multiple arrays of range distances with each value in
the array being the distance from the robot to the closest obstacle at a
certain angle. The localization problem with LiDAR is: given a LiDAR scan,
find the robot’s $[x,y]$ coordinates on the map and its heading $\theta$
relative to the $x$ axis of the map.

Figure 2: Network Structure of the Local_INN. The forward path (solid arrows)
is from pose to LiDAR scan. The reverse path (dashed arrows) is from LiDAR
scan to robot pose. Conditional input is calculated from the robot’s previous
pose. The INN used in this paper has 6 coupling layers and the VAE encoder and
decoder have 2 layers of MLPs for 2D LiDARs and plus 6 layers of 2D
convolutions for 3D LiDARs.

We use normalizing flow to find a bijective mapping between a robot’s pose
vector $\mathbf{x}\in\mathbb{R}^{3}$ on the map and LiDAR scan vector
$\mathbf{y}\in\mathbb{R}^{\text{angle}}$ with a latent vector
$\mathbf{z}\in\mathbb{R}^{6}$. The forward path of the localization problem is
easy, so we can simulate an infinite amount of pose-scan data pairs for
training by randomly sampling the state space. We use a rounded pose (as in
equation 2) of the robot to produce the conditional input
$\mathbf{c}\in\mathbb{R}^{3}$ in the INN. This rounded pose can be computed
during testing by rounding the robot’s previous pose. Because INN requires the
same input and output dimension, we use positional encoding to augment the
pose vector $\mathbf{x}$ to $\mathbf{\hat{x}}\in\mathbb{R}^{6L}$, where $L$ is
the level of the sine-cosine positional encoding [17]. On the LiDAR scan side,
we use a VAE[16] to encode the LiDAR scan $\mathbf{y}$ to
$\mathbf{\hat{y}}\in\mathbb{R}^{6L-6}$, which is concatenated with latent
vector $\mathbf{z}\sim\mathcal{N}(0,\,1)$. We use the latent vector to catch
the full posterior distribution of $\mathbf{x}$ conditioned on $\mathbf{c}$
given $\mathbf{y}$. This can later be used to sample the covariance of the
inferred pose vector.

###  III-A Conditional Normalizing Flow

Normalizing flows contain a series of invertible transformations. We use the
affine coupling block architecture introduced in Real-NVP[7] and extended by
[11, 37] to incorporate a conditional input. The forward path of a single
coupling block is:

$\begin{split}\mathbf{v}_{1}&=\mathbf{u}_{1}\odot\exp(s_{2}(\mathbf{u}_{2},\mathbf{\hat{c}}))+t_{2}(\mathbf{u}_{2},\mathbf{\hat{c}}),\\\
\mathbf{v}_{2}&=\mathbf{u}_{2}\odot\exp(s_{1}(\mathbf{v}_{1},\mathbf{\hat{c}}))+t_{1}(\mathbf{v}_{1},\mathbf{\hat{c}}).\end{split}$
(1)

The input $\mathbf{u}$ is split into two halves $\mathbf{u}_{1}$ and
$\mathbf{u}_{2}$, which undergo affine transformations with scale coefficient
$s_{i}$ and translation coefficient $t_{i}$ for $i\in\\{1,2\\}$. Here $\odot$
is element-wise multiplication. The outputs $\mathbf{v}_{1}$ and
$\mathbf{v}_{2}$ are then concatenated together before exiting this coupling
block. The exponential function here is to eliminate zero outputs, which
ensures invertibility. In the reverse direction, given $\mathbf{v}_{1}$ and
$\mathbf{v}_{2}$, this structure is easily invertible without any
computational overheads. Therefore, $s_{i}$ and $t_{i}$ are not required to be
invertible and can be learned with neural networks. Multiple coupling blocks
are connected to increase the expressiveness of the normalizing flows. After
each coupling block, there is a predefined random permutation to shuffle the
variables so that the splitting of the input vector is different for each
block. We followed [15] to use two layers of MLP with ReLU activation in each
affine coupling block and used a parameterized soft clamping mechanism to
prevent instabilities. Let’s denote the forward and reverse path of the INN
network with $h_{\text{inn}}^{\mathit{forward}}$,
$h_{\text{inn}}^{\mathit{reverse}}$.

To deal with the inverse ambiguity due to map symmetry, a rounded pose
computed from the robot’s previous pose $\mathbf{x}^{\text{pre}}$ is passed
through a positional encoding $\gamma(\cdot)$, then encoded by a separate MLP
$h_{\text{cond}}$ before concatenating to $\mathbf{u}_{i}$ or $\mathbf{v}_{i}$
in the coupling block:

$\mathbf{c}=\frac{\lceil
N\mathbf{x^{\text{pre}}}\rfloor}{N},\mathbf{\hat{c}}=h_{\text{cond}}(\gamma(\mathbf{c})).$
(2)

During training, $\mathbf{x}^{\text{pre}}$ is approximated by adding a zero
mean Gaussian noise to the ground truth pose:

$\mathbf{x}^{\text{pre}}_{\text{training}}=\mathbf{x}+\delta,\delta\sim\mathcal{N}(0,\,\sigma^{2}).$
(3)

The rounded previous states essentially divide the state space into $N^{3}$
zones, and which zone the robot previously existed in is provided to the INN
as conditional input. The Gaussian noise during training ensures that it’s
okay for the $\mathbf{x^{\text{pre}}}$ near zone boundaries to be rounded into
either neighboring zones. Depending on the map, $\sigma^{2}$ for
$[x,y,\theta]$ and integer parameter $N$ needs to be picked. We picked
$\sigma^{2}$ around 0.5 meters and $N=10$ for all our experiments, which means
the zones are quite large for the 3D maps.

###  III-B Positional Encoding

Positional encoding was used in [17] [31] to boost the performance of the
neural network in fitting high-frequency information. The positional encoding
$\gamma(\cdot)$ we used maps from $\mathbb{R}$ to $\mathbb{R}^{2L}$ with
increasing frequencies:

$\begin{split}\gamma(p)=(&\sin(2^{0}\pi p),\cos(2^{0}\pi p),\dotsc,\\\
&\sin(2^{L-1}\pi p),\cos(2^{L-1}\pi p)).\end{split}$ (4)

When applied to pose vectors, function $\gamma(\cdot)$ is applied separately
to $[x,y,\theta]$. Pose vector $\mathbf{x}$ is encoded with $L=10$ and the
conditional input is encoded with $L=1$. All variables are normalized to
$[0,1)$ before being applied to $\gamma(\cdot)$. We observed that adding
positional encoding directly helps the forward path by augmenting the
3-dimensional input, which in turn helps the reverse training as well.

###  III-C Variational Autoencoder

LiDARs produce hundreds to thousands of range data points per channel. Due to
the input and output dimension requirement of the INN, putting everything into
the INN would vastly increase the size of the network without proportional
benefit. On the other hand, sub-sampling LiDAR scans increase susceptibility
to noisy or invalid LiDAR points. Therefore, to fully utilize the LiDAR scans
points and simultaneously limit the network size, we use a VAE to first encode
the LiDAR scans into a multivariate Gaussian latent space with mean
$\boldsymbol{\mu}_{\text{vae}}$ and variance
$\boldsymbol{\sigma}^{2}_{\text{vae}}$.

The encoder $h_{\text{vae}}^{\mathit{encode}}$ of the VAE has one-layer MLP
with ReLU that is connected to the input, and two separate one-layer MLPs for
encoding $\boldsymbol{\mu}_{\text{vae}}$ and
$\boldsymbol{\sigma}^{2}_{\text{vae}}$. Then, the encoder outputs by random
sampling the encoded distribution:

$\mathbf{\hat{y}}\sim\mathcal{N}(\boldsymbol{\mu}_{\text{vae}},\,\boldsymbol{\sigma}^{2}_{\text{vae}})$
(5)

The decoder $h_{\text{vae}}^{\mathit{decode}}$ of the VAE has two layers of
MLP. The first one is with ReLU and the second is with Sigmoid.

###  III-D Optimization

The guaranteed invertibility of INN means that we can do bi-directional
training by optimizing loss from both sides of the network. We train both the
forward and reverse paths with supervised losses. In each epoch, the forward
and reverse paths are both calculated and gradients are added together before
an optimizer step.

The VAE network is responsible for encoding and reconstructing the LiDAR
scans:

$\begin{split}\mathbf{\hat{y}}_{\text{vae}}&=h_{\text{vae}}^{\mathit{encode}}(\mathbf{y}_{\text{gt}}),\\\
\mathbf{y}_{\text{vae}}&=h_{\text{vae}}^{\mathit{decode}}(\mathbf{\hat{y}}_{\text{vae}}),\end{split}$
(6)

which is optimized for the commonly used ELBO loss:

$\mathcal{L}_{\text{vae}}=\|\mathbf{y}_{\text{gt}}-\mathbf{y}_{\text{vae}}\|_{1}+\lambda_{\text{KL}}\text{KL}(\mathcal{N}(\boldsymbol{\mu}_{\text{vae}},\boldsymbol{\sigma}^{2}_{\text{vae}}),\mathcal{N}(0,1)),$
(7)

where $\lambda_{\text{KL}}$ is a weight for the KL divergence term. The VAE is
trained together with the INN.

Each epoch of training the INN starts with evaluating the encoder of the VAE
with ground truth LiDAR scans $\mathbf{y}_{\text{gt}}$ to get the encoded
scans $\mathbf{\hat{y}}_{\text{vae}}$, and evaluating the decoder of the VAE
with the output of the encoder, as in (6). $\mathcal{L}_{\text{vae}}$ is
calculated as in (7). The next step is to evaluate the forward path of the INN
with $\mathbf{\hat{x}}_{\text{gt}}$ to get the forward output:

$[\mathbf{\hat{y}}_{\text{inn}},\mathbf{z}_{\text{inn}}]=h_{\text{inn}}^{\mathit{forward}}(\mathbf{\hat{x}}_{\text{gt}},\mathbf{\hat{c}}).$
(8)

We then evaluate the decoder of VAE again with output from the INN forward
path:

$\mathbf{y}_{\text{inn}}=h_{\text{vae}}^{\mathit{decode}}(\mathbf{\hat{y}}_{\text{inn}}),$
(9)

and calculate a loss on the LiDAR scan output:

$\mathcal{L}_{\mathbf{y}}=\|\mathbf{y}_{\text{gt}}-\mathbf{y}_{\text{inn}}\|_{1}.$
(10)

We also calculate a loss that matches the output of the INN forward path with
the output of the VAE encoder:

$\mathcal{L}_{\mathbf{\hat{y}}}=\|\mathbf{\hat{y}}_{\text{vae}}-\mathbf{\hat{y}}_{\text{inn}}\|_{1}.$
(11)

For the reverse path of the INN, we first evaluate with the encoded scan from
VAE encoder concatenated by the latent vector generated by the forward path:
[$\mathbf{\hat{y}}_{\text{vae}}$, $\mathbf{z}_{\text{inn}}$]. This produces a
predicted pose we call $\mathbf{\hat{x}}_{\text{inn},\text{0}}$:

$\mathbf{\hat{x}}_{\text{inn},\text{0}}=h_{\text{inn}}^{\mathit{reverse}}([\mathbf{\hat{y}}_{\text{vae}},\mathbf{z}_{\text{inn}}],\mathbf{\hat{c}}).$
(12)

We calculated a L1 loss between $\mathbf{\hat{x}}_{\text{inn},\text{0}}$ and
the ground truth:

$\mathcal{L}_{\mathbf{\hat{x}},0}=\|\mathbf{\hat{x}}_{\text{gt}}-\mathbf{\hat{x}}_{\text{inn},\text{0}}\|_{1}.$
(13)

Following [15], the intuition of this reverse evaluation is to link the
encoded scans plus the predicted latent vector to the single corresponding
pose in the ambiguous inverse problem.

To capture the full posterior, we then sample $m$ latent vectors
$\mathbf{z}\sim\mathcal{N}(0,\,1)$ and evaluate the reverse path using the
sampled latent vectors combined with $\mathbf{\hat{y}}_{\text{vae}}$.

$\mathbf{\hat{x}}_{\text{inn},i}=h_{\text{inn}}^{\mathit{reverse}}([\mathbf{\hat{y}}_{\text{vae}},\mathbf{z}_{i}],\mathbf{\hat{c}}),\text{for
}i=1\dots m.$ (14)

This generates $m$ poses and we select the minimum of the L1 losses as the
second part of the reverse loss:

$\mathcal{L}_{\mathbf{\hat{x}},i}=\min_{i=1\dots
m}\|\mathbf{\hat{x}}_{\text{gt}}-\mathbf{\hat{x}}_{\text{inn},i}\|_{1}.$ (15)

Overall, the training loss of the whole network is:

$\mathcal{L}_{\text{all}}=\mathcal{L}_{\text{vae}}+\mathcal{L}_{\mathbf{y}}+\lambda_{\mathbf{\hat{y}}}\mathcal{L}_{\mathbf{\hat{y}}}+\mathcal{L}_{\mathbf{\hat{x}},0}+\mathcal{L}_{\mathbf{\hat{x}},i},$
(16)

where $\lambda_{\mathbf{\hat{y}}}$ is the weight for
$\mathcal{L}_{\mathbf{\hat{y}}}$.

##  IV Experiments

###  IV-A 2D LiDAR Localization and real-world robot

TABLE I: Map Reconstruction and Localization Errors with 2D LiDAR | Race Track (Simulation) | Hallway (Real) | Outdoor (Real)
---|---|---|---
| Original Map
---
Reconstruction
Test Trajectory
|  |
|  $xy$(m) |  $\theta$(∘) |  $xy$(m) |  $\theta$(∘) |  $xy$(m) |  $\theta$(∘)
Online PF (1m/s) | $0.045\pm\mathbf{0.058}$ | $0.400\pm 0.512$ | $\mathbf{0.039}\pm\mathbf{0.066}$ | $\mathbf{0.482}\pm 0.808$ | $\mathbf{0.013}\pm\mathbf{0.018}$ | $\mathbf{0.358}\pm\mathbf{0.456}$
Local_INN (1m/s) | $0.050\pm 0.102$ | $0.201\pm 0.532$ | $0.196\pm 0.433$ | $0.528\pm\mathbf{0.792}$ | $0.034\pm 0.047$ | $0.924\pm 1.130$
↑ + EKF | $\mathbf{0.039}\pm 0.077$ | $0.182\pm 0.464$ | $0.093\pm 0.139$ | $0.536\pm 0.797$ | $0.034\pm 0.047$ | $0.917\pm 1.129$
↑ + TensorRT | $0.039\pm 0.076$ | $\mathbf{0.177}\pm\mathbf{0.443}$ | $0.104\pm 0.159$ | $0.547\pm 0.802$ | $0.033\pm 0.046$ | $0.930\pm 1.142$
Online PF (5m/s) | $0.139\pm 0.168$ | $1.463\pm 2.107$ | $\mathbf{0.071}\pm\mathbf{0.117}$ | $0.943\pm 1.738$ | $0.033\pm 0.047$ | $0.940\pm 1.371$
Local_INN+EKF (5m/s) | $\mathbf{0.034}\pm\mathbf{0.056}$ | $\mathbf{0.133}\pm\mathbf{0.284}$ | $0.100\pm 0.147$ | $\mathbf{0.565}\pm\mathbf{0.900}$ | $\mathbf{0.032}\pm\mathbf{0.046}$ | $\mathbf{0.915}\pm\mathbf{1.130}$

We first validate the proposed method of localization with three different 2D
LiDAR maps. The first map is a race track in simulation, and the second and
third maps are real-world indoor hallway and outdoor environments mapped using
the ROS SLAM toolbox with an F1TENTH racecar[38], which is a 1/10 scale
autonomous racing car equipped with a Hokuyo 30LX LiDAR and a NVIDIA Jetson
Xavier NX board. To collect training data, we uniformly sample $[x,y,\theta]$
on the drivable surface of each map, and use a 2D LiDAR simulator to find the
corresponding LiDAR ranges. This means the trained network will be able to
localize everywhere on the map. We collect 100k data pairs and train a
separate network for each map.

To test the localization performance, we localize a car robot following a test
trajectory in each environment and compare the inferred pose with the ground
truth. For the real maps, we train with simulated data but test using real
LiDAR data on the F1TENTH car driving in indoor and outdoor environments. We
approximate the ground truth poses using a particle filter with the full LiDAR
inputs and running it offline on a desktop with an infinite compute budget.
For a baseline, we configured a GPU-accelerated particle filter [39], so that
it can run around the same frequency as the Local_INN on the Jetson NX.

In these experiments, we use 270 points for each LiDAR scan $\mathbf{y}$
covering 270 degrees in front of the LiDAR, following the FoV of the Hokuyo
LiDARs. $\mathbf{\hat{y}}$ is set to have 54 dimensions and $\mathbf{z}$ to
have 6 dimensions. The encoder of the VAE has one layer of MLP before
regressing the $\boldsymbol{\mu}_{\text{vae}}$ and
$\boldsymbol{\sigma}^{2}_{\text{vae}}$ with separate MLP layers. The decoder
has two layers of MLP converting 54-dimension $\mathbf{\hat{y}_{\text{vae}}}$
back to 270 ranges points. The INN network has 6 coupling layers, each having
separate MLP layers for scale and translation coefficients. We trained the
network with batchsize of 500 and with a learning rate that starts from
$1\text{\times}{10}^{-3}$ and exponentially decays to
$5\text{\times}{10}^{-5}$ in 600 epochs.

The map reconstruction is qualitatively evaluated by calculating the forward
path with additionally random sampled test poses. The inferred LiDAR ranges
are then converted into the map frame and accumulated to produce an occupancy
map. The orange dots in table I are reconstructed maps. We can see the
reconstructed map largely overlaps with the real map with some losses in high
spatial-frequency information at hard corners. The red bar on the upper right
corner of each map is an indicator for 1 meter.

During the inference of the reverse path, we sample latent vector $\mathbf{z}$
and calculate a batch of inferred poses. We can use the covariance of each
batch as the confidence of the network. To demonstrate this, we use an
Extended Kalman Filter to fuse the network outputs with vehicle odometry. The
EKF uses a kinematic bicycle model as the motion model, and the pose output
and covariance from the INN as the observation model.

Table I presents localization absolute mean and RMS errors in each
environment. We see that not only the localization performance is comparable
to particle filter, but the error and RMS also do not increase with vehicle
speed. On the contrary, we see the error increase with the particle filter.
This is because Local_INN does not directly rely on the smoothness of the
state’s history, but only relies on the zoning provided by the previous state.
Table II compares the runtime of the Local_INN with the GPU-accelerated
particle filter we used. We are comparing the latency of Local_INN and
particle filter. Other latencies are not accounted for. With runtime
optimizations like TensorRT, Local_INN can output localization results with
much lower latency than particle filter with almost no decrease in
performance, which is crucial in latency-sensitive applications such as high-
speed racing [40].

TABLE II: Runtime Comparisons on NVIDIA Jetson NX Online PF |  $45$ Hz
---|---
Local_INN (Pytorch) |  $48$ Hz
Local_INN+TensorRT |  $\mathbf{270}$ Hz

###  IV-B 3D Open Space LiDAR Localization

We then extended our experiments to using 3D LiDAR data, for which we also
have three different environments: Town 10 in the CARLA simulator[41], KAIST
in Mulran dataset[42], and Columbia Park in Apollo dataset[22]. For CARLA, we
used the simulator to sample all drivable surfaces in the town. To fully train
the Local_INN, simulating a large amount of data from the map is preferred.
But for comparison with existing works, we just used provided data points for
Mulran and Apollo datasets. When testing the network, we report numbers from
in-session and out-session localization. For in-session results, in CARLA, we
have additionally sampled points; in Mulran and Apollo, we randomly picked and
set aside 20% of the dataset for testing. For out-session tests, the network
is tested with sequences that are captured at another date. We provide in-
session performances to show that the network is able to interpolate between
the training data.

We treat 3D LiDAR scans as range images for the 3D experiments. To correctly
reconstruct the out-of-range LiDAR points, we added a mask layer to the range
images, and an L2 loss on it. The structure and dimension of the network are
mostly unchanged for the 3D experiments. The only additions are 6 layers of 2D
convolution and transpose convolution layers to the encoder and decoder of the
VAE for the range images.

Figure 3: 3D Map Reconstruction for Mulran and CARLA. Orange boxed thumbnails
are original maps. Reconstructions are produced by evaluating the forward path
of the Local_INN with poses exterior to the training set.

The quality of the map reconstruction is again qualitatively examined as some
examples are shown in Fig. 3. Because we simulated many more data points from
the CARLA environment, we can see the reconstruction is very close to the
original point cloud.

Table III shows a comparison of RMS errors between our method and existing
works in localization experiments with 3D LiDARs. Due to the simplicity of our
3D setup, we are comparing to a method from Chen et al[19] that only uses
range images from 3D LiDARs, and a method from Yin et al[43] that also uses a
neural network with convolution layers to treat LiDAR information. We can see
that our results on par with the state-of-the-art.

TABLE III: Comparison of Localization RMS Errors with 3D LiDAR Methods ($xy$[m], $\theta$[∘]) | CARLA | Mulran | Apollo
---|---|---|---
Local_INN in-session | $0.27,0.12$ | $0.29,0.24$ | $0.50,0.26$
Local_INN out-session | $-,-$ | $1.41,1.00$ | $1.22,0.53$
Chen et al. | $0.48,3.87$ | $0.83,3.14$ | $0.57,3.40$
RaLL (Yin et al.) | $-,-$ | $1.27,1.50$ | $-,-$

###  IV-C Global Localization

Global localization is needed when a robot starts with an unknown pose or when
the robot encounters the kidnapping problem. MCL algorithms usually do global
localization by spreading the covariance all around the map and using
iterations of control inputs and measurements to decrease the covariance. For
Local_INN, the global localization process mainly involves simultaneously
tracking multiple assumptions of zoning on the map and a selection process to
narrow down the assumptions.

Algorithm 1 shows our global localization process. We track a set
$\mathcal{C}$ of $n$ conditional inputs, each with a weight $w_{i}$ for
$i=1\dots n$. The set $\mathcal{C}$ is initialized by randomly sample $N$
states in the state space $\mathcal{S}$. We set the total number of latent
vectors $\mathbf{z}$ sampled from normal distribution as $nM$ for a constant
$M$. Initially, every $\mathbf{c}_{i}\in\mathcal{C}$ has the same weight
$w_{i}$, so each one gets $M$ samples of $\mathbf{z}$.

Algorithm 1 Local_INN Global Localization

1:$n\leftarrow N,m_{i}\leftarrow M,w_{i}\leftarrow 1/M$ for $i=1\dots n$

2:$\mathcal{X^{\text{rand}}}\leftarrow$ random_sample($\mathcal{S},n$)

3:$\mathcal{C}_{0}\leftarrow$
convert_to_cond_inputs($\mathcal{X^{\text{rand}}}$)

4:while new LiDAR scan $\mathbf{y}_{t+1}$ coming do

5: for $i=1\dots n_{t}$ do

6: $\mathbf{x}_{t+1,i}\leftarrow$
Local_INN_reverse($\mathbf{y}_{t+1},\mathbf{c}_{i},m_{i}$)

7: $\mathcal{X}_{t+1}\text{.append(}\mathbf{x}_{t+1,i}\text{)}$

8: $\mathbf{y}_{\text{inn},i}\leftarrow$
Local_INN_forward($\mathbf{x}_{t+1,i},\mathbf{c}_{i}$)

9: $w_{i}\leftarrow 1/\|\mathbf{y}_{\text{inn},i}-\mathbf{y}_{t+1}\|_{1}$

10: end for

11: $\mathcal{C}_{t+1}\leftarrow$ convert_to_cond_inputs($\mathcal{X}_{t+1}$)

12: $n_{t+1}\leftarrow|\mathcal{C}_{t+1}|$

13: for $i=1\dots n_{t+1}$ do

14: $m_{i}\leftarrow\text{normalized(}w_{i}\text{)}n_{t+1}M$

15: end for

16:end while

When a new LiDAR scan arrives, for each $\mathbf{c}_{i}\in\mathcal{C}$, we
evaluate the reverse path of the Local_INN with $m_{i}$ samples of latent
vector $\mathbf{z}$. The output poses from Local_INN become the next
$\mathcal{C}$. We then update the weight $w_{i}$ for every $\mathbf{c}_{i}$
using the reciprocal of the scan error, calculated with the current sensor
measurement, and inferred LiDAR scan from evaluating the forward path of the
Local_INN. We favor the $\mathbf{c}_{i}\in$ $\mathcal{C}$ that have higher
weights by redistributing $\mathbf{z}$ samples based on the weights. Those
with higher weights will have more $\mathbf{z}$ samples, which in turn may
result in better pose estimations. It also should be noted that the size of
$\mathcal{C}$ will decrease as iterations go because repeated elements in
$\mathcal{C}$ are combined. Hence, we design a selection process to find the
best-fit candidate. Lastly, we record the accumulated weights for every
iteration and the $\mathbf{c}_{i}$ with the highest accumulated weights will
be the most likely zone that the robot exists in.

We test out the above algorithm with different environments. We define a
Converged as the correct pose having the highest weight and a Tracking as the
correct pose within the top 5 on the tracking list. Table IV presents the
percentage of Converged cases, Tracking cases, and the absolute mean errors if
the correct pose is picked or in tracking at the 10th iteration. The starting
poses are randomly picked and the rates are out of 2k tests in each
environment. We use the test trajectory for the 2D maps and out-of-session
test sets for the 3D maps. The result shows the neural network can quickly
identify correct poses with only 10 LiDAR scans. We can also see in the
Hallway map, that the convergence of the assumptions is slower, which is
expected in this highly symmetrical environment. As the algorithm keeps
iterating with new LiDAR data, it will eventually converge to the correct
pose.

TABLE IV: Global Localization Success Rates in Different Environments at Iteration 10 Map | Converged | Tracking |  $\Delta_{xy}$, $\Delta_{\theta}$
---|---|---|---
Race Track | 79.5% | 99.5% | 0.075, 0.274
Hallway | 66.4% | 91.1% | 0.258, 0.538
Outdoor | 98.5% | 100% | 0.049, 0.911
Mulran | 93.5% | 95.0% | 0.884, 0.454
Apollo | 82.5% | 83.0% | 1.569, 0.122

##  V Conclusion

In this paper, we present a normalizing flow-based framework to solve the
robot localization problem. The trained INN provides a bijective mapping
between map information and robot poses. While localizing, sampling the latent
space gives us a mean and covariance, which can be used as uncertainty
estimation for the fusing with other data sources. In our 2D experiments,
Local_INN surpasses particle filer by providing localization with errors as
low as 0.032 m and 0.915∘, while running 270Hz on an embedded platform. Such
low latency combined with the fact that its error does not significantly
increase with robot velocity makes it suitable for high-speed applications. We
also show that Local_INN has great potential in 3D LiDAR localization with
errors of 0.29 m, 0.24∘ in-session, and 1.41 m, 1.00∘ out-of-session on the
Mulran dataset. Moreover, with our global localization algorithm, Local_INN
has a convergence rate of 93.5% in the Mulran dataset at the 10th iteration.

## References

  * [1] F. Dellaert, D. Fox, W. Burgard, and S. Thrun, “Monte carlo localization for mobile robots,” in _Proceedings 1999 IEEE international conference on robotics and automation (Cat. No. 99CH36288C)_ , vol. 2. IEEE, 1999, pp. 1322–1328.
  * [2] S. Thrun, “Probabilistic robotics,” _Communications of the ACM_ , vol. 45, no. 3, pp. 52–57, 2002.
  * [3] V. Fox, J. Hightower, L. Liao, D. Schulz, and G. Borriello, “Bayesian filtering for location estimation,” _IEEE pervasive computing_ , vol. 2, no. 3, pp. 24–33, 2003.
  * [4] R. Dube, A. Cramariuc, D. Dugas, H. Sommer, M. Dymczyk, J. Nieto, R. Siegwart, and C. Cadena, “Segmap: Segment-based mapping and localization using data-driven descriptors,” _The International Journal of Robotics Research_ , vol. 39, no. 2-3, pp. 339–355, 2020.
  * [5] P.-E. Sarlin, C. Cadena, R. Siegwart, and M. Dymczyk, “From coarse to fine: Robust hierarchical localization at large scale,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 12 716–12 725.
  * [6] E. G. Tabak and E. Vanden-Eijnden, “Density estimation by dual ascent of the log-likelihood,” _Communications in Mathematical Sciences_ , vol. 8, no. 1, pp. 217–233, 2010.
  * [7] L. Dinh, J. Sohl-Dickstein, and S. Bengio, “Density estimation using real nvp,” _arXiv preprint arXiv:1605.08803_ , 2016.
  * [8] D. P. Kingma and P. Dhariwal, “Glow: Generative flow with invertible 1x1 convolutions,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [9] G. Papamakarios, E. T. Nalisnick, D. J. Rezende, S. Mohamed, and B. Lakshminarayanan, “Normalizing flows for probabilistic modeling and inference.” _J. Mach. Learn. Res._ , vol. 22, no. 57, pp. 1–64, 2021.
  * [10] L. Ardizzone, J. Kruse, S. Wirkert, D. Rahner, E. W. Pellegrini, R. S. Klessen, L. Maier-Hein, C. Rother, and U. Köthe, “Analyzing inverse problems with invertible neural networks,” _arXiv preprint arXiv:1808.04730_ , 2018.
  * [11] L. Ardizzone, C. Lüth, J. Kruse, C. Rother, and U. Köthe, “Guided image generation with conditional invertible neural networks,” _arXiv preprint arXiv:1907.02392_ , 2019.
  * [12] T. J. Adler, L. Ardizzone, A. Vemuri, L. Ayala, J. Gröhl, T. Kirchner, S. Wirkert, J. Kruse, C. Rother, U. Köthe _et al._ , “Uncertainty-aware performance assessment of optical imaging modalities with invertible neural networks,” _International journal of computer assisted radiology and surgery_ , vol. 14, no. 6, pp. 997–1007, 2019.
  * [13] M. Xiao, S. Zheng, C. Liu, Y. Wang, D. He, G. Ke, J. Bian, Z. Lin, and T.-Y. Liu, “Invertible image rescaling,” in _European Conference on Computer Vision_. Springer, 2020, pp. 126–144.
  * [14] R. Zhao, T. Liu, J. Xiao, D. P. Lun, and K.-M. Lam, “Invertible image decolorization,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 6081–6095, 2021.
  * [15] T. Wehrbein, M. Rudolph, B. Rosenhahn, and B. Wandt, “Probabilistic monocular 3d human pose estimation with normalizing flows,” in _International Conference on Computer Vision (ICCV)_ , Oct. 2021.
  * [16] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” _arXiv preprint arXiv:1312.6114_ , 2013.
  * [17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [18] C. Zhang, M. H. Ang, and D. Rus, “Robust lidar localization for autonomous driving in rain,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 3409–3415.
  * [19] X. Chen, I. Vizzo, T. Läbe, J. Behley, and C. Stachniss, “Range Image-based LiDAR Localization for Autonomous Vehicles,” in _Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA)_, 2021.
  * [20] X. Chen, T. Läbe, L. Nardi, J. Behley, and C. Stachniss, “Learning an overlap-based observation model for 3d lidar localization,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 4602–4608.
  * [21] I. A. Barsan, S. Wang, A. Pokrovsky, and R. Urtasun, “Learning to localize using a lidar intensity map,” _arXiv preprint arXiv:2012.10902_ , 2020.
  * [22] W. Lu, Y. Zhou, G. Wan, S. Hou, and S. Song, “L3-net: Towards learning based lidar localization for autonomous driving,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 6389–6398.
  * [23] R. Clark, S. Wang, A. Markham, N. Trigoni, and H. Wen, “Vidloc: A deep spatio-temporal model for 6-dof video-clip relocalization,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 6856–6864.
  * [24] M. A. Uy and G. H. Lee, “Pointnetvlad: Deep point cloud based retrieval for large-scale place recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4470–4479.
  * [25] Y. Cho, G. Kim, S. Lee, and J.-H. Ryu, “Openstreetmap-based lidar global localization in urban environment without a prior lidar map,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 4999–5006, 2022.
  * [26] L. Sun, D. Adolfsson, M. Magnusson, H. Andreasson, I. Posner, and T. Duckett, “Localising faster: Efficient and precise lidar-based robot localisation in large-scale environments,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 4386–4392.
  * [27] M. Cai, C. Shen, and I. Reid, “A hybrid probabilistic model for camera relocalization,” 2019.
  * [28] A. Kendall and R. Cipolla, “Modelling uncertainty in deep learning for camera relocalization,” in _2016 IEEE international conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 4762–4769.
  * [29] H. Deng, M. Bui, N. Navab, L. Guibas, S. Ilic, and T. Birdal, “Deep bingham networks: Dealing with uncertainty and ambiguity in pose estimation,” _International Journal of Computer Vision_ , pp. 1–28, 2022.
  * [30] X. Wei, I. A. Bârsan, S. Wang, J. Martinez, and R. Urtasun, “Learning to localize through compressed binary maps,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 10 316–10 324.
  * [31] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” _Communications of the ACM_ , vol. 65, no. 1, pp. 99–106, 2021\.
  * [32] E. Sucar, S. Liu, J. Ortiz, and A. J. Davison, “imap: Implicit mapping and positioning in real-time,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 6229–6238.
  * [33] Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys, “Nice-slam: Neural implicit scalable encoding for slam,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 12 786–12 796.
  * [34] C. Durkan, A. Bekasov, I. Murray, and G. Papamakarios, “Neural spline flows,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [35] H. Wu, J. Köhler, and F. Noé, “Stochastic normalizing flows,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 5933–5944, 2020.
  * [36] Q. Zhang and Y. Chen, “Diffusion normalizing flow,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 16 280–16 291, 2021.
  * [37] C. Winkler, D. Worrall, E. Hoogeboom, and M. Welling, “Learning likelihoods with conditional normalizing flows,” _arXiv preprint arXiv:1912.00042_ , 2019\.
  * [38] M. O’Kelly, H. Zheng, D. Karthik, and R. Mangharam, “F1tenth: An open-source evaluation environment for continuous control and reinforcement learning,” in _Proceedings of the NeurIPS 2019 Competition and Demonstration Track_ , ser. Proceedings of Machine Learning Research, H. J. Escalante and R. Hadsell, Eds., vol. 123. PMLR, 08–14 Dec 2020, pp. 77–89.
  * [39] C. Walsh and S. Karaman, “Cddt: Fast approximate 2d ray casting for accelerated localization,” vol. abs/1705.01167, 2017. [Online]. Available: http://arxiv.org/abs/1705.01167
  * [40] J. Betz, H. Zheng, A. Liniger, U. Rosolia, P. Karle, M. Behl, V. Krovi, and R. Mangharam, “Autonomous Vehicles on the Edge: A Survey on Autonomous Vehicle Racing,” _IEEE Open Journal of Intelligent Transportation Systems_ , vol. 3, pp. 458–488, 2022.
  * [41] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” in _Conference on robot learning_. PMLR, 2017, pp. 1–16.
  * [42] G. Kim, Y. S. Park, Y. Cho, J. Jeong, and A. Kim, “Mulran: Multimodal range dataset for urban place recognition,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 6246–6253.
  * [43] H. Yin, R. Chen, Y. Wang, and R. Xiong, “Rall: end-to-end radar localization on lidar map using differentiable measurement model,” _IEEE Transactions on Intelligent Transportation Systems_ , 2021.