11institutetext:  Oak Ridge National Laboratory, Oak Ridge, TN, USA111This
manuscript has been authored by UT-Battelle, LLC, under contract DE-
AC05-00OR22725 with the US Department of Energy (DOE). The publisher
acknowledges the US government license to provide public access under the DOE
Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).
11email<EMAIL_ADDRESS>
# Unveiling User Behavior on Summit
Login Nodes as a User

Sean R. Wilkinson  0000-0002-1443-7479 Ketan Maheshwari  0000-0001-6852-5653
Rafael Ferreira da Silva  0000-0002-1720-0928

###### Abstract

We observe and analyze usage of the login nodes of the leadership class Summit
supercomputer from the perspective of an ordinary user—not a system
administrator—by periodically sampling user activities (job queues, running
processes, etc.) for two full years (2020–2021). Our findings unveil key usage
patterns that evidence misuse of the system, including gaming the policies,
impairing I/O performance, and using login nodes as a sole computing resource.
Our analysis highlights observed patterns for the execution of complex
computations (workflows), which are key for processing large-scale
applications.

###### Keywords:

High Performance Computing Workload Characterization User Behavior.

##  1 Introduction

HPC systems have been designed to address computing, storage, and networking
needs for complex, high-profile applications. Specifically, leadership class
supercomputers [16] meet the needs of applications that require high-speed
interconnects, low latency, high I/O throughput, and fast processing
capabilities (currently petascale, and soon exascale) [6]. Understanding the
performance of these systems and applications is a cornerstone for the design
and development of efficient, reliable, and scalable systems. To this end,
several works have focused on the development of system- and application-level
monitoring and profiling tools that can provide fine-grained characterizations
of systems’ and applications’ performance.

The current landscape of HPC systems performance research is mostly focused on
the system’s performance—which is utterly valuable for systems design [15, 9,
4]. However, the _user perception_ of the system is often disregarded, and
there is a common misconception that application execution performance is the
only consideration for user satisfaction. Although application performance is
one of the chief goals of HPC, there are several additional factors that
impact user experience. More specifically, before experiencing the
capabilities of the HPC nodes, users’ first interactions are with the _login
nodes_ , where users share resources like CPU, memory, storage, and network
bandwidth while performing basic tasks like compiling code, designing
experiments, and orchestrating services. The login nodes on an HPC system
represent a gateway to the system which is often overlooked when considering
the capabilities and performance of the overall system. We argue that the
experience on the login nodes may impact a user’s perception of and behavior
on the system, thus influencing whether and how the user continues to utilize
that system in future work.

In this paper, we attempt to identify long-term usage patterns by collecting
observational data on the login nodes from the Summit leadership class
supercomputer hosted at the Oak Ridge Leadership Computing Facility (OLCF) at
Oak Ridge National Laboratory (ORNL). Every hour for two years (2020–2021), we
have collected data about the login nodes’ performance with respect to CPU,
memory, and disk usage, and we also collected data about their activity with
respect to logged-in users, the programs they were running on the login nodes,
and the status of all user jobs. Figure 1 shows the number of users per login
node within this time window. In addition to examining traditional metrics
like system usage and distribution of jobs and users across the nodes, we also
seek to (i) highlight atypical usage and user misconducts, (ii) relate these
behaviors to potential performance issues, (iii) identify usage patterns of a
complex class of applications such as scientific workflows, and (iv) establish
relationships between users’ sessions length and system load.

Figure 1: Distribution of users over 5 Summit login nodes (Jan 2020–Dec 2021).
Gaps in the time series indicate outages or system downtime.

##  2 Characteristics of the Summit Login Node Data

Table 1 summarizes the main characteristics of the collected data. The dataset
represents activity from 1,967 unique users, who connected using 9,841 unique
IPs and submitted 1,783,867 jobs, of which 1,073,754 completed successfully
while 705,103 had a non-zero exit code. Figure 2 shows the distribution of
users’ geolocations, which were resolved through an IP geolocation tool [1].
For the sake of privacy, any user-specific data had been previously anonymized
and not retained.

Table 1: Characteristics of Summit login node data for a period of two years (Jan 2020–Dec 2021). Totals for “Unique Users” and “Unique IPs” do not sum additively due to Summit users whose use spanned both years. Additionally, the total number of jobs may not coincide with the sum of individual years because jobs may be carried over from one year to the next. Year | # Unique Users | # Unique IPs | # Jobs
---|---|---|---
completed | suspended | exited | total
2020 | 1,509 | 5,094 | 480,550 | 1,869 | 313,257 | 795,676
2021 | 1,514 | 5,467 | 668,580 | 3,264 | 410,493 | 1,082,337
Total | 1,967 | 9,841 | 1,073,754 | 5,010 | 705,103 | 1,783,867
Figure 2: Users’ geolocation distribution obtained with IP lookup ($\sim$93%
of total users).

_System Characteristics and Data Collection –_ Summit is equipped with 5 login
nodes [18]. Each login node runs Red Hat Enterprise Linux v8.2 and comes
equipped with two 3.8 GHz 16-core IBM POWER9 CPUs (4 threads per core), 512
GiB of DDR4-2667, 4 NVidia V100 GPUs each with 16 GiB of HBM2, and connection
to a 250 PB GPFS scratch filesystem. Users usually log into Summit via SSH to
the load-balanced summit.olcf.ornl.gov hostname, but they can optionally
connect to a specific login node. Data were collected hourly, starting January
1, 2020, on all five login nodes. A shell script ran in the user space as a
while loop within a Linux tmux session because user cronjobs are not allowed,
and it collected traditional system usage performance metrics as well as user
behavior (e.g., running processes and jobs). One caveat is that the hourly
sampling frequency may have failed to capture fine grained behavior, as many
things can happen between samples. Nevertheless, we believe that the large
volume of samples sufficiently captures most of the representative system and
user activity. More precisely, each sample collects the following data:

  * •

List of currently logged-in users using the w command;

  * •

CPU and memory usage using the top and ps commands (which also provides the
list of running processes), and statistics from meminfo and vmstat in the
/proc filesystem;

  * •

Status of users’ batch jobs via the bjobs command;

  * •

Disk usage statistics using the df -h command and disk throughput by measuring
the timespan for writing a 1GB data file to GPFS.

_Data Preparation –_ Real-world data may be incomplete, noisy, and
inconsistent, which can obscure useful patterns [20]. Data preparation
techniques cannot be fully automated; it is necessary to apply them with
knowledge of their effect on the data being prepared. We used our prior
knowledge about the execution of scientific applications on HPC to extract and
combine relevant information from each source of data. We have then pre-
processed the dataset by removing redundancies and missing data (e.g., due to
outages and system downtimes), sanitizing lists of programs and users for
long-running processes and jobs, and resolving IP addresses for filtering and
identifying individual users and their locations, among other things.

##  3 System Metrics

In this section, we examine overall characteristics and performance metrics
from Summit. The assessed set of metrics are restricted to an ordinary user’s
perspective of the system, as viewed from a login node. Although these metrics
are often reported and analyzed in-depth from the system’s perspective by
using system-wide monitoring and profiling tools, here we have used a subset
of these metrics to support our claims regarding user experience and behavior.

###  3.1 Users Access

Figure 3: Distribution of unique IPs across Summit login nodes (Jan 2020–Dec
2021).

Figure 1 shows the distribution of user sessions per login node. The average
percentage of user distribution is 21.9% ($\pm$9.2%), 20.7% ($\pm 9.2\%$),
17.9% ($\pm$7.8%), 20.2% ($\pm$8.8%), and 19.3% ($\pm$8.1%) for login nodes
1–5, respectively. Although this distribution is relatively balanced among
login nodes, by inspecting the distribution of unique IPs per login session
(Figure 3) we observe that there is an imbalance on the disposition of
individual users among the nodes. Specifically, the average percentage of
unique IPs distribution is 20.5% ($\pm$8.8%), 20.8% ($\pm$9.2%), 16.7%
($\pm$8.6%), 23.5% ($\pm$10.1%), and 18.5% ($\pm$8.1%) for login nodes 1–5,
respectively. This indicates that a subset of users may be (involuntarily)
benefitting from an increased number of concurrent login sessions; thus, their
perceived experience of the system may be more favorable when compared with
users who share resources with a larger number of individual users.

To evaluate the above claim, we examined CPU utilization and the number of
user processes per login node (Figures 4 and 5). Overall, CPU utilization is
relatively balanced among nodes (around 15% in average across nodes) with some
spikes on login nodes 2, 3, and 5. Unsurprisingly, the number of user
processes follows similar trends as for the distribution of unique IPs. Both
of these results corroborate the claim that a small subset of users have been
benefited from lower concurrency. More precisely, the balanced distribution of
CPU utilization on login nodes 3 and 5 indicates that this small set of users
consumes as many resources on these nodes as the larger set of users on the
other nodes.

Figure 4: CPU utilization on Summit login nodes (Jan 2020–Dec 2021). Figure 5:
User processes on Summit login nodes (Jan 2020–Dec 2021).

###  3.2 I/O Throughput

Every hour, we have measured the I/O throughput of Summit’s GPFS for writing a
1 GB randomly generated binary data file to a shared folder. Notice that we do
not aim to assess peak write speeds; instead our goal is to identify potential
low performance caused by user-related I/O operations within the login nodes.
Figure 6 shows the distribution of the number of user processes running per
login node in relation to the I/O throughput for writing a 1 GB file. Note
that the performance of the GPFS filesystem may also be affected by I/O
operations occurring on the compute nodes; thus a weak correlation is expected
with processes running on the login nodes. That said, we can observe that a
low performance is highly correlated with an increased number of user
processes running on the login nodes. Specifically, throughput values as low
as 42MB/s are reported when more than 3,000 user processes are running for
more than 3 consecutive hours. For the same set of datapoints, user processes
running on login5 run for more than 5 consecutive hours, which coincides with
the timespan in which the filesystem yields low performance (recall that
login5 has, on average, a reduced number of concurrent unique users).
Analogously, impaired performance (around 250 MB/s) is observed for a very
small group of users who run more than 2,500 processes on login3 for more than
7 hours.

Figure 6: I/O throughput of Summit’s GPFS for writing a 1 GB file in relation
to the number of user processes.

###  3.3 Computational Jobs

The fundamental purpose of leadership-class supercomputers is to improve
science by running the largest-scale computational jobs. It is expected that
user satisfaction is mostly dictated by the ability to execute batch jobs
successfully with good performance and without long waits in the queue. Figure
7-_top_ shows the percentage distribution of jobs based on their status. The
workload average jobs submitted, running, and completed, as shown by the LSF
scheduler, are 529 ($\pm$271), 81 ($\pm$24), and 90 ($\pm$69), respectively.
Given that the number of individual users (see Table 1) is orders of magnitude
higher than the average number of running jobs, the variation in the number of
running jobs seems relatively low.

Figure 7: Distribution of jobs’ statuses (_top_) and jobs’ sizes (_bottom_).
Each “violin” represents the distribution of jobs’ sizes in a given month in
terms of node-hours as a rotated kernel density plot on top of a box-plot that
shows the first and third quartiles of the distribution; the width of the
violin corresponds to the number of jobs, and the dots indicate outliers in
the tails.

Figure 7-_bottom_ shows the distribution of node-hours consumed per job.
Intriguingly, the shape of the distributions are alike across years and
months. More precisely, the average root mean square error (RMSE) is below 6
for every month comparison between the two years, with most jobs consuming
between 1,000 and 10,000 node-hours. This result suggests that jobs are mostly
submitted by a small set of users running similar, yet large, workloads.
Indeed, by examining the number of jobs submitted per user, we observe that
_only 29 users ( $\sim$1.4% of total number of users in the dataset),
submitted more than 50% of all Summit jobs over the measured period of time._
These jobs represent more than 82% of the total consumed node-hours in the
dataset (Figure 8). As expected, most individual jobs consume between 1,000
and 10,000 node-hours, which corroborates the findings asserted from Figure
7-_bottom_. Most users submitted a very small number of jobs, though they span
a wide range of node-hours consumption, with a few jobs consuming nearly all
available compute resources. We can also observe that specific users submitted
sets of individual jobs with a wide range of node-hours (e.g., from 96 up to
193,537), but also submitted more than 1,100 jobs with the same size (e.g.,
$\sim$4,300 node-hours).

Figure 8: Distribution of total node-hours consumed per user. (The dashed
vertical line delineates the total node-hours consumed by 29 users, which
represents more than 82% of the entire dataset.)

##  4 User Behavior

HPC performance metrics are traditionally associated with success metrics such
as high system utilization and large number of users and jobs, which correlate
to wide system adoption by the community and fulfillment of scientific goals.
Understanding and modeling user behavior in HPC environments is key to
exhibiting usage patterns that may help improve the design of the system,
relate performance bottlenecks to specific behaviors, and ascertain violations
of policies and best practices, among other things. Previous studies have
mainly focused on job characteristics (performance metrics as presented in
Section 3.3) and scheduling (queuing time, wall time, etc.) [13, 12, 19]. In
this section, we examine user behavior from the standpoint of (i) the average
user session length, (ii) misuses, and (iii) usage patterns of a complex class
of applications such as scientific workflows.

###  4.1 User Sessions

In this section, we investigate the length of user sessions in an attempt to
characterize user behavior by relating the time users spend logged into the
system with the number and size (in terms of number of nodes) of jobs
submitted. We define a _session_ as a time interval indicated by activity
which begins and ends with inactivity. We use batch job submission as the
indicator of activity, and for inactivity, we leverage _think time_ [7], which
quantifies the time between the completion of a job and the submission of the
next job by the same user. Thus, a session is the time period that complements
two subsequent think times for the same user. In this work, we assume that a
think time is characterized by an interval of more than 24 hours. We do not
consider weekends, holidays, or system downtimes or outages as think times.

We identified 27,789 sessions, the longest of which spans 123 days and runs 68
jobs over a maximum of 64 nodes. Most of the users (about 92%) established
more than one session, and most of the user sessions (about 84%) span less
than one day; also, more than 50% of these sessions request only 1 or 2 nodes
per job. Sessions with large-scale jobs that use nearly all of Summit’s
compute nodes span only a few hours, with only 3 spanning slightly more than
one day. This supports the idea that user experience on login nodes
significantly impacts user satisfaction, because users spend most of their
time testing and debugging while using the login nodes. Figure 9 shows the
distribution of user sessions’ lengths in relation to the total number of
nodes used by all jobs within a session.

Figure 9: Users’ session lengths (in days) in relation to the number of jobs
submitted and maximum number of nodes requested.

###  4.2 Misuse

Typically, HPC systems balance users across the set of login nodes to improve
the overall user experience and limit any potential performance impact due to
heavy user processes (see Section 3.1). To prevent low quality of service,
most HPC systems provide guidance and best practices for operations that
should not be performed on login nodes because they are shared resources. For
instance, it is discouraged to run long-term and/or heavy services (e.g.,
databases) on such nodes. In this section, we examine whether users run
processes that could harm the overall performance of these shared resources.
To this end, we mined the dataset for processes that did not represent
typical, system-related tasks, that consumed a substantial amount of resources
(CPU / GPU / memory), or that ran for a long period of time. We limit our
discussion in this section to two representative use cases: (i) execution of
tightly coupled applications using mpirun and mpiexec, and (ii) execution of
high-throughput applications.

_Tightly Coupled Applications –_ We have identified 1,172 uses of mpirun and
mpiexec by 74 users for running tightly coupled applications in the login
node. (Our filtering process removed mentions to compiling operations and
flags, environment variables, etc.) In further investigation, we noticed that
816 out of the 1,172 instances of mpirun and mpiexec spawned only a single
process for less than one hour – which suggests that those executions were
simple tests. Figure 10-_left_ shows execution times for the mpirun and
mpiexec instances, their associated CPU utilization, and the number of
processes spawned. The longest execution runs for 204 hours and spawns 16
processes, followed by a dozen of executions that run for about 100 hours.
There is also a cluster of instances that consume more than 90% of CPU for an
average of 12 hours, with two instances running for 47 and 49 hours each. A
detailed look at these instances unveiled that they use up to 4 cores from the
login nodes and up to 4 GB of RAM each, which could then considerably impact
the performance of sound processes (compilation, (de)compression, file
synchronization, etc.) from other users.

Figure 10: _Left:_ Execution of mpirun and mpiexec on login nodes (1,172
instances from 74 users). _Right:_ 56 executions of GROMACS (gmx_mpi) on login
nodes by two users. Each user runs instances of GROMACS on every login node,
which occupy the available GPUs for several hours.

Figure 10-_right_ shows a subset of the executions shown in Figure 10-_left_ ,
which corresponds to executions of GROMACS [17], a widely used molecular
dynamics package, on GPUs in the login nodes. Specifically, we highlight a use
case in which two users attempt to “game the system” by launching concurrent
executions of the GPU-enabled version of GROMACS (gmx_mpi), configured to
spawn one CPU process and as many GPU processes as available in the system. To
prevent such behaviors, Summit enforces limits on the login nodes to ensure
resource availability by leveraging the Linux kernel feature cgroups: each
user is limited to 16 hardware threads, 16 GB of memory, and 1 GPU; and after
4 hours of CPU-time all login sessions are limited to 0.5 hardware threads;
after 8 hours, the process is automatically killed. These limits are reset as
new login sessions are started. These two users consumed 50% of all GPU
resources across login nodes for about 84 consecutive hours, however, through
a synchronized process in which each of them re-initiated a session
periodically, so the limits would be reset. This behavior is not only
substantially harmful to other users by preventing a fair share of resources,
but also it conflicts with best practices of not running scientific
applications within login nodes.

_High-Throughput Applications –_ We have identified a substantial number of
executions of high-throughput applications on the login nodes. Here, we focus
on a subset of these executions that consumes more than 90% of CPU per
process, which comprises 8,014 instances executed by 549 users (27.9% of total
users). Figure 11-_left_ shows the distribution of user processes _vs._ their
length, in hours, that run user codes (i.e., scientific applications) on the
login nodes. As for the tightly-coupled applications above, we have filtered
out all instances related to sound processes (compilation, (de)compression,
file synchronization, etc.). Users ran a wide range of codes—495 unique
programs—in which $\sim$78% of them run for less than an hour; thus, we
consider them as execution tests. Some instances span 16 threads (cgroups
limit) and run up to 7 hours, while others (about 7% of the dataset) use more
than 8 threads and run between 3 and 8 hours. We then consider these instances
as misuse of the login nodes. Due to the limits imposed by Summit, we do not
observe any attempt to “game the system”; these processes are mostly evenly
distributed across login nodes, with a slightly higher number for login3 (405
instead of 330 on average for the other login nodes).

Figure 11: _Left:_ Execution of user processes (high-throughput applications)
on login nodes. _Right:_ Execution of Python programs on login nodes.
_Bottom:_ Execution of Python programs by users that have never submitted a
batch job to the system. (Note that 1600% CPU utilization means that a process
comprising 16 threads consumed 100% CPU utilization each.)

In spite of the large variation of user programs, we have identified that
4,478 instances (from 329 users) are running Python programs (Figure
11-_right_). These instances represent 72.6% of the instances shown in Figure
11-_left_ , which run for more than an hour. This result indicates that some
users may tend to use these login nodes as additional computing resources, or
even as their sole computing node. In order to assert the latter, we attempted
to isolate the list of users that ran any of these codes without ever
submitting a single job to the batch queue. Astonishingly, we identified 41
users that fall into this category, which comprises 1,012 instances, i.e.,
12.6% of the original dataset (Figure 11-_bottom_). Although running user
programs on login nodes as an extension of computing resources is against best
practices, using a leadership-class HPC system for running user-based codes
uniquely on login nodes must be prevented—strict policies and processes should
then be defined to impede similar misuse of resources.

While the cgroups mechanism protects the overall login node resources, it
falls short in “low key” and “gaming the system” misuse, as shown above.
Several measures may be taken to mitigate these issues. For example, the data
collected by this work can be used to identify misusers, either to educate
them about best practices or perhaps to introduce punitive actions. We will
not conjecture about potential new policies here, however.

###  4.3 Scientific Workflows

Scientific workflows are used almost universally across scientific domains for
solving complex and large-scale computing and data analysis problems. The
importance of workflows is highlighted by the fact that they have underpinned
some of the most significant discoveries of the past few decades [3]. Many of
these workflows have significant demands for computation, storage, and
communcation, and thus they have been increasingly executed on large-scale
computer systems [14]. In this section, we seek to identify how and to what
extent workflows have been used on Summit. Typically, workflow systems run a
coordinator process that manages workflow tasks’ dependencies, launches jobs
to the batch queue as their dependencies are satisfied, monitors their jobs’
execution, and performs data movement operations on behalf of the user. Table
2 shows the total number of processes run by workflow systems in Summit login
nodes. In total, 71 users utilized workflow technologies for automating the
execution of their scientific applications. These processes often refer to
agents that manage the workflow execution and they can take several formats:
from single orchestration components (e.g., Swift/T) to the management of
ensembles (e.g., RADICAL/EnTK). The former leverages batch jobs for defining
workflows within a parallel, tightly coupled application (thus the lower
number of processes), while the latter manages sets of tasks as high-
throughput applications, i.e. the so-called _pilot jobs_ [2].

Table 2: Total number of workflow management systems’ processes observed across Summit login nodes (Jan 2020–Dec 2021). | parsl | swift/t | pegasus | fireworks | dask | maestro | cylc | dagman | snakemake | radical
---|---|---|---|---|---|---|---|---|---|---
processes | 3,807 | 88 | 5,399 | 319 | 40,875 | 2,225 | 106 | 4 | 15,797 | 2,113,192
users | 7 | 3 | 3 | 5 | 27 | 6 | 1 | 1 | 5 | 13

Figure 12 shows the cumulative number of workflow-related processes across
Summit login nodes for our dataset. Overall, workflow technology adoption has
gradually increased throughout these past two years. A notable growth in
workflow usage is observed in the first two quarters of 2020, which coincides
with research conducted to understand the COVID-19 pandemic through the use of
HPC. Specifically, this research leveraged the RADICAL/EnTK framework for
investigating spike dynamics in a variety of complex environments, including
within a complete SARS-CoV-2 viral envelope simulation [5]. This research has
been awarded the 2020 ACM Gordon Bell Special Prize for High Performance
Computing-Based COVID-19 Research.

Figure 12: Cumulative number of workflow management systems’ processes
observed across Summit login nodes (Jan 2020–Dec 2021), shown with square root
scale.

##  5 Related Work

Analyzing and characterizing HPC workloads is a common practice for measuring
system and application performance metrics and thus identifying potential
bottlenecks and atypical behaviors [7]. For example, the National Energy
Research Scientific Computing Center (NERSC) has profiled and characterized
three generations of their supercomputing systems [11]. In these studies, HPC
benchmarks are used to obtain performance measurements, which are then used
for the procurement process of machines. Similarly, a characterization of the
workload of Tianhe-1A at the National Supercomputer Center in Tianjin presents
equivalent system-level metrics [8]. In [10], a characterization of a parallel
filesystem unveils I/O bottlenecks for different classes of applications.
Conversely, our analyses in this paper target users’ experience and behavior
on login nodes—the interface to HPC systems.

In [13], user behavior is studied with regards to think time, the time between
the completion of a job and the submission of the next job by the same user.
Although this work leverages this same concept for defining user sessions, the
study conducted in [13] attempted to understand and characterize patterns of
job submissions. Our work, instead, seeks to understand user behavior on login
nodes and relate their actions to misuses of the system or performance issues.
To the best of our knowledge, this is the first work that conducts such a
study.

##  6 Conclusion and Future Work

We examined observation data from the login nodes of the leadership-class
Summit supercomputer at OLCF. We analyzed traditional system performance
metrics such as user access, I/O throughput, and job characteristics, as well
as user behavior regarding session lengths, misuse of login nodes, and how
users have leveraged workflows to perform complex, distributed computing. Our
findings identified key usage patterns that we believe will shed light on the
usage of login nodes on contemporary clusters and supercomputers. As immediate
future work, we will continue to collect this observation data for the rest of
the life of Summit, and we will start data collection for the upcoming
exascale Frontier supercomputer at OLCF. We also intend to analyze the data
further into other dimensions, including resource usage balancing and
correlation of external events (e.g., conference deadlines, call for proposals
deadlines, etc.), as well as the impact of the COVID-19 pandemic on the user
behavior.

_Acknowledgments._ This research used resources of the Oak Ridge Leadership
Computing Facility at the Oak Ridge National Laboratory, which is supported by
the Office of Science of the U.S. Department of Energy under Contract No. DE-
AC05-00OR22725. We acknowledge Suzanne Parete-Koon for early brainstorming of
some of the ideas presented here. We thank Scott Atchley, Bronson Messer, and
Sarp Oral for their thorough revision of this paper.

## References

  * [1] IP Geolocation API. https://www.abstractapi.com/ip-geolocation-api (2022)
  * [2] Ananthraj, V., De, K., Jha, S., Klimentov, A., Oleynik, D., Oral, S., Merzky, A., Mashinistov, R., Panitkin, S., Svirin, P., Turilli, M., Wells, J., Wilkinson, S.: Towards exascale computing for high energy physics: The atlas experience at ornl. In: 2018 IEEE 14th International Conference on e-Science (e-Science). pp. 341–342 (2018). https://doi.org/10.1109/eScience.2018.00086
  * [3] Badia Sala, R.M., Ayguadé Parra, E., Labarta Mancho, J.J.: Workflows for science: A challenge when facing the convergence of hpc and big data. Supercomputing frontiers and innovations 4(1) (2017). https://doi.org/10.14529/jsfi170102
  * [4] Bang, J., Kim, C., Wu, K., Sim, A., Byna, S., Kim, S., Eom, H.: Hpc workload characterization using feature selection and clustering. In: Proceedings of the 3rd International Workshop on Systems and Network Telemetry and Analytics (2020). https://doi.org/10.1145/3391812.3396270
  * [5] Casalino, L., Dommer, A.C., Gaieb, Z., Barros, E.P., Sztain, T., Ahn, S.H., Trifan, A., Brace, A., Bogetti, A.T., Clyde, A., et al.: Ai-driven multiscale simulations illuminate mechanisms of sars-cov-2 spike dynamics. The International Journal of High Performance Computing Applications (2021). https://doi.org/10.1177/10943420211006452
  * [6] Dongarra, J., Gottlieb, S., Kramer, W.T.: Race to exascale. Computing in Science & Engineering 21(1) (2019). https://doi.org/10.1109/MCSE.2018.2882574
  * [7] Feitelson, D.G.: Looking at data. In: 2008 IEEE International Symposium on Parallel and Distributed Processing (2008). https://doi.org/10.1109/IPDPS.2008.4536092
  * [8] Feng, J., Liu, G., Zhang, J., Zhang, Z., Yu, J., Zhang, Z.: Workload characterization and evolutionary analyses of tianhe-1a supercomputer. In: International Conference on Computational Science (2018). https://doi.org/10.1007/978-3-319-93698-7_44
  * [9] Liu, Z., Lewis, R., Kettimuthu, R., Harms, K., Carns, P., Rao, N., Foster, I., Papka, M.E.: Characterization and identification of hpc applications at leadership computing facility. In: Proceedings of the 34th ACM International Conference on Supercomputing (2020). https://doi.org/10.1145/3392717.3392774
  * [10] Lockwood, G.K., Snyder, S., Wang, T., Byna, S., Carns, P., Wright, N.J.: A year in the life of a parallel file system. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis. IEEE Press (2018). https://doi.org/10.1109/SC.2018.00077
  * [11] Nersc benchmarking and workload characterization. https://www.nersc.gov/research-and-development/benchmarking-and-workload-characterization (2021)
  * [12] Rodrigo, G.P., Östberg, P.O., Elmroth, E., Antypas, K., Gerber, R., Ramakrishnan, L.: Towards understanding hpc users and systems: a nersc case study. Journal of Parallel and Distributed Computing 111 (2018). https://doi.org/10.1016/j.jpdc.2017.09.002
  * [13] Schlagkamp, S., Ferreira da Silva, R., Deelman, E., Schwiegelshohn, U.: Understanding user behavior: from hpc to htc. Procedia Computer Science (2016). https://doi.org/10.1016/j.procs.2016.05.397, international Conference on Computational Science 2016, ICCS 2016
  * [14] Ferreira da Silva, R., Filgueira, R., Pietri, I., Jiang, M., Sakellariou, R., Deelman, E.: A characterization of workflow management systems for extreme-scale applications. Future Generation Computer Systems 75 (2017). https://doi.org/10.1016/j.future.2017.02.026
  * [15] Ferreira da Silva, R., Rynge, M., Juve, G., Sfiligoi, I., Deelman, E., Letts, J., Würthwein, F., Livny, M.: Characterizing a high throughput computing workload: The compact muon solenoid (CMS) experiment at LHC. Procedia Computer Science 51 (2015). https://doi.org/10.1016/j.procs.2015.05.190, international Conference On Computational Science, ICCS 2015 Computational Science at the Gates of Nature
  * [16] Top 500. https://www.top500.org (2021)
  * [17] Van Der Spoel, D., Lindahl, E., Hess, B., Groenhof, G., Mark, A.E., Berendsen, H.J.: Gromacs: fast, flexible, and free. Journal of computational chemistry 26(16) (2005). https://doi.org/GROMACS: fast, flexible, and free
  * [18] Vazhkudai, S.S., De Supinski, B.R., Bland, A.S., Geist, A., Sexton, J., Kahle, J., Zimmer, C.J., Atchley, S., Oral, S., Maxwell, D.E., et al.: The design, deployment, and evaluation of the coral pre-exascale systems. In: SC18: International Conference for High Performance Computing, Networking, Storage and Analysis. pp. 661–672. IEEE (2018)
  * [19] Wolter, N., McCracken, M.O., Snavely, A., Hochstein, L., Nakamura, T., Basili, V.: What’s working in hpc: Investigating hpc user behavior and productivity. CTWatch Quarterly 2(4A) (2006)
  * [20] Zhang, S., Zhang, C., Yang, Q.: Data preparation for data mining. Applied artificial intelligence 17(5-6) (2003). https://doi.org/10.1080/713827180


# A Quick Guide for the Iterated Extended Kalman Filter on Manifolds

Jianzhu Huai1, Xiang Gao2 1Jianzhu Huai is with the State Key Lab of
Surveying, Mapping, and Remote Sensing, Wuhan University2Xiang Gao was with
the Technical University of Munich.

###### Abstract

The extended Kalman filter (EKF) is a common state estimation method for
discrete nonlinear systems. It recursively executes the propagation step as
time goes by and the update step when a set of measurements arrives. In the
update step, the EKF linearizes the measurement function only once. In
contrast, the iterated EKF (IEKF) refines the state in the update step by
iteratively solving a least squares problem. The IEKF has been extended to
work with state variables on manifolds which have differentiable $\boxplus$
and $\boxminus$ operators, including Lie groups. However, existing
descriptions are often long, deep, and even with errors. This note provides a
quick reference for the IEKF on manifolds, using freshman-level matrix
calculus. Besides the bare-bone equations, we highlight the key steps in
deriving them.

##  I Problem and Assumptions

The problem is to estimate the state vector $\mathbf{x}$ and its covariance
$\mathbf{P}$ given its dynamic equation $\mathbf{f}()$ and the external
observations $\mathbf{z}$. We assume that the state vector $\mathbf{x}$ is an
element of a differentiable manifold with a boxplus operator $\boxplus$ and a
boxminus operator $\boxminus$, defined as

$\displaystyle\mathbf{x}\boxplus\boldsymbol{\delta}=\mathbf{y},$ (1)
$\displaystyle\mathbf{y}\boxminus\mathbf{x}=\boldsymbol{\delta}.$ (2)

where $\mathbf{y}$ is another manifold element close to $\mathbf{x}$ with a
distance vector $\boldsymbol{\delta}$. For an Euclidean space, $\boxplus$ is
the common plus(+) and $\boxminus$ is the common minus (-). For elements of
SO3, example definitions of $\boxplus$ and $\boxminus$ are given in (23) and
(28). In the state estimation problem, we usually denote the estimate of
$\mathbf{x}$ by $\widehat{\mathbf{x}}$, which is usually defined relative to
$\mathbf{x}$ by

$\displaystyle\widehat{\mathbf{x}}\boxplus\boldsymbol{\delta}=\mathbf{x},$ (3)
$\displaystyle\mathbf{x}\boxminus\widehat{\mathbf{x}}=\boldsymbol{\delta}.$
(4)

An alternative definition is
$\mathbf{x}\boxplus\boldsymbol{\delta}=\widehat{\mathbf{x}}$, _e.g_., in [1].
This definition leads to a less-intuitive Kalman update step and is uncommonly
used.

For the extended Kalman filter (EKF), we use the dynamic equation discretized
at time steps $t_{1},\cdots,t_{k}$. Time steps are usually chosen according to
the stamp of the measurement vector $\mathbf{z}$, _e.g_.,
$\mathbf{z}_{k}=\mathbf{z}(t_{k})$ is acquired at time $t_{k}$. Let’s consider
one EKF time step from $t_{k-1}$ until $t_{k}$, which involves propagation
with the discrete dynamic equation and update with the measurement
$\mathbf{z}_{k}$. The discrete dynamic equation from $t_{k-1}$ to $t_{k}$ is
given by

$\mathbf{x}_{k}=\mathbf{f}(\mathbf{x}_{k-1},\mathbf{u}_{k-1},\mathbf{w}_{k-1}),$
(5)

where $\mathbf{w}_{k-1}$ is the discrete noise at $t_{k-1}$ with a Gaussian
distribution $N(\mathbf{0},\mathbf{Q}_{k-1})$. The observation equation at
$t_{k}$ is

$\mathbf{z}_{k}=\mathbf{h}(\mathbf{x}_{k})+\mathbf{n}_{k},$ (6)

where $\mathbf{n}_{k}$ is the discrete noise at $t_{k}$ with a Gaussian
distribution $N(\mathbf{0},\mathbf{R}_{k})$. For simplicity, we assume that
the observation noise is additive and $\mathbf{z}_{k}$ is in a vector space.
Indeed, many state estimation problems meet this assumption.

##  II Iterated Extended Kalman Filter

Now we describe the estimation process of the iterated EKF (IEKF). Denote the
a-posteriori state vector estimate at $t_{k-1}$ by $\mathbf{x}_{k-1}^{+}$ and
its covariance by $\mathbf{P}_{k-1}^{+}$. The propagation step propagates the
state estimate to $t_{k}$ by

$\displaystyle\mathbf{x}_{k}^{-}$
$\displaystyle=\mathbf{f}(\mathbf{x}_{k-1}^{+},\mathbf{u}_{k-1},\mathbf{0}),$
(7) $\displaystyle\mathbf{P}_{k}^{-}$
$\displaystyle=\mathbf{F}_{k-1}\mathbf{P}_{k-1}^{+}\mathbf{F}_{k-1}^{\mkern-1.5mu\mathsf{T}}+\mathbf{G}_{k-1}\mathbf{Q}_{k-1}\mathbf{G}_{k-1}^{\mkern-1.5mu\mathsf{T}},$
(8)

where the propagation Jacobians are

$\displaystyle\mathbf{F}_{k-1}$
$\displaystyle=\lim_{\boldsymbol{\epsilon}\rightarrow\mathbf{0}}\frac{\mathbf{f}(\mathbf{x}_{k-1}^{+}\boxplus\boldsymbol{\epsilon},\mathbf{u},\mathbf{w})\boxminus\mathbf{f}(\mathbf{x}_{k-1}^{+},\mathbf{u},\mathbf{w})}{\boldsymbol{\epsilon}}$
(9) $\displaystyle\mathbf{G}_{k-1}$
$\displaystyle=\frac{\partial\mathbf{f}(\mathbf{x}_{k-1}^{+},\mathbf{u},\mathbf{w})}{\partial\mathbf{w}}\Big{|}_{\mathbf{w}=\mathbf{0}}$
(10)

With the measurement $\mathbf{z}_{k}$, the EKF solves for the state vector
$\mathbf{x}_{k}^{+}$ that minimizes the weighted square sum of the deviation
from the predicted state vector $\mathbf{x}_{k}^{-}$ and the innovation term
$\mathbf{r}_{k}=\mathbf{z}_{k}-\mathbf{h}(\mathbf{x}_{k}^{+})$, _i.e_.,

$\begin{split}\min_{\mathbf{x}_{k}^{+}}&(\mathbf{x}_{k}^{+}\boxminus\mathbf{x}_{k}^{-})^{\mkern-1.5mu\mathsf{T}}{\mathbf{P}_{k}^{-}}^{-1}(\mathbf{x}_{k}^{+}\boxminus\mathbf{x}_{k}^{-})+\\\
&(\mathbf{z}_{k}-\mathbf{h}(\mathbf{x}_{k}^{+}))^{\mkern-1.5mu\mathsf{T}}\mathbf{R}_{k}^{-1}(\mathbf{z}_{k}-\mathbf{h}(\mathbf{x}_{k}^{+})).\end{split}$
(11)

In contrast, the IEKF iteratively updates the state vector starting from
$\mathbf{x}_{k,0}^{+}=\mathbf{x}_{k}^{-}$. In iteration $j$, it solves for an
increment $\boldsymbol{\delta}_{k,j}$ to the current state vector estimate
$\mathbf{x}_{k,j}^{+}$ that minimizes the weighted square sum of the deviation
from the prediction $\mathbf{x}_{k}^{-}$ and the innovation
$\mathbf{r}_{k,j}=\mathbf{z}_{k}-\mathbf{h}(\mathbf{x}_{k,j}^{+})$,

$\begin{split}\min_{\boldsymbol{\delta}_{k,j}}&\left\lVert\mathbf{x}_{k,j}^{+}\boxminus\mathbf{x}_{k}^{-}+\mathbf{J}_{k,j}\boldsymbol{\delta}_{k,j}\right\rVert_{{\mathbf{P}_{k}^{-}}^{-1}}+\\\
&\left\lVert\mathbf{z}_{k}-\mathbf{h}(\mathbf{x}_{k,j}^{+})-\mathbf{H}_{k,j}\boldsymbol{\delta}_{k,j}\right\rVert_{\mathbf{R}_{k}^{-1}}\end{split}$
(12)

where the Jacobian matrices are defined as

$\displaystyle\mathbf{H}_{k,j}$
$\displaystyle=\lim_{\boldsymbol{\epsilon}\rightarrow\mathbf{0}}\frac{\mathbf{h}(\mathbf{x}_{k,j}^{+}\boxplus\boldsymbol{\epsilon})-\mathbf{h}(\mathbf{x}_{k,j}^{+})}{\boldsymbol{\epsilon}}$
(13) $\displaystyle\mathbf{J}_{k,j}$
$\displaystyle=\lim_{\boldsymbol{\epsilon}\rightarrow\mathbf{0}}\frac{(\mathbf{x}_{k,j}^{+}\boxplus\boldsymbol{\epsilon}\boxminus\mathbf{x}_{k}^{-})-(\mathbf{x}_{k,j}^{+}\boxminus\mathbf{x}_{k}^{-})}{\boldsymbol{\epsilon}}$
(14)

$\mathbf{J}_{k,j}$ is a square identity matrix if $\mathbf{x}$ is in a vector
space. Otherwise, it can be well approximated by the square identity matrix.
Setting the derivative of (12) to zero, we can get the solution for
$\boldsymbol{\delta}_{k,j}$, which are used for the iterative update111Thanks
to Yarong Luo from WHU for checking the update equations.,

$\displaystyle\mathbf{S}_{k,j}$
$\displaystyle=\mathbf{H}_{k,j}\mathbf{L}_{k,j}\mathbf{P}_{k}^{-}\mathbf{L}_{k,j}^{\mkern-1.5mu\mathsf{T}}\mathbf{H}_{k,j}^{\mkern-1.5mu\mathsf{T}}+\mathbf{R}_{k},$
(15) $\displaystyle\mathbf{K}_{k,j}$
$\displaystyle=\mathbf{L}_{k,j}\mathbf{P}_{k}^{-}\mathbf{L}_{k,j}^{\mkern-1.5mu\mathsf{T}}\mathbf{H}_{k,j}^{\mkern-1.5mu\mathsf{T}}\mathbf{S}_{k,j}^{-1},$
(16)
$\displaystyle\begin{split}\boldsymbol{\delta}_{k,j}&=\mathbf{K}_{k,j}\big{[}\mathbf{H}_{k,j}\mathbf{L}_{k,j}(\mathbf{x}_{k,j}^{+}\boxminus\mathbf{x}_{k}^{-})+\\\
&\mathbf{z}_{k}-\mathbf{h}(\mathbf{x}_{k,j}^{+})\big{]}-\mathbf{L}_{k,j}(\mathbf{x}_{k,j}^{+}\boxminus\mathbf{x}_{k}^{-}),\end{split}$
(17) $\displaystyle\mathbf{x}_{k,j+1}^{+}$
$\displaystyle=\mathbf{x}_{k,j}^{+}\boxplus\boldsymbol{\delta}_{k,j}.$ (18)

where $\mathbf{L}_{k,j}=\mathbf{J}_{k,j}^{-1}$. The covariance will be updated
once at the end of all iterations,

$\mathbf{P}_{k}^{+}=(\mathbf{I}-\mathbf{K}_{k,n}\mathbf{H}_{k,n})\mathbf{L}_{k,n}\mathbf{P}_{k}^{-}\mathbf{L}_{k,j}^{\mkern-1.5mu\mathsf{T}}$
(19)

In summary, the IEKF algorithm is given in Algorithm 1.

Algorithm 1 The iterated extended Kalman filter algorithm

Input at time $t_{k-1}$: $\mathbf{x}_{k-1}^{+}$, $\mathbf{P}_{k-1}^{+}$,
$\mathbf{u}_{k-1}$, $\mathbf{z}_{k}$, max iterations $n$, termination
threshold $\epsilon$

Output at time $t_{k}$: $\mathbf{x}_{k}^{+}$, $\mathbf{P}_{k}^{+}$

Propagate state and covariance with (7)(8) to $\mathbf{x}_{k}^{-}$,
$\mathbf{P}_{k}^{-}$

$j\leftarrow 0$

$\mathbf{x}_{k,0}^{+}\leftarrow\mathbf{x}_{k}^{-}$

while $j\neq n$ do

update the state vector estimate with (15)-(18)

$j\leftarrow j+1$

if $\left\lVert\boldsymbol{\delta}_{k,j}\right\rVert<\epsilon$ then

break

end if

end while

$\mathbf{x}_{k}^{+}=\mathbf{x}_{k,n}^{+}$

update the covariance with (19)

When the dimension of observations is high, the inversion in (16) can be
intensive. There are two approaches to deal with this issue. One is to perform
QR decomposition of $\mathbf{H}_{k,j}$ as done in MSCKF [2, (4.61)],

$\displaystyle\mathbf{H}_{k,j}$
$\displaystyle=\begin{bmatrix}\mathbf{Q}_{1}&\mathbf{Q}_{2}\end{bmatrix}\begin{bmatrix}\mathbf{T}_{H}\\\
\mathbf{0}\end{bmatrix},$ (20)
$\displaystyle\underbrace{\mathbf{Q}_{1}^{\mkern-1.5mu\mathsf{T}}(\mathbf{z}-\mathbf{h}(\mathbf{x}_{k,j}^{+}))}_{\mathbf{r}_{q}}$
$\displaystyle=\mathbf{T}_{H}\boldsymbol{\delta}_{k,j}+\underbrace{\mathbf{Q}_{1}^{\mkern-1.5mu\mathsf{T}}\mathbf{n}_{k}}_{\mathbf{n}_{q}},$
(21)

where $\mathbf{n}_{q}$ is a Gaussian noise,
$N(\mathbf{0},\mathbf{R}_{q}=\mathbf{Q}_{1}^{\mkern-1.5mu\mathsf{T}}\mathbf{R}_{k}\mathbf{Q}_{1})$.
Then, substituting $\mathbf{r}_{q}$, $\mathbf{T}_{H}$, and $\mathbf{R}_{q}$
for $\mathbf{r}_{k,j}$, $\mathbf{H}_{k,j}$, and $\mathbf{R}_{k}$, the IEKF
update proceeds as in (15)-(19). The other approach is to rewrite $\mathbf{K}$
as done in FAST-LIO [3], _i.e_.,

$\mathbf{K}_{k,j}=[\mathbf{H}_{k,j}^{\mkern-1.5mu\mathsf{T}}\mathbf{R}_{k}^{-1}\mathbf{H}_{k,j}+(\mathbf{L}_{k,j}\mathbf{P}_{k}^{-}\mathbf{L}_{k,j}^{\mkern-1.5mu\mathsf{T}})^{-1}]^{-1}\mathbf{H}_{k,j}^{\mkern-1.5mu\mathsf{T}}\mathbf{R}_{k}^{-1}.$
(22)

The expression is equivalent to (16), but needs an extra covariance inversion.

##  III Example Manifold SO3

As an example, we compute $\mathbf{F}_{k-1}$ and $\mathbf{J}_{k,j}$ when
$\mathbf{x}$ is an SO3 element. Denote the orientation of $\\{B\\}$ frame
relative to the $\\{W\\}$ frame by $\mathbf{R}_{WB}$, and its estimate by
$\widehat{\mathbf{R}}_{WB}$. If we define $\boxplus$ and $\boxminus$ like

$\displaystyle\widehat{\mathbf{R}}_{WB}\boxplus\delta\theta_{WB}$
$\displaystyle=\widehat{\mathbf{R}}_{WB}\mathrm{Exp}(\delta\theta_{WB})=\mathbf{R}_{WB},$
(23) $\displaystyle\mathbf{R}_{WB}\boxminus\widehat{\mathbf{R}}_{WB}$
$\displaystyle=\mathrm{Log}(\widehat{\mathbf{R}}_{WB}^{\mkern-1.5mu\mathsf{T}}\mathbf{R}_{WB})=\delta\theta_{WB},$
(24)

as done in FAST-LIO [3], then $\mathbf{F}_{k-1}$ and $\mathbf{J}_{k,j}$ are

$\displaystyle\begin{split}\mathbf{F}_{k-1}&=\lim_{\boldsymbol{\epsilon}\rightarrow\mathbf{0}}\frac{1}{\boldsymbol{\epsilon}}\mathrm{Log}\Big{[}(\mathbf{R}_{WB_{k-1}}\mathrm{Exp}(\int_{t_{k-1}}^{t_{k}}\boldsymbol{\omega}_{WB}^{B}dt))^{\mkern-1.5mu\mathsf{T}}\cdot\\\
&\mathbf{R}_{WB_{k-1}}\mathrm{Exp}(\boldsymbol{\epsilon})\mathrm{Exp}(\int_{t_{k-1}}^{t_{k}}\boldsymbol{\omega}_{WB}^{B}dt)\Big{]}\\\
&=\mathrm{Exp}(-\int_{t_{k-1}}^{t_{k}}\boldsymbol{\omega}_{WB}^{B}dt),\end{split}$
(25)
$\displaystyle\begin{split}\mathbf{J}_{k,j}&=\lim_{\boldsymbol{\epsilon}\rightarrow\mathbf{0}}\frac{\mathrm{Log}({\mathbf{x}_{k}^{-}}^{\mkern-1.5mu\mathsf{T}}\mathbf{x}_{k,j}^{+}\mathrm{Exp}(\boldsymbol{\epsilon}))-\mathrm{Log}({\mathbf{x}_{k}^{-}}^{\mkern-1.5mu\mathsf{T}}\mathbf{x}_{k,j}^{+})}{\boldsymbol{\epsilon}}\\\
&=\mathbf{J}_{r}^{-1}(\delta\phi),\end{split}$ (26)
$\displaystyle\delta\phi_{j}$
$\displaystyle=\mathbf{x}_{k,j}^{+}\boxminus\mathbf{x}_{k}^{-},$ (27)

where we use the angular velocity $\boldsymbol{\omega}_{WB}^{B}$ relationship
$\mathbf{R}_{WB_{k}}=\mathbf{R}_{WB_{k-1}}\mathrm{Exp}(\int_{t_{k-1}}^{t_{k}}\boldsymbol{\omega}_{WB}^{B}dt)$,
and $\mathbf{x}$ denotes the rotation matrix $\mathbf{R}_{WB}$ for clarity.
The above results agree with FAST-LIO equations.

Otherwise, if we define $\boxplus$ and $\boxminus$ like

$\displaystyle\widehat{\mathbf{R}}_{WB}\boxplus\delta\theta_{WB}$
$\displaystyle=\mathrm{Exp}(\delta\theta_{WB})\widehat{\mathbf{R}}_{WB}=\mathbf{R}_{WB}$
(28) $\displaystyle\mathbf{R}_{WB}\boxminus\widehat{\mathbf{R}}_{WB}$
$\displaystyle=\mathrm{Log}(\mathbf{R}_{WB}\widehat{\mathbf{R}}_{WB}^{\mkern-1.5mu\mathsf{T}})$
(29)

as in [4], then $\mathbf{F}_{k-1}$ and $\mathbf{J}_{k,j}$ will be

$\displaystyle\mathbf{F}_{k-1}$ $\displaystyle=\mathbf{I}_{3},$ (30)
$\displaystyle\begin{split}\mathbf{J}_{k,j}&=\lim_{\boldsymbol{\epsilon}\rightarrow\mathbf{0}}\frac{\mathrm{Log}(\mathrm{Exp}(\boldsymbol{\epsilon})\mathbf{x}_{k,j}^{+}{\mathbf{x}_{k}^{-}}^{\mkern-1.5mu\mathsf{T}})-\mathrm{Log}(\mathbf{x}_{k,j}^{+}{\mathbf{x}_{k}^{-}}^{\mkern-1.5mu\mathsf{T}})}{\boldsymbol{\epsilon}}\\\
&=\mathbf{J}_{l}^{-1}(\delta\phi),\end{split}$ (31)
$\displaystyle\delta\phi_{j}$
$\displaystyle=\mathbf{x}_{k,j}^{+}\boxminus\mathbf{x}_{k}^{-}.$ (32)

##  IV Related Work

The IEKF for the vector space had been discussed in _e.g_., [5]. Later, it was
extended to the state vector on differentiable manifolds, _e.g_., [6].
However, the definition of the boxplus operator Jacobian $\mathbf{L}$ [6,
(49)] is vague, and $\mathbf{L}$ was erroneously transposed in [6, (50-52)].
In fact, $\mathbf{L}$ relates the multiplicative increment on the manifold to
the additive increment in the tangent space (see also (35)), _i.e_.,

$\mathbf{L}(\phi)=\lim_{\boldsymbol{\epsilon}\rightarrow\mathbf{0}}\frac{\mathrm{Exp}(\phi+\boldsymbol{\epsilon})\boxminus\mathrm{Exp}(\phi)}{\boldsymbol{\epsilon}}$
(33)

where $\mathrm{Exp}$ is the exponential map [7] for the differentiable
manifold. For vectors, $\mathrm{Exp}(\mathbf{x})=\mathbf{x}$. For SO3
elements,
$\mathrm{Exp}(\phi)=\mathbf{I}_{3}+\sum_{k=1}^{\infty}\frac{1}{k!}\lfloor\phi\rfloor_{\times}^{k}$.
Authors of FAST-LIO [8] also formulate the IEKF on manifolds, but at great
length with many new notations. The lack of a quick-start reference for the
IEKF on manifolds motivates this note. Note that X. Gao’s recent book “SLAM
technology in autonomous driving and robotics” also presents a simplified IEKF
for lidar-inertial odometry in chapter 8.3.

Here are a few helpful equations for the above derivations.

$\frac{\partial(\mathbf{a}+\mathbf{Bx})^{\mkern-1.5mu\mathsf{T}}\mathbf{P}^{-1}(\mathbf{a}+\mathbf{Bx})}{\partial\mathbf{x}}=2(\mathbf{a+Bx})^{\mkern-1.5mu\mathsf{T}}\mathbf{P}^{-1}\mathbf{B}$
(34)

Handy Jacobians for SO3 elements are copied from [9],

$\displaystyle\mathrm{Exp}(\phi+\delta\phi)\approx\mathrm{Exp}(\phi)\mathrm{Exp}(\mathbf{J}_{r}(\phi)\delta\phi)$
(35)
$\displaystyle\mathbf{J}_{r}(\phi)=\mathbf{I}_{3}-\frac{1-\cos\theta}{\theta^{2}}\lfloor\phi\rfloor_{\times}+\frac{\theta-\sin{\theta}}{\theta^{3}}\lfloor\phi\rfloor_{\times}^{2}$
(36)
$\displaystyle\mathrm{Exp}(\phi+\delta\phi)\approx\mathrm{Exp}(\mathbf{J}_{l}(\phi)\delta\phi)\mathrm{Exp}(\phi)$
(37)
$\displaystyle\mathbf{J}_{r}^{-1}(\phi)=\mathbf{I}_{3}+\frac{1}{2}\lfloor\phi\rfloor_{\times}+\left(\frac{1}{\theta^{2}}-\frac{1+\cos\theta}{2\theta\sin\theta}\right)\lfloor\phi\rfloor_{\times}^{2}$
(38)
$\displaystyle\mathbf{J}_{r}^{\mkern-1.5mu\mathsf{T}}(\phi)=\mathbf{J}_{l}(\phi)=\mathbf{J}_{r}(-\phi)$
(39)
$\displaystyle\mathbf{J}_{r}^{\mkern-1.5mu\mathsf{-T}}(\phi)=\mathbf{J}_{l}^{-1}(\phi)=\mathbf{J}_{r}^{-1}(-\phi)$
(40)

## References

  * [1] C. Jekeli, _Inertial Navigation Systems with Geodetic Applications_. de Gruyter. [Online]. Available: https://www.degruyter.com/view/books/9783110800234/9783110800234/9783110800234.xml
  * [2] J. Huai, “Collaborative SLAM with crowdsourced data.” [Online]. Available: https://etd.ohiolink.edu/pg˙10?0::NO:10:P10˙ACCESSION˙NUM:osu1483669256597152
  * [3] W. Xu and F. Zhang, “FAST-LIO: A fast, robust LiDAR-inertial odometry package by tightly-coupled iterated Kalman filter,” vol. 6, no. 2, pp. 3317–3324. [Online]. Available: https://ieeexplore.ieee.org/document/9372856/
  * [4] J. Huai, Y. Lin, Y. Zhuang, C. Toth, and D. Chen, “Observability analysis and keyframe-based filtering for visual inertial odometry with full self-calibration,” vol. 38, no. 5, pp. 3219–3237. [Online]. Available: http://arxiv.org/abs/2201.04989
  * [5] J. Havlík and O. Straka, “Performance evaluation of iterated extended Kalman filter with variable step-length,” vol. 659, no. 1, p. 012022, publisher: IOP Publishing. [Online]. Available: https://dx.doi.org/10.1088/1742-6596/659/1/012022
  * [6] M. Bloesch, M. Burri, S. Omari, M. Hutter, and R. Siegwart, “Iterated extended Kalman filter based visual-inertial odometry using direct photometric feedback,” vol. 36, no. 10, pp. 1053–1072. [Online]. Available: https://doi.org/10.1177/0278364917728574
  * [7] Manfredo P. do Carmo, _Differential geometry of curves and surfaces_ , 2nd ed. Dover Publications, Inc.
  * [8] D. He, W. Xu, and F. Zhang, “Kalman filters on differentiable manifolds.” [Online]. Available: http://arxiv.org/abs/2102.03804
  * [9] M. R. Hartley, “Contact-aided state estimation on Lie groups for legged robot mapping and control,”<EMAIL_ADDRESS>[Online]. Available: https://deepblue.lib.umich.edu/handle/2027.42/151657