# CLOi-Mapper: Consistent, Lightweight, Robust, and Incremental Mapper With
Embedded Systems for Commercial Robot Services

DongKi Noh1,2, Member, IEEE, Hyungtae Lim1, Member, IEEE, Gyuho Eoh3, Member,
IEEE, Duckyu Choi1, Jeongsik Choi2, Hyunjun Lim1, SeungMin Baek2, and Hyun
Myung1∗, Senior Member, IEEE ∗Corresponding author: Hyun Myung1DongKi Noh,
Hyungtae Lim, Duckyu Choi, Hyunjun Lim, and Hyun Myung are with School of
Electrical Engineering at KAIST (Korea Advanced Institute of Science and
Technology), Daejeon, 34141, Republic of Korea. {dongki.noh, shapelim, duckyu,
tp02134<EMAIL_ADDRESS>Noh, Jeongsik Choi, and SeungMin Baek are
with the Advanced Robotics Lab., CTO Division, LG Electronics, Seoul, 06772,
Republic of Korea. {dongki.noh, jeongs.choi<EMAIL_ADDRESS>Eoh
is with Department of Mechatronics Engineering, Tech University of Korea,
Siheung-si, Gyeonggi-do, 15073, Republic of Korea<EMAIL_ADDRESS>supplementary on our study are available at https://github.com/Multiplanet-
Robot.

###### Abstract

In commercial autonomous service robots with several form factors,
simultaneous localization and mapping (SLAM) is an essential technology for
providing proper services such as cleaning and guidance. Such robots require
SLAM algorithms suitable for specific applications and environments. Hence,
several SLAM frameworks have been proposed to address various requirements in
the past decade. However, we have encountered challenges in implementing
recent innovative frameworks when handling service robots with low-end
processors and insufficient sensor data, such as low-resolution 2D LiDAR
sensors. Specifically, regarding commercial robots, consistent performance in
different hardware configurations and environments is more crucial than the
performance dedicated to specific sensors or environments. Therefore, we
propose a) a multi-stage approach for global pose estimation in embedded
systems; b) a graph generation method with zero constraints for synchronized
sensors; and c) a robust and memory-efficient method for long-term pose-graph
optimization. As verified in in-home and large-scale indoor environments, the
proposed method yields consistent global pose estimation for services in
commercial fields. Furthermore, the proposed method exhibits potential
commercial viability considering the consistent performance verified via mass
production and long-term ($>$ 5 years) operation.

###### Index Terms:

Commercial robots, embedded systems, global-pose estimation, mapping, SLAM
framework.

##  I Introduction

Simultaneous localization and mapping (SLAM) has long been an essential
technique in robot navigation and autonomous platform fields. To apply SLAM
algorithms to real-world robots, various methods with exteroceptive sensors,
such as range, vision, and depth sensors, have been widely utilized [1, 2, 3,
4, 5, 6, 7, 8, 9, 10, 11, 12, 13]. Recently, there has been a growing need to
apply SLAM algorithms to service robots with diverse sensor configurations, as
illustrated in Fig. 1(a). Moreover, consistent real-time service has become
crucial at both the mapping and localization stages. Regarding the failure of
local consistency, as illustrated in Fig. 1(b), robot services might be
temporarily halted owing to significant position corrections. However, despite
encountering limitations in implementing innovative algorithms within embedded
systems, the proposed method has been successfully applied to real service
robots, thus enabling consistent pose estimation and mapping, as illustrated
in Fig. 1(c).

(a)

(b) ORB-SLAM [9] with our embedded system (R9)

(c) CLOi-Mapper with our embedded system (R9)

Figure 1: (a) Illustration of our SLAM applications arranged from bottom-left
to top-right. Airstar is the world’s first commercially available airport
guidance robot utilized in an international airport. Although each robot has a
different sensor configuration, computational performance, and operational
condition, they all utilize the proposed framework. (b)–(c) Visualization of
the global pose tracking output in an indoor space. Empirically, it was
demonstrated that CLOi-Mapper exhibits less delayed pose tracking performance
with the proposed embedded processor ($\leq$ 1 GFLOPS) (depicted in the red
dotted rectangle), thus enabling the proposed system to enhance trajectory
smoothness by mitigating duplicated trajectories and large corrections. In
addition, the colors of nodes represent nodes within the same local map.

Accordingly, this study introduces various relevant requirements for applying
SLAM algorithms to real-world service robots and presents approaches for
handling them. Considering commercial service robots, we collected several
requirements from researchers, developers, and marketers in service robot
communities, and summarized them into three: a) SLAM algorithms should be
extensible to include complementary sensors and ensure stable services in
different environments, b) locally consistent global poses and a globally
consistent map should be provided in both localization-only and SLAM stages,
and c) SLAM algorithms applied to service robots should exhibit stable
performance even in resource-restricted systems. In our case, we have a lack
of memory ($\leq$ 200 MB), low-frequency inertial measurement unit (IMU) data
transfer ($\leq$ 20 Hz) due to narrow bandwidth between an application
processor (AP) and a micro controller unit, and a low-cost processor ($\leq$ 1
GFLOPS).

Various methods can be employed to satisfy local consistency of pose
estimation in real-time, such as the LiDAR-inertial odometry (LIO) [2, 3, 4,
5], visual-inertial odometry (VIO) [7, 6, 8, 9], and sensor fusion methods [1,
14, 15, 16]. However, these frameworks require computational capabilities for
tracking poses across multiple sequential frames (e.g. bundle adjustment
(BA)), which could be challenging to implement in embedded systems.

Accordingly, this study focuses on extensible methods to fuse sensors
complementarily in different combinations; unlike existing studies, this
letter presents more practical methods for providing locally consistent global
poses in real-time for location-based services with mobile robots. In
particular, we leveraged the Kalman-filter-like tracking method based on the
Bayesian framework with a minimum number of nodes, and verified that the
proposed method can be applied to systems with a low-frequency IMU and
embedded processor. In addition, regarding globally consistent poses, we
employ an efficient graph generation method with temporal node integration and
node pruning to mitigate the degradation of measurements from low-cost sensors
and handle memory limitations in resource-constrained systems.

As verified in myriad in-home and large-scale indoor environments, the
proposed method, named as CLOi-Mapper, an abbreviation of Consistent,
Lightweight rObust incremental Mapper, has been applied to various
commercialized services. In particular, to the best of our knowledge, Airstar
is the first commercialized robot (since 2018) that has been operational as an
airport guidance robot in the real world for more than five years. Although
European researchers conducted a study on a guidance robot in the Spencer
project111http://www.spencer.eu at Schiphol International Airport,
Netherlands, it is yet to be commercialized. In conclusion, the framework
presented herein represents the culmination of a 5-year development process,
continuously improving the initial version established five years ago. The
contributions of this letter are as follows:

  * •

A novel mapping system is proposed, which comprises a simplified Bayesian
framework-based method for consistent pose estimation and efficient back-end
suitable for resource-constrained systems, e.g. low-end sensors and processors
with insufficient memory.

  * •

Moreover, the zero-constraints-based graph structure is presented, which can
handle different combinations of sensor types and form factors with minimal
tuning.

  * •

In particular, the generalization capability of the proposed method has been
verified via mass production.

The remainder of this paper is organized as follows. Section II introduces
SLAM algorithms in terms of commercialized service robots. Section III
comprehensively elucidates the proposed CLOi-Mapper. Subsequently, Sections IV
and V present the experimental setups and results for real-world robots,
respectively. Finally, a summary of our findings and conclusions are presented
in Section VI.

##  II Related Works

###  II-A SLAM Applications to Service Robots

According to World Robotics Reports [17], several sensor configurations
related to SLAM are utilized in service robots, e.g. cameras, RGB-D, LiDAR,
and radar sensors. In particular, SLAMTek is well-known for LiDAR-based
solutions employed in various service robots. Dyson and iRobot introduced a
visual feature-based SLAM with an omni-view and forward-view camera.
Artificial markers, such as QR codes, can be partially utilized to ensure that
indoor delivery services are reliable in complex environments.

Although the algorithms applied to robots introduced in the World Robotics
Reports have rarely been released with source codes, several approaches with
low-cost sensors and IMUs are expected to be suitable for embedded systems.
ORB-SLAM [10] series is one of the most popular visual feature-based SLAM
systems that can operate with monocular, stereo, and RGB-D cameras. Grid map-
based algorithms similar to Hector-SLAM [11] have been applied to various
commercialized cleaning robots with 2D LiDAR sensors. Recently, RGB-D sensors
and algorithms similar to RTAB-Map [12] have been utilized to generate dense
3D maps of indoor environments. In addition, Google Cartographer [13]
functions as a real-time SLAM system that enables robots to accurately map
large-scale environments with a 2D LiDAR sensor and an affordable embedded
system.

Until recently, low-cost hardware remains prevalent in mass-produced consumer
robotic products. In this context, this study focuses on algorithms capable of
managing diverse SLAM applications with combinations of low-cost hardwares and
sensors.

###  II-B Studies on Sensor Fusion and Real-Time Operation

Recently, Shan et al. [15] proposed efficient sensor fusion with multiple
sensors using the factor-graph. Moreover, Zhao et al. [16] presented an easier
and more flexible method to fuse multiple sensors robustly in a mini computer
with sub-factor graphs. However, these algorithms were not applied to
commercialized service robots with an embedded processor, but to a platform
with an Intel NUC onboard computer; hence, this approach may be unsuitable for
low-cost embedded systems. Moreover, even service robot systems with hardware
described in [16] inherently exhibit temporal latency characteristics in low-
priority tasks because they simultaneously handle several devices and tasks,
including SLAM. Consequently, recent studies [18, 19] indicate that
synchronization can be a challenging problem in real robotic systems.

##  III CLOi-Mapper

###  III-A System Overview

The proposed method addresses locally and globally consistent pose estimation
of ground mobile robots with resource-restricted systems targeted to a
commercial level. Based on the aforementioned requirements, the proposed CLOi-
Mapper primarily comprises three parts, as illustrated in Fig. 2.

First, the proposed method begins with extensible graph generation to handle
various sensor configurations without modifying the framework. At this stage,
frame nodes related to visual and LiDAR sensors are generated by the method
introduced in Section III.B. In particular, we propose a zero-constraint to
handle synchronized sensors (see Section III.B.3).

Second, given these frame nodes, this study aims to estimate locally and
globally consistent global poses. In general, batch optimization applied to a
graph-based SLAM can provide accurate state estimation results. Nevertheless,
it requires high computational complexity, which potentially leads to system
latency. Hence, we propose a Kalman-filter-like tracking method with an anchor
node to yield locally consistent poses for real-time robot operation, even in
the worst-case scenario (see Section III.C).

Finally, to improve memory efficiency and simplify graph optimization for
real-time operation, this study involves pruning and merging the graph
comprising locally consistent poses. Specifically, our proposed method uses
both geometric and information weights derived from a grid map to maintain a
geometrically uniform distribution of nodes (see Section III.D.2).

We adopted these modules and proposed a multi-stage method for globally
consistent pose estimation. These modules are explained in the following
subsections.

Figure 2: Block diagram presenting an overview of CLOi-Mapper. The dashed box
represents the novel and modified blocks proposed by this study. The
functionalities of each block are described as follows: (a) Front-end of our
framework with visual and LiDAR odometry, (b) semi-real-time global pose
tracker with a simplified graph, (c) global mapper with pose-graph
optimization, including a temporal node and the novel pruning method.

###  III-B Front-End in CLOi-Mapper

####  III-B1 Visual Odometry for Visual Frames

Visual odometry aims to estimate relative poses between keyframes using the
geometrical relationship between 2D points ($\mathbf{p}_{p}$) in pixel
coordinates and 3D ($\mathbf{p}_{w}$) points in world coordinates. Keyframes
can be selected among given frames with several rules, such as sufficient
3D/2D points for feature matching, redundancy check of a frame based on the
$\chi^{2}$-test, and distance traveled. We utilized a general projective
camera model [20] to determine the relationship between 2D and 3D points. The
3D points in the $(k-1)$-th frame can be reprojected into the 2D points in the
$k$-th frame by
$\mathbf{p}_{p,k}={\pi}(\mathbf{\mathcal{M}}_{k},\mathbf{p}_{w,k-1}),$ where
$\mathbf{p}_{p,k}$, $k$, $\mathbf{\mathcal{M}}_{k}$, and ${\pi}(\cdot)$ denote
a reprojected point on the pixel coordinate in the $k$-th frame, a keyframe
index, $k$-th projection matrix, and the reprojection function of 3D point,
respectively. By employing $\mathbf{p}_{p,k}$ and the reprojection error
estimated via sparse bundle adjustment (SBA) [21], the odometry between
sequential visual frames in a sliding window can be estimated.

Empirically, the number of reliable features may be limited owing to the
utilization of a cost-effective camera in our system. To address this
limitation and ensure consistent visual odometry (VO) performance in scenarios
with limited features, we suggest three methods: a) we projected 3D points
from the $k$-th and $(k+1)$-th keyframes onto the $(k-1)$-th keyframe as 2D
points. This projection allows the $(k-1)$-th keyframe to incorporate
supplementary measurements (2D/3D features) from the $k$-th and $(k+1)$-th
keyframes, thereby enhancing the accuracy of 3D points and poses, b) we merged
duplicated 3D points between consecutive keyframes to enhance accuracy by
averaging the corresponding points, and c) to eliminate less informative
features and improve the accuracy of features, we implemented a filter to
discard 3D points observed in only a few keyframes, typically less than four
in indoor scenarios, based on empirical observations.

####  III-B2 LiDAR Odometry for LiDAR Frames and 2D Mapping

We leveraged iterative closest point (ICP) [22] to estimate odometry between
sequential LiDAR frames and generate loop constraints. LiDAR frames are
generated independently at every timestamp. However, not all LiDAR frames can
be utilized owing to computational limitations in real robotic systems.
Therefore, LiDAR keyframes are selected based on two conditions: completing
the optimization processes of VO and LO, and being farther than a predefined
distance from the center of an adjacent LiDAR frame in the global coordinate.

Figure 3: Overview of a structurally extensible pose graph with cameras and a
2D LiDAR sensor. Visual and LiDAR odometry parts contain visual features and
local maps, respectively, and the two parts are interconnected via the frame
nodes set $\mathcal{V}$. Frame nodes comprise a combination of wheel odometry
with IMU, visual odometry, and LiDAR odometry.

####  III-B3 Extensible Graph Generation for Handling Various Sensor
Configurations

The pose-graph is defined as $\mathcal{G}$ = $(\mathcal{V},\mathcal{E})$,
where
$\mathcal{V}=\\{\mathcal{V}_{1},\mathcal{V}_{2},\dots,\mathcal{V}_{N}\\}$ is a
set of frame nodes $\mathcal{V}_{a}\leavevmode\nobreak\ (1\leq a\leq N)$ and
$\mathcal{E}\subset\mathcal{V}\crossproduct\mathcal{V}$ is a set of ordered
pairs $\\{e_{a,b}\\}_{a\neq b}$ between nodes $\mathcal{V}_{a}$ and
$\mathcal{V}_{b}\leavevmode\nobreak\ (1\leq b\leq N)$, where $N$ is the total
number of frame nodes. Using visual and LiDAR frames, the proposed method
indirectly connects nodes generated by distinct sensors by linking through the
intermediary frame nodes set denoted as $\mathcal{V}$, as illustrated in Fig.
3.

####  III-B4 Edge Generation in the Case of Synchronized Sensors

We propose a method for handling various sensor configurations without
altering the framework. Let us assume that cameras are already synchronized;
then, we should handle the sensor data of cameras simultaneously. Accordingly,
data stacking methods, such as image stitching, should be employed in such
cases, which leads to the framework’s modification. This could be a limitation
when managing various robots under a unified framework for efficiency. To
address this limitation, we assume that synchronized sensor data are obtained
sequentially, as illustrated in Fig. 4. Hence, we set a relative pose between
them to the identity matrix, which we called the zero-constraint. To employ
this approach with numerical stability, this study adopts a strategy where we
set a covariance value close to zero and employ an information matrix based on
a heuristic approach, which is associated with the determinant-based method
described in [23] as follows:

$I(\mathbf{x}_{i};\mathbf{x}_{j})\approx\begin{cases}\|\mathbf{\Sigma}_{i,j}^{-1}\|&\text{
if }\mathbf{\Sigma}_{i,j}^{-1}\leavevmode\nobreak\ \text{exists}\\\
|1/\operatorname{det}(\mathbf{\Sigma}_{i,j})|&\text{ if
}\mathbf{\Sigma}_{i,j}^{-1}\rightarrow\infty\end{cases},\vspace{-0.1cm}$ (1)

where $I$, $\boldsymbol{\Sigma}_{i,j}$, and $\operatorname{det}(\cdot)$ denote
mutual information for generating Chow-Liu trees [24, 25] in our back-end,
covariance between $i$-th and $j$-th nodes, and a determinant, respectively.

###  III-C Semi-Real-Time Global Pose Estimation in CLOi-Mapper

####  III-C1 Bayesian-Framework-Based Simplified Global Pose Estimation

Pose graph optimization (PGO) for global pose estimation could be
computationally intensive in embedded systems. Hence, PGO is executed with the
lowest priority in practical robotic systems equipped with an embedded
processor to mitigate total system delay. Consequently, the execution of PGO
for global pose estimation is inevitably delayed, leading to pose drift.

Figure 4: Illustration of the proposed zero-constraint concept. (a) Regarding
the robot with synchronized cameras, various studies employ image merging or
fusion of odometry estimated by each camera. (b) In contrast, we propose a
zero-constraint, setting it as a relative pose to enable the synchronization
effect without altering the framework.

Nevertheless, in order to track a desired trajectory consistently in time-
delayed systems, this letter proposes a simplified Bayesian approach for
optimization. Let the conditional probability density function (PDF) of the
latest global pose ($\mathbf{x}_{k}\in\mathbb{R}^{3}$) be
$p(\mathbf{x}_{k}|\mathbf{x}_{0:k-1},\mathbf{m},\mathbf{z}_{1:k})$, where
$\mathbf{x}_{0:k-1}$, $\mathbf{z}_{1:k}$, and $\mathbf{m}$ represent the set
of poses, the set of measurements, and the entire SLAM map, respectively. We
employ a previously optimized node as an anchor node, denoted as
$\mathbf{x}^{\ast}=\mathop{\mathrm{argmin}}_{\mathbf{x}}\text{SMD}(\mathbf{x}_{f},\mathbf{x})$,
where SMD and $\mathbf{x}_{f}$ represent the squared Mahalanobis distance and
last pose optimized by PGO, respectively, as illustrated in Fig. 5. The anchor
node enables the new robot pose to maintain consistency with the previously
optimized robot poses. By using a Bayesian approach and leveraging the local
adjustment module [26],
$p(\mathbf{x}_{k}|\mathbf{x}_{0:k-1},\mathbf{m},\mathbf{z}_{1:k})$ can be
approximated as

$\displaystyle
p(\mathbf{x}_{k}|\mathbf{x}_{k-1},\mathbf{x}_{f},\mathbf{x}^{\ast},\mathbf{x}_{0},\mathbf{m}_{s,k},\mathbf{z}_{1:k}),$
(2)

where $\mathbf{x}_{0}$, $\mathbf{x}_{k}$, and $\mathbf{m}_{s,k}$ denote a
reference pose, the pose of the $k$-th frame node $\mathcal{V}_{k}$, and a
SLAM submap that is a covisibility map around $\mathbf{x}_{k}$, respectively.
Compared with LIO and VIO [1, 2, 3, 4, 27], the proposed method exclusively
depends on the minimum number of nodes such as $\mathbf{x}_{0}$ and
$\mathbf{x}_{k}$, resulting in real-time pose estimation. As illustrated in
Fig. 5, the conditional PDF in (2) can be represented with odometry,
measurements, and loop constraints within a submap $\mathbf{m}_{s,k}$. By
using this representation, (2) is factorized as follows:

$\displaystyle\underbrace{p(\mathbf{x}_{k}|\mathbf{\bar{x}}_{k},\mathbf{z}_{k-1:k})}_{\text{Measurement}}\underbrace{p(\mathbf{\bar{x}}_{k}|\mathbf{x}_{f},\mathbf{o}_{k})}_{\text{Motion
model}}p(\mathbf{x}_{0})\boldsymbol{\alpha}_{k}\prod_{q}\rho(\boldsymbol{\beta}_{q,k})$
(3) $\displaystyle\text{s.t.}\leavevmode\nobreak\
\boldsymbol{\alpha}_{k}=\exp_{t}(\mathbf{\bar{x}}_{k},\mathbf{x}^{\ast})\leavevmode\nobreak\
\text{and}\leavevmode\nobreak\
\boldsymbol{\beta}_{q,k}=\exp_{t}(\mathbf{\bar{x}}_{k},\mathbf{x}_{q}),$

where $\mathbf{\bar{x}}_{k}$, $\mathbf{o}_{k}$, and ${\rho}$ denote a
predicted pose, wheel-based odometry fused by IMU’s angular rate, and a Huber
norm, respectively; pairs ($q,k$) belong to a set $\gamma$ of visual and LiDAR
loop constraints. In addition, $\exp_{t}(\mathbf{x}_{m},\mathbf{x}_{n})$ is
the Gaussian uncertainty model between a pose $\mathbf{x}_{m}$ and
$\mathbf{x}_{n}$, defined as
$\frac{1}{\eta}\exp{-\text{SMD}(\mathbf{x}_{m},\mathbf{x}_{n})/2}$, where
$\eta$ denotes a normalization factor. Finally, (2) can be simplified to an
optimization problem between the current node ($\mathbf{x}_{k}$) and the
reference node ($\mathbf{x}_{0}$), which is graphically illustrated in Fig. 5.
By manipulating (3) with a negative logarithm, the solution to (3) can be
obtained by minimizing the sum of residuals as follows:

$\displaystyle\mathbf{\hat{x}}_{k}=\mathop{\mathrm{argmin}}_{\mathbf{x}_{k}}\Big{\\{}\boldsymbol{\zeta}_{k}+\log(\boldsymbol{\alpha}_{k})+\sum_{(q,k)\in\mathbf{\gamma}}\log\big{(}{\rho}(\boldsymbol{\beta}_{q,k})\big{)}\Big{\\}}$
(4) $\displaystyle\text{s.t.}\leavevmode\nobreak\
\boldsymbol{\zeta}_{k}=\|h(\mathbf{x}_{k},\mathbf{x}_{0})-\mathbf{z}_{k,0}\|^{2}_{\mathbf{\Sigma}_{k,0}},$

where ${h}(\cdot)$, $\mathbf{z}_{k,0}$, and $\mathbf{\Sigma}_{k,0}$ represent
the function of odometry prediction between two sequential nodes, observations
pertaining to odometry, and combined covariance matrices [28], respectively.

Figure 5: Illustration of simplified pose graph generation between the
reference pose ($\mathbf{x}_{0}$) and current pose ($\mathbf{x}_{k}$) in
global coordinates. (L-R) By utilizing the metric embedding method [26], we
transform the original pose graph into the simplified pose graph with
previously optimized nodes, odometry $\mathbf{o}_{k}$, measurements
$\mathbf{z}_{k-1:k}$, and loop constraints in the SLAM submap
($\mathbf{m}_{s,k}$).

####  III-C2 Pose Prediction in the Worst Case

In the worst case, computational limitations could delay the aforementioned
global pose estimation, thereby increasing the bound of robot pose error and
leading to local consistency degradation. To address this situation, we assume
that the $(k+1)$-th transform matrix $\delta\mathbf{T}_{k+1}$ for correction
of wheel-based odometry is very similar to the previous one
$\delta\mathbf{T}_{k}$. Consequently, if the global pose estimation is not
conducted within the predefined time, we apply the previous correction
transform $\delta\mathbf{T}_{k}$ to adjust the odometry at the current step as
follows:

$\displaystyle\bar{\mathbf{T}}_{k+1}$
$\displaystyle=\delta\mathbf{T}_{k+1}\oplus\mathbf{T}_{\text{od},k+1}\approx\delta\mathbf{T}_{k}\oplus\mathbf{T}_{\text{od},k+1},$
(5)

where $\bar{\mathbf{T}}$, $\mathbf{T}_{\text{od}}$, and the operator $\oplus$
denote a predicted pose transform matrix, wheel-based odometry pose transform
matrix, and the composition, respectively. With the previous corrected pose
$\hat{\mathbf{T}}_{k}=\delta\mathbf{T}_{k}\oplus\mathbf{T}_{\text{od},k}$ and
odometry propagation
$\mathbf{T}_{\text{od},k+1}=\mathbf{T}_{\text{od},k}\oplus(\Delta\mathbf{T}_{\text{od},k+1})$,
we manipulate (5) as follows:

$\displaystyle\delta\mathbf{T}_{k}\oplus\mathbf{T}_{\text{od},k+1}$
$\displaystyle=(\hat{\mathbf{T}}_{k}\ominus\cancel{\mathbf{T}}_{\text{od},k})\oplus(\cancel{\mathbf{T}}_{\text{od},k}\oplus\Delta\mathbf{T}_{\text{od},k+1})$
(6)
$\displaystyle=\hat{\mathbf{T}}_{k}\oplus\Delta\mathbf{T}_{\text{od},k+1},$

where the operators $\ominus$ and $\Delta$ denote the inverse composition and
a relative pose, respectively.

Utilizing a predicted pose transform matrix in (6) as an alternative to the
corrected pose transform matrix during time-delayed situations might be a
viable approach to maintain the desired trajectory without significant
discontinuity.

###  III-D Graph Pruning-Based Robust Back-End in CLOi-Mapper

To enhance memory efficiency and graph optimization robustness in real robotic
systems, this letter proposes a back-end with a pruning and merging method.

####  III-D1 Graph Generation With a Temporal Node

Figure 6: Illustration of an example of the pose tracking failure (a dotted
orange line) of ORB-SLAM3, depicting in-home conditions featuring zigzag
motion and pure rotation. Black arrows and accompanying images represent the
robot’s trajectory and forward views captured by its onboard camera.

Owing to various harsh environments, pure rotation caused by zigzag motions,
and low-cost sensors, reliable keyframes may not be consistently generated in
practical SLAM applications. Typically, these unreliable keyframes, such as a
frame with few features, are prone to skipping due to their lower measurement
quality, potentially leading to pose tracking failures, as illustrated in Fig.
6. To mitigate such failures within sequential keyframes, we propose a graph
generation approach to best exploit unreliable keyframes (e.g. those with
insufficient features). We refrain from immediately skipping keyframes with
insufficient features. Instead, we subject them to optimization and
subsequently assess their quality to determine their usefulness in the node
pruning stage.

We define sets of unregistered (new) and optimized frame nodes as
$\mathcal{V}_{\text{new}}=\\{\mathcal{V}_{k-M+1},\mathcal{V}_{k-M+2},\dots,\mathcal{V}_{k}\\}$
and
$\mathcal{V}_{\text{opt}}=\\{\mathcal{V}_{0},\mathcal{V}_{1},\dots,\mathcal{V}_{k-M}\\}$
in the global map, respectively, where $M$ represents the window size of a
pose graph. Let $\mathcal{V}_{f}=\mathcal{V}_{k-M+1}$ be the first frame of a
set of unregistered frames as a temporal frame node. $\mathcal{V}_{f}$
contains a pose transform matrix ${}^{w}\mathbf{T}_{f}$ and covariance
$\mathbf{\Sigma}_{f}$. ${}^{w}\mathbf{T}_{f}$ can be obtained from a pose
$\textbf{x}_{f}$ as ${}^{w}\mathbf{T}_{f}$ $=$ $\Psi(\textbf{x}_{f})$, where
$\Psi$ denotes a function that converts a pose into a pose transform matrix.

Next, a relative pose transform matrix $\Delta\mathbf{T}$ between
$\mathcal{V}_{k-M}$ and $\mathcal{V}_{k-M+1}=\mathcal{V}_{f}$ is generated and
outliers of matching pairs are rejected at the same time. Suppose that the
mean and covariance of $\mathcal{V}_{k-M}$ are $\bar{\mathbf{T}}_{k-M}$ and
$\bar{\mathbf{\Sigma}}_{k-M}$, respectively. By using the propagation method,
the mean and covariance of $\mathcal{V}_{f}$ are calculated as follows:

$\displaystyle{}^{w}\mathbf{T}_{f}\leftarrow\bar{\mathbf{T}}_{k-M+1}=\bar{\mathbf{T}}_{k-M}\Delta\mathbf{T},$
(7)
$\displaystyle\mathbf{\Sigma}_{f}\leftarrow\bar{\mathbf{\Sigma}}_{k-M+1}=\mathbf{\bar{\Sigma}}_{0,k-M+1}+\mathbf{\bar{\Sigma}}_{k-M,k-M+1},$
(8)

where $\mathbf{\bar{\Sigma}}_{i,j}$ denotes the covariance between the $i$-th
and $j$-th nodes. In particular, $\mathbf{\bar{\Sigma}}_{1}$ is
$\mathbf{0}_{4\crossproduct 4}$. We set the temporal node $\mathcal{V}_{f}$
using (7) and (8), and then a temporal pose graph
$\mathcal{G}^{\prime}=(\mathcal{V}^{\prime},\mathcal{E}^{\prime})$ can be
generated with
$\mathcal{V}^{\prime}=\mathcal{V}_{\text{opt}}\leavevmode\nobreak\
\cup\leavevmode\nobreak\ {\mathcal{V}_{f}}$ and
$\mathcal{E}^{\prime}\subset\mathcal{V}^{\prime}\crossproduct\mathcal{V}^{\prime}$.
Next, the temporal pose graph $\mathcal{G}^{\prime}$ is incrementally pruned
and optimized using an efficient online pruning algorithm.

####  III-D2 Node Pruning and Optimization

Aiming to maintain a single node in a cell, optimize memory efficiency, and
expedite pose optimization, the algorithm selects the most informative node,
as illustrated in Fig. 7(a). By leveraging uncertainty representation [23] and
using a geometrical weight as shown in Fig. 7(b), we define the total weight
$w_{(c_{j},l)}$ as

$\displaystyle
w_{(c_{j},l)}=s\operatorname{Tr}\left(\mathbf{\Lambda}_{(c_{j},l)}\right)+(1-s)\sum_{i=1}^{n}(d_{i}^{x_{l},c_{j}})^{2},$
(9)

where $\mathbf{\Lambda}$, $s$, $n$, and $d_{i}^{x_{l},c_{j}}$ denote an
information matrix, a scale factor, the number of neighbor nodes, and distance
between the $l$-th node of the $j$-th cell and the $i$-th nodes in adjacent
cells, respectively. The subscript $(\cdot)_{(c_{j},l)}$ denotes the $l$-th
node in the $i$-th cell. The first and second terms in (9) represent the
information and geometric weights, respectively. This enables appropriate node
distributions for commercial service robots with different purposes: guide
robots that prioritize information-based navigation using points of interest
(PoI), and cleaning robots that require geometrical coverage through zigzag
navigation. Furthermore, to ensure real-time operability, we adopt a method
that utilizes information on edges instead of nodes, as in [25, 24].
Accordingly, we modified (9) as follows:

$\displaystyle
w_{(c_{j},l)}=s\sum_{k=1}^{m}\left(\Lambda_{(e_{k},c_{j},l)}\right)+(1-s)\sum_{i=1}^{n}(d_{i}^{x_{l},c_{j}})^{2},$
(10)

where $\Lambda_{(e_{k},c_{j},l)}$ and $m$ denote the information of the $k$-th
edge connected to the $l$-th node in the $j$-th cell and the number of edges,
respectively.

Figure 7: Example of node elimination and weights. (a) Node displacement
before and after redundant node elimination; (b) Geometric weights for
$\mathbf{x}_{1}$ and $\mathbf{x}_{2}$ in cell $c_{j}$ (e.g. the geometric
weight of $\mathbf{x}_{1}$ is $\sum d_{i}^{x_{1},c_{j}}$ $(i=1,\cdots,4)$).

Given the cell size of a grid map, the highest weighted pose $\mathbf{x}_{l}$
in the cell is selected by
$\mathbf{x}_{l}=\mathop{\mathrm{argmax}}_{\mathbf{x}_{l}}w_{(c_{j},l)}$ and
others are eliminated. Covariances of selected nodes are updated by compound
approximate transformations [28]. Accordingly, new edges are generated among
selected nodes. At this stage, the number of generated edges inevitably
increases up to
$(\sum_{k=1}^{{}_{{N}_{\text{edge}}-1}}k)-{{N}_{\text{edge}}}$, where
${N}_{\text{edge}}$ denotes the number of edges connected to the eliminated
node. To prune newly added edges, we employ data compression methods [25, 29],
and Chow-Liu trees [24] with mutual information (1). In addition, we evaluated
several minimum spanning tree (MST) methods such as Chazelle [30, 31], and
Kruskal and Prime [32] to determine a set of edges with a minimum joint
probability distribution. Empirically, in our study, an average elimination of
4 nodes and 6 edges per node was observed. Based on these metrics and the
computational complexity expressed as
$O(N_{\text{edge}}\log(N_{\text{node}}))$ [30, 31, 32], where
${N}_{\text{node}}$ is the number of nodes, we chose the Kruskal method, which
is known to be suitable for handling simple and sparse trees.

At the optimization stage, we leverage the max-mixture (MM) approach [33] with
the pruned graph defined as $\mathcal{G}^{\prime\prime}$ =
$(\mathcal{V}^{\prime\prime},\mathcal{E}^{\prime\prime})$ for a robust SLAM
back-end. Finally, by applying $\mathcal{V}^{\prime\prime}$ and
$\mathcal{E}^{\prime\prime}$ to the nonlinear least-squares problem, our cost
function is defined as

$\displaystyle\mathbf{\mathcal{X}}^{*}=\mathop{\mathrm{argmin}}_{\mathbf{\mathcal{X}}\in\mathcal{V}^{\prime\prime}}\Big{\\{}\sum_{i}\|{h}(\mathbf{x}_{s,i-1},\mathbf{x}_{s,i})-\mathbf{z}_{i-1,i}\|^{2}_{\mathbf{\Sigma}_{s,i}}+$
(11)
$\displaystyle\sum_{(j,k)\in\mathcal{E}^{\prime\prime}}{\rho}\big{(}\|{g}(\mathbf{x}_{s,j},\mathbf{x}_{s,k})-\boldsymbol{\omega}_{j,k}\|^{2}_{\mathbf{\Omega}_{s,j,k}}\big{)}\Big{\\}},$

where ${h}(\cdot)$ and ${g}(\cdot)$ are odometry predictions and loop
constraints, respectively; $\mathbf{z}_{i-1,i}$ and
$\boldsymbol{\omega}_{j,k}$ represent observations pertaining to odometry and
loop closure, respectively; $\mathbf{\Sigma}_{s,i}$ and
$\mathbf{\Omega}_{s,j,k}$ are combined covariance matrices [28]. Our multi-
stage approach, named as CLOi-Mapper, has been implemented in several embedded
systems and deployed in the commercial products.

##  IV Experiments

###  IV-A Experimental Setups and Environments

We conducted experiments in diverse environments, spanning in-home settings,
expansive indoor spaces, offices characterized by repetitive layouts, and
varied in-house scenarios across Korea, Germany, and the USA. Our primary
objectives included refining pose tracking to ensure local consistency,
optimized mapping for global consistency, and systematically testing the
applicability of our methodology in real-world settings.

First, we studied an airport guide robot named
AirStar222https://www.youtube.com/watch?v=ztdARyV-Njg at Incheon International
Airport, Republic of Korea. The study encompassed areas such as airside (A/S),
landside (L/S), and baggage claim (B/C) zones within Terminals 1 and 2.
AirStar is equipped with two monochromatic cameras with $704\crossproduct 478$
resolution and a 2D LiDAR sensor that provides measurements ranging from 30 cm
to 30 m with 1.0 deg angle resolution at a rate of 1Hz.

Second, we developed an office and home cleaning robot equipped with a single
camera ($1280\crossproduct 960$ resolution) and an embedded processor
(Cortex-A9333https://developer.arm.com/Processors/Cortex-A9). In particular,
the office cleaning robot is equipped with a 2D LiDAR sensor with a range of
10 m and 1.0 deg angle resolution.

Table I: Quantitative performance comparison of locally consistent trajectories. The sensor combinations C, 2C, L, and C+L denote a single camera, two cameras, one LiDAR, and one camera + LiDAR, respectively. Place | # of | Travelled | Average pose error ($t_{\text{rel}}$, m)
---|---|---|---
(Incheon, Korea) | nodes | distance (m) | C | 2C | L | C+L
Terminal 1 L/S | 10,732 | 6,608 | 0.48 | 0.48 | 0.07 | 0.06
Terminal 1 A/S | 11,938 | 3,449 | 1.09 | 1.28 | 0.1 | 0.1
Terminal 1 B/C | 9,992 | 2,701 | 0.57 | 0.56 | 0.11 | 0.09
Terminal 2 B/C | 4,275 | 1,306 | 1.14 | 1.05 | 0.28 | 0.28
Terminal 2 A/S | 8,913 | 6,335 | 83.2 | 1.62 | 0.78 | 0.22

###  IV-B Parameters and Hardware Settings

Empirically, we set $s=0.5$ in (10) and a submap (see Fig. 5) size of
$4.1\crossproduct 4.1\leavevmode\nobreak\ \text{m}$ for in-home environments,
while we set a submap size of $10\crossproduct 10\leavevmode\nobreak\
\text{m}$ for large-scale environments. The cameras were tilted up at an angle
of $45^{\circ}$ for the airport guidance robot and $60^{\circ}$ for the other
robots.

###  IV-C Error Metrics

As quantitative metrics, average relative pose errors ($t_{\text{rel}}$) are
defined as

  * •

$t_{\text{rel}}=\sqrt{\sum_{n=1}^{N}(\mathbf{t}_{n,\text{GT}}-{\hat{\mathbf{t}}}_{n})^{2}/N}$,

where $\mathbf{t}_{n,\text{GT}}$ and $\hat{\mathbf{t}}_{n}$ denote the ground
truth position vector and estimated position vector, respectively; the
subscripts $o$ and ${}_{\text{GT}}$ denote the origin pose and the value from
the ground truth, respectively; $N$ represents the number of selected nodes.
In addition, to quantify the degree of change from the original map, we
propose a metric called the Average Ratio of Pose Shift (ARPS), defined as

  * •

$\text{ARPS}=\\{(\sum_{i=1}^{M}\frac{|\mathbf{t}_{i,\text{ori}}-\mathbf{t}_{i,\text{pru}}|}{|\mathbf{t}_{i,\text{ori}}|})/M\\}\crossproduct
100$,

where $\mathbf{t}_{i,\text{ori}}$ denotes the original position vector of the
$i$-th node in the map, and $\mathbf{t}_{i,\text{pru}}$ denotes the position
vector of the same node after the pruning and optimization stage.

##  V Experimental Results and Discussion

We show the performance evaluation results in terms of our extensible
framework, locally consistent poses, and globally consistent poses.

###  V-A Performances Relative to Combinations of Sensors

Figure 8: Qualitative comparison of locally consistent trajectories for
various sensor combinations at various positions: (a) Terminal 1 L/S, (b)
Terminal 1 A/S (c) Terminal 1 B/C, (d) Terminal 2 B/C. Figure 9: Qualitative
comparison of locally consistent trajectories for various sensor combinations
at Terminal 2 A/S are presented in Table I. Our algorithm demonstrates
consistent pose estimation across multiple case studies involving limited
sensor combinations. The case studies are as follows: (a) The trajectory of
the 2-camera setup (red line) crosses the moving walkway. (b) Local pose
oscillations occur even in the 1 camera + 1 LiDAR scenario (purple line). (c)
The trajectory from a single camera’s data yields inaccurate results. (d) The
trajectory of the LiDAR sensor (green line) intersects the moving walkway.
Figure 10: Qualitative comparison of the global map generated by several SLAM
algorithms in a section of Terminal 2 A/S, Incheon International Airport. (a)
Our result demonstrates consistent performance. (b) Cartographer subtly
underperformed compared with our result, which is highlighted by red dotted
ellipses indicating wall alignment performance. (c)–(d) Hector-SLAM and RTAB-
MAP failed to map after passing the moving walkway. Furthermore, ORB-SLAM3
encountered failures in our case, potentially due to limitations in the number
of features ($\leq$ 20) and frequent directional changes.

First, performance changes in terms of locally consistent pose estimation were
analyzed. This letter presents the performance of locally consistent pose
estimation for various combinations of sensors across five locations, as
presented in Table I. Here, the robot has multiple sensors for localization,
such as two cameras and a LiDAR sensor. The combination of sensors could be
altered owing to various causes, such as a contaminated lens causing a blurry
image. Our approach has been validated in complex and expansive environments
to address these variations, as illustrated in Figs. 8 and 9. Meanwhile,
obtaining ground-truth poses is challenging, as it requires high-quality and
high-cost equipment. In order to assess the consistency of our algorithm, we
conducted trajectory comparisons between the estimations of limited sensor
combinations and the trajectory determined using all sensors on our robot
(depicted as the black line in Fig. 9).

Table II: Comparison of performance with widely used SLAM methods suitable for
each commercially affordable sensor.

(@Terminal 2 A/S) | Average pose error ($t_{\text{rel}}$, m)
---|---
Category | Algorithm | C | 2C | L | C+L
| Ours | $\triangle$ | 1.62 | 0.78 | 0.22
Vision | ORB-SLAM | ✕ | ✕ |
LiDAR | Cartographer |  | 0.83 |
Hector-SLAM | $\triangle$
Fusion | RTAB-MAP |  | ✕ | $\triangle$ | $\triangle$

$\triangle$ and ✕ denote map generation with pose errors exceeding 2 meters
and map generation failed, respectively.

Specifically, the result obtained from a single camera at Terminal 2 A/S (83.2
m) is inconsistent with the others, as depicted by the blue lines in Fig. 9.
This is caused by a significant amount of wrongly associated poses, which
could be due to similarity in captured images in repeatedly patterned
environments. Consequently, the proposed algorithm effectively achieved pose
consistency by seamlessly integrating multiple sensors within a large-scale
environment. Furthermore, the proposed algorithm’s generalizability in service
robots was comprehensively evaluated across diverse in-home environments in
three countries, namely, Korea, the USA, and Germany, with home cleaning
robots. Accordingly, the proposed algorithm was verified to estimate a current
pose in real-time. The performance of local consistency related to cleaning
robots in in-home environments has been proven in real-world cases (with
approximately 500,000+ units sold) by users. Therefore, we believe that the
evaluation is sufficient. This method was proven highly valuable for coverage
navigation, particularly in scenarios that require precise path following,
such as navigating a zigzag path, even during the mapping stage.

###  V-B Global Consistency Compared With Widely Used Methods Suitable for
Commercially Affordable Sensors

Regarding global consistency, compared with widely used SLAM algorithms in the
industry fields, the proposed algorithm can be more effective and applicable
within crowded and large-scale environments, as illustrated in Fig. 10.
Despite numerous attempts, even the widely used algorithms face challenges in
completing the entire map, often resulting in failure, as presented in Table
II. The results for each sensor using the proposed method were compared
separately to highlight its effectiveness. This issue appears to emerge from
the algorithm’s inability to satisfy the requirements related to parameter
settings (e.g. lack of features).

(a)

(b)

(c)

Figure 11: Qualitative results of pose-graph pruning and optimization for
CSAIL dataset. (a) Our result with a grid cell size of 0.3$\crossproduct$0.3
m; (b) Our result with a grid cell size of 1$\crossproduct$1 m set by the
method described in Fig. 7; (c) The result based on the recent data-
compression algorithm [34]. Table III: Performance comparison of node pruning
with various public datasets and our dataset.

| # of Node | # of Edge | NPC | $\text{ARPS}^{4}$
---|---|---|---|---
CSAIL |  327 (1,045) |  354 (1,172) | 1 | 0.78%
FR079 |  718 (989) |  861 (1,217) | 1 | 0.60%
M3500 | 1,113 (3,500) | 1,762 (5,453) | 1 | 4.20%
R9-home§ |  317 (600) |  663 (1,038) | 1 | 1.30%

Note: The numbers in parentheses are the results before pruning. NPC denotes
the average numbers of nodes per grid cell. $\mathsection$ represents a
dataset with our cleaning robot called R9 (https://github.com/Multiplanet-
Robot).

Furthermore, we qualitatively compared our SLAM back-end with the recent
pruning algorithm [34], as illustrated in Fig. 11. The proposed pruning
algorithm was evaluated on various datasets as represented in Table III, and
verified that the results were aligned with our objectives in terms of
commercialization; specifically, the criteria were satisfied: the number of
nodes per cell $\leq$ 2, number of edges connected to a node $\leq$ 3, and
ARPS after pruning and optimization was less than 10 %. We found that the
method using (10) significantly enhanced the computation time on public
datasets rather than using (9), such as the Manhattan (M3500; 3,551 ms
$\rightarrow$ 10.36 ms) and CSAIL datasets (195.67 ms $\rightarrow$ 2.79 ms).
Moreover, the average ratio of pose shift after optimization was similar to
the results obtained using (9) (M3500: 3.62 % $\rightarrow$ 4.20 %, CSAIL: 3.3
% $\rightarrow$ 0.78 %), with smaller values indicating better performance.

In addition, the proposed algorithm was evaluated in terms of the consistency
of global maps by applying it to a robot (B9) in various real office
environments. This letter presents only the qualitative results, as
illustrated in Fig. 12. Consequently, the applicability of the proposed
algorithm in offices has been verified successfully.

###  V-C Resource Usage in the Embedded System

Furthermore, in this study, our SLAM algorithm is operated on an embedded
processor called Cortex-A9, with processor usage below 25%. In addition,
memory usage is less than 180 MB to cover a 330 $\text{m}^{2}$ space. For
example, the GPE is operated at over 3 Hz on the specified processor.

##  VI Conclusions

This study has presented a multi-stage approach to global pose estimation and
mapping, particularly suitable for low-cost embedded systems. The proposed
method seamlessly integrates various sensors into our SLAM algorithm without
requiring significant structural modifications. We proposed a memory-efficient
back-end with pruning and optimization techniques, which were validated across
multiple mass-produced commercial robots in terms of consistency and
stability. Future research will extend this framework to multi-robot SLAM.

Figure 12: Illustration of qualitative mapping results corresponding to large-
scale office environments in Korea with repeated patterns such as partitions;
(a) Nexen building 4F, (b) LG Science Park 7F, (c) LGE Seocho 14F, (d) Science
Park 6F, (e) SNI building 3F, (f) LGE Seocho 1B, and (g) Seoul building 13F.

## References

  * [1] P. Gu and Z. Meng, “S-VIO: Exploiting structural constraints for RGB-D visual inertial odometry,” _IEEE Robot. Automat. Lett._ , vol. 8, no. 6, pp. 3542–3549, 2023.
  * [2] Z. Wang, X. Liu, L. Yang, and F. Gao, “SW-LIO: A sliding window based tightly coupled LiDAR-inertial odometry,” _IEEE Robot. Automat. Lett._ , vol. 8, no. 10, pp. 6675–6682, 2023.
  * [3] C. Bai, T. Xiao, Y. Chen, H. Wang, F. Zhang, and X. Gao, “Faster-LIO: Lightweight tightly coupled LiDAR-inertial odometry using parallel sparse incremental voxels,” _IEEE Robot. Automat. Lett._ , vol. 7, no. 2, pp. 4861–4868, 2022.
  * [4] W. Xu, Y. Cai, D. He, J. Lin, and F. Zhang, “FAST-LIO2: Fast direct LiDAR-inertial odometry,” _IEEE Trans. Robot._ , vol. 38, no. 4, pp. 2053–2073, 2022.
  * [5] C. Sung, S. Jeon, H. Lim, and H. Myung, “What if there was no revisit? Large-scale graph-based SLAM with traffic sign detection in an HD map using LiDAR inertial odometry,” _Intell. Serv. Robot._ , pp. 1–10, 2022.
  * [6] S. Song, H. Lim, A. J. Lee, and H. Myung, “DynaVINS: A visual-inertial SLAM for dynamic environments,” _IEEE Robot. Automat. Lett._ , vol. 7, no. 4, pp. 11 523–11 530, 2022.
  * [7] T. Qin, P. Li, and S. Shen, “VINS-Mono: A robust and versatile monocular visual-inertial state estimator,” _IEEE Trans. Robot._ , vol. 34, no. 4, pp. 1004–1020, 2018.
  * [8] S. Leutenegger, S. Lynen, M. Bosse, R. Siegwart, and P. Furgale, “Keyframe-based visual–inertial odometry using nonlinear optimization,” _Int. J. Robot. Res._ , vol. 34, no. 3, pp. 314–334, 2015.
  * [9] C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. Montiel, and J. D. Tardós, “ORB-SLAM3: An accurate open-source library for visual, visual–inertial, and multimap SLAM,” _IEEE Trans. Robot._ , vol. 37, no. 6, pp. 1874–1890, 2021.
  * [10] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos, “ORB-SLAM: A versatile and accurate monocular SLAM system,” _IEEE trans. Robot._ , vol. 31, no. 5, pp. 1147–1163, 2015.
  * [11] S. Kohlbrecher, J. Meyer, O. von Stryk, and U. Klingauf, “A flexible and scalable SLAM system with full 3D motion estimation,” in _Proc. IEEE Int. Symp. Safety, Security and Rescue Robotics_ , 2011, pp. 8729–8736.
  * [12] M. Labbé and F. Michaud, “RTAB-Map as an open-source LiDAR and visual simultaneous localization and mapping library for large-scale and long-term online operation,” _Journal of Field Robotics_ , vol. 36, no. 2, pp. 416–446, 2019.
  * [13] W. Hess, D. Kohler, H. Rapp, and D. Andor, “Real-time loop closure in 2D LiDAR SLAM,” in _Proc. IEEE Int. Conf. Robot. Autom._ , 2016, pp. 1271–1278.
  * [14] A. J. Lee, S. Song, H. Lim, W. Lee, and H. Myung, “(LC)2: LiDAR-camera loop constraints for cross-modal place recognition,” _IEEE Robot. Automat. Lett._ , vol. 8, no. 6, pp. 3589–3596, 2023.
  * [15] T. Shan, B. Englot, C. Ratti, and D. Rus, “LVI-SAM: Tightly-coupled LiDAR-visual-inertial odometry via smoothing and mapping,” in _Proc. IEEE Int. Conf. Robot. Autom._ , 2021, pp. 5692–5698.
  * [16] S. Zhao, H. Zhang, P. Wang, L. Nogueira, and S. Scherer, “Super odometry: IMU-centric LiDAR-visual-inertial estimator for challenging environments,” in _Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst._ , 2021, pp. 8729–8736.
  * [17] “World robotics–service robots,” World Robotics Reports, Accessed on: Nov. 29, 2023. [Online]. Available: http://ifr.org/worldrobotics/.
  * [18] J. Sola, J. Vallvé, J. Casals, J. Deray, M. Fourmy, D. Atchuthan, A. Corominas-Murtra, and J. Andrade-Cetto, “WOLF: A modular estimation framework for robotics based on factor graphs,” _IEEE Robot. Automat. Lett._ , vol. 7, no. 2, pp. 4710–4717, 2022.
  * [19] A. Cramariuc, L. Bernreiter, F. Tschopp, M. Fehr, V. Reijgwart, J. Nieto, R. Siegwart, and C. Cadena, “maplab 2.0 – A modular and multi-modal mapping framework,” _IEEE Robot. Automat. Lett._ , vol. 8, no. 2, pp. 520–527, 2022.
  * [20] J. Kannala and S. S. Brandt, “A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses,” _IEEE Trans. on Pattern Anal. Mach. Intell._ , vol. 28, no. 8, pp. 1335–1340, 2006.
  * [21] M. I. Lourakis and A. A. Argyros, “SBA: A software package for generic sparse bundle adjustment,” _ACM Trans. Math. Software_ , vol. 36, no. 1, pp. 1–30, 2009.
  * [22] P. J. Besl and N. D. McKay, “Method for registration of 3-D shapes,” in _Sens. Fusion IV: Control Paradigms and Data Struct._ , vol. 1611, 1992, pp. 586–606.
  * [23] M. L. Rodríguez-Arévalo, J. Neira, and J. A. Castellanos, “On the importance of uncertainty representation in active SLAM,” _IEEE Trans. Robot._ , vol. 34, no. 3, pp. 829–834, 2018.
  * [24] C. Chow and C. Liu, “Approximating discrete probability distributions with dependence trees,” _IEEE Trans. Information Theory_ , vol. 14, no. 3, pp. 462–467, 1968.
  * [25] H. Kretzschmar and C. Stachniss, “Information-theoretic compression of pose graphs for laser-based SLAM,” _Int. J. Robot. Res._ , vol. 31, no. 11, pp. 1219–1230, 2012.
  * [26] J. Lim, J.-M. Frahm, and M. Pollefeys, “Online environment mapping,” in _Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR)_ , 2011, pp. 3489–3496.
  * [27] M. Bloesch, M. Burri, S. Omari, M. Hutter, and R. Siegwart, “Iterated extended Kalman filter based visual-inertial odometry using direct photometric feedback,” _Int. J. Robot. Res._ , vol. 36, no. 10, pp. 1053–1072, 2017.
  * [28] R. C. Smith and P. Cheeseman, “On the representation and estimation of spatial uncertainty,” _Int. J. Robot. Res._ , vol. 5, no. 4, pp. 56–68, 1986.
  * [29] N. Carlevaris-Bianco, M. Kaess, and R. M. Eustice, “Generic node removal for factor-graph SLAM,” _IEEE Trans. Robot._ , vol. 30, no. 6, pp. 1371–1385, 2014.
  * [30] B. Chazelle, “A minimum spanning tree algorithm with inverse-Ackermann type complexity,” _J. ACM_ , vol. 47, no. 6, pp. 1028–1047, 2000.
  * [31] D. R. Karger, P. N. Klein, and R. E. Tarjan, “A randomized linear-time algorithm to find minimum spanning trees,” _J. ACM_ , vol. 42, no. 2, pp. 321–328, 1995.
  * [32] R. L. Graham and P. Hell, “On the history of the minimum spanning tree problem,” _IEEE Ann. Hist. Comput._ , vol. 7, no. 1, pp. 43–57, 1985.
  * [33] E. Olson and P. Agarwal, “Inference on networks of mixtures for robust robot mapping,” _Int. J. Robot. Res._ , vol. 32, no. 7, pp. 826–840, 2013.
  * [34] K. J. Doherty, D. M. Rosen, and J. J. Leonard, “Spectral measurement sparsification for pose-graph SLAM,” in _Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst._ , 2022, pp. 01–08.


# The Chai Platform’s AI Safety Framework

Xiaoding Lu &Aleksey Korshuk &Zongyi Liu &William Beauchamp
Chai Research

###### Abstract

Chai empowers users to create and interact with customized chatbots, offering
unique and engaging experiences. Despite the exciting prospects, the work
recognizes the inherent challenges of a commitment to modern safety standards.
Therefore, this paper presents the integrated AI safety principles into Chai
to prioritize user safety, data protection, and ethical technology use. The
paper specifically explores the multidimensional domain of AI safety research,
demonstrating its application in Chai’s conversational chatbot platform. It
presents Chai’s AI safety principles, informed by well-established AI research
centres and adapted for chat AI. This work proposes the following safety
framework: Content Safeguarding; Stability and Robustness; and Operational
Transparency and Traceability. The subsequent implementation of these
principles is outlined, followed by an experimental analysis of Chai’s AI
safety framework’s real-world impact. We emphasise the significance of
conscientious application of AI safety principles and robust safety measures.
The successful implementation of the safe AI framework in Chai indicates the
practicality of mitigating potential risks for responsible and ethical use of
AI technologies. The ultimate vision is a transformative AI tool fostering
progress and innovation while prioritizing user safety and ethical standards.

##  1 Introduction

With the rapid improvement in the quality and fluency of virtual
conversational AI agents, there has been growing integration of chat-based
based AI systems (Chat AIs) into a range of real-world applications Caldarini
et al. (2022); Almansor and Hussain (2020). This has led to the development of
new technologies and opportunities, as seen in platforms such as the Chai
research platform. The Chai research platform is an innovative online platform
that empowers users to design and interact with chatbots that emulate friends,
mentors, or fictional characters Irvine et al. (2023).

However, with this flexibility comes potential challenges. If unregulated,
there’s a risk that users, intentionally or unintentionally, might design
chatbots that do not align to desired ethical and safe AI standards. This
potential issue highlights the importance of robust safety measures to prevent
harmful outcomes, such as promoting inappropriate content or negatively
impacting user well-being Lu et al. (2023). This paper addresses safety
considerations for Chat AI and introduces the AI safety principles integrated
into the Chai platform to ensure that deployed chat ai technologies best align
with modern safe AI practices Schuett et al. (2023). Safe AI is about
developing and managing AI systems that work in a manner that is beneficial to
humanity and encourages generation of safe content. It involves prioritising
user safety, protecting data, and ensuring that the technology behaves
ethically and responsibly Mohseni et al. (2022). In this paper, we detail our
three main chat ai safety pillars; content safeguarding Awad et al. (2018),
system stability and robustness Drenkow et al. (2021), and operational
transparency and traceability Wanner et al. (2022). The second part of this
paper then discusses how these considerations are practically integrated into
chat ai platforms, where we detail the strategies taken that reinforces Chai’s
commitment to creating a safe AI conversational platform.

##  2 Background

AI safety, as a field of study, is dedicated to understanding and mitigating
the potential risks associated with the development and deployment of
artificial intelligence technologies Bostrom (2014); Schuett et al. (2023).
The primary concern revolves around ensuring alignment Christian (2020);
Gabriel and Ghazavi (2021) of these technologies with human values, ethics,
and societal norms Yudkowsky (2011). This involves designing AI systems that
can understand, learn from, and act in accordance with the principles, values,
and goals that humans hold Gabriel (2020); Sotala and Yampolskiy (2014). The
objective is to create artificial intelligence that not only improves
efficiency and productivity but also respects human autonomy, privacy, and
other core ethical principles Dignum (2019).

In order to achieve this, AI safety research Mohseni et al. (2022) is
multidimensional, involving elements of computer science, cognitive science,
ethics, and social science Fishbein and Ajzen (2005); Sarma and Hay (2017).
Researchers investigate technical aspects like robustness, interpretability,
and transparency of AI systems, while also exploring philosophical questions
about moral and ethical considerations regarding how we define human values
Turchin (manuscript); Muehlhauser and Helm (2012); Friedman and Hendry (2019).
A range of approaches explore how to effectively incorporate defined human
values within reward functions used to train AI systems Hendrycks et al.
(2020); Soares et al. (2015); Ng and Russell (2000); Riedl and Harrison
(2016); Armstrong ; Etzioni and Etzioni (2016). The ultimate aim is to ensure
that the vast transformative power of AI is harnessed in a manner that is
safe, responsible, and beneficial for all of humanity Taylor et al. (2020).

In this work, we outline the approach used to incoporate modern AI safety
standards within generative AI technology for Chai’s conversational chatbot
platform Irvine et al. (2023).

##  3 AI Safety Principles

In this section we analyse the different important considerations that various
well-established AI research centres have proposed, and then use this to
propose three main pillars that define Chai’s AI safety principles for a
conversational chatbot platform.

Deepmind outline three core AI safety principles Leike et al. (2017);
specification, robustness and assurance. Meta’s pillars of responsible AI
Pesenti (2021) are privacy, security, fairness, transparency and
accountability. While OpenAI’s tenants Willner  include the reduction of harm,
fostering trust, and continuous improvement. We analyse the different proposed
pillars and aggregate them into the following main categories, with a
consideration of how to align these definitions best for chat AI:

  1. 1.

Content Safeguarding Content safeguarding aims to encourage chat AI systems to
generate responses (content) that are aligned with human values such that they
do not cause any risk to users. This includes ensuring that the system
generates any appropriate, ethical content and that the system remains
respectful to users, as well as adhering to modern standards of morality and
ethics Awad et al. (2018).

  2. 2.

Stability and Robustness Stable and Robust systems can handle diverse
situations without failing- whether that be possible domain/distributional
shifts Liusie et al. (2022); Malinin et al. (2021); Liang et al. (2023) or
user perturbations, such as adversarial attacks Raina and Gales (2023);
Chakraborty et al. (2018), that it might encounter when deployed. Robust
systems act in a predictable manner and do not exhibit any unexpected
behaviour, even in environments that are different from the standard training
domain Shafique et al. (2021).

  3. 3.

Operational Transparency and Traceability This refers to creators having the
ability to interpret, observe, audit and track all system activity Mora-
Cantallops et al. (2021); Räuker et al. (2023); Wanner et al. (2022). For
example, this can be achieved by having logs of all past system activity which
can be analysed for quality control, allowing developers to determine the
cause of any possible undesirable behaviour at an early stage. Early detection
of unsafe AI behaviour allows for intervention, e.g. via human action or
activation/tripping of safety mechanisms before the undesired AI behaviour
progresses further Schwalbe and Schels (2020).

##  4 Chai Safety Framework

Figure 1: AI Safety System Content Safeguarding Pipeline.

In this section, we outline the methods used to achieve the three safety
pillars presented for Chai’s AI platform in the previous section.

###  4.1 The Chai platform

The Chai platform is an online framework where thousands of chat AIs with
distinct personas are hosted, which other users can freely interact with. The
lifecycle begins with users having the ability to create new chat ai personas
by selecting keywords that describe the characteristics of a particular chat
ai. Keywords such as bubbly or intelligent are then used to condition the
responses of a particular chat AI persona. Users can then upload an image that
represents the chat AI, which for example can be the cover picture of a
fictional character that the keywords attempt to describe, and select a name
that describes the persona. The chat persona is then uploaded to the platform,
where other users are able to explore all existing chat AIs and converse with
any selected persona.

####  4.1.1 Content Safeguarding

To ensure that the content generated by any chat AI aligns with human values
while performing the task, we apply multiple stages of safety alignment to all
created chat AI personas as well as uploaded images.

One concern is that a particular set of selected keywords may lead to personas
that could display antisocial behaviour. We therefore develop a moderation
system that is applied to all chatbots before deployment, where the moderation
system filters out any chatbots that may exhibit negative properties. This is
achieved by running thousands of logged conversational histories into each new
persona, and then sampling the responses of chat AI personas under the
different conversational histories. We then apply an automatic safety
classifier to all outputs of the chat persona, and if the analysis flags that
the chat AI may not be sufficiently safe, then the persona is discarded and
not included within the set of accessible personas on the chai AI platform.
Further beyond moderation of deployed personas, our base model is trained to
generate safe responses, as detailed in Lu et al. (2023).

Similarly, users may upload images that do not adhere to safety standards. We
therefore use an image moderation classification system to block any images
which are not deemed safe. Our entire Content Safeguarding pipeline is
presented in Figure 1.

####  4.1.2 Stability and Robustness

To ensure that the deployed Chai system is robust and can handle diverse
outputs, the base chat AI language model is frequently retrained with recent
user conversations from diverse geographical locations. Continuously
retraining our system on current user conversations ensures that the model can
rapidly adapt to any implicit distributional shift of user queries.
Additionally, the platform provides users with the opportunity to rate the
chatbot’s responses or provide suggestions on possible better responses. This
information can then be frequently used to update our reward function Irvine
et al. (2023), to better align our systems with users’ preferences and be
better equipped for any new emerging domain that the system may encounter.

####  4.1.3 Operational Transparency and Traceability

The Chai Platform logs all user conversational exchanges which we store in a
private and secure server. If users flag any responses from particular
personas, we have the full trace of exchanges with the persona which can be
used to analyse the system. Further, by periodically reviewing the exchanges
for quality and safety, we can better understand how the AI chatbot interacts
with users, identify potential risks, and monitor the chat AI’s compliance
with our content guidelines and policies. This helps enable early detection of
any unsafe practices which can be remedied.

##  5 Experiments

The aim of this experimental section is to determine the impact of Chai’s AI
safety framework on the real-world safety attributes of deployed AI chatbots
on the Chai platform. As detailed in Lu et al. (2023), the base large language
model used in all deployed chatbots is a GPT-J 6B Wang and Komatsuzaki (2021)
model fine-tuned on novels and literature
111https://huggingface.co/hakurei/lit-6B. Since the inception of the deployed
platform on 5th May 2022, the base model has undergone various iterations of
development to align with the AI safety framework, under the categories of
content safeguarding, stability & robustness and Operational Transparency &
Traceability. Section 4 indicates the specific details of the approaches used
to achieve these desired AI safety goals.

Although qualitative analysis of randomly sampled user conversations has
revealed that the latest iterations of the deployed chat AIs adhere well to
modern safety standards, as desired, it is critical to measure this progress
quantiatively. However, it can be challenging to define a single metric to
measure the safety of a chatbot model. Inspired by OpenAI’s moderation tool
222https://platform.openai.com/docs/guides/moderation, in this work, we define
a single-value safety score, $\bar{s}$ to act as a proxy measure for platform
safety. Specifically we report the Note Safe for Work (NSFW) words ratio by
day, which gives the fraction of all words, $w$ appearing in real user
conversations (totaling $N$ words) 333There are typically 2 million user-
chatbot conversations per day with an average length of 500 words in a
conversation., classed as NSFW in a single day for deployed models. We define
a dictionary, $\mathcal{V}$, for all the NSFW words in the English language,
aligning to OpenAI’s moderation categories: hate speech, self-harm, sexual
content and violence.

$\bar{s}=\frac{1}{N}\sum_{n}^{N}\mathbbm{1}(w_{n}\in\mathcal{V})$ (1)

With this definition of model safety we observe how new iterations of the chat
AIs impact the fraction of NSFW words. Figure 2 demonstrates that key major
updates in the base models and platform safety design in June 2022, October
2022 and March 2023 result in significant improvements in model safety as per
our proxy metric. This encourages us to believe that with an active effort to
align the Chai research platform with our proposed AI safety principles, we
are able to have significant measurable and quantifiable improvements in model
safety. This leads us to the understanding that our model is safe to be
deployed in real-world settings.

Figure 2: AI Safety Progression

##  6 Conclusions

In conclusion, the emergence and rapid evolution of conversational AI
platforms like Chai heralds an exciting era of personalized and democratized
AI interactions. However, this powerful tool also necessitates careful
stewardship to ensure that its use aligns with ethical standards and does not
inadvertently lead to harmful consequences. As this paper outlines, through
the conscientious application of AI safety principles and robust safety
measures, platforms like Chai can provide not just powerful and unique AI
interactions, but also a safe and nurturing environment for users to explore
the landscape of conversational AI.

With a multifaceted approach that includes safety-oriented training, rigorous
moderation, and consistent audits within Chai’s AI safety framwork (content
safeguarding, stability & robustness and operational transparency &
traceability), we demonstrate that the implementation of AI safety is both an
attainable and practical endeavor. This study highlights the successful
application of the safe AI framework, indicating how potential risks can be
mitigated to ensure the responsible and ethical use of AI technologies. It is
clear that, when approached with due diligence and responsibility, AI can be a
transformative tool that fosters progress and innovation, while always
prioritizing user safety and ethical standards.

## References

  * Almansor and Hussain (2020) Ebtesam Almansor and Farookh Hussain. 2020.  _Survey on Intelligent Chatbots: State-of-the-Art and Future Research Directions_, pages 534–543.
  * (2) Stuart Armstrong.  Research agenda v0.9: Synthesising a human’s preferences into a utility function.
  * Awad et al. (2018) Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean-François Bonnefon, and Iyad Rahwan. 2018.  The moral machine experiment.  _Nature_ , 563(7729):59–64.
  * Bostrom (2014) Nick Bostrom. 2014.  _Superintelligence_.  Dunod.
  * Caldarini et al. (2022) Guendalina Caldarini, Sardar F. Jaf, and Kenneth McGarry. 2022.  A literature survey of recent advances in chatbots.  _CoRR_ , abs/2201.06657.
  * Chakraborty et al. (2018) Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopadhyay. 2018.  Adversarial attacks and defences: A survey.  _CoRR_ , abs/1810.00069.
  * Christian (2020) Brian Christian. 2020.  _The alignment problem: Machine Learning and human values_.  Norton & Company.
  * Dignum (2019) Virginia Dignum. 2019.  Ensuring responsible ai in practice.  _Responsible Artificial Intelligence_ , page 93–105.
  * Drenkow et al. (2021) Nathan Drenkow, Numair Sani, Ilya Shpitser, and Mathias Unberath. 2021.  Robustness in deep learning for computer vision: Mind the gap? _CoRR_ , abs/2112.00639.
  * Etzioni and Etzioni (2016) Amitai Etzioni and Oren Etzioni. 2016.  Designing ai systems that obey our laws and values.  _Commun. ACM_ , 59(9):29–31.
  * Fishbein and Ajzen (2005) Martin Fishbein and Icek Ajzen. 2005.  Theory-based behavior change interventions: Comments on hobbis and sutton.  _Journal of Health Psychology_ , 10(1):27–31.
  * Friedman and Hendry (2019) Batya Friedman and David G. Hendry. 2019.  _Value sensitive design: Shaping technology with moral imagination_.  MIT Press.
  * Gabriel (2020) Iason Gabriel. 2020.  Artificial intelligence, values and alignment.  _CoRR_ , abs/2001.09768.
  * Gabriel and Ghazavi (2021) Iason Gabriel and Vafa Ghazavi. 2021.  The challenge of value alignment: from fairer algorithms to AI safety.  _CoRR_ , abs/2101.06060.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2020.  Aligning AI with shared human values.  _CoRR_ , abs/2008.02275.
  * Irvine et al. (2023) Robert Irvine, Douglas Boubert, Vyas Raina, Adian Liusie, Ziyi Zhu, Vineet Mudupalli, Aliaksei Korshuk, Zongyi Liu, Fritz Cremer, Valentin Assassi, Christie-Carol Beauchamp, Xiaoding Lu, Thomas Rialan, and William Beauchamp. 2023\.  Rewarding chatbots for real-world engagement with millions of users.
  * Leike et al. (2017) Jan Leike, Miljan Martic, Victoria Krakovna, Pedro A. Ortega, Tom Everitt, Andrew Lefrancq, Laurent Orseau, and Shane Legg. 2017.  Ai safety gridworlds.
  * Liang et al. (2023) Jian Liang, Ran He, and Tieniu Tan. 2023.  A comprehensive survey on test-time adaptation under distribution shifts.
  * Liusie et al. (2022) Adian Liusie, Vatsal Raina, Vyas Raina, and Mark Gales. 2022.  Analyzing biases to spurious correlations in text classification tasks.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 78–84, Online only. Association for Computational Linguistics.
  * Lu et al. (2023) Xiaoding Lu, Aleksey Korshuk, Zongyi Liu, William Beauchamp, and Chai Research. 2023\.  Safer conversational ai as a source of user delight.
  * Malinin et al. (2021) Andrey Malinin, Neil Band, Alexander Ganshin, German Chesnokov, Yarin Gal, Mark J. F. Gales, Alexey Noskov, Andrey Ploskonosov, Liudmila Prokhorenkova, Ivan Provilkov, Vatsal Raina, Vyas Raina, Mariya Shmatova, Panos Tigas, and Boris Yangel. 2021.  Shifts: A dataset of real distributional shift across multiple large-scale tasks.  _CoRR_ , abs/2107.07455.
  * Mohseni et al. (2022) Sina Mohseni, Haotao Wang, Chaowei Xiao, Zhiding Yu, Zhangyang Wang, and Jay Yadawa. 2022.  Taxonomy of machine learning safety: A survey and primer.  _ACM Comput. Surv._ , 55(8).
  * Mora-Cantallops et al. (2021) Marçal Mora-Cantallops, Salvador Sánchez-Alonso, Elena García-Barriocanal, and Miguel-Angel Sicilia. 2021.  Traceability for trustworthy ai: A review of models and tools.  _Big Data and Cognitive Computing_ , 5(2).
  * Muehlhauser and Helm (2012) Luke Muehlhauser and Louie Helm. 2012.  The singularity and machine ethics.  _The Frontiers Collection_ , page 101–126.
  * Ng and Russell (2000) Andrew Y. Ng and Stuart J. Russell. 2000.  Algorithms for inverse reinforcement learning.  In _Proceedings of the Seventeenth International Conference on Machine Learning_ , ICML ’00, page 663–670, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
  * Pesenti (2021) Jerome Pesenti. 2021.  Facebook’s five pillars of responsible ai, de, jun. 2021.  _URL https://ai. facebook. com/blog/facebooks-five-pillars-of-responsible-ai_.
  * Raina and Gales (2023) Vyas Raina and Mark Gales. 2023.  Identifying adversarially attackable and robust samples.
  * Riedl and Harrison (2016) Mark O. Riedl and Brent Harrison. 2016.  Using stories to teach human values to artificial agents.  In _AAAI Workshop: AI, Ethics, and Society_.
  * Räuker et al. (2023) Tilman Räuker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell. 2023.  Toward transparent ai: A survey on interpreting the inner structures of deep neural networks.
  * Sarma and Hay (2017) Gopal P. Sarma and Nick J. Hay. 2017.  _Mammalian value systems_.
  * Schuett et al. (2023) Jonas Schuett, Noemi Dreksler, Markus Anderljung, David McCaffary, Lennart Heim, Emma Bluemke, and Ben Garfinkel. 2023.  Towards best practices in agi safety and governance: A survey of expert opinion.
  * Schwalbe and Schels (2020) Gesina Schwalbe and Martin Schels. 2020.  A survey on methods for the safety assurance of machine learning based systems.
  * Shafique et al. (2021) Muhammad Shafique, Mahum Naseer, Theocharis Theocharides, Christos Kyrkou, Onur Mutlu, Lois Orosa, and Jungwook Choi. 2021.  Robust machine learning systems: Challenges, current trends, perspectives, and the road ahead.  _CoRR_ , abs/2101.02559.
  * Soares et al. (2015) Nate Soares, Benja Fallenstein, Stuart Armstrong, and Eliezer Yudkowsky. 2015.  Corrigibility.  In _Artificial Intelligence and Ethics, Papers from the 2015 AAAI Workshop, Austin, Texas, USA, January 25, 2015_ , volume WS-15-02 of _AAAI Technical Report_. AAAI Press.
  * Sotala and Yampolskiy (2014) Kaj Sotala and Roman V Yampolskiy. 2014.  Responses to catastrophic agi risk: a survey.  _Physica Scripta_ , 90(1):018001.
  * Taylor et al. (2020) Jessica Taylor, Eliezer Yudkowsky, Patrick LaVictoire, and Andrew Critch. 2020.  Alignment for advanced machine learning systems.  _Ethics of Artificial Intelligence_ , page 342–382.
  * Turchin (manuscript) Alexey Turchin. manuscript.  Ai alignment problem: ?human values? don?t actually exist.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wanner et al. (2022) Jonas Wanner, Lukas-Valentin Herm, Kai Heinrich, and Christian Janiesch. 2022.  The effect of transparency and trust on intelligent system acceptance: Evidence from a user-based study.  _Electronic Markets_ , 32(4):2079–2102.
  * (40) Dave Willner.  [link].
  * Yudkowsky (2011) Eliezer Yudkowsky. 2011.  Complex value systems are required to realize valuable futures.


# A Novel Autoencoders-LSTM Model for Stroke Outcome Prediction using
Multimodal MRI Data

###### Abstract

Patient outcome prediction is critical in management of ischemic stroke. In
this paper, a novel machine learning model is proposed for stroke outcome
prediction using multimodal Magnetic Resonance Imaging (MRI). The proposed
model consists of two serial levels of Autoencoders (AEs), where different AEs
at level 1 are used for learning unimodal features from different MRI
modalities and a AE at level 2 is used to combine the unimodal features into
compressed multimodal features. The sequences of multimodal features of a
given patient are then used by an LSTM network for predicting outcome score.
The proposed AE2-LSTM model is proved to be an effective approach for better
addressing the multimodality and volumetric nature of MRI data. Experimental
results show that the proposed AE2-LSTM outperforms the existing state-of-the
art models by achieving highest AUC=0.71 and lowest MAE=0.34.

Index Terms—  Multimodal image fusion, Long Short-Term Memory (LSTM),
Autoencoder (AE), Stroke outcome prediction, Magnetic Resonance Imaging (MRI),
modified Rankin Scale (mRS).

##  1 Introduction

Stroke is the second-leading cause of death and the third-leading cause of
death and disability combined [1]. About 87% of all strokes are classified as
ischemic. In this case, an artery that supplies blood to the brain is blocked.
Although reperfusion therapies (intravenous thrombolysis and thrombectomy)
have revolutionized ischemic stroke management, outcome remains highly
heterogeneous across patients. Improving the prediction of outcome in stroke
patients would contribute to tailor treatment decisions and evaluate novel
therapeutic strategies.

Application of machine learning models on stroke outcome prediction is a field
of growing interest. Some authors proposed different 2D and 3D Convolutional
Neural Networks (CNNs) for predicting patient outcomes from MR/CT images and
clinical data [2, 3, 4, 5, 6, 7]. However, none of them perfectly match the
multimodal and volumetric nature of MRI/CT data or contain too many parameters
(in case of 3D-CNNs) when the number of image modalities increases.

A two-level Autoencoders followed by a Long Short-Term Memory (AE2-LSTM) is
proposed for stroke outcome prediction using multimodal MRI data. Five AEs are
first trained to learn unimodal features from five MRI modalities. Another AE
is used to combine these five unimodal features into one main multimodal
representation. And finally, an LSTM network is applied to predict the 3-month
modified Rankin Scale (mRS), a 7-point disability scale, from the compressed
multimodal feature series.

The rest of the paper is organized as follows: a review of recent machine
learning approaches on stroke outcome prediction is presented in the next
section. The proposed AE2-LSTM framework is described in detail in section 3.
A description of the dataset used in this study and experimental results are
reported in section 4. Finally, section 5 concludes the paper and outlines the
future directions.

##  2 Related Work

Fig. 1: The general block diagram of a machine learning model for stroke
outcome prediction using multimodal MRI.

The general block diagram of a machine learning-based stroke outcome
prediction using multimodal MRI is shown in Figure 1. In one of the early
efforts to predict patient outcome using machine learning, Ramos et al. [2]
applied Random Forest (RF), Support Vector Machine (SVM) and Artficial Neural
Networks (ANN) algorithms to predict mRS scores. Both clinical/biological
variables – e.g. age, pre-stroke mRS, glucose level and NIH Stroke Scale
(NIHSS) at baseline – and radiological parameters – presence of leukoaraiosis,
old infarctions, hyperdense vessel sign, and hemorrhagic transformation – were
used as input features. The main limitation of such models is that the
features are hand-crafted, unlike the state-of the art end-to-end deep
learning models that learned dynamically and automatically features from raw
images.

Bacchi et al. [3] proposed to combine CNN and ANN models to predict NIHSS or
mRS scores. Noncontrast CT images are processed via a CNN model while clinical
data (such as age, sex, blood pressure) are processed in an ANN model. The
outputs of both models are then merged through Fully-Connected (FC) layers to
generate the final binary (poor vs. good) outcomes.

Zihni et al. [4] proposed a deep learning-based multimodal fusion strategy for
integration of neuroimaging information with clinical metadata. First, a
3D-CNN and a ANN models are built for processing neuroimaging and clinical
data, respectively. Then, features from two models are fused using FC layer
for prediction of mRS scores.

Nishi et al. [5] applied deep learning for extracting neuroimaging features in
order to predict clinical outcomes (mRS) for patients with large vessel
occlusion. They proposed a 2-output deep encoder-decoder architecture (3D
U-Net) with DWI data as an input. The U-Net prediction mask (output 1) is for
the ischemic lesion segmentation task, and the high-level feature maps (output
2) are also extracted from the deepest middle layers. Then, they used the
obtained features in a 2-layer ANN to predict the binarized 3-month mRS
scores.

Recently, Hatami et al. [7] proposed a multimodal CNN-LSTM model to
automatically encode the spatio-temporal context of MR images in a deep
learning architecture. They applied a curriculum learning, a family of
learning algorithms in which first starts with only ”easy” patients (from the
model’s perspective) and then gradually includes more ”difficult” patients.
The training has two main steps: first a CNN-LSTM ranks the patients from 1 to
10 based on their ”difficulty”, and then another CNN-LSTM (similar
architecture) uses the ranking information to progressively learn the mRS
scores by including different types of patients ranked according to their
difficulty at different training epochs.

A multimodal CNN-LSTM based ensemble model is proposed in [6] for processing
five MR imaging modalities. For each modality, a dedicated network provides
preliminary prediction of the mRS. The final mRS score is obtained by merging
the preliminary probabilities of each module dedicated to a specific type of
MR image weighted by a clinical metadata, i.e. age or NIHSS.

The main limitation of the previous models is that none of them perfectly fits
the multimodal and volumetric nature of the MRI data. For instance, 3D CNNs in
[3, 4, 5] result in too many parameters if they use a dedicated 3D-CNN for
each image modality. Also, CNN-LSTM models proposed by [6, 7] use CNNs pre-
trained on natural image dataset (ImageNet [8]). The authors used ImageNet
features of off-the-shelf models such as ResNet [9], DenseNet [10] and VGG
[11] with no fine-tuning on MRI data which is not ideal. The proposed AE2-LSTM
Model tries to address these challenges.

##  3 Method

Fig. 2: The proposed AE2-LSTM model for stroke outcome prediction using
multimodal MRI. Left: slice-level training of AEs for learning unimodal and
multimodal features. Right: patient-level training and testing of LSTM for
predicting mRS score.

###  3.1 Autoencoders (AE)

An AE is a special type of neural network with an unsupervised learning
technique for the task of representation learning. It is trained to attempt to
copy its input to its output. The network consists of two parts: an encoder
function $h=f(x)$ and a decoder that produces a reconstruction $r=g(h)$.
Blocks of AEs can be seen in Figure 2 left. The AE learns a representation
(encoding) for a set of data, typically for dimensionality reduction, by
training the network to ignore insignificant information (“noise”). Different
variations of AEs are proposed aiming to force the learned representations to
assume useful properties [12]. For example regularized AEs (Sparse, Denoising
and Contractive) are proposed for learning representations (features) for
classification tasks [13] and Variational AEs are used as generative models
[14].

###  3.2 Long Short Term Memory Networks (LSTM)

LSTM [15] is a type of recurrent neural network to classify, process and
predict time series data. Unlike standard feedforward neural networks, LSTM
has feedback connections. A typical LSTM unit is composed of a cell, an input
gate, an output gate and a forget gate. The cell remembers values over
arbitrary time intervals and the gates regulate the flow of information into
and out of the cell. A block of the LSTM network can be seen in Figure 2
right.

###  3.3 Proposed AE2-LSTM Model

The proposed AE2-LSTM model for stroke outcome prediction using multimodal MRI
data is shown in Figure 2. This novel architecture is composed of two levels
of AEs (an AE at level 1 is specific to each of the imaging modality given as
input, and level 2 AE is used to combine the unimodal representations into
multimodal features) followed by an LSTM network. It is proposed to encode the
volumetric nature of MRI sequences into a series of compressed features, each
feature vector representing an MRI slice. Each patient is therefore
represented by a series of multimodal feature series, which is later used by
an LSTM network to map them into the patient outcome. The training phase has
two main steps:

i) training of AEs for learning unimodal and multimodal features. In this step
which is shown in Figure 2 left, there are five AEs at level 1, one for each
MRI modality, and one AE at level 2 for combining them. AEs at level 1 are
responsible for learning unimodal features ($Z_{1-5}$) from MRI slices. These
feature vectors are compressed representations of each modality. The AE at
level 2 is receiving concatenated $Z_{1-5}$ features as input and output, and
trying to compress it further into the final multimodal features ($Z$). This
is the vector representing each slice from five modalities. This step can be
considered as unsupervised multimodal feature learning of MRI sequences. It is
important to note that the AEs can be trained in two manners: AEs viewed
isolated and independent from each other and minimize the reconstruction loss
functions separately (which is the case in this paper), or all AEs being
considered as one big model and minimize the losses simultaneously (e.g.
minimizing sum of the losses).

ii) training of LSTM for predicting mRS score. In this step which is shown in
Figure 2 right, a series of multimodal features for each patient are generated
by inferring the AE encoders. Afterwards, the LSTM network is trained in order
to map each sequential feature into a final mRS score.

In the testing phase and for each given patient, AE encoders are applied to
the MRI sequences in order to obtain the feature series $Z_{s}$, and the LSTM
network is used to predict the final mRS score (Figure 2 right).

In this research, we used five MR imaging modalities, i.e. Apparent Diffusion
Coefficient (ADC), Cerebral Blood Flow (CBF), Cerebral Blood Volume (CBV),
Diffusion-Weighted Imaging (DWI) and Time-to-Maximum (Tmax). However, the
model is flexible, and any number of modalities or other types of imaging
(e.g. CT) can be used.

##  4 Experiments

Table 1: Performance (mean $\pm$ std over 10 runs) of the proposed AE2-LSTM compared to different baselines. | Model | Input data | AUC | MAE | Accuracy | Specificity | Sensitivity | F1-score
---|---|---|---|---|---|---|---
Random Forest [2] | Clinical variables | 0.65$\pm$0.02 | 0.43$\pm$0.01 | 0.65$\pm$0.02 | 0.64$\pm$0.02 | 0.64$\pm$0.02 | 0.63$\pm$0.02
3D-CNN [16] | MRI modalities | 0.66$\pm$0.02 | 0.43$\pm$0.00 | 0.65$\pm$0.02 | 0.64$\pm$0.02 | 0.64$\pm$0.02 | 0.64$\pm$0.02
CNN-LSTM [6] | MRI modalities | 0.62$\pm$0.04 | 0.42$\pm$0.01 | 0.63$\pm$0.04 | 0.61$\pm$0.04 | 0.61$\pm$0.04 | 0.60$\pm$0.03
CNN-LSTM CL [7] | MRI modalities | 0.68$\pm$0.05 | 0.38$\pm$0.01 | 0.69$\pm$0.02 | 0.65$\pm$0.03 | 0.66$\pm$0.03 | 0.64$\pm$0.02
AE2-LSTM | MRI modalities | 0.71$\pm$0.03 | 0.34$\pm$0.01 | 0.72$\pm$0.03 | 0.67$\pm$0.03 | 0.67$\pm$0.03 | 0.68$\pm$0.03

###  4.1 Data Source and Preprocessing

Patients were included from the HIBISCUS-STROKE cohort [16, 6], which is an
ongoing monocentric observational cohort enrolling patients with an ischemic
stroke due to a proximal intracranial artery occlusion treated by
thrombectomy. In total 119 patients were analyzed. Inclusion criteria were:
(1) patients with an anterior circulation stroke related to a proximal
intracranial occlusion; (2) diffusion and perfusion MRI as baseline imaging;
(3) patients treated by thrombectomy with or without intravenous thrombolysis.

All patients underwent the following protocol (IRB number:00009118) at
admission: DWI, and dynamic susceptibility-contrast perfusion imaging (DSC-
PWI). Final clinical outcome was assessed at 3-month during a face-to-face
follow-up visit using the mRS. The distribution of the final mRS scores and
its associated binarization into poor and good outcomes are shown in Figure 3.
In this paper, we used the binarized mRS for classifying patients’ outcomes.

Parametric maps were extracted from the DSC-PWI by circular singular value
decomposition of the tissue concentration curves (Olea Sphere, Olea Medical,
France): CBF, CBV and Tmax. DSC-PWI parametric maps were coregistered within
subjects to DWI using linear registration with Ants [17] and all MRI slices
were of size $192\times 192$. The skull from all patients was removed using
FSL [18]. Finally, images were normalized between 0 and 1 to ensure inter-
patient standardization.

Fig. 3: Top) Distribution of the 3-month mRS scores, bottom) associated
binarization into good {0,1,2} and poor outcome {3,4,5,6}. 66% vs. 34% good
and poor outcomes respectively within our cohort.

###  4.2 Analysis and Results

Six different measures are used to evaluate the performance of the models:
classification accuracy (recognition rate), F1 score, sensitivity,
specificity, Mean-Absolute-Error (MAE) and the Area Under the Curve (AUC). Ten
independent runs with random seeds are performed, and means and standard
deviations of the measures are reported.

To compare our results, we proposed four baseline models. First baseline is a
RF classifier inspired by Ramos et al. [2]. The inputs for the RF are the
following clinical data: NIHSS baseline, age, door-to-puncture time and
Fazekas scale. Second baseline is inspired by [16, 4] where the input MR
images are identical to our proposed model. It is an early fusion 3D-CNNs
model, where the five MR modalities are concatenated and used as an unique
input. In order to represent the architecture, we use $C_{(size)}$ and
$F_{(size)}$ where $C$ is a 3D convolution ($3\times 3\times 3$) followed by a
ReLU activation function, a batch-normalization and 3D max-pooling layers and
$F$ is FC. Therefore, the CNN architecture is
$C_{8}$-$C_{16}$-$C_{32}$-$C_{64}$-$C_{128}$-$C_{256}$-$F_{100}$. The third
and fourth baselines are two similar CNN-LSTM models, one is trained in the
standard Stochastic Gradient Descent (SGD) manner [6] and the other using
Curriculum Learning (CL) strategy [7]. In SGD mini-batches are randomly
sampled from the training set to feed into the learner to update its weights
at each training iteration, while CL organizes the mini-batches according to
an ascending level of difficulty.

We carried out experiments in PyTorch. Five independent stacked AEs are
trained, one for each MRI modality, to learn the module-specific features and
another AE for learning multimodal features from the unimodal features. For
simplicity, the architecture and parameters of the AEs are chosen similar and
set as followings: _Max Epochs:400_ , _L2 Weight Regularization:0.004_ ,
_Sparsity Regularization:4_ , _Sparsity Proportion:0.05_. In order to obtain
the best results, different _feature sizes:{200,500,1000,2000}_ were tried.
The LSTM architecture consists of a sequence of input layers, two LSTM layers,
a FC and a sigmoid layer with the half-mean-squared-error loss. In order to
obtain the best results, Adam and SGD optimizers with learning rate of $1e-4$
were tried with a maximum number of epochs of $1000$ with early stopping and
batch size of $32$. In our experiments, we explored different hidden nodes for
LSTM $nh:\\{50,100,500,1000\\}$ and the best $nh:500$ was selected for optimal
feature size of 1000 based on 5-fold cross-validation performance.

Table 1 reports the performance of the proposed AE2-LSTM compared to different
baselines and state-of-the-art models. The AE2-LSTM model outperforms the
other models in all of the six measures with AUC=0.71, MAE=0.34, Accuracy=72%,
Specificity=0.67, Sensitivity=0.67 and F1 score=0.68.

##  5 Conclusions and Future Work

A novel deep learning based architecture is proposed for predicting stroke
patient outcome. The proposed AE2-LSTM model outperformed the state-of-the-art
models showing that it is a better fit for multimodal and volumetric data such
as MRI. The AEs in this model carefully learn the multimodal features from
different image modalities and the LSTM learns the sequential information in
MRI sequences and map it to the final mRS score.

As for future research, the proposed AE2-LSTM model should be further
validated on larger multi-center cohorts. Effects of different AE types and
different imaging modalities should also be investigated. Adapting the model
in order to use the clinical and biological variables together with the
imaging modalities is another important future direction.

##  6 Acknowledgments

This work was supported by the RHU MARVELOUS (ANR-16-RHUS-0009) of Universite
Claude Bernard Lyon-1 (UCBL) and by the RHU BOOSTER (ANR-18-RHUS-0001), within
the program ”Investissements d’Avenir” operated by the French National
Research Agency (ANR).

##  7 Compliance with Ethical Standards

This study followed the principles of the Declaration of Helsinki, and was
approved by the local ethics committee (IRB number: 00009118, September 8th,
2016). All subjects or their relatives signed an informed consent form.

## References

  * [1] V. Feigin et al.,  “Global, regional, and national burden of stroke and its risk factors, 1990–2019: A systematic analysis for the global burden of disease study 2019,”  The Lancet Neurology, vol. 20, no. 10, pp. 795–820, 2021.
  * [2] L. A. Ramos et al.,  “Predicting poor outcome before endovascular treatment in patients with acute ischemic stroke,”  Frontiers in neurology, vol. 11, pp. 1215, 2020.
  * [3] S. Bacchi et al.,  “Deep learning in the prediction of ischaemic stroke thrombolysis functional outcomes: a pilot study,”  Academic radiology, vol. 27, no. 2, pp. 19–23, 2020.
  * [4] E. Zihni et al.,  “Multimodal fusion strategies for outcome prediction in stroke,”  HEALTHINF, pp. 421–428, 2020.
  * [5] H. Nishi et al.,  “Deep learning–derived high-level neuroimaging features predict clinical outcomes for large vessel occlusion,”  Stroke, vol. 51, no. 5, pp. 1484–1492, 2020.
  * [6] N. Hatami et al.,  “CNN-LSTM based multimodal MRI and clinical data fusion for predicting functional outcome in stroke patients,”  44th IEEE Engineering in Medicine and Biology Conference (EMBC), 2022.
  * [7] N. Hatami et al.,  “Patient rank-based curriculum learning for predicting functional outcome in stroke after thrombectomy,”  8th European Stroke Organisation Conference (ESOC), 2022.
  * [8] O. Russakovsky et al.,  “ImageNet Large Scale Visual Recognition Challenge,”  International Journal of Computer Vision (IJCV), vol. 115, no. 3, pp. 211–252, 2015.
  * [9] K. He et al.,  “Deep residual learning for image recognition,”  CVPR, 2016.
  * [10] G. Huang et al.,  “Densely connected convolutional networks,”  CVPR, 2017.
  * [11] K. Simonyan and A. Zisserman,  “Very deep convolutional networks for large-scale image recognition,”  arXiv preprint arXiv:1409.1556, 2014.
  * [12] I. Goodfellow et al.,  Deep learning,  MIT press, 2016.
  * [13] P. Vincent et al.,  “Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion.,”  Journal of machine learning research, vol. 11, no. 12, 2010.
  * [14] D. Kingma and M. Welling,  “An introduction to variational autoencoders,”  Foundations and Trends in Machine Learning, vol. 12, no. 4, pp. 307–392, 2019.
  * [15] S. Hochreiter and J. Schmidhuber,  “Long short-term memory,”  Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
  * [16] N. Debs et al.,  “Impact of the reperfusion status for predicting the final stroke infarct using deep learning,”  NeuroImage: Clinical, vol. 29, pp. 102548, 2021.
  * [17] B. Avants et al.,  “A reproducible evaluation of ants similarity metric performance in brain image registration,”  Neuroimage, vol. 54, no. 3, pp. 2033–2044, 2011.
  * [18] S. Smith et al.,  “Fsl: New tools for functional and structural brain image analysis,”  NeuroImage, vol. 6, no. 13, pp. 249, 2001.