#  SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM
spla-tam . github . io

Nikhil Keetha1, Jay Karhade1, Krishna Murthy Jatavallabhula2, Gengshan Yang1,
Sebastian Scherer1, Deva Ramanan1, and Jonathon Luiten1
1CMU 2MIT

###### Abstract

Dense simultaneous localization and mapping (SLAM) is pivotal for embodied
scene understanding. Recent work has shown that 3D Gaussians enable high-
quality reconstruction and real-time rendering of scenes using multiple posed
cameras. In this light, we show for the first time that representing a scene
by $3D$ Gaussians can enable dense SLAM using a single unposed monocular RGB-D
camera. Our method, SplaTAM, addresses the limitations of prior radiance
field-based representations, including fast rendering and optimization, the
ability to determine if areas have been previously mapped, and structured map
expansion by adding more Gaussians. We employ an online tracking and mapping
pipeline while tailoring it to specifically use an underlying Gaussian
representation and silhouette-guided optimization via differentiable
rendering. Extensive experiments show that SplaTAM achieves up to $2\times$
state-of-the-art performance in camera pose estimation, map construction, and
novel-view synthesis, demonstrating its superiority over existing approaches,
while allowing real-time rendering of a high-resolution dense 3D map.

Figure 1: SplaTAM enables _precise camera tracking_ and _high-fidelity
reconstruction_ for dense simultaneous localization and mapping (SLAM) in
challenging real-world scenarios. SplaTAM achieves this by online optimization
of an explicit volumetric representation, $3$D Gaussian Splatting [14], using
differentiable rendering. Left: We showcase the high-fidelity $3$D Gaussian
Map along with the train (SLAM-input) & novel view camera frustums. It can be
noticed that SplaTAM achieves sub-cm localization despite the large motion
between subsequent cameras in the texture-less environment. This is
particularly challenging for state-of-the-art baselines leading to the failure
of tracking. Right: SplaTAM enables _photo-realistic_ rendering of both train
& novel views at _400 FPS_ for a resolution of $876\times 584$.

##  1 Introduction

Visual simultaneous localization and mapping (SLAM)—the task of estimating the
pose of a vision sensor (such as a depth camera) and a map of the
environment—is an essential capability for vision or robotic systems to
operate in previously unseen 3D environments. For the past three decades, SLAM
research has extensively centered around the question of _map representation_
– resulting in a variety of sparse [2, 23, 3, 7], dense [8, 4, 34, 25, 26, 13,
15, 41, 42, 6], and neural scene representations [21, 37, 45, 54, 30, 29]. Map
representation is a fundamental choice that dramatically impacts the design of
every processing block within the SLAM system, as well as of the downstream
tasks that depend on the outputs of SLAM.

In terms of dense visual SLAM, the most successful _handcrafted_
representations are points, surfels/flats, and signed distance fields. While
systems based on such map representations have matured to production level
over the past years, there are still significant shortcomings that need to be
addressed. Tracking explicit representations relies crucially on the
availability of rich 3D geometric features and high-framerate captures.
Furthermore, these approaches only reliably explain the observed parts of the
scene; many applications, such as mixed reality and high-fidelity 3D capture,
require techniques that can also explain/synthesize unobserved/novel camera
viewpoints [32].

The shortcomings of _handcrafted_ representations, combined with the emergence
of radiance field representations [21] for high-quality image synthesis, have
fueled a recent class of methods that attempt to encode the scene into the
weight space of a neural network. These _radiance-field_ -based SLAM
algorithms [53, 30] benefit from high-fidelity global maps and image
reconstruction losses that capture dense photometric information via
differentiable rendering. However, current methods use implicit neural
representations to model the volumetric radiance fields, which causes a number
of issues in the SLAM setting - they are computationally inefficient, not easy
to edit, do not model spatial geometry explicitly, and suffer from
catastrophic forgetting.

In this context, we explore the question, “How can one use an explicit
volumetric representation to design a SLAM solution?” Specifically, we use a
radiance field based on 3D Gaussians [14] to _Splat (Render)_ , _Track_ , and
_Map_ for SLAM. We believe that this representation has the following benefits
over existing map representations:

  * •

Fast rendering and rich optimization: Gaussian Splatting can be rendered at
speeds up to $400$ FPS, making it significantly faster to visualize and
optimize than implicit alternatives. The key enabling factor for fast
optimization is the rasterization of 3D primitives. We introduce several
simple modifications that make splatting even faster for SLAM, including the
removal of view-dependent appearance and the use of isotropic Gaussians.
Furthermore, this allows us to use dense photometric loss for SLAM in real-
time, in contrast to traditional & implicit map representations that rely
respectively on sparse 3D geometric features or pixel-sampling to maintain
efficiency.

  * •

Maps with explicit spatial extent: The spatial frontier of the existing map
can be easily controlled by adding Gaussians only in parts of the scene that
have been observed in the past. Given a new image frame, this allows one to
efficiently identify which portions of the scene are new content (outside the
map’s spatial frontier) by rendering a silhouette. This is crucial for camera
tracking as we only want to compare mapped areas of the scene to new images.
However, this is difficult for implicit map representations, as the network is
subject to global changes during gradient-based optimization for the unmapped
space.

  * •

Explicit map: We can arbitrarily increase the map capacity by simply adding
more Gaussians. Furthermore, this explicit volumetric representation enables
us to edit parts of the scene, while still allowing photo-realistic rendering.
Implicit approaches cannot easily increase their capacity or edit their
represented scene.

  * •

Direct gradient flow to parameters: As the scene is represented by Gaussians
with physical 3D locations, colors, and sizes, there is a direct, almost
linear (projective) gradient flow between the parameters and the rendering.
Because camera motion can be thought of as keeping the camera still and moving
the scene, we also have a direct gradient into the camera parameters, which
enables fast optimization. Neural-based representations don’t have this as the
gradient needs to flow through (potentially many) non-linear neural network
layers.

Given all of the above advantages, an explicit volumetric representation is a
natural way for efficiently inferring a high-fidelity spatial map while
simultaneously estimating camera motion, as is also visible from SplaTAM:
Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM  spla-tam . github . io.
We show across all our experiments on simulated and real data that our
approach, SplaTAM, achieves state-of-the-art results compared to all previous
approaches for camera pose estimation, map estimation, and novel-view
synthesis.

Figure 2: Overview of SplaTAM. Top-Left: The input to our approach at each
timestep is the current RGB-D frame and the 3D Gaussian Map Representation
that we have built representing all of the scene seen so far. Top-Right: Step
(1) estimates the camera pose for the new image, using silhouette-guided
differentiable rendering. Bottom-Right: Step (2) increases the map capacity by
adding new Gaussians based on the rendered silhouette and input depth. Bottom-
Left: Step (3) updates the underlying Gaussian map with differentiable
rendering.

##  2 Related Work

In this section, we briefly review various approaches to dense SLAM, with a
particular emphasis on recent approaches that leverage implicit
representations encoded in overfit neural networks for tracking and mapping.
For a more detailed review of traditional SLAM methods, we refer the
interested reader to this excellent survey [2].

Traditional approaches to dense SLAM have explored a variety of explicit
representations, including 2.5D images [34, 15], (truncated) signed-distance
functions [25, 41, 26, 6], gaussian mixture models [9, 10] and circular
surfels (flats) [13, 42, 33]. Of particular relevance to this work is point-
based fusion by Keller  _et al_. [13], which uses a _flat_ as the atomic unit
in its map representation. Flats, or _surfels_ , are colored circular surface
elements that are optimized in real-time from RGB-D image inputs. Each surfel
is capable of encoding multiple points on a surface, allowing for a more
compact map representation. Surfels may be easily rasterized, as leveraged in
[42, 33] for tracking camera poses via differentiable rasterization. While the
aforementioned SLAM approaches do not assume the visibility function to be
differentiable, there exist modern differentiable rasterizers that enable
gradient flow through depth discontinuities [49]. In this work, we use
volumetric (as opposed to surface-only) scene representations in the form of
3D Gaussians, which enable fast and accurate tracking and mapping.

Pretrained neural network representations have been integrated with
traditional SLAM techniques, largely focusing on predicting depth from RGB
images. These approaches range from directly integrating depth predictions
from a neural network into SLAM [38], to learning a variational autoencoder
that decodes compact optimizable latent codes to depth maps [1, 51], to
approaches that simultaneously learn to predict a depth cost volume and
tracking [52].

Implicit scene representations: iMAP [37] first performed both tracking and
mapping using neural implicit representations. To improve scalability, NICE-
SLAM [53] proposed the use of hierarchical multi-feature grids. On similar
lines, iSDF [28] used implicit representations to efficiently compute signed-
distances, compared to [25, 27]. Following this, a number of works [54, 40,
19, 22, 31, 50, 11, 18, 29], have recently advanced implicit-based SLAM
through a number of ways - by reducing catastrophic forgetting via continual
learning(experience replay), capturing semantics, incorporating uncertainty,
employing efficient resolution hash-grids and encodings, and using improved
losses. More recently, Point-SLAM [30], proposes an alternative route, similar
to [44] by using neural point clouds and performing volumetric rendering with
feature interpolation, offering better 3-D reconstruction, especially for
robotics. However, like other implicit representations, volumetric ray
sampling greatly limits its efficiency, thereby resorting to optimization over
a sparse set of pixels as opposed to per-pixel dense photometric error.
Instead, SplaTAM’s explicit volumetric radiance model leverages fast
rasterization, enabling complete use of per-pixel dense photometric errors.

3D Gaussian Splatting: Recently, 3D Gaussians have emerged as a promising 3D
scene representation [14, 16, 39, 17], in particular with the ability to
differentiably render them extremely quickly via splatting [14]. This approach
has also been extended to model dynamic scenes [20, 43, 46, 47] with dense
6-DOF motion [20]. Such approaches for both static and dynamic scenes require
that each input frame has an accurately known 6-DOF camera pose, in order to
successfully optimize the representation. For the first time our approach
removes this constraint, simultaneously estimating the camera poses while also
fitting the underlying Gaussian representation.

##  3 Method

SplaTAM is the first dense RGB-D SLAM solution to use $3$D Gaussian Splatting
[14]. By modeling the world as a collection of 3D Gaussians which can be
rendered into high-fidelity color and depth images, we are able to directly
use differentiable-rendering and gradient-based-optimization to optimize both
the camera pose for each frame and an underlying volumetric discretized map of
the world.

### Gaussian Map Representation.

We represent the underlying map of the scene as a set of 3D Gaussians. We made
a number of simplifications to the representation proposed in [14], by using
only view-independent color and forcing Gaussians to be isotropic. This
implies each Gaussian is parameterized by only $8$ values: three for its RGB
color ${\bf c}$, three for its center position $\bm{\mu}\in\mathbb{R}^{3}$,
one for its radius $r$ and one for its opacity $o\in[0,1]$. Each Gaussian
influences a point in 3D space $\mathbf{x}\in\mathbb{R}^{3}$ according to the
standard (unnormalized) Gaussian equation weighted by its opacity:

$\displaystyle
f(\mathbf{x})=o\exp\left(-\frac{\|\mathbf{x}-\bm{\mu}\|^{2}}{2r^{2}}\right).$
(1)

### Differentiable Rendering via Splatting.

The core of our approach is the ability to render high-fidelity color, depth,
and silhouette images from our underlying Gaussian Map into any possible
camera reference frame in a differentiable way. This differentiable rendering
allows us to directly calculate the gradients in the underlying scene
representation (Gaussians) and camera parameters with respect to the error
between the renders and provided RGB-D frames, and update both the Gaussians
and camera parameters to minimize this error, thus fitting both accurate
camera poses and an accurate volumetric representation of the world.

Gaussian Splatting [14] renders an RGB image as follows: Given a collection of
3D Gaussians and camera pose, first sort all Gaussians from front-to-back. RGB
images can then be efficiently rendered by alpha-compositing the splatted 2D
projection of each Gaussian in order in pixel space. The rendered color of
pixel ${\bf p}=(u,v)$ can be written as:

$C(\mathbf{p})=\sum_{i=1}^{n}\mathbf{c}_{i}f_{i}(\mathbf{p})\prod_{j=1}^{i-1}(1-f_{j}(\mathbf{p})),$
(2)

where $f_{i}(\bf{p})$ is computed as in Eq. (1) but with $\bm{\mu}$ and $r$ of
the splatted 2D Gaussians in pixel-space:

$\bm{\mu}^{\textrm{2D}}=K\frac{E_{t}\bm{\mu}}{d},\;\;\;\;\;\;\;\;\;\;r^{\textrm{2D}}=\frac{fr}{d},\quad\text{where}\quad
d=(E_{t}\bm{\mu})_{z}.$ (3)

Here, $K$ is the (known) camera intrinsic matrix, $E_{t}$ is the extrinsic
matrix capturing the rotation and translation of the camera at frame $t$, $f$
is the (known) focal length, and $d$ is the depth of the $i^{th}$ Gaussian in
camera coordinates.

We propose to similarly differentiably render depth:

$D(\mathbf{p})=\sum_{i=1}^{n}d_{i}f_{i}(\mathbf{p})\prod_{j=1}^{i-1}(1-f_{j}(\mathbf{p})),$
(4)

which can be compared against the input depth map and return gradients with
respect to the 3D map.

We also render a silhouette image to determine visibility – e.g. if a pixel
contains information from the current map:

$S(\mathbf{p})=\sum_{i=1}^{n}f_{i}(\mathbf{p})\prod_{j=1}^{i-1}(1-f_{j}(\mathbf{p})).$
(5)

### SLAM System.

We build a SLAM system from our Gaussian representation and differentiable
renderer. We begin with a brief overview and then describe each module in
detail. Assume we have an existing map (represented via a set of 3D Gaussians)
that has been fitted from a set of camera frames 1 to t. Given a new RGB-D
frame $t+1$, our SLAM system performs the following steps (see Fig. 2):

  1. (1)

Camera Tracking. We minimize the image and depth reconstruction error of the
RGB-D frame with respect to camera pose parameters for $t+1$, but only
evaluate errors over pixels within the visible silhouette.

  2. (2)

Gaussian Densification. We add new Gaussians to the map based on the rendered
silhouette and input depth.

  3. (3)

Map Update. Given the camera poses from frame 1 to $t+1$, we update the
parameters of all the Gaussians in the scene by minimizing the RGB and depth
errors over all images up to $t+1$. In practice, to keep the batch size
manageable, a selected subset of keyframes that overlap with the most recent
frame are optimized.

### Initialization.

For the first frame, the tracking step is skipped, and the camera pose is set
to identity. In the densification step, since the rendered silhouette is
empty, all pixels are used to initialize new Gaussians. Specifically, for each
pixel, we add a new Gaussian with the color of that pixel, the center at the
location of the unprojection of that pixel depth, an opacity of $0.5$, and a
radius equal to having a one-pixel radius upon projection into the 2D image
given by dividing the depth by the focal length:

$\displaystyle r=\frac{D_{\textrm{GT}}}{f}$ (6)

### Camera Tracking.

Camera Tracking aims to estimate the camera pose of the current incoming
online RGB-D image. The camera pose is initialized for a new timestep by a
constant velocity forward projection of the pose parameters in the camera
center + quaternion space. E.g. the camera parameters are initialized using
the following:

$\displaystyle E_{t+1}=E_{t}+(E_{t}-E_{t\text{-}1})$ (7)

The camera pose is then updated iteratively by gradient-based optimization
through differentiably rendering RGB, depth, and silhouette maps, and updating
the camera parameters to minimize the following loss while keeping the
Gaussian parameters fixed:

$\displaystyle
L_{\textrm{t}}=\sum_{\textbf{p}}\Bigl{(}S(\textbf{p})>0.99\Bigr{)}\Bigl{(}\textrm{L}_{1}\bigl{(}D(\textbf{p})\bigr{)}+0.5\textrm{L}_{1}\bigl{(}C(\textbf{p})\bigr{)}\Bigr{)}$
(8)

This is an L1 loss on both the depth and color renders, with color weighted
down by half. We only apply the loss over pixels that are rendered from well-
optimized parts of the map by using our rendered visibility _silhouette_ which
captures the epistemic uncertainty of the map. This is very important for
tracking new camera poses, as often new frames contain new information that
hasn’t been captured or well optimized in our map yet. The L1 loss also gives
a value of 0 if there is no ground-truth depth for a pixel.

### Gaussian Densification.

Gaussian Densification aims to initialize new Gaussians in our Map at each
incoming online frame. After tracking we have an accurate estimate for the
camera pose for this frame, and with a depth image we have a good estimate for
where Gaussians in the scene should be. However, we don’t want to add
Gaussians where the current Gaussians already accurately represent the scene
geometry. Thus we create a densification mask to determine which pixels should
be densified:

$M(\textbf{p})=\Bigl{(}S(\textbf{p})<0.5\Bigr{)}+\\\
\Bigl{(}D_{\textrm{GT}}(\textbf{p})<D(\textbf{p})\Bigr{)}\Bigl{(}\textrm{L}_{1}\bigl{(}D(\textbf{p})\bigr{)}>50\textrm{MDE}\Bigr{)}$
(9)

This mask indicates where the map isn’t adequately dense ($S<0.5$), or where
there should be new geometry in front of the current estimated geometry (the
ground-truth depth is in front of the predicted depth, and the depth error is
greater than 50 times the median depth error (MDE)). For each pixel, based on
this mask, we add a new Gaussian following the same procedure for first-frame
initialization.

### Gaussian Map Updating.

This aims to update the parameters of the 3D Gaussian Map given the set of
online camera poses estimated so far. This is done again by differentiable-
rendering and gradient-based-optimization, however unlike tracking, in this
setting the camera poses are fixed, and the parameters of the Gaussians are
updated.

This is equivalent to the “classic” problem of fitting a radiance field to
images with known poses. However, we make two important modifications. Instead
of starting from scratch, we warm-start the optimization from the most
recently constructed map. We also do not optimize over all previous
(key)frames but select frames that are likely to influence the newly added
Gaussians. We save each $n^{\text{th}}$ frame as a keyframe and select $k$
frames to optimize, including the current frame, the most recent keyframe, and
$k-2$ previous keyframes which have the highest overlap with the current
frame. Overlap is determined by taking the point cloud of the current frame
depth map and determining the number of points inside the frustum of each
keyframe.

This phase optimizes a similar loss as during tracking, except we don’t use
the silhouette mask as we want to optimize over all pixels. Furthermore, we
add an SSIM loss to RGB rendering and cull useless Gaussians that have near 0
opacity or are too large as done in [14].

Figure 3: Visualization of the reconstruction, estimated and ground truth
camera poses on ScanNet++ [48] S2. It can be observed that the estimated poses
from SplaTAM (green frustums & red trajectory) precisely align with the ground
truth (blue frustums & trajectory) while providing a high-fidelity
reconstruction.

&

Methods | Avg. | R0 | R1 | R2 | Of0 | Of1 | Of2 | Of3 | Of4
---|---|---|---|---|---|---|---|---|---
Vox-Fusion [45] | 3.09 | 1.37 | 4.70 | 1.47 | 8.48 | 2.04 | 2.58 | 1.11 | 2.94
NICE-SLAM [53] | 1.06 | 0.97 | 1.31 | 1.07 | 0.88 | 1.00 | 1.06 | 1.10 | 1.13
ESLAM [12] | 0.63 | 0.71 | 0.70 | 0.52 | 0.57 | 0.55 | 0.58 | 0.72 | 0.63
Point-SLAM [30] | 0.52 | 0.61 | 0.41 | 0.37 | 0.38 | 0.48 | 0.54 | 0.69 | 0.72
SplaTAM | 0.36 | 0.31 | 0.40 | 0.29 | 0.47 | 0.27 | 0.29 | 0.32 | 0.55
Replica [35]

Methods | Avg. | fr1/ | fr1/ | fr1/ | fr2/ | fr3/
---|---|---|---|---|---|---
desk | desk2 | room | xyz | off.
Kintinuous [41] | 4.84 | 3.70 | 7.10 | 7.50 | 2.90 | 3.00
ElasticFusion [42] | 6.91 | 2.53 | 6.83 | 21.49 | 1.17 | 2.52
ORB-SLAM2 [23] | 1.98 | 1.60 | 2.20 | 4.70 | 0.40 | 1.00
1-7 NICE-SLAM [53] | 15.87 | 4.26 | 4.99 | 34.49 | 31.73 | 3.87
Vox-Fusion [45] | 11.31 | 3.52 | 6.00 | 19.53 | 1.49 | 26.01
Point-SLAM [30] | 8.92 | 4.34 | 4.54 | 30.92 | 1.31 | 3.48
SplaTAM | 5.48 | 3.35 | 6.54 | 11.13 | 1.24 | 5.16
TUM-RGBD [36]

Methods | Avg. | 0000 | 0059 | 0106 | 0169 | 0181 | 0207
---|---|---|---|---|---|---|---
Vox-Fusion [45] | 26.90 | 68.84 | 24.18 | 8.41 | 27.28 | 23.30 | 9.41
NICE-SLAM [53] | 10.70 | 12.00 | 14.00 | 7.90 | 10.90 | 13.40 | 6.20
Point-SLAM [30] | 12.19 | 10.24 | 7.81 | 8.65 | 22.16 | 14.77 | 9.54
SplaTAM | 11.88 | 12.83 | 10.10 | 17.72 | 12.08 | 11.10 | 7.46
Orig-ScanNet [5]

Table 1:  Online Camera-Pose Estimation Results on Four Datasets (ATE RMSE
$\downarrow$[cm]). Our method consistently outperforms all the SOTA-dense
baselines on ScanNet++, Replica, and TUM-RGBD, while providing competitive
performance on Orig-ScanNet [5]. Best results are highlighted as first,
second, and third. Numbers for the baselines on Orig-ScanNet [5], TUM-RGBD
[36] and Replica [35] are taken from Point-SLAM [30].

##  4 Experimental Setup

### Datasets and Evaluation Settings.

We evaluate our approach on four datasets: ScanNet++ [48], Replica [35], TUM-
RGBD [36] and the original ScanNet [5]. The last three are chosen in order to
follow the evaluation procedure of previous radiance-field-based SLAM methods
Point-SLAM [30] and NICE-SLAM [53]. However, we also add ScanNet++ [48]
evaluation because none of the other three benchmarks have the ability to
evaluate rendering quality on hold out novel views and only evaluate camera
pose estimation and rendering on the training views.

Replica [35] is the simplest benchmark as it contains synthetic scenes, highly
accurate and complete (synthetic) depth maps, and small displacements between
consecutive camera poses. TUM-RGBD [36] and the original ScanNet are harder,
especially for dense methods, due to poor RGB and depth image quality as they
both use old low-quality cameras. Depth images are quite sparse with lots of
missing information, and the color images have a very high amount of motion
blur. For ScanNet++ [48] we use the DSLR captures from two scenes (8b5caf3398
(S1) and b20a261fdf (S2)), where complete dense trajectories are present. In
contrast to other benchmarks, ScanNet++ color and depth images are very high
quality, and provide a second capture loop for each scene to evaluate
completely novel hold-out views. However, each camera pose is very far apart
from one another making pose-estimation very difficult. The difference between
consecutive frames on ScanNet++ is about the same as a 30-frame gap on
Replica. For all benchmarks except ScanNet++, we take baseline numbers from
Point-SLAM [30]. Similar to Point-SLAM, we evaluate every $5$th frame for the
training view rendering benchmarking on Replica. Furthermore, for all
comparisons to prior baselines, we present results as the average of $3$ seeds
(0-2) and use seed 0 for the ablations.

### Evaluation Metrics.

We follow all evaluation metrics for camera pose estimation and rendering
performance as [30]. For measuring RGB rendering performance we use PSNR, SSIM
and LPIPS. For depth rendering performance we use Depth L1 loss. For camera
pose estimation tracking we use the average absolute trajectory error (ATE
RMSE).

### Baselines.

The main baseline method we compare to is Point-SLAM [30] as it is the
previous state-of-the-art (SOTA) method for dense radiance-field-based SLAM.
We also compare to older dense SLAM approaches such as NICE-SLAM [53], Vox-
Fusion [45], and ESLAM [12] where appropriate. On TUM-RGBD, we also compare
against three traditional SLAM systems Kintinuous [41], ElasticFusion [42],
and ORB-SLAM2 [23]. On the difficult ScanNet++, in addition to Point-SLAM, we
also evaluate ORB-SLAM3 [3] as a representative feature-based traditional SLAM
system.

##  5 Results & Discussion

In this section, we first discuss our evaluation results on camera pose
estimation for the four benchmark datasets. Then, we further showcase our
high-fidelity $3$D Gaussian reconstructions and provide qualitative and
quantitative comparisons of our rendering quality (both for novel views and
input training views). Finally, we discuss pipeline ablations and provide a
runtime comparison.

### Camera Pose Estimation Results.

In Table 3, we compare our method’s camera pose estimation results to a number
of baselines on four datasets [5, 48, 35, 36].

On ScanNet++ [48], both SOTA SLAM approaches Point-SLAM [30] and ORB-SLAM3 [3]
(RGB-D variant) completely fail to correctly track the camera pose due to the
very large displacement between contiguous cameras, and thus give very large
pose-estimation errors. In particular, for ORB-SLAM3, we observe that the
texture-less ScanNet++ scans cause the tracking to re-initialize multiple
times due to the lack of features. In contrast, our approach successfully
manages to track the camera over both sequences giving an average trajectory
error of only 1.2cm.

On the relatively easy synthetic Replica [35] dataset, the previous de-facto
evaluation benchmark, our approach reduces the trajectory error over the
previous SOTA [30] by more than $30\%$ from $0.52$cm to $0.36$cm.

On TUM-RGBD [36], all the volumetric methods struggle immensely due to both
poor depth sensor information (very sparse) and poor RGB image quality
(extremely high motion blur). However, compared to previous methods in this
category [30, 53] our approach still significantly outperforms, decreasing the
trajectory error of the previous SOTA in this category [30] by almost $40\%$,
from $8.92$cm to $5.48$cm. However, on this benchmark, feature-based sparse
tracking methods such as ORB-SLAM2 [23] still outperform dense methods.

The original ScanNet [5] benchmark has similar issues to TUM-RGBD and as such
no dense volumetric method is able to obtain results with less than $10$cm
trajectory error. On this benchmark, our approach performs similarly to the
two prior SOTA approaches [30, 53].

Overall these camera pose estimation results are extremely promising and show
off the strengths of our SplaTAM method. The results on ScanNet++ show that if
you have high-quality clean input images, our approach can successfully and
accurately perform SLAM even with extremely large motions between camera
positions, which is not something that is possible with previous SOTA
approaches such as Point-SLAM [30] and ORB-SLAM3 [3].

### Visualization of Gaussian Map Reconstruction and Estimated Camera Poses.

In Fig. 3, we show visual results of our reconstructed Gaussian Map on the two
sequences from ScanNet++. As can be seen, these reconstructions are incredibly
high quality both in terms of their geometry and their visual appearance. This
is one of the major benefits of using a 3D Gaussian Splatting-based map
representation.

We also show the camera trajectories and camera pose frustums estimated by our
method for these two sequences overlaid on the map. One can easily see the
large displacement often occurring between sequential camera poses, making
this a very difficult SLAM benchmark, and yet one that our approach manages to
solve extremely accurately.

Methods | Metrics | Avg. | R0 | R1 | R2 | Of0 | Of1 | Of2 | Of3 | Of4
---|---|---|---|---|---|---|---|---|---|---
Vox-Fusion [45] |  PSNR $\uparrow$ | 24.41 | 22.39 | 22.36 | 23.92 | 27.79 | 29.83 | 20.33 | 23.47 | 25.21
SSIM $\uparrow$ | 0.80 | 0.68 | 0.75 | 0.80 | 0.86 | 0.88 | 0.79 | 0.80 | 0.85
LPIPS $\downarrow$ | 0.24 | 0.30 | 0.27 | 0.23 | 0.24 | 0.18 | 0.24 | 0.21 | 0.20
1-11 NICE-SLAM [53] |  PSNR $\uparrow$ | 24.42 | 22.12 | 22.47 | 24.52 | 29.07 | 30.34 | 19.66 | 22.23 | 24.94
SSIM $\uparrow$ | 0.81 | 0.69 | 0.76 | 0.81 | 0.87 | 0.89 | 0.80 | 0.80 | 0.86
LPIPS $\downarrow$ | 0.23 | 0.33 | 0.27 | 0.21 | 0.23 | 0.18 | 0.24 | 0.21 | 0.20
1-11 Point-SLAM [30] |  PSNR $\uparrow$ | 35.17 | 32.40 | 34.08 | 35.50 | 38.26 | 39.16 | 33.99 | 33.48 | 33.49
SSIM $\uparrow$ | 0.98 | 0.97 | 0.98 | 0.98 | 0.98 | 0.99 | 0.96 | 0.96 | 0.98
LPIPS $\downarrow$ | 0.12 | 0.11 | 0.12 | 0.11 | 0.10 | 0.12 | 0.16 | 0.13 | 0.14
1-11 SplaTAM |  PSNR $\uparrow$ | 34.11 | 32.86 | 33.89 | 35.25 | 38.26 | 39.17 | 31.97 | 29.70 | 31.81
SSIM $\uparrow$ | 0.97 | 0.98 | 0.97 | 0.98 | 0.98 | 0.98 | 0.97 | 0.95 | 0.95
LPIPS $\downarrow$ | 0.10 | 0.07 | 0.10 | 0.08 | 0.09 | 0.09 | 0.10 | 0.12 | 0.15

Table 2: Quantitative Train View Rendering Performance on Replica [35].
SplaTAM is comparable to the SOTA baseline, Point-SLAM [30] and consistently
outperforms the other dense SLAM methods by a large margin. Numbers for the
baselines are taken from Point-SLAM [30]. Note that Point-SLAM uses ground-
truth depth for rendering.

### Rendering Quality Results.

In Table 5, we evaluate our method’s rendering quality on the input views of
Replica dataset [35] (as is the standard evaluation approach from Point-SLAM
[30] and NICE-SLAM [53]). Our approach achieves similar PSNR, SSIM, and LPIPS
results as Point-SLAM [30], although the comparison is not fair as Point-SLAM
has the unfair advantage as it takes the ground-truth depth of these images as
an input for where to sample its 3D volume for rendering. Our approach
achieves much better results than the other baselines Vox-Fusion [45] and
NICE-SLAM [53], improving over both by around $10$dB in PSNR.

In general, we believe that the rendering results on Replica in Table 5 are
irrelevant because the rendering performance is evaluated on the same training
views that were passed in as input, and methods can simply have a high
capacity and overfit to these images. We only show this as this is the de-
facto evaluation for prior methods, and we wish to have some way to compare
against them.

Hence, a better evaluation is to evaluate novel-view rendering. However, all
current SLAM benchmarks don’t have a hold-out set of images separate from the
camera trajectory that the SLAM algorithm estimates, and so they cannot be
used for this purpose. Therefore, we set up a novel benchmark for this using
the new high-quality ScanNet++ dataset.

The results for both novel-view and training-view rendering on this ScanNet++
benchmark can be found in Table 5. Our approach achieves a good novel-view
synthesis result of $24.41$ PSNR on average and slightly higher on training
views with $27.98$ PSNR. Note that this is while estimating the unknown camera
poses, i.e., completely unposed novel-view synthesis results. Since Point-SLAM
[30] fails to successfully estimate the camera poses, it also completely fails
on the task of novel-view synthesis.

We can also evaluate the geometric reconstruction of the scene by evaluating
the rendered depth and comparing it to the ground-truth depth, again for both
training and novel views. Our method obtains an incredibly accurate
reconstruction with a depth error of only around $2$cm in novel views and
$1.3$cm in training views. These excellent reconstruction results can also be
seen in Fig. 3.

Visual results of both novel-view and training-view rendering for RGB and
depth can be seen in Fig. 4. As can be seen, our methods achieve visually
excellent results over both scenes for both novel and training views. In
contrast, Point-SLAM [30] fails at camera-pose tracking and overfits to the
training views, and isn’t able to successfully render novel views at all.
Point-SLAM also takes the ground-truth depth as an input for rendering to
determine where to sample, and as such the depth maps look similar to the
ground-truth, while color rendering is completely wrong.

With this novel benchmark that is able to correctly evaluate Novel-View
Synthesis and SLAM simultaneously, as well as our approach as a strong initial
baseline for this benchmark, we hope to inspire many future approaches that
improve upon these results for both tasks.

Methods | Metrics | Novel View | Training View
---|---|---|---
Avg. | S1 | S2 | Avg. | S1 | S2
Point-SLAM [30] |  PSNR [dB] $\uparrow$ | 11.91 | 12.10 | 11.73 | 14.46 | 14.62 | 14.30
SSIM $\uparrow$ | 0.28 | 0.31 | 0.26 | 0.38 | 0.35 | 0.41
LPIPS $\downarrow$ | 0.68 | 0.62 | 0.74 | 0.65 | 0.68 | 0.62
Depth L1 [cm] $\downarrow$ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗
1-8 SplaTAM |  PSNR [dB] $\uparrow$ | 24.41 | 23.99 | 24.84 | 27.98 | 27.82 | 28.14
SSIM $\uparrow$ | 0.88 | 0.88 | 0.87 | 0.94 | 0.94 | 0.94
LPIPS $\downarrow$ | 0.24 | 0.21 | 0.26 | 0.12 | 0.12 | 0.13
Depth L1 [cm] $\downarrow$ | 2.07 | 1.91 | 2.23 | 1.28 | 0.93 | 1.64

Table 3: Novel & Train View Rendering Performance on ScanNet++ [48]. SplaTAM
provides high-fidelity performance on both training views seen during SLAM and
held-out novel views.

PS [30] |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
Ours |  |  |  |  |  |  |  |
GT |  |  |  |  |  |  |  |
| S1 | S2 | S1 | S2
| Novel View | Train View
Figure 4: Renderings on ScanNet++ [48]. Our method, SplaTAM, renders color &
depth for the novel & train views with fidelity comparable to the ground
truth. It can also be observed that Point-SLAM [30] fails to provide good
renderings on both novel & train view images. Note that Point-SLAM (PS) uses
the ground-truth depth to render the train & novel view images. Despite the
use of ground-truth depth, the failure of PS can be attributed to the failure
of tracking as shown in Sec. 3.

### Color and Depth Loss Ablation.

Our SplaTAM involves fitting the camera poses (during tracking) and the scene
map (during mapping) using both a photometric (RGB) and depth loss. In Table
5, we ablate the decision to use both and investigate the performance of only
one or the other for both tracking and mapping. We do this using Room 0 of
Replica. With only depth our method completely fails to track the camera
trajectory, because the L1 depth loss doesn’t provide adequate information in
the x-y image plane. Using only an RGB loss successfully tracks the camera
trajectory (although with more than 5x the error as using both). Both the RGB
and depth work together to achieve excellent results. With only the color
loss, reconstruction PSNR is really high, only $1.5$ PSNR lower than the full
model. However, the depth L1 is much higher using only color vs directly
optimizing this depth error. In the color-only experiments, the depth is not
used as a loss for either tracking or mapping, but it is used for the
densification and initialization of Gaussians.

### Camera Tracking Ablation.

In Table 4 we ablate three aspects of our camera tracking: (1) the use of
forward velocity propagation, (2) the use of a silhouette mask to mask out
invalid map areas in the loss, and (3) setting the silhouette threshold to
0.99 instead of 0.5. All three are critical to excellent results. Without
forward velocity propagation tracking still works, but the overall error is
more than 10x higher, which subsequently degrades the map and rendering.
Silhouette is critical as without it tracking completely fails. Setting the
silhouette threshold to 0.99 allows the loss to be applied on well-optimized
pixels in the map, thereby leading to an important 5x reduction in the error
compared to the threshold of 0.5, which is used for the densification.

Velo. | Sil. | Sil. | ATE | Dep. L1 | PSNR
---|---|---|---|---|---
Prop. | Mask | Thresh. |  [cm]$\downarrow$ |  [cm]$\downarrow$ |  [dB]$\uparrow$
✗ | ✓ | 0.99 | 2.95 | 2.15 | 25.40
✓ | ✗ | 0.99 | 115.80 | 0.29 | 14.16
✓ | ✓ | 0.5 | 1.30 | 0.74 | 31.36
✓ | ✓ | 0.99 | 0.27 | 0.49 | 32.81
Table 4: Camera Tracking Ablations on Replica/Room 0. Track. | Map. | Track. | Map. | ATE | Dep. L1 | PSNR
---|---|---|---|---|---|---
Color | Color | Depth | Depth |  [cm]$\downarrow$ |  [cm]$\downarrow$ |  [dB]$\uparrow$
✗ | ✗ | ✓ | ✓ | 86.03 | ✗ | ✗
✓ | ✓ | ✗ | ✗ | 1.38 | 12.58 | 31.30
✓ | ✓ | ✓ | ✓ | 0.27 | 0.49 | 32.81
Table 5: Color & Depth Loss Ablation on Replica/Room 0.

### Runtime Comparison.

In Table 6, we compare our runtime to NICE-SLAM [53] and Point-SLAM [30] on a
Nvidia RTX 3080 Ti. Each iteration of our approach renders a full 1200x980
pixel image ($\sim$1.2 mil pixels) to apply the loss for both tracking and
mapping. Other methods use only $200$ pixels for tracking and $1000$ pixels
for mapping each iteration (but attempt to cleverly sample these pixels). Even
though we differentiably render 3 orders of magnitude more pixels, our
approach incurs similar runtime, primarily due to the efficiency of
rasterizing 3D Gaussians. Additionally, we show a version of our method with
fewer iterations and half-resolution densification, SplaTAM-S, which works 5x
faster with only a minor degradation in performance. In particular, SplaTAM
uses $40$ and $60$ iterations per frame for tracking & mapping respectively on
Replica, while SplaTAM-S uses $10$ and $15$ iterations per frame.

Methods | Tracking | Mapping | Tracking | Mapping | ATE RMSE
---|---|---|---|---|---
/Iteration | /Iteration | /Frame | /Frame |  [cm] $\downarrow$
NICE-SLAM [53] | 30 ms | 166 ms | 1.18 s | 2.04 s | 0.97
Point-SLAM [30] | 19 ms | 30 ms | 0.76 s | 4.50 s | 0.61
SplaTAM | 25 ms | 24 ms | 1.00 s | 1.44 s | 0.27
SplaTAM-S | 19 ms | 22 ms | 0.19 s | 0.33 s | 0.39
Table 6: Runtime on Replica/R0 using a RTX 3080 Ti.

### Limitations & Future Work.

Although SplaTAM achieves state-of-the-art performance, we find our method to
show some sensitivity to motion blur, large depth noise, and aggressive
rotation. We believe a possible solution would be to temporally model these
effects and wish to tackle this in future work. Furthermore, SplaTAM can be
scaled up to large-scale scenes through efficient representations like OpenVDB
[24]. Finally, our method requires known camera intrinsics and dense depth as
input for performing SLAM, and removing these dependencies is an interesting
avenue for the future.

##  6 Conclusion

In this paper, we have presented SplaTAM, a novel SLAM system that leverages a
3D Gaussian Splatting radiance field as its underlying map representation,
enabling faster rendering and optimization, explicit knowledge of map spatial
extent, and streamlined map densification. We demonstrate its effectiveness in
achieving state-of-the-art results in camera pose estimation, scene
reconstruction, and novel-view synthesis. We believe that SplaTAM not only
sets a new benchmark in both the SLAM and novel-view synthesis domains but
also opens up exciting avenues, where the integration of 3D Gaussian Splatting
with SLAM offers a robust framework for further exploration and innovation in
scene understanding. Our work underscores the potential of this integration,
paving the way for more sophisticated and efficient SLAM systems in the
future.

Acknowledgments. This work was supported in part by the Intelligence Advanced
Research Projects Activity (IARPA) via Department of Interior/ Interior
Business Center (DOI/IBC) contract number 140D0423C0074. The U.S. Government
is authorized to reproduce and distribute reprints for Governmental purposes
notwithstanding any copyright annotation thereon. Disclaimer: The views and
conclusions contained herein are those of the authors and should not be
interpreted as necessarily representing the official policies or endorsements,
either expressed or implied, of IARPA, DOI/IBC, or the U.S. Government. This
work used Bridges-2 at PSC through allocation cis220039p from the Advanced
Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS)
program which is supported by NSF grants #2138259, #2138286, #2138307,
#2137603, and #213296, and also supported by a hardware grant from Nvidia.

The authors thank Yao He for his support with testing ORB-SLAM3 & Jeff Tan for
testing the code & demos. We also thank Chonghyuk (Andrew) Song, Margaret
Hansen & Michael Kaess for their insightful feedback & discussion on initial
versions of the work.

## References

  * Bloesch et al. [2018] Michael Bloesch, Jan Czarnowski, Ronald Clark, Stefan Leutenegger, and Andrew J Davison.  Codeslam—learning a compact, optimisable representation for dense visual slam.  In _CVPR_ , 2018.
  * Cadena et al. [2016] Cesar Cadena, Luca Carlone, Henry Carrillo, Yasir Latif, Davide Scaramuzza, José Neira, Ian Reid, and John J Leonard.  Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age.  _IEEE Transactions on robotics_ , 32(6):1309–1332, 2016.
  * Campos et al. [2021] Carlos Campos, Richard Elvira, Juan J Gómez Rodríguez, José MM Montiel, and Juan D Tardós.  Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam.  _IEEE Transactions on Robotics_ , 37(6):1874–1890, 2021.
  * Curless and Levoy [1996] Brian Curless and Marc Levoy.  A volumetric method for building complex models from range images.  In _Proceedings of the 23rd annual conference on Computer graphics and interactive techniques_ , pages 303–312, 1996.
  * Dai et al. [2017a] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  ScanNet: Richly-annotated 3D reconstructions of indoor scenes.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE/CVF, 2017a.
  * Dai et al. [2017b] Angela Dai, Matthias Nießner, Michael Zollöfer, Shahram Izadi, and Christian Theobalt.  Bundlefusion: Real-time globally consistent 3d reconstruction using on-the-fly surface re-integration.  _ACM Transactions on Graphics 2017 (TOG)_ , 2017b.
  * Davison et al. [2007] Andrew J Davison, Ian D Reid, Nicholas D Molton, and Olivier Stasse.  Monoslam: Real-time single camera slam.  _IEEE transactions on pattern analysis and machine intelligence_ , 29(6):1052–1067, 2007.
  * Engel et al. [2014] Jakob Engel, Thomas Schöps, and Daniel Cremers.  Lsd-slam: Large-scale direct monocular slam.  In _European conference on computer vision_ , pages 834–849. Springer, 2014.
  * Goel and Tabib [2023] Kshitij Goel and Wennie Tabib.  Incremental multimodal surface mapping via self-organizing gaussian mixture models, 2023.
  * Goel et al. [2023] Kshitij Goel, Nathan Michael, and Wennie Tabib.  Probabilistic point cloud modeling via self-organizing gaussian mixture models.  _IEEE Robotics and Automation Letters_ , 8(5):2526–2533, 2023.
  * Han et al. [2023] Xiao Han, Houxuan Liu, Yunchao Ding, and Lu Yang.  Ro-map: Real-time multi-object mapping with neural radiance fields.  _arXiv preprint arXiv:2304.05735_ , 2023.
  * Johari et al. [2023] Mohammad Mahdi Johari, Camilla Carta, and François Fleuret.  Eslam: Efficient dense slam system based on hybrid representation of signed distance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 17408–17419, 2023.
  * Keller et al. [2013] Maik Keller, Damien Lefloch, Martin Lambers, Shahram Izadi, Tim Weyrich, and Andreas Kolb.  Real-time 3d reconstruction in dynamic scenes using point-based fusion.  In _2013_ , 2013.
  * Kerbl et al. [2023] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis.  3d gaussian splatting for real-time radiance field rendering.  _ACM Transactions on Graphics_ , 42(4), 2023.
  * Kerl et al. [2013] Christian Kerl, Jürgen Sturm, and Daniel Cremers.  Robust odometry estimation for rgb-d cameras.  In _ICRA_ , 2013.
  * Keselman and Hebert [2022] Leonid Keselman and Martial Hebert.  Approximate differentiable rendering with algebraic surfaces.  In _European Conference on Computer Vision_ , pages 596–614. Springer, 2022.
  * Keselman and Hebert [2023] Leonid Keselman and Martial Hebert.  Flexible techniques for differentiable rendering with 3d gaussians.  _arXiv preprint arXiv:2308.14737_ , 2023.
  * Kong et al. [2023] Xin Kong, Shikun Liu, Marwan Taher, and Andrew J Davison.  vmap: Vectorised object mapping for neural field slam.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 952–961, 2023.
  * Li et al. [2023] Heng Li, Xiaodong Gu, Weihao Yuan, Luwei Yang, Zilong Dong, and Ping Tan.  Dense rgb slam with neural implicit maps.  _arXiv preprint arXiv:2301.08930_ , 2023.
  * Luiten et al. [2023] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan.  Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis.  _arXiv preprint arXiv:2308.09713_ , 2023.
  * Mildenhall et al. [2021] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Communications of the ACM_ , 65(1):99–106, 2021.
  * Ming et al. [2022] Yuhang Ming, Weicai Ye, and Andrew Calway.  idf-slam: End-to-end rgb-d slam with neural implicit mapping and deep feature tracking.  _arXiv preprint arXiv:2209.07919_ , 2022.
  * Mur-Artal and Tardos [2017] Raul Mur-Artal and Juan D. Tardos.  ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras.  _IEEE Transactions on Robotics_ , 33(5):1255–1262, 2017.
  * Museth et al. [2013] Ken Museth, Jeff Lait, John Johanson, Jeff Budsberg, Ron Henderson, Mihai Alden, Peter Cucka, David Hill, and Andrew Pearce.  Openvdb: An open-source data structure and toolkit for high-resolution volumes.  In _ACM SIGGRAPH Courses_ , 2013.
  * Newcombe et al. [2011a] Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohli, Jamie Shotton, Steve Hodges, and Andrew W Fitzgibbon.  Kinectfusion: Real-time dense surface mapping and tracking.  In _ISMAR_ , 2011a.
  * Newcombe et al. [2011b] Richard A Newcombe, Steven J Lovegrove, and Andrew J Davison.  Dtam: Dense tracking and mapping in real-time.  In _2011 international conference on computer vision_ , pages 2320–2327. IEEE, 2011b.
  * Oleynikova et al. [2017] Helen Oleynikova, Zachary Taylor, Marius Fehr, Roland Siegwart, and Juan Nieto.  Voxblox: Incremental 3d euclidean signed distance fields for on-board mav planning.  In _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 1366–1373. IEEE, 2017.
  * Ortiz et al. [2022] Joseph Ortiz, Alexander Clegg, Jing Dong, Edgar Sucar, David Novotny, Michael Zollhoefer, and Mustafa Mukadam.  isdf: Real-time neural signed distance fields for robot perception.  _arXiv preprint arXiv:2204.02296_ , 2022.
  * Rosinol et al. [2022] Antoni Rosinol, John J Leonard, and Luca Carlone.  Nerf-slam: Real-time dense monocular slam with neural radiance fields.  _arXiv preprint arXiv:2210.13641_ , 2022.
  * Sandström et al. [2023a] Erik Sandström, Yue Li, Luc Van Gool, and Martin R Oswald.  Point-slam: Dense neural point cloud-based slam.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 18433–18444, 2023a.
  * Sandström et al. [2023b] Erik Sandström, Kevin Ta, Luc Van Gool, and Martin R Oswald.  Uncle-slam: Uncertainty learning for dense neural slam.  _arXiv preprint arXiv:2306.11048_ , 2023b.
  * Sarlin et al. [2022] Paul-Edouard Sarlin, Mihai Dusmanu, Johannes L Schönberger, Pablo Speciale, Lukas Gruber, Viktor Larsson, Ondrej Miksik, and Marc Pollefeys.  Lamar: Benchmarking localization and mapping for augmented reality.  In _European Conference on Computer Vision_ , pages 686–704. Springer, 2022.
  * Schops et al. [2019] Thomas Schops, Torsten Sattler, and Marc Pollefeys.  BAD SLAM: Bundle adjusted direct RGB-D SLAM.  In _CVF/IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Steinbrücker et al. [2011] Frank Steinbrücker, Jürgen Sturm, and Daniel Cremers.  Real-time visual odometry from dense rgb-d images.  In _ICCV Workshops_ , 2011.
  * Straub et al. [2019] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, et al.  The replica dataset: A digital replica of indoor spaces.  _arXiv preprint arXiv:1906.05797_ , 2019.
  * Sturm et al. [2012] Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers.  A benchmark for the evaluation of RGB-D SLAM systems.  In _International Conference on Intelligent Robots and Systems (IROS)_. IEEE/RSJ, 2012.
  * Sucar et al. [2021] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew J Davison.  imap: Implicit mapping and positioning in real-time.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 6229–6238, 2021.
  * Tateno et al. [2017] K. Tateno, F. Tombari, I. Laina, and N. Navab.  Cnn-slam: Real-time dense monocular slam with learned depth prediction.  In _CVPR_ , 2017.
  * Wang et al. [2022] Angtian Wang, Peng Wang, Jian Sun, Adam Kortylewski, and Alan Yuille.  Voge: a differentiable volume renderer using gaussian ellipsoids for analysis-by-synthesis.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Wang et al. [2023] Hengyi Wang, Jingwen Wang, and Lourdes Agapito.  Co-slam: Joint coordinate and sparse parametric encodings for neural real-time slam.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13293–13302, 2023.
  * Whelan et al. [2015a] Thomas Whelan, Michael Kaess, Hordur Johannsson, Maurice Fallon, John J Leonard, and John McDonald.  Real-time large-scale dense rgb-d slam with volumetric fusion.  _The International Journal of Robotics Research_ , 34(4-5):598–626, 2015a.
  * Whelan et al. [2015b] Thomas Whelan, Stefan Leutenegger, Renato Salas-Moreno, Ben Glocker, and Andrew Davison.  Elasticfusion: Dense slam without a pose graph.  In _Robotics: Science and Systems_ , 2015b.
  * Wu et al. [2023] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang.  4d gaussian splatting for real-time dynamic scene rendering.  _arXiv preprint arXiv:2310.08528_ , 2023.
  * Xu et al. [2022] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann.  Point-nerf: Point-based neural radiance fields.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5438–5448, 2022.
  * Yang et al. [2022] Xingrui Yang, Hai Li, Hongjia Zhai, Yuhang Ming, Yuqian Liu, and Guofeng Zhang.  Vox-fusion: Dense tracking and mapping with voxel-based neural implicit representation.  In _IEEE International Symposium on Mixed and Augmented Reality (ISMAR)_ , pages 499–507. IEEE, 2022.
  * Yang et al. [2023a] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin.  Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction.  _arXiv preprint arXiv:2309.13101_ , 2023a.
  * Yang et al. [2023b] Zeyu Yang, Hongye Yang, Zijie Pan, Xiatian Zhu, and Li Zhang.  Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting.  _arXiv preprint arXiv:2310.10642_ , 2023b.
  * Yeshwanth et al. [2023] Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai.  Scannet++: A high-fidelity dataset of 3d indoor scenes.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 12–22, 2023.
  * Yifan et al. [2019] Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung.  Differentiable surface splatting for point-based geometry processing.  _ACM Transactions on Graphics (TOG)_ , 38(6):1–14, 2019.
  * Zhang et al. [2023] Wei Zhang, Tiecheng Sun, Sen Wang, Qing Cheng, and Norbert Haala.  Hi-slam: Monocular real-time dense mapping with hybrid implicit fields.  _arXiv preprint arXiv:2310.04787_ , 2023.
  * Zhi et al. [2019] Shuaifeng Zhi, Michael Bloesch, Stefan Leutenegger, and Andrew J Davison.  Scenecode: Monocular dense semantic reconstruction using learned encoded scene representations.  In _CVPR_ , 2019.
  * Zhou et al. [2018] Huizhong Zhou, Benjamin Ummenhofer, and Thomas Brox.  Deeptam: Deep tracking and mapping.  In _ECCV_ , 2018.
  * Zhu et al. [2022] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R Oswald, and Marc Pollefeys.  Nice-slam: Neural implicit scalable encoding for slam.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12786–12796, 2022.
  * Zhu et al. [2023] Zihan Zhu, Songyou Peng, Viktor Larsson, Zhaopeng Cui, Martin R Oswald, Andreas Geiger, and Marc Pollefeys.  Nicer-slam: Neural implicit scene encoding for rgb slam.  _arXiv preprint arXiv:2302.03594_ , 2023.


# InterPrompt: Interpretable Prompting for Interrelated Interpersonal Risk
Factors in Reddit Posts

MSVPJ Sathvik  IIIT DharwadIndia<EMAIL_ADDRESS>,  Surjodeep Sarkar
UMBC Baltimore, MDUSA<EMAIL_ADDRESS>,  Chandni Saxena  The Chinese
University of Hong KongHong Kong<EMAIL_ADDRESS>,  Sunghwan Sohn
Mayo ClinicRochester, MNUSA<EMAIL_ADDRESS>and  Muskan Garg  Mayo
ClinicRochester, MNUSA<EMAIL_ADDRESS>
(2018)

###### Abstract.

Mental health professionals and clinicians have observed the upsurge of mental
disorders due to Interpersonal Risk Factors (IRFs). To simulate the human-in-
the-loop triaging scenario for early detection of mental health disorders, we
recognized textual indications to ascertain these IRFs: Thwarted Belongingness
(TBe) and Perceived Burdensomeness (PBu) within personal narratives. In light
of this, we use N-shot learning with GPT-3 model on the IRF dataset, and
underscored the importance of fine-tuning GPT-3 model to incorporate the
context-specific sensitivity and the interconnectedness of textual cues that
represent both IRFs.

In this paper, we introduce an Interpretable Prompting (InterPrompt) method to
boost the attention mechanism by fine-tuning the GPT-3 model. This allows a
more sophisticated level of language modification by adjusting the pre-trained
weights. Our model learns to detect usual patterns and underlying connections
across both the IRFs, which leads to better system-level explainability and
trustworthiness. The results of our research demonstrate that all four
variants of GPT-3 model, when fine-tuned with InterPrompt, perform
considerably better as compared to the baseline methods, both in terms of
classification and explanation generation.

fine-tuning, interpersonal risk factor, interpretable prompting, mental health

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Background

Every year, over 700,000 lives are lost to suicide, as reported by the World
Health Organization. The number of attempted suicides stemming from each
completed suicide surpasses this figure,111https://www.who.int/news-room/fact-
sheets/detail/suicide emphasizing the significant role that interpersonal
relationships play in an individual’s life. The Interpersonal-Psychological
Theory of Suicide (IPTS) (Joiner Jr et al., 2009), asserts the severity of
suicidal tendencies as the result of three interconnected factors: (i)
Acquired Capability for committing suicide, (ii) Thwarted Belongingness (TBe)
characterized by feelings of social isolation and disconnectedness, and (iii)
Perceived Burdensomeness (PBu) involving a sense of being a burden on society.
Consequently, TBe and PBu together act as Interpersonal Risk Factors (IRFs),
signifying adequate immediate causes of suicide. Social media platforms,
especially those offering anonymity (e.g., subreddit in Reddit), are
increasingly becoming a refuge for individuals whose psychological stigma
deters them from seeking professional mental healthcare (Tay et al., 2018).
These platforms facilitate easy and anonymous sharing of personal experiences,
creating a secure, empathetic environment for self-expression (Roy et al.,
2023). The growing demand for quality mental healthcare, coupled with a
scarcity of clinical psychologists, highlights the importance of computational
models in analyzing mental health through social media posts (Torous et al.,
2021).

Figure 1. Schema of the proposed work for identifying textual cues while
determining Interpersonal Risk Factors in Reddit Post.

To address the above issues, we draw inspiration from the notion of Safety,
Explainability, and Trustworthiness advocated by the social NLP community (Roy
et al., 2023), that emphasizes “interpretability” and “system-level
explainability” in computational models for mental health analysis. In this
paper, we define “Explainability” as the linguistic cues that shed light on
the presence of TBe and PBu within text. We take into consideration of GPT-3’s
capability to generate sentiment-driven text with a human-like quality,
leading us to regard it as an exceptionally fitting model for mental health
analysis (Brown et al., 2020; Xie et al., 2022; Yang et al., 2023). However,
in order to build more comprehensive and explainable model that can
effectively understand the linguistic cues, we introduce “N-shot learning”
(Wei et al., [n. d.]) followed by our “Interpretable Prompting (InterPrompt)”
approach for task-specific fine-tuning of the GPT-3 model. As such, we aim to
enhance the performance of GPT-3 model and tailor it to a downstream task of
explainable identification of IRFs, while still benefiting from the knowledge
and understanding, the model gained during the initial pre-training phase (see
Figure 1).

Figure 2. Overview of the InterPrompt-driven fine-tuning of GPT-3 model. TBe:
Thwarted Belongingness, and PBu: Perceived Burdensomeness.

We present our contributions in three-fold ways: (i) we provide an in-depth
analysis that demonstrates the inefficiency of GPT-3 model, when employing
“N-shot learning” for task-specific mental health analysis, (ii) we introduce
InterPrompt, which effectively incorporates task-specific context to enhance
the interpretability and System-level explainabilty of the model, and (iii) we
compare the InterPrompt-driven fine-tuned GPT-3 model with baselines to
provide empirical evidence of the advantages and advancements offered by our
approach. To the best of our knowledge, we are the first to design prompt to
comprehend interrelated IRFs in a given text.

##  2\. Proposed Method

#### Problem Definition

Let $\mathcal{D}$ be the given IRF dataset. Given a text
$T=\\{S_{1},S_{2},S_{3},...,S_{N},$ $S_{sep}\\}$ which is a sequence of $N$
sentences, where $S_{sep}=[SEP]$ represents “`\n\nIntent:\n\n`” as illustrated
in Figure 2. In our problem formulation, we first define the textual cues for
TBe in the given text $T$ which is denoted as $\mu_{i}(\epsilon)$, where
$\mu_{i}$ represents textual cues and $\epsilon$ represents the TBe. We define
the label for TBe as ($y_{i}(\epsilon)$), where $y_{i}$ represents the label
for a given factor. Secondly, we define the textual cues for PBu in the given
text $T$ which is denoted as ($\mu_{i}(\upsilon)$), where $\mu_{i}$ represents
textual cues and $\upsilon$ represents the PBu. We define the label for PBu as
($y_{i}(\epsilon)$), where $y_{i}$ represents the label for a given task. We
initially transform the factor labels from their binary representation into
text format. We further classify our approach in three phases: (A)
InterPrompt, (B) fine-tuning GPT-3 model, and (C) testing phase as illustrated
in Figure 2.

For InterPrompt, we define notations for text, prompts, and completion. We
consider prompts denoted as
$\rho=\\{\rho_{1},\rho_{2},\rho_{3},...,\rho_{m}\\}$ where $m=3$ and we define
$\rho=\\{\rho_{1},\rho_{2},\rho_{3}\\}$ as a set of (i) labeling factors
($\rho_{1}$): This given sentence represents {belong, burden, both belong and
burden, neither belong nor burden}, (ii) textual cues for TBe ($\rho_{2}$):
‘word those indicate belong expression in the sentence...’, (iii) textual cues
for PBu ($\rho_{3}$): ‘word those indicate burden expression in the
sentence...’. The idea of InterPrompt is to generate the story completion
($C_{i}$) the process of generating or completing text ($T_{i}$) based on the
given prompts ($\rho$) as shown in Equation 1.

(1)
$C_{i}=f(\rho_{1}+y_{i}(\epsilon,\upsilon)))+f(\rho_{2}+\mu_{i}(\epsilon))+f(\rho_{3}+\mu_{i}(\upsilon))$

Here, $f(.)$ is a function that generates multi-task story completion using
the three prompts. $\rho_{1}$ is used for labeling factors, $\rho_{2}$ is used
for textual cues for TBe, and $\rho_{3}$ is used for textual cues for PBu.

For fine-tuning GPT-3 model, we construe the story completion to fine-tune
GPT-3 model ($G$) using OpenAI servers. GPT-3 model finds the probability of
obtaining required output for the given input. We define the probabilities
$P=\\{P_{1},P_{2},P_{3}\\}$ for three tasks of identifying
$\hat{y_{i}}(\epsilon,\upsilon),\hat{\mu_{i}}(\epsilon),\hat{\mu_{i}}(\upsilon)$,
respectively. We train our model with $k$ samples:

(2)
$\displaystyle\hat{y_{i}}(\epsilon,\upsilon)=P_{1}(y_{i}(\epsilon,\upsilon)|T_{i},\rho_{1});\quad
i\in k,$
$\displaystyle\hat{\mu_{i}}(\epsilon)=P_{2}(\mu_{i}(\epsilon)|T_{i},\rho_{2});\quad
i\in k,$
$\displaystyle\hat{\mu_{i}}(\upsilon)=P_{3}(\mu_{i}(\upsilon)|T_{i},\rho_{3});\quad
i\in k,$

where $k$ is the number of samples used to fine-tune different variants of
GPT-3 model using OpenAI servers. The story completion is a prompt designed to
capture multi-tasking of the model as:

(3)
$C_{i}=\rho_{1}+\hat{y_{i}}(\epsilon,\upsilon)+\rho_{2}+\hat{\mu_{i}}(\epsilon)+\rho_{3}+\hat{\mu_{i}}(\upsilon)$

#### Loss Function for fine-tuning GPT-3

: To achieve story completion by taking into account multiple prompts and
focusing on a combined loss function that incorporates both the structured
extraction of entities and the generation of coherent text. We define the
combined loss for our task as:

(4) $L_{combined}=\lambda_{1}\cdot L_{1}+\lambda_{2}\cdot
L_{2}+\lambda_{3}\cdot L_{3}$

where $L\\_1,L\\_2,L\\_3$ are defined as loss functions for predicting
$\hat{y_{i}}(\epsilon,\upsilon),$
$\hat{\mu_{i}}(\epsilon),\hat{\mu_{i}}(\upsilon)$, respectively. For a given
sequence $S_{i}$ of text $T_{i}$ and the number of generated tokens as
$E_{i}$, we define loss functions as:

(5) $\displaystyle
L_{1}(y_{i},\hat{y_{i}})=\left(-\sum_{t=1}^{S_{i}}\sum_{i=1}^{E_{i}}y_{t,i}\log(\hat{y}_{t,i}|\rho_{1})\right)$
$\displaystyle
L_{2}(\mu_{i}(\epsilon),\hat{\mu_{i}}(\epsilon))=\left(-\sum_{t=1}^{S_{i}}\log
P(\hat{\mu_{i}}(\epsilon)|\rho_{2})\right)$ $\displaystyle
L_{3}(\mu_{i}(\upsilon),\hat{\mu_{i}}(\upsilon))=\left(-\sum_{t=1}^{S_{i}}\log
P(\hat{\mu_{i}}(\upsilon)|\rho_{3})\right)$

Here, $L_{1}$ is a token level cross-entropy loss functions for entity
extraction; $L_{2}$ and $L_{3}$ are the maximum likelihood estimation loss for
text generation. $\lambda_{1},\lambda_{2}$ and $\lambda_{3}$ are the weights
that balance the contribution of all three loss components for identifying
explainable IRFs during fine-tuning GPT-3 model. We define the process of
fine-tuning $G$ on OpenAI servers as a function ($O(G)$). The model $G$ is
fine-tuned by optimizing a loss function $L$ based on the output of the GPT-3
model on the story completion $C_{i}$ and the true labels $C_{i}$. The
equation can be represented in a simplified form as:

(6) $O(G)=\arg\min_{k}(L_{combined}(C_{i},\hat{C_{i}}))$

where our GPT model is fine-tuned by adjusting the pre-trained weights to
minimize loss function.

For testing phase, we label a given text ($T_{i}$) combined with input $\rho$
as $\sum_{i=1}^{k}(C_{i}|T_{i},\rho);$ where $k\subseteq N$. We decode the
single story completion $\hat{C_{i}}$ to find labels ($y_{i}$) and textual
cues ($\mu$) for both TBe ($\epsilon$) and PBu ($\upsilon$). The textual cues
is a single most expressive continuous expression of words obtained against
$\mu$. Labels ($y_{i}$) are mapped back from string to binary labels.

##  3\. Experiments

#### Dataset

Central to the IPTS, we emphasize the exploration of textual cues related to
interconnected interpersonal risk factors in a given text. This approach
upholds our fine-grained observations, robust arguments and are reinforced by
statistical analysis of IRF dataset (Garg et al., 2023). We observe that the
ratio of instances where PBu equals 1 to instances where TBe equals 1 is
approximately 32% higher than the increase in the ratio of PBu equals 1 to TBe
equals 0, as illustrated in Table 1. Our finding emphasizes a stronger
correlation between TBe and PBu.

| PBu: 0 | PBu: 1
---|---|---
TBe: 0 | 1123 | 472
TBe: 1 | 1252 | 675
%$\Delta$ |  129/1123 $=0.1148$ |  203/472 $=0.4301$
Table 1. Dataset statistics for Thwarted Belongingness (TBe) and Perceived Burdensomeness (PBu) representing the number of samples in IRF dataset for corresponding labels. Table 2. Statistical significance test for classifiers. Here, p-val is p-value, IPG is InterPrompt-driven fine-tuned GPT-3 model, A is ada variant, B is babbage variant, C is curie variant, D is davinci variant Models | IPG [B] | IPG [C] | IPG [D]
---|---|---|---
| t-test | p-val | t-test | p-val | t-test | p-val
IPG [A] | 0.053 | 0.957 | -0.639 | 0.523 | -0.426 | 0.670
IPG [B] | - | - | -0.693 | 0.489 | -0.479 | 0.632
IPG [C] | - | - | - | - | 0.213 | 0.831
Table 3. Comparison of InterPrompt driven fine-tuned GPT-3 variants with baseline models over IRF dataset Model | Thwarted Belongingness | Perceived Burdensomeness
---|---|---
| Precision | Recall | F1-score | Accuracy | Precision | Recall | F1-score | Accuracy
BERT | 69.70 | 76.97 | 72.30 | 68.97 | 56.47 | 53.00 | 52.20 | 72.56
RoBERTa | 71.23 | 73.54 | 71.35 | 68.97 | 67.27 | 37.52 | 45.51 | 74.93
DistilBERT | 70.24 | 74.08 | 71.15 | 68.50 | 51.15 | 31.89 | 36.93 | 71.71
MentalBERT | 77.97 | 77.40 | 76.73 | 75.12 | 64.22 | 65.75 | 62.77 | 78.33
OpenAI+LR | 79.00 | 83.59 | 81.23 | 78.62 | 82.66 | 63.08 | 71.55 | 84.58
OpenAI+RF | 79.06 | 80.68 | 79.86 | 77.48 | 83.33 | 49.23 | 61.90 | 81.36
OpenAI+SVM | 81.31 | 80.34 | 80.83 | 78.90 | 79.15 | 74.77 | 76.90 | 86.19
OpenAI+MLP | 81.40 | 75.56 | 78.37 | 76.92 | 72.08 | 77.85 | 74.85 | 83.92
OpenAI+XGB | 81.22 | 79.83 | 80.52 | 78.62 | 80.36 | 68.00 | 73.67 | 85.05
GPT-3 Zero-shot | 63.78 | 21.54 | 32.21 | 51.63 | 27.56 | 16.28 | 20.47 | 61.42
GPT-3 One-shot | 61.15 | 84.57 | 70.98 | 63.12 | 34.84 | 86.05 | 49.60 | 46.67
GPT-3 Few-shot | 57.42 | 94.68 | 71.49 | 59.72 | 32.16 | 98.14 | 48.45 | 36.31
Fine-tuned GPT-3 [A] | 86.14 | 83.93 | 85.02 | 83.63 | 76.72 | 79.08 | 77.88 | 86.19
Fine-tuned GPT-3 [B] | 86.38 | 84.21 | 85.12 | 84.29 | 81.24 | 80.69 | 80.96 | 87.08
Fine-tuned GPT-3 [C] | 85.17 | 87.35 | 86.24 | 84.58 | 79.94 | 80.92 | 80.43 | 87.89
Fine-tuned GPT-3 [D] | 84.59 | 83.01 | 83.80 | 82.83 | 80.12 | 84.17 | 82.13 | 87.54
Table 4. Performance evaluation of generated explanations through similarity measures Model Name | Thwarted Belongingness | Perceived Burdensomeness
---|---|---
| Rouge-1 | Rouge-L | BLEU-1 | EM | Rouge-1 | Rouge-L | BLEU-1 | EM
MentalBERT+LIME | 0.2202 | - | 0.1509 | - | 0.2425 | - | 0.1706 | -
MentalBERT+ SHAP | 0.2415 | - | 0.1593 | - | 0.2601 | - | 0.1718 | -
Fine-tuned GPT-3 [A] | 0.6597 | 0.6591 | 0.6373 | 0.5787 | 0.7738 | 0.7738 | 0.7556 | 0.6993
Fine-tuned GPT-3 [B] | 0.6631 | 0.6628 | 0.6412 | 0.5816 | 0.7832 | 0.7831 | 0.7693 | 0.7163
Fine-tuned GPT-3 [C] | 0.6637 | 0.6633 | 0.6377 | 0.5603 | 0.7989 | 0.7989 | 0.7846 | 0.7348
Fine-tuned GPT-3 [D] | 0.6809 | 0.6805 | 0.6532 | 0.5801 | 0.7771 | 0.7771 | 0.7579 | 0.6965

#### N-Shot Learning

Central to the sensitivity to exemplars in prompting approaches, we
investigate the permutations of N-shot learning in GPT-3 prompting to evaluate
the effectiveness of GPT-3 classifiers in the mental health domain. The
results indicate that the accuracy of GPT-3 classifiers for TBe and PBu tasks
falls within the range of near chance, typically around 40% to 60% (see Table
3). We first employ the “zero-shot prompting” approach yielding an accuracy of
51.63% for TBe and 61.42% for PBu. As such, we proceed to implement the “one-
shot learning” and “few-shot learning” approaches, using a single initial
input example and eight initial input examples, respectively. The decline in
performance observed with N-shot learning can be attributed to the contextual
sensitivity, underscoring the importance of task-specific fine-tuning of pre-
trained language models (PLMs).

###  3.1. Experimental Setup

For our experimental purposes and evaluation, we utilize the IRF, a Reddit
dataset comprising 3522 posts which is divided into three sets: training (1972
posts), validation (493 posts), and testing (1057 posts) (Garg et al., 2023).
We fine-tune all four variants of GPT-3 using the training dataset at OpenAI
server, namely, (i) ada $\rightarrow$ [A], (ii) babbage $\rightarrow$ [B],
curie $\rightarrow$ [C], davinci $\rightarrow$ [D]. We keep the parameters at
their default settings. We compare our model with baseline models: (i) pre-
trained lanaguage models: BERT (Devlin et al., 2018), RoBERTa (Liu et al.,
2019), DistilBERT (Sanh et al., 2019), MentalBERT (Ji et al., 2021), (ii)
OpenAI + machine learning models (Garg et al., 2023).

#### Evaluation Metrics

The best performing baseline among pre-trained language models, MentalBERT,
generate explanations using two explainable NLP methods: LIME (Garreau and von
Luxburg, 2020) and SHAP (Lundberg and Lee, 2017). Our fine-tuned GPT-3 models
generate explanations along with classified labels. We assess the
classification performance of all models using standard metrics such as
Precision, Recall, F1-Score, and Accuracy. To evaluate the quality of the
generated explanations, we compare them with ground-truth explanations using
similarity check algorithms including Recall-Oriented Understudy for Gisting
Evaluation (ROUGE) (Lin, 2004), BiLingual Evaluation Understudy (BLEU)
(Papineni et al., 2002), and Exact Match (EM) scores, comparing them with
ground-truth explanations. The goal was to determine the faithfulness and
accuracy of the explanations generated by our fine-tuned GPT-3 models.

###  3.2. Results and Discussion

#### Classification Model

In our study, we found that our models consistently outperformed all existing
baselines as illustrated in Table 3. Notably, we observed higher recall values
compared to precision, indicating that our models accurately recognized the
correct labels. Based on our experiments, ”fine-tuned GPT-3 [B]” gives best
performance in terms of accuracy, and ”fine-tuned GPT-3 [C]” and ”fine-tuned
GPT-3 [D]” outperforms all the other classifiers for TBe and PBu in terms of
F1-score, respectively.

#### Statistical Significance

As we observe minor variation in accuracy and F1-score among all the variants,
we perform the statistical significance test (James et al., 2013; Hastie et
al., 2009). We used the student’s t-test (Kalpić et al., 2011) to assess the
statistical significance with $p-value=0.05$ (see Table 2). We notice
significant differences in the performance of different variants of fine-tuned
GPT-3 models, indicating the best performance classifiers with highest values
of accuracy.

#### Generated Explanations

We further evaluate the explanations generated and compare them with ground-
truth explanations to examine increased reliability and robustness in fine-
tuned GPT-3 models. The explanations generated by LIME and SHAP lack
continuity, which restricts their ability to generate ROUGE-L and EM scores.
Our observations indicate that the fine-tuned GPT-3 models labeled as ”fine-
tuned GPT-3 [D]” and ”fine-tuned GPT-3 [C]” exhibit the highest performance
for TBe and PBu, respectively, surpassing all other methods considered. This
finding suggests an enhanced level of reliability and efficacy in these
particular models. More specifically, our analysis reveals notable performance
by the ”fine-tuned GPT-3 [D]” model in the TBe task, achieving a ROUGE-1 score
of 68.09% and a BLEU score of 65.32%. Similarly, the ”fine-tuned GPT-3 [C]”
model showcases impressive performance in the PBu task, attaining a ROUGE-1
score of 79.89% and a BLEU score of 78.46% (see Table 4). Overall, the system-
level explainability for PBu demonstrates superior performance compared to
TBe, leading to higher F1-scores and Accuracy measures.

#### Ethics and Broader Impact

We are dedicated to upholding ethical principles and protecting user privacy
(Yoo et al., 2023; Sawhney et al., 2023). In accordance with this commitment,
we rigorously comply with OpenAI’s terms and conditions and refrain from
divulging our models or OpenAI key. In order to safeguard user privacy and
minimize the risk of misuse, all examples presented in this paper have been
anonymized, obfuscated, and paraphrased (Zirikly et al., 2019). This NLP-
focused task finds its practical utility in the preliminary assessment of
individuals on social media platforms during in-person sessions of mental
health triaging, and clinical diagnostic interviewing (Daws, 2020; Westra et
al., 2011).

##  4\. Conclusion

As people express their thoughts on social media with ease, we observe an
opportunity to simulate human-in-the-loop triaging for by examining the
interpersonal risk factors (IRF) in human writing for early detection of
depression. In this research, we introduce Interpretable Prompting
(InterPrompt) as a method to fine-tune the GPT-3 model, resulting in the
generation of responses that are both coherent and contextually relevant. We
achieve this by combining the multi-task prompting in story completion,
enabling it to learn higher-level ways of manipulating the language. As such,
we emphasise the importance of combining the text labels and the explanations
that aid the system-level explanability with a special focus on comprehending
textual cues projecting reasons behind the presence of TBe and PBu. Our model
outperforms all the baselines both as a classifier and explanation generator.
In the near future, it will be interesting to determine the extent of
associations among both IRFs in the near future. Also, our task-specific fine-
tuning opens up novel avenues for conducting fine-grained analysis of mental
health.

## References

  * (1)
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Daws (2020) Ryan Daws. 2020.  Babylon Health lashes out at doctor who raised AI chatbot safety concerns.  _https://www.artificialintelligence-news.com/2020/02/26/babylon-health-doctor-ai-chatbot-safety-concerns/_ (2020).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Garg et al. (2023) Muskan Garg, Amirmohammad Shahbandegan, Amrit Chadha, and Vijay Mago. 2023.  An Annotated Dataset for Explainable Interpersonal Risk Factors of Mental Disturbance in Social Media Posts.  _arXiv preprint arXiv:2305.18727_ (2023).
  * Garreau and von Luxburg (2020) Damien Garreau and Ulrike von Luxburg. 2020.  Explaining the Explainer: A First Theoretical Analysis of LIME.  arXiv:2001.03447 [cs.LG]
  * Hastie et al. (2009) Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. 2009.  _The elements of statistical learning: data mining, inference, and prediction_. Vol. 2.  Springer.
  * James et al. (2013) Gareth James, Daniela Witten, Trevor Hastie, and Robert Tibshirani. 2013.  _An introduction to statistical learning_. Vol. 112.  Springer.
  * Ji et al. (2021) Shaoxiong Ji, Tianlin Zhang, Luna Ansari, Jie Fu, Prayag Tiwari, and Erik Cambria. 2021.  Mentalbert: Publicly available pretrained language models for mental healthcare.  _arXiv preprint arXiv:2110.15621_ (2021).
  * Joiner Jr et al. (2009) Thomas E Joiner Jr, Kimberly A Van Orden, Tracy K Witte, and M David Rudd. 2009.  _The interpersonal theory of suicide: Guidance for working with suicidal clients._ American Psychological Association.
  * Kalpić et al. (2011) Damir Kalpić, Nikica Hlupić, and Miodrag Lovrić. 2011.  _Student’s t-Tests_.  Springer Berlin Heidelberg, Berlin, Heidelberg, 1559–1563.  https://doi.org/10.1007/978-3-642-04898-2_641
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries. In _Text summarization branches out_. 74–81.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Lundberg and Lee (2017) Scott Lundberg and Su-In Lee. 2017.  A Unified Approach to Interpreting Model Predictions.  arXiv:1705.07874 [cs.AI]
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation. In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_. 311–318.
  * Roy et al. (2023) Kaushik Roy, Manas Gaur, Misagh Soltani, Vipula Rawte, Ashwin Kalyan, and Amit Sheth. 2023.  ProKnow: Process knowledge for safety constrained and explainable question generation for mental health diagnostic assistance.  _Frontiers in big Data_ 5 (2023), 1056728.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_ (2019).
  * Sawhney et al. (2023) Ramit Sawhney, Atula Neerkaje, Ivan Habernal, and Lucie Flek. 2023.  How Much User Context Do We Need? Privacy by Design in Mental Health NLP Applications. In _Proceedings of the International AAAI Conference on Web and Social Media_ , Vol. 17. 766–776.
  * Tay et al. (2018) Stacie Tay, Kat Alcock, and Katrina Scior. 2018.  Mental health problems among clinical psychologists: Stigma and its impact on disclosure and help-seeking.  _Journal of Clinical Psychology_ 74, 9 (2018), 1545–1555.
  * Torous et al. (2021) John Torous, Sandra Bucci, Imogen H Bell, Lars V Kessing, Maria Faurholt-Jepsen, Pauline Whelan, Andre F Carvalho, Matcheri Keshavan, Jake Linardon, and Joseph Firth. 2021.  The growing field of digital psychiatry: current evidence and the future of apps, social media, chatbots, and virtual reality.  _World Psychiatry_ 20, 3 (2021), 318–335.
  * Wei et al. ([n. d.]) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. [n. d.].  Finetuned Language Models are Zero-Shot Learners. In _International Conference on Learning Representations_.
  * Westra et al. (2011) Henny A Westra, Adi Aviram, and Faye K Doell. 2011.  Extending motivational interviewing to the treatment of major mental health problems: current directions and evidence.  _The Canadian Journal of Psychiatry_ (2011).
  * Xie et al. (2022) Qianqian Xie, Jennifer Amy Bishop, Prayag Tiwari, and Sophia Ananiadou. 2022.  Pre-trained language models with domain knowledge for biomedical extractive summarization.  _Knowledge-Based Systems_ 252 (2022), 109460.
  * Yang et al. (2023) Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, and Sophia Ananiadou. 2023.  On the Evaluations of ChatGPT and Emotion-enhanced Prompting for Mental Health Analysis.  _arXiv preprint arXiv:2304.03347_ (2023).
  * Yoo et al. (2023) Dong Whi Yoo, Aditi Bhatnagar, Sindhu Kiranmai Ernala, Asra Ali, Michael L Birnbaum, Gregory D Abowd, and Munmun De Choudhury. 2023.  Discussing Social Media During Psychotherapy Consultations: Patient Narratives and Privacy Implications.  _Proceedings of the ACM on Human-Computer Interaction_ 7, CSCW1 (2023), 1–24.
  * Zirikly et al. (2019) Ayah Zirikly, Philip Resnik, Ozlem Uzuner, and Kristy Hollingshead. 2019.  CLPsych 2019 shared task: Predicting the degree of suicide risk in Reddit posts. In _Proceedings of the sixth workshop on computational linguistics and clinical psychology_. 24–33.