# Lifelong LERF: Local 3D Semantic Inventory Monitoring
Using FogROS2

Adam Rashid1*, Chung Min Kim1*, Justin Kerr1*, Letian Fu1, Kush Hari1, Ayah
Ahmad1, Kaiyuan Chen2,
Huang Huang1, Marcus Gualtieri3, Michael Wang3, Christian Juette3, Nan Tian3,
Liu Ren3, Ken Goldberg1 *Equal contribution, 1The AUTOLab at 2UC Berkeley,
3Bosch

###### Abstract

Inventory monitoring in homes, factories, and retail stores relies on
maintaining data despite objects being swapped, added, removed, or moved. We
introduce Lifelong LERF, a method that allows a mobile robot with minimal
compute to jointly optimize a dense language and geometric representation of
its surroundings. Lifelong LERF maintains this representation over time by
detecting semantic changes and selectively updating these regions of the
environment, avoiding the need to exhaustively remap. Human users can query
inventory by providing natural language queries and receiving a 3D heatmap of
potential object locations. To manage the computational load, we use Fog-ROS2,
a cloud robotics platform, to offload resource-intensive tasks. Lifelong LERF
obtains poses from a monocular RGBD SLAM backend, and uses these poses to
progressively optimize a Language Embedded Radiance Field (LERF) for semantic
monitoring. Experiments with 3-5 objects arranged on a tabletop and a
Turtlebot with a RealSense camera suggest that Lifelong LERF can persistently
adapt to changes in objects with up to 91% accuracy.

##  I Introduction

The ability to monitor inventory over time and respond to natural language
queries such as “where did I leave my keys?” or “where is the yellow electric
screwdriver?” can be useful in homes, retail shops, repair shops, hospitals,
and factories. Indoor scenes evolve over time, with objects moving, changing,
appearing, or disappearing. To remain useful, systems must keep track of the
current environment state. We consider how a mobile robot (wheeled or legged)
can navigate using Simultaneous Localization and Mapping (SLAM) [1, 2, 3] to
collect images, build, query, and update semantic 3D models over time.

Lifelong LERF enables a mobile robot with limited on-board compute to create a
dense semantic representation that supports natural language queries, and
maintain this representation by automatically detecting semantic differences
in 3D as it navigates, updating the reconstruction to reflect these changes.
Lifelong LERF uses DROID-SLAM as its monocular pose estimation backend, and
uses these poses to progressively optimize a Language Embedded Radiance Field
[4] (LERF) which densely embeds CLIP [5] embeddings within a Neural Radiance
Field (NeRF) [6].

Figure 1: Lifelong LERF Example. Top: A mobile robot takes a scan of the scene
and builds a Language Embedded Radiance Field (LERF). Then, the scene is
altered, for example the “Clorox” wipes are replaced by a cookie can. The
robot periodically rescans the scene and identifies what has changed by
rendering semantic features stored in LERF and comparing them against those
extracted from the newly captured images. Bottom: Lifelong LERF efficiently
updates the LERF using new images of the scene, progressively changing local
geometry and semantics.

Prior work Evo-NeRF [7] demonstrates NeRFs can update a scene by discarding
stale images and replacing them with fresh images, similar to how many SLAM
models can update an occupancy distribution. However, for inventory tracking
it is impractical to throw away old images and remap the entire environment.
In this work, we present a new method for localizing changed regions in an
environment to selectively update.

Lifelong LERF detects 3D differences based on the deviation between natural
language embeddings computed over a freshly captured image and a view
previously rendered from the LERF at the same pose, leveraging the novel view
synthesis capabilities of LERF to generate pixel-aligned difference heatmaps.
Given these scene changes, the robot can update the LERF by masking out stale
regions of previous input images and ignoring these pixels during future ray
sampling. As fresh images are collected to the updated scene, the underlying
LERF evolves to match new observations.

The compute required for running SLAM, optimizing a LERF, and computing
semantic differences is too demanding for on-board robot hardware. We address
this issue with cloud computing by using FogROS2 [8]. We conduct experiments
on a Turtlebot 4 and evaluate Lifelong LERF’s ability to maintain an updated
language representation as changes are made to local tabletop scenes. We find
that experiments in a tabletop environment suggest Lifelong LERF can
persistently localize objects through scene changes up to 91% of the time,
with the proposed semantic difference method outperforming a depth baseline in
robustness to false positives and object swapping. This method could be
complementary to methods using semantics for downstream applications such as
language-conditioned grasping [9].

This paper contributes:

  1. 1.

A novel method for detecting changes in a scene by comparing rendered language
embeddings against those calculated from captured images.

  2. 2.

An approach for incorporating detected scene changes progressively into an
evolving LERF.

  3. 3.

A system that allows Lifelong LERF to function on a robot with lightweight
compute (Turtlebot 4 with Raspberry Pi 4) by leveraging FogROS2.

##  II Related Work

###  II-A Semantic SLAM

SLAM algorithms allow robots to localize themselves while building a map of
their surroundings. While SLAM primarily focuses on geometric reconstruction,
recent variants provide impressive tracking and reconstruction abilities.
Several prior systems give robots the ability to model the semantic meaning of
the objects in their environment. Many works used explicit scene
representations, such as volumes, point clouds, scene graphs, and truncated
signed distance functions (TSDF), for semantic scene understanding. [10]
proposes an object-oriented online SLAM system to produce semantically labeled
TSDF object instances reconstructions and 3D foreground masks. Object
segmentation [11, 12, 13, 14, 15] and detection [16] have been leveraged on
volume and points clouds for semantic scene understanding. [17] provides a
library for semantic SLAM by constructing 3D semantic meshes from semantic
hand labeled images. 3D scene graph has also been used to describe a static
scene either on a given mesh [18] or on the incoming frames of observations
[19, 20]. [19] incrementally fuses the semantic prediction of the current
observation into a global semantic graph. [20] constructs a 3D scene graph
incrementally based on topological maps of locally built Euclidean Signed
Distance Functions (ESDFs).

Recent research in open vocabulary vision models [21, 22, 23, 24, 25] leads to
increasing interest in creating 3D representations that incorporate semantic
features. To reconstruct a static semantic SLAM, ConceptFusion [26] combines
SLAM and semantic features by projecting CLIP [5] and features into 3D, where
the labels are refined by 2D unsupervised segmentation.

###  II-B Dynamic Scene Representation

Environments often undergo changes over time. Extensive study has been done on
detecting 2D scene changes [27, 28, 29]. For 3D environments, [30] represents
dynamic scenes with moving agents with 3D dynamic scene graphs by integrating
object and human detection and pose estimation model. [31] uses the Variable
Scene Graph (VSG), an augmentation of 3D scene graph, to represent semantic
scene changes. The variability of VSG is estimated in a supervised way. Prior
work [32, 33, 34, 35] has shown success using TSDFs in capturing long-term
changes. Other works have used 2D point cloud data [36] or spatiotemporal grid
[37] to learn the long-term environment changes but don’t provide semantic
understanding. These systems also require huge storage overhead.

Neural Radiance Fields (NeRF) [6] are attractive alternative representation
for high-quality scene reconstruction from pose RGB images, with an explosion
of recent work on visual quality [38, 39, 40, 41, 42, 43, 44], large-scale
scenes [45, 46, 47], optimization speed [48, 49, 50, 51], dynamic scenes [52,
53, 54], combining SLAM and NeRF [55, 56, 57, 58, 59] and more.

Most similar to this work are methods which seek to adapt NeRF to a continual
framework, such as Evo-NeRF [7], ICNGP [60], [61], CLNeRF [62], and [63].
However, prior works either don’t consider changing scenes or naively discard
stale images. Note that this goal is different from dynamic NeRF
reconstruction [64, 65], which must also recover the full scene motion, and 3D
inpainting [66], which must hallucinate the scene without an object. In
Lifelong LERF, we leverage DROID-SLAM [3] to obtain camera poses. We then
actively detect changed regions and selectively update them to avoid
unnecessarily discarding images.

###  II-C Cloud and Fog Robotics

Cloud Robotics [67] is an emerging computational paradigm for robotics that
enables robots with limited onboard computing capability to gain on-demand
computing hardware resources. Exemplary cloud robotics applications include
SLAM, motion planning [68] and grasp planning [69, 70]. Major Cloud service
providers, such as Amazon Web Services (AWS) and Google Cloud Platform (GCP),
provide proprietary interfaces for robotics applications to interact with
cloud interfaces, such as AWS Greengrass [71] and Google Cloud Robotics Core.
Rapyuta [72] is a platform for centralized management and deployment of a
pipeline of robotics applications. FogROS [73] is the first open-source cloud
robotics platform that offloads robotics applications to public cloud. FogROS
enables robots to interact with the Cloud through the Robot Operating System
(ROS) interfaces. [74] present FogROS2 that supports ROS2 and more major cloud
service providers. FogROS2-SGC [75, 8] secure and globally connects
distributed robots with a peer-to-peer network. In this work, we apply FogROS2
to a low-powered and inexpensive mobile robot.

##  III Problem Statement

Figure 2: _Experiment setup and FOG-ROS2 Integration._ Left: We use a
TurtleBot 4 as the mobile robot. We mount a RealSense D457 RGBD camera on top
of the robot via a monopod. Right: We use FOG-ROS2 to execute both DROID-SLAM
and LERF on a cloud machine. In particular, after the DROID-SLAM node obtains
paired RGB and depth observations, it computes the camera pose. The LERF node
reconstructs the LERF and calculates if the new observation is semantically
inconsistent with the stored representation.

For our algorithm, we assume that:

  1. (i)

Objects are distributed on a local flat surface with sufficient visibility
during a trajectory surrounding them.

  2. (ii)

During any scan of the scene, the objects in the scene are static.

We consider a typical SLAM scenario, with a wheeled, mobile robot equipped
with an RGBD camera on a pole mounted on the robot. We consider several time
periods: in the first time period, the robot constructs an initial model of
the static environment. Between subsequent time periods, a set of objects may
have been moved, added, or removed, and the goal of the robot is to detect
these unknown changes and update the model accordingly. After each time
period, the robot should be able to localize objects specified through natural
language queries, even when objects are moved between time periods.

##  IV Method

Lifelong LERF builds a LERF while the robot moves, updates it through
environment changes, and localizes natural language queries. It has three main
modules:

  1. 1.

Scene reconstruction takes in a stream of RGBD images and executes DROID-SLAM,
concurrently building a LERF with camera poses from SLAM and RGB image
observations.

  2. 2.

The semantic differencing module compares incoming scene observations to the
current reconstruction, producing 3D bounding boxes around changed regions.

  3. 3.

The LERF updating module takes in these bounding boxes and masks stale regions
of images to ensure the reconstruction adapts to the new scene.

###  IV-A Constructing LERF

####  IV-A1 Camera pose estimation

To construct the LERF representation, we need accurate camera poses from an
RGBD camera stream. To provide these, we use DROID-SLAM [3] to estimate camera
poses. DROID-SLAM is a hybrid deep learning and optimization-based SLAM system
that takes in monocular, stereo or RGB-D video and outputs per-keyframe
disparity maps and keyframe poses. It iteratively updates predicted camera
poses and pixel-wise depths through a Dense Bundle Adjustment layer, a
differentiable module that computes a Gauss-Newton update to camera poses and
pixel-wise depth. We choose to use DROID-SLAM for its strong pose tracking
performance, and also for its ability to function without IMU and at lower
camera fps (10hz), making it more robust to network latency.

We directly feed the output keyframes and poses into the LERF to use as input
views. We first initialize the pose prediction process with 4 RGBD images to
ensure DROID-SLAM’s scene scale matches physical scale, then switch to pure
RGB estimation, which we find qualitatively drifts less. After adding initial
poses, camera poses are additionally refined over the course of NeRF
optimization by Nerfstudio’s camera optimization [45], a process which uses
the differentiable rendering capabilities of NeRF to fine-tune poses, as
proposed in BARF [76].

####  IV-A2 Scene box determination

Camera poses generated by DROID-SLAM are not necessarily axis-aligned with
real-world coordinates, primarily because they are determined through an
optimization process that aims for internal consistency rather than alignment
with an external coordinate system. However, the hashgrid encoding [48]
assumes all cameras fit within a pre-defined scene box, an important step for
ensuring parameters are sufficiently distributed throughout the scene. To
account for this, we automatically set the scene scaling based on the first
time period of mapping after the whole scene has been traversed.

####  IV-A3 Concurrency in LERF Computation

Training for LERF typically takes minutes to complete, which is not sufficient
in this context as the robot must actively detect environmental changes and
decide whether to update the underlying LERF model. In the standard LERF
training process, the input images are first pre-processed by extracting CLIP
and DINO features, a process that takes minutes. To enable continual LERF
optimization, we introduce two modifications to the LERF training process.
First, we continuously optimize the LERF model as the robot scans the scene,
dynamically adding images during training as opposed to training on a fixed
set. Second, we compute CLIP and DINO features in parallel with the ongoing
LERF optimization, computing features in a callback and offloading completed
images lazily within the train loop. This requires significant computation,
about 500ms on a 4090 GPU per image, motivating the usage of cloud robotics to
offload this process.

###  IV-B Scene Change Detection

Figure 3: Semantic differencing. The semantic differencing module calculates
2D feature maps from the fresh observation (top), the 3D LERF embeddings
(middle), and the 2D CLIP embeddings of a NeRF-rendered image. $\phi^{rend}$
approximates the distribution shift from 2D to 3D, resulting in higher-quality
semantic difference heatmaps (bottom). See Sec IV-B for details.

Motivated by the constantly changing environments in homes and warehouses,
Lifelong LERF detects changes in the environment over time so that it can
selectively update these regions. Given an image and approximate camera pose
as input, the output of this module is a set of 3D bounding boxes of changed
regions. We explore a semantic differencing method based on language
embeddings rendered from CLIP for computing 3D changed regions. This method
leverages the novel view synthesis ability of implicit neural fields for
comparing the current reconstruction to new input views.

####  IV-B1 3D LERF Feature Difference

Since the primary use case of a semantic map is for semantic queries, this
approach uses the underlying language embeddings to identify scene changes. To
detect semantic changes, we first render the 3D field from the same camera
pose as the new image to obtain a 2D feature map $\phi^{LERF}$. Since LERF is
a multi-scale representation, we pick an image scale of 0.25 for querying the
underlying field with and use the same scale to generate 2D embeddings by
sliding a crop over the input image and passing the crops through the CLIP
image encoder to obtain $\phi^{2D}$.

Naively computing the difference $\phi^{LERF}-\phi^{2D}$ falsely activates on
edges of the scene. This is because the CLIP embeddings in 3D experience a
distribution shift from their respective input views. To understand this,
consider a scene with a tall narrow object: input views of this object contain
primarily the background so their CLIP embeddings are dominated by background
features. However, the background is inconsistent between different viewing
angles, meaning the average embedding in 3D will be different than each 2D
view. To overcome this, we propose a method for estimating this 2D-to-3D
distribution shift and apply it to obtain a semantic re-normalization of
differences obtained. Let $\phi^{rend}$ be the 2D grid of embeddings obtained
by calculating CLIP embeddings of sliding a crop window over a rendered RGB
image from NeRF. Then, the distribution shift can be quantified as
$\phi^{LERF}-\phi^{rend}$. We subtract off this shift from the 2D image
embeddings, re-normalize, then compute the dot product with 3D embeddings to
find the difference. Mathematically, this is

$\texttt{renorm}(\phi^{2D}-(\phi^{LERF}-\phi^{rend}))\cdot\phi^{LERF}$

This re-normalization is critical; visualizations of an ablation are provided
in Fig.3.

####  IV-B2 3D Box Detection

We utilize the semantic differencing module while the robot drives to estimate
3D changed regions. To do this, we binarize heatmaps with a threshold of 0.9,
with values below 0.9 cosine similarity registering as different. We then run
Tarjan’s connected component’s algorithm [77] to parse the heatmap to discrete
heatmaps that do not intersect. We then filter out heatmaps with less than 30
pixels and deproject each heatmap into 3D by using the minimum of NeRF depth
and RealSense depth, to account for the potential of adding or removing
objects. We concatenate point clouds from 15 images at a time, then compute
clusters using DBSCAN and fit oriented bounding boxes to each cluster. We
constrain boxes’ Z axes to align with the world axis.

###  IV-C Updating LERF

Once changes have been identified in 3D, we must selectively update both the
language and visual properties of the LERF to match the scene state. To do
this, we mask input NeRF views from the previous stage by projecting the 3D
bounding boxes into each training view. We then prevent rays from being
sampled inside image masks, which effectively prevents the LERF from
incorporating information from stale regions of images. During each time
period of exploration, we only mask images from prior time periods to avoid
over-masking images. NeRF leverages proposal networks [40] to guide the
volumentric sampling process, which represents a low-frequency 3D distribution
of where geometry lies in the scene. We modify the proposal sampling procedure
by adding a constant probability mass of 0.02 to each ray sample, determined
empirically, ensuring that samples are always taken in free space. This is
important for convergence in locations where objects are added as otherwise
the field would never be sampled where objects are added.

###  IV-D Fog-ROS2 Integration

Wheeled mobile robots possess limited computation power designed for basic
controllers and sensors. Consequently, they cannot execute Lifelong LERF on
their single-board computer, while both DROID-SLAM and LERF necessitate
distinct GPUs. To overcome this limitation, we incorporate Fog-ROS2 to
facilitate access to on-demand cloud computation via a separate workstation.
We encapsulate DROID-SLAM and LERF in ROS2 nodes and implement the
publish/subscribe ROS2 interfaces; consequently, the robot can publish images
to DROID-SLAM in the cloud, which consecutively publishes the maps and
keyframe poses with LERF.

##  V Experiments

To evaluate Lifelong LERF we consider a local table with up to 5 objects lying
on it (see Fig. 2). We use a Turtlebot 4 equipped with a RealSense D457 camera
which faces toward the starboard side. SLAM and LERF optimization executes on
a workstation with 2 RTX 4090s (Fig. 2). This direction is chosen to maximize
camera parallax to enhance SLAM pose estimation and NeRF quality. In addition,
it makes mapping inward-facing scenes easier with a differential drive base.
Experiments progress in time periods: we first capture the entire scene by
driving the robot in a fixed trajectory around the table and train the LERF
for 2000 steps (2 minutes) before initiating the next time period. All
consecutive queries happen without additional idle training time, only
updating the LERF during robot motion.

Figure 4: Experiment setup (Left): Test objects; (Right): Three types of scene
changes included in evaluation. Red box denotes the changed scene region.

In subsequent time periods, we randomly select one or two objects to add,
remove, or replace within the scene (Fig. 5), and the robot executes a partial
trajectory to detect changes and decide whether to remap or not. If it detects
changes, it completes its trajectory around the table and adds new images to
the LERF, otherwise it early-terminates. Between each time period, a language
query for each object is sent to the LERF; both updated and static objects. A
language query is considered a success if the argmax language activation in
the 3D scene lies on the correct object. For objects that were removed, we
consider the localization a success if the argmax language activation no
longer lies on its previous location, and for swapped objects if a query for
the removed object no longer activates on swapped in object. LERF is well
known to struggle with answering “existence” questions [4], so to evaluate how
well the language embeddings are replaced we ensure removed objects’ language
queries no longer remain in the same location, and maintain the same
activation as the background.

The experiments divide possible changes in a scene into 4 categories:

  1. 1.

No Change: after the initial scan, no objects are removed from the scene. This
measures a scene difference detector’s bias towards false positive prediction
(i.e. model predicting that the scene is changed and needs to be remapped but
instead is unchanged).

  2. 2.

Removal: after the initial scan, one or two objects are removed from the
scene.

  3. 3.

Addition: after the initial scan, one or two objects are added to the scene.

  4. 4.

Swap: one or two objects are placed in the scene after the initial scan.

For every experiment with scene changes, we execute 2 difficulty tiers per
category and 2 trials per tier. Tier 1 moves 1 object between each time
period, and Tier 2 moves 2 objects between each time period. All experiments
consist of 3 time periods, except the no change experiment which has 2.

During experiments for add, remove, and swap we measure 1) the percent of
changed objects detected by Lifelong LERF, defined by outputting bounding
boxes fully containing them, 2) the number of pixels masked in the original
image, and 3) language query accuracy for each object on the table after each
time period. We aim for recall to be as close to 100% while minimizing the
number of masked pixels, preserving the original input images wherever
possible. We separately calculate language query accuracy for moved and static
objects, to separately evaluate the robustness of the LERF updating method for
adapting language embeddings without affecting existing regions of the scene.
Finally, to test the false positive rate of the system, we execute the same
trajectory twice and measure whether the system falsely detects a change in
the scene.

| Add | Remove | Swap
---|---|---|---
Change Detection Method | Baseline | Lifelong LERF | Baseline | Lifelong LERF | Baseline | Lifelong LERF
Language Query Accuracy (Moved Objects) (%) | 75 | 92 | 89 | 90 | 67 | 92
Language Query Accuracy (Static Objects)(%) | 83 | 90 | 97 | 97 | 63 | 75
Pixel Mask Ratio (%) | 13 | 21 | 20 | 17 | 17 | 24
Change Detection Recall (%) | 92 | 92 | 83 | 92 | 67 | 92

Table I: Results: We evaluate the two scene change detection methods across 8
scenes. Lifelong LERF has on average an 91% language query on accuracy objects
moved throughout the trial, highlighting its ability to adapt both geometry
and language as the scene changes. Both depth and semantic differencing can
detect scene changes with high recall with similar pixel mask ratios, but
semantic differencing is more robust against false positives (see Tab II).

###  V-A Baseline: Depth-Camera Image Differencing

We compare against depth-based image differencing where the robot renders
depth from the stored LERF at the same pose as the incoming image and
calculates the pixel-wise absolute difference between the two depth images,
then filter differences which are larger than 30cm or smaller than 10cm, and
rejects depth values outside the range 10cm to 150cm. The resulting binary
heatmap is passed through the same pipeline described in Sec IV-B. Note that
both the baseline and lifelong LERF models have the same language query time
and change detection time, since they share the same LERF architecture and
make the same number of 2D renders.

Figure 5: Sequential scene update. Scene reconstructions are shown in the
middle, and human queries are shown at the bottom. As the scene updates, the
heatmap for human queries updates accordingly.

##  VI Results

| No Change
---|---
Change Detection Method | Baseline | Lifelong LERF
Decision Accuracy | 0% | 80%
Pixel Mask Ratio | 8% | 1.2%

Table II: False Positive Evaluation: Depth over-predicts due to SLAM pose
drift and NeRF frame misalignment, leading to high false positives. Lifelong
LERF is more pose-error tolerant.

We average the detection percent and the localization accuracy across 4
different scenes for each category of scene change and summarize the results
in Tables I, II. Empirically, the depth-differencing baseline tends to
mistakenly identify most objects in the scene as changed because of subtle
camera pose drift over time. Any pose misalignment produces false positives in
the depth difference and results in a heatmap that over-segments objects.

###  VI-A No Changes

Because depth is sensitive to pose misalignment, the depth difference method
falsely detects changes every time. Lifelong LERF successfully predicts that
there are no changes within the first 15 images 80% of the time. This saves
substantial time since the robot does not need to fully remap the environment.
Lifelong LERF is more robust to pose misalignment because this language
difference is computed at a lower spatial frequency due to CLIP’s [5]
receptive field.

###  VI-B Adding and Removing

When adding objects, the depth-differencing baseline averages 79% language
query accuracy and a 92% change detection recall rate. Lifelong LERF averages
91% language query accuracy and detects changes in the environment correctly
92% of the time. The depth-based baseline tends to under-select added objects,
leading to insufficient updates to the LERF to update the 3D semantic
features, resulting in slightly worse language performance. In the object
removal setting, the depth-based baseline achieves comparable language query
performance with Lifelong LERF. Prior work [7] has also demonstrated that it
is comparatively easier to remove information from NeRF than to add new
information.

###  VI-C Swapping Objects

The depth baseline suffers significantly because swapping an object largely
preserves scene geometry, while the semantic differencing method is able to
capture the differences in the scene since the language embeddings between new
and old objects differ significantly. This is reflected by the fact that the
baseline localizes moved objects with 67% accuracy, while semantic
differencing localizes a 92%.

Figure 6: Lifelong LERF failure modes:  (Left): NeRF’s spurious density
triggers false difference detection and alters CLIP feature differences
through incorrect activations. (Right): If semantic changes are not correctly
detected, dataset inconsistency will form fuzzy density filled with holes and
fail to fully add or remove the object.

##  VII Limitations

Some of the failure modes are presented in Fig. 6. Lifelong LERF operates
under the assumption of a known scene and uses pre-defined trajectories for
mapping. It is not practical for highly dynamic environments and large-scale
scenes, which may benefit from autonomous exploration and multi-camera setups.
The algorithm also shows sensitivity to object size in semantic change
detection and has a tendency to over-segment scenes. More sophisticated
approach to difference detection which combines segmentation, semantics,
geometry, and perhaps a learned difference detection module may be interesting
future work.

##  VIII Conclusion

In this study, we introduce Lifelong LERF, a system that builds and updates a
LERF to dynamically adapt to semantic changes in a scene. Using the dense
semantics of LERF, we propose a method for detecting scene changes that is
more robust to pose errors or reconstruction inaccuracy compared to a depth
camera differencing baseline. Results suggest high accuracy in change
detection with a minimal rate of false positives in local tabletop settings,
and the ability to rapidly query the updated LERF with natural language to
inventory objects. We offload computation to a local server using Fog-ROS2 to
enable deployment on an inexpensive Turtlebot robot. Future work will study
scaling this method to larger scale room environments, where more complex
mapping is necessary.

## References

  * [1] Johannes Lutz Schönberger and Jan-Michael Frahm  “Structure-from-Motion Revisited”  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016
  * [2] Raul Mur-Artal, Jose Maria Martinez Montiel and Juan D Tardos  “ORB-SLAM: a versatile and accurate monocular SLAM system”  In _IEEE transactions on robotics_ 31.5 IEEE, 2015, pp. 1147–1163
  * [3] Zachary Teed and Jia Deng  “DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras”  In _Advances in neural information processing systems_ , 2021
  * [4] Justin Kerr et al.  “LERF: Language Embedded Radiance Fields”  In _International Conference on Computer Vision (ICCV)_ , 2023
  * [5] Alec Radford et al.  “Learning transferable visual models from natural language supervision”  In _International conference on machine learning_ , 2021, pp. 8748–8763  PMLR
  * [6] Ben Mildenhall et al.  “Nerf: Representing scenes as neural radiance fields for view synthesis”  In _Communications of the ACM_ 65.1 ACM New York, NY, USA, 2021, pp. 99–106
  * [7] Justin Kerr et al.  “Evo-nerf: Evolving nerf for sequential robot grasping of transparent objects”  In _6th Annual Conference on Robot Learning_ , 2022
  * [8] Kaiyuan Chen et al.  “FogROS2-SGC: A ROS2 Cloud Robotics Platform for Secure Global Connectivity”  In _2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2021, pp. 2035–2042  IEEE
  * [9] Adam Rashid et al.  “Language Embedded Radiance Fields for Zero-Shot Task-Oriented Grasping”  In _7th Annual Conference on Robot Learning_ , 2023  URL: https://openreview.net/forum?id=k-Fg8JDQmc
  * [10] John McCormac et al.  “Fusion++: Volumetric object-level slam”  In _2018 international conference on 3D vision (3DV)_ , 2018, pp. 32–41  IEEE
  * [11] Angela Dai et al.  “Scannet: Richly-annotated 3d reconstructions of indoor scenes”  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 5828–5839
  * [12] Francis Engelmann et al.  “3d-mpa: Multi-proposal aggregation for 3d semantic instance segmentation”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 9031–9040
  * [13] Ji Hou, Angela Dai and Matthias Nießner  “3d-sis: 3d semantic instance segmentation of rgb-d scans”  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 4421–4430
  * [14] Jean Lahoud, Bernard Ghanem, Marc Pollefeys and Martin R Oswald  “3d instance segmentation via multi-task metric learning”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 9256–9266
  * [15] Charles Ruizhongtai Qi, Li Yi, Hao Su and Leonidas J Guibas  “Pointnet++: Deep hierarchical feature learning on point sets in a metric space”  In _Advances in neural information processing systems_ 30, 2017
  * [16] Charles R Qi, Or Litany, Kaiming He and Leonidas J Guibas  “Deep hough voting for 3d object detection in point clouds”  In _proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 9277–9286
  * [17] Antoni Rosinol, Marcus Abate, Yun Chang and Luca Carlone  “Kimera: an open-source library for real-time metric-semantic localization and mapping”  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 1689–1696  IEEE
  * [18] Iro Armeni et al.  “3d scene graph: A structure for unified semantics, 3d space, and camera”  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 5664–5673
  * [19] Shun-Cheng Wu et al.  “Scenegraphfusion: Incremental 3d scene graph prediction from rgb-d sequences”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 7515–7525
  * [20] Nathan Hughes, Yun Chang and Luca Carlone  “Hydra: A real-time spatial perception system for 3D scene graph construction and optimization”  In _arXiv preprint arXiv:2201.13360_ , 2022
  * [21] Boyi Li et al.  “Language-driven Semantic Segmentation”  In _International Conference on Learning Representations_ , 2022  URL: https://openreview.net/forum?id=RriDjddCLN
  * [22] Golnaz Ghiasi, Xiuye Gu, Yin Cui and Tsung-Yi Lin  “Scaling open-vocabulary image segmentation with image-level labels”  In _European Conference on Computer Vision_ , 2022, pp. 540–557  Springer
  * [23] Feng Liang et al.  “Open-vocabulary semantic segmentation with mask-adapted clip”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 7061–7070
  * [24] Alexander Kirillov et al.  “Segment Anything”  In _arXiv:2304.02643_ , 2023
  * [25] Jiarui Xu et al.  “ODISE: Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models”  In _arXiv preprint arXiv: 2303.04803_ , 2023
  * [26] Krishna Murthy Jatavallabhula et al.  “ConceptFusion: Open-set Multimodal 3D Mapping”  In _rss_ , 2023
  * [27] Jin-Man Park et al.  “Changesim: Towards end-to-end online scene change detection in industrial indoor environments”  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2021, pp. 8578–8585  IEEE
  * [28] Lixiang Ru, Bo Du and Chen Wu  “Multi-temporal scene classification and scene change detection with correlation based fusion”  In _IEEE Transactions on Image Processing_ 30 IEEE, 2020, pp. 1382–1394
  * [29] Rodrigo Caye Daudt, Bertr Le Saux and Alexandre Boulch  “Fully convolutional siamese networks for change detection”  In _2018 25th IEEE International Conference on Image Processing (ICIP)_ , 2018, pp. 4063–4067  IEEE
  * [30] Antoni Rosinol et al.  “3D dynamic scene graphs: Actionable spatial perception with places, objects, and humans”  In _arXiv preprint arXiv:2002.06289_ , 2020
  * [31] Samuel Looper et al.  “3d vsg: Long-term semantic scene change prediction through 3d variable scene graphs”  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , 2023, pp. 8179–8186  IEEE
  * [32] Lukas Schmid et al.  “Panoptic multi-tsdfs: a flexible representation for online multi-resolution volumetric mapping and long-term dynamic scene consistency”  In _2022 International Conference on Robotics and Automation (ICRA)_ , 2022, pp. 8018–8024  IEEE
  * [33] Ross Finman, Thomas Whelan, Michael Kaess and John J Leonard  “Toward lifelong object segmentation from change detection in dense rgb-d maps”  In _2013 European Conference on Mobile Robots_ , 2013, pp. 178–185  IEEE
  * [34] Marius Fehr et al.  “TSDF-based change detection for consistent long-term dense reconstruction and dynamic object discovery”  In _2017 IEEE International Conference on Robotics and automation (ICRA)_ , 2017, pp. 5237–5244  IEEE
  * [35] Edith Langer, Timothy Patten and Markus Vincze  “Robust and efficient object change detection by combining global semantic information and local geometric verification”  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2020, pp. 8453–8460  IEEE
  * [36] María T Lázaro, Roberto Capobianco and Giorgio Grisetti  “Efficient long-term mapping in dynamic environments”  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2018, pp. 153–160  IEEE
  * [37] Tomáš Krajník, Jaime Pulido Fentanes, Marc Hanheide and Tom Duckett  “Persistent localization and life-long mapping in changing environments using the frequency map enhancement”  In _2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2016, pp. 4558–4563  IEEE
  * [38] Michal Adamkiewicz et al.  “Vision-only robot navigation in a neural radiance world”  In _IEEE Robotics and Automation Letters_ 7.2 IEEE, 2022, pp. 4606–4613
  * [39] Jonathan T Barron et al.  “Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 5855–5864
  * [40] Jonathan T Barron et al.  “Mip-nerf 360: Unbounded anti-aliased neural radiance fields”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5470–5479
  * [41] Li Ma et al.  “Deblur-nerf: Neural radiance fields from blurry images”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 12861–12870
  * [42] Xin Huang et al.  “Hdr-nerf: High dynamic range neural radiance fields”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 18398–18408
  * [43] Sara Sabour et al.  “RobustNeRF: Ignoring Distractors with Robust Losses”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 20626–20636
  * [44] Julien Philip and Valentin Deschaintre  “Radiance Field Gradient Scaling for Unbiased Near-Camera Training”  In _arXiv preprint arXiv:2305.02756_ , 2023
  * [45] Matthew Tancik et al.  “Nerfstudio: A modular framework for neural radiance field development”  In _SIGGRAPH_ , 2023
  * [46] Peng Wang et al.  “F2-NeRF: Fast Neural Radiance Field Training with Free Camera Trajectories”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 4150–4159
  * [47] Jonathan T Barron et al.  “Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields”  In _arXiv preprint arXiv:2304.06706_ , 2023
  * [48] Thomas Müller, Alex Evans, Christoph Schied and Alexander Keller  “Instant neural graphics primitives with a multiresolution hash encoding”  In _ACM Transactions on Graphics (ToG)_ 41.4 ACM New York, NY, USA, 2022, pp. 1–15
  * [49] Anpei Chen et al.  “Tensorf: Tensorial radiance fields”  In _Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII_ , 2022, pp. 333–350  Springer
  * [50] Sara Fridovich-Keil et al.  “K-planes: Explicit radiance fields in space, time, and appearance”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 12479–12488
  * [51] Alex Yu et al.  “Plenoxels: Radiance fields without neural networks”  In _arXiv preprint arXiv:2112.05131_ , 2021
  * [52] Keunhong Park et al.  “HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields”  In _ACM Trans. Graph._ 40.6 ACM, 2021
  * [53] Zhengqi Li et al.  “Dynibar: Neural dynamic image-based rendering”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 4273–4284
  * [54] Albert Pumarola, Enric Corona, Gerard Pons-Moll and Francesc Moreno-Noguer  “D-NeRF: Neural Radiance Fields for Dynamic Scenes”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020
  * [55] Edgar Sucar, Shikun Liu, Joseph Ortiz and Andrew J Davison  “iMAP: Implicit mapping and positioning in real-time”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 6229–6238
  * [56] Zihan Zhu et al.  “Nice-slam: Neural implicit scalable encoding for slam”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 12786–12796
  * [57] Daniil Lisus, Connor Holmes and Steven Waslander  “Towards Open World NeRF-Based SLAM”  In _2023 20th Conference on Robots and Vision (CRV)_ , 2023, pp. 37–44  IEEE
  * [58] Antoni Rosinol, John J Leonard and Luca Carlone  “Nerf-slam: Real-time dense monocular slam with neural radiance fields”  In _arXiv preprint arXiv:2210.13641_ , 2022
  * [59] Chi-Ming Chung et al.  “Orbeez-slam: A real-time monocular visual slam with orb features and nerf-realized mapping”  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , 2023, pp. 9400–9406  IEEE
  * [60] Ryan Po, Zhengyang Dong, Alexander W. Bergman and Gordon Wetzstein  “Instant Continual Learning of Neural Radiance Fields”, 2023  URL: https://api.semanticscholar.org/CorpusID:261531118
  * [61] Zike Yan et al.  “Continual neural mapping: Learning an implicit scene representation from sequential observations”  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 15782–15792
  * [62] Zhipeng Cai and Matthias Mueller  “CLNeRF: Continual Learning Meets NeRF”, 2023  arXiv:2308.14816 [cs.CV]
  * [63] Jiahui Fu et al.  “Robust Change Detection Based on Neural Descriptor Fields”  In _IROS_ , 2022
  * [64] Yu-Lun Liu et al.  “Robust dynamic radiance fields”  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 13–23
  * [65] Mert Asim Karaoglu et al.  “Dynamon: Motion-aware fast and robust camera localization for dynamic nerf”  In _arXiv preprint arXiv:2309.08927_ , 2023
  * [66] Youtan Yin, Zhoujie Fu, Fan Yang and Guosheng Lin  “OR-NeRF: Object Removing from 3D Scenes Guided by Multiview Segmentation with Neural Radiance Fields”  In _arXiv preprint arXiv:2305.10503_ , 2023
  * [67] Ben Kehoe, Sachin Patil, Pieter Abbeel and Ken Goldberg  “A survey of research on cloud robotics and automation”  In _IEEE Transactions on Automation Science and Engineering_ 12.2, 2015, pp. 398–409
  * [68] Jeffrey Ichnowski et al.  “Fog Robotics Algorithms for Distributed Motion Planning Using Lambda Serverless Computing”  In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 4232–4238
  * [69] Nan Tian et al.  “A cloud robot system using the dexterity network and Berkeley robotics and automation as a service (BRASS)”  In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2017, pp. 1615–1622
  * [70] Pusong Li et al.  “Dex-Net as a service (DNaaS): A cloud-based robust robot grasp planning system”  In _IEEE Conference on Automation Science and Engineering (CASE)_ , 2018, pp. 1420–1427
  * [71] “AWS IoT Greengrass” Accessed: 2021-02-15, https://aws.amazon.com/greengrass/
  * [72] Gajamohan Mohanarajah, Dominique Hunziker, Raffaello D’Andrea and Markus Waibel  “Rapyuta: A cloud robotics platform”  In _IEEE Transactions on Automation Science and Engineering_ 12.2, 2014, pp. 481–493
  * [73] Kaiyuan Eric Chen et al.  “FogROS: An Adaptive Framework for Automating Fog Robotics Deployment”  In _2021 IEEE 17th International Conference on Automation Science and Engineering (CASE)_ , 2021, pp. 2035–2042  IEEE
  * [74] Jeffrey Ichnowski et al.  “FogROS2: An Adaptive and Extensible Platform for Cloud and Fog Robotics Using ROS 2”  In _arXiv preprint arXiv:2205.09778_ , 2022
  * [75] Kaiyuan Chen et al.  “FogROS G: Enabling Secure, Connected and Mobile Fog Robotics with Global Addressability”  In _arXiv preprint arXiv:2210.11691_ , 2022
  * [76] Chen-Hsuan Lin, Wei-Chiu Ma, Antonio Torralba and Simon Lucey  “BARF: Bundle-Adjusting Neural Radiance Fields”  In _IEEE International Conference on Computer Vision (ICCV)_ , 2021
  * [77] Robert Tarjan  “Depth-first search and linear graph algorithms”  In _SIAM journal on computing_ 1.2 SIAM, 1972, pp. 146–160