# FastLGS: Speeding up Language Embedded Gaussians with Feature Grid Mapping

Yuzhou Ji  East China Normal UniversityShanghaiChina ,  He Zhu  East China
Normal UniversityShanghaiChina ,  Junshu Tang  Shanghai Jiao Tong
UniversityShanghaiChina ,  Wuyi Liu  East China Normal UniversityShanghaiChina
,  Zhizhong Zhang  East China Normal UniversityShanghaiChina ,  Yuan Xie  East
China Normal UniversityShanghaiChina ,  Lizhuang Ma  Shanghai Jiao Tong
UniversityShanghaiChina and  Xin Tan  East China Normal
UniversityShanghaiChina<EMAIL_ADDRESS>
###### Abstract.

The semantically interactive radiance field has always been an appealing task
for its potential to facilitate user-friendly and automated real-world 3D
scene understanding applications. However, it is a challenging task to achieve
high quality, efficiency and zero-shot ability at the same time with semantics
in radiance fields. In this work, we present FastLGS, an approach that
supports real-time open-vocabulary query within 3D Gaussian Splatting (3DGS)
under high resolution. We propose the semantic feature grid to save multi-view
CLIP features which are extracted based on Segment Anything Model (SAM) masks,
and map the grids to low dimensional features for semantic field training
through 3DGS. Once trained, we can restore pixel-aligned CLIP embeddings
through feature grids from rendered features for open-vocabulary queries.
Comparisons with other state-of-the-art methods prove that FastLGS can achieve
the first place performance concerning both speed and accuracy, where FastLGS
is 98 $\times$ faster than LERF and 4 $\times$ faster than LangSplat.
Meanwhile, experiments show that FastLGS is adaptive and compatible with many
downstream tasks, such as 3D segmentation and 3D object inpainting, which can
be easily applied to other 3D manipulation systems. The project page is at
https://george-attano.github.io/FastLGS/.

open-vocabulary detection, zero-shot learning, semantic 3D field

††journal: TOG††ccs: Computing methodologies Reconstruction††ccs: Computing
methodologies Scene understanding††ccs: Computing methodologies Object
detection

Figure 1. Open-vocabulary query results from FastLGS in reconstructed scenes,
which has the ability to generate high quality masks of single or multiple
objects having the same semantics in novel views under interactive time
consumption. The displayed query times are from rendering resolution 1297x840
(line 1) and 989x731 (line 2).

Teaser.

##  1\. Introduction

With the potential in fields such as robotics and augmented reality, 3D scene
understanding has always been a conspicuous topic in the research community.
Given a set of posed images, the goal is to learn an effective and efficient
3D semantic representation along with scene reconstruction, which should
naturally support a wide range of downstream tasks in scene manipulation.
However, it remains a vital challenge to simultaneously achieve accuracy,
efficiency and the grounding of open-vocabulary level semantics together.

Predominantly, existing methods usually only focus on the accuracy of
differentiating objects, which normally assign points with labels or low-level
features. For example, Semantic-NeRF (Zhi et al., 2021), N3F (Tschernezki et
al., 2022) and Panoptic Lifting (Siddiqui et al., 2023) successfully lift
labels or noisy 2D features into 3D radiance fields, providing semantic
segmentations in certain scenes. However, their labels and low-level features
(e.g., DINO (Caron et al., 2021) features) are not sufficient for natural
modality interactions since some methods require extra position information
(e.g., N3F(Tschernezki et al., 2022)) and others even do not provide any
interaction (e.g., Semantic-NeRF(Zhi et al., 2021) and Panoptic
Lifting(Siddiqui et al., 2023)), and they also suffer from poor quality
concerning in-the-wild scenes due to the lack of zero-shot ability. One
possible solution for introducing zero-shot semantics with friendly
interactions is to use pre-trained text-image models with high-level features.
LERF (Kerr et al., 2023), as one of the state-of-the-art examples, enables
pixel-aligned queries of the distilled 3D CLIP (Radford et al., 2021)
embeddings, supporting long-tail open-vocabulary queries. Nevertheless,
despite solving the problem of open-vocabulary interactions, LERF’s results
are completely not object-centric (in particular, it has been difficult to
selectively edit specific regions or objects(Kobayashi et al., 2022)) and can
only extract fuzzy relevancy maps, which might degrade in novel views because
its non-object-centric CLIP ground truth under interpolation is neither 3D
consistent nor accurate. The above discussion shows the dilemma of balancing
open-vocabulary ability and localizing accuracy when building a semantic
field, not to mention these NeRF-based methods also suffer from the slow
rendering speed.

To get out of such a predicament, one may naturally think of achieving object-
centric along with CLIP embeddings, but this brings numerous problems. First,
we need to achieve the extraction of objects across all views to ensure object
level 3D-consistency, where the popular Segment Anything Model (SAM) (Kirillov
et al., 2023) is strong enough to serve. However, to be a zero-shot method
without any other model guidance in the whole process, the raw multi-
granularity SAM masks can be confusing for the many-to-many masks mapping of
even a single object. Recent LangSplat (Qin et al., 2024) trains an auto-
encoder to handle this issue, but the reconstructed features are degraded and
not sufficient for demanding query tasks. Meanwhile, directly building a high-
level CLIP field is not only time-consuming but also inconsistent across views
because different views of one object can provide close but distinct CLIP
features. Moreover, the query strategy is also largely determined by the
building of a semantic field. If we query by computing the CLIP relevancy of
each pixel as in LERF, it is still not affordable. These are all key
challenges for building the interactive 3D semantic field.

Accordingly, in this paper, we present the FastLGS, which speeds up to build
accurate 3D open-vocabulary semantic fields within 3D Gaussian Splatting
(3DGS) (Kerbl et al., 2023) via the feature grid mapping strategy. Given a set
of posed images, we ensure object-centric by using a mask-based method as the
basis. FastLGS first extracts all SAM masks and sent through the CLIP encoder
to extract object-level image features. While directly training a CLIP feature
field in high dimensionality could be time-consuming, we build a feature grid
to map CLIP features to lower 3D space, but which inherently creates a many-
to-one mapping problem. To tackle the problem of many-to-one mapping problem
in 3D-consistent mapping, we propose a cross-view feature matching strategy to
assign feature grids that represent scene objects with all sides of semantics.
The low-dimensional mappings are attached to Gaussians to build a feature
field. During the inference stage, pixel-aligned features will be rendered and
we restore their semantics within feature grids to generate results upon open-
vocabulary queries.

During experiments, we found FastLGS can generate competitive target masks
compared with state-of-the-art 3D segmentation and semantic field methods,
grounding much more accurate language embeddings than LangSplat (see Figure
2). Meanwhile, the FastLGS is fast and supports real-time interactions under
high resolution (98 $\times$ faster than LERF and 4 $\times$ faster than
LangSplat, see Table 4), which can also be queried for single or multiple
objects with similar queries in an adjustable way. Notably, FastLGS is
adaptive and compatible with many downstream tasks that can be easily applied
to systems like 3D segmentation and manipulation.

In summary, the principal contributions of this work include:

  * •

To our best knowledge, this is the first work that can interactively generate
scene target masks upon natural language queries under high resolution
(1440$\times$1080) in real-time.

  * •

We propose the grid mapping strategy with the zero-shot open-vocabulary
prompted paradigm for building a 3D semantic field in 3DGS. As far as we know,
our method is one of the fastest and most accurate methods in generating
pixel-aligned semantic features in the 3D space.

  * •

We provide an efficient 3D semantic basis and find it adaptive to easily
integrate with several downstream tasks like 3D segmentation and 3D object
inpainting.

Figure 2. Visualized relevancy (turbo) of query “Xbox wireless controller”.
Our result is much more accurate in having higher relevancy within queried
areas and lower relevancy among other regions compared with LangSplat.

.

##  2\. Related works

###  2.1. NeRF and 3D Gaussian Splatting (3DGS)

Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have
revolutionized 3D scene modeling and rendering. Introduced by (Mildenhall et
al., 2021), NeRF synthesizes photorealistic scenes from sparse images by
modeling volumetric density and color, further enhanced by innovations like
mip-nerf (Barron et al., 2021) and TensoRF (Chen et al., 2022). 3D Gaussian
Splatting, detailed by (Kerbl et al., 2023) and expanded upon in work (Luiten
et al., 2024) on dynamic 3D Gaussians, optimizes the rendering of point clouds
with Gaussian kernels for real-time applications. These developments continue
to advance the field, highlighting the dynamic growth and application breadth
of these technologies. Additional contributions by (Tang et al., 2023b) and
others (Fang et al., 2022) underscore the ongoing enhancements and versatility
of 3DGS in handling increasingly complex rendering tasks. Noting the
advantages of 3D Gaussian Splatting in rendering speed and quality, our paper
chooses it as a representation for building semantic field.

###  2.2. 2D and 3D Segmentation

The field of 2D image segmentation has undergone remarkable advancements since
the introduction of Fully Convolutional Networks (FCN) (Long et al., 2015),
which transformed traditional Convolutional Neural Networks (CNN) into
architectures capable of pixel-wise predictions, revolutionizing segmentation
tasks. This innovation spurred expansive growth, with significant
contributions like Mask R-CNN (He et al., 2017), which added a branch for
object mask prediction to Faster R-CNN, and the adoption of Transformer
architecture, notably through SEgmentation TRansformer (SETR) (Zheng et al.,
2021), alongside studies (Cheng et al., 2021; Strudel et al., 2021; Xie et
al., 2021; Cheng et al., 2022). These works demonstrated the Transformer’s
potential in capturing complex dependencies for enhanced semantic
segmentation. Recent innovations such as SAM (Kirillov et al., 2023) and SEEM
(Zou et al., 2023) utilize various kinds of prompts for segmentation, inspired
many mask based researches.

Advancements in 3D segmentation have paralleled those in 2D, with significant
breakthroughs such as PointNet (Qi et al., 2017a) and PointNet++ (Qi et al.,
2017b), which revolutionized the direct processing of point sets for
classification and segmentation. These technologies enhanced the handling of
3D data by adapting and scaling to capture intricate local structures. Further
innovations include Recurrent Slice Networks (RSNets) (Huang et al., 2018) for
improved volumetric data segmentation, and Cylinder3D (Zhou et al., 2020),
tailored for LiDAR semantic segmentation in driving scenes. Following
developments in radiance fields have also refined the precision and
applicability of 3D segmentation techniques (Goel et al., 2023; Tang et al.,
2023a; Cen et al., 2023b), but open-vocabulary text guided 3D segmentation
remains a challenge for the lack of compatible semantic scene construction.

###  2.3. Semantic 3D scene

Advancements in semantic integration have significantly enhanced 3D modeling,
notably through NeRF and 3DGS. (Zhi et al., 2021) introduced semantic layers
into NeRF, setting the stage for enriched scene understanding. This has been
further developed in studies such as LERF (Kerr et al., 2023), ISRF (Goel et
al., 2023), DFF (Yen-Chen et al., 2022), and N3F (Tschernezki et al., 2022),
which integrate detailed semantic segmentation to refine 3D visualizations.

In parallel, enhancements in 3DGS have also advanced semantic capabilities.
The foundational work (Kerbl et al., 2023) and further developments (Gu et
al., 2024; Ye et al., 2023) have refined the application of 3DGS in semantic
segmentation tasks. The SAGA framework (Cen et al., 2023a) demonstrates the
integration of detailed 2D segmentation outcomes into 3D models, enhancing
semantic accuracy. LangSplat (Qin et al., 2024) explores incorporating
language processing into 3DGS, adding a new layer to semantic interpretations
in 3D scene modeling. However, with inconsistent or fuzzy results in varying
views and slow query speed, the quality of semantic fields built by these
approaches still limits their further applications.

Figure 3. FastLGS pipeline. Left: Initialization. Mid: Feature grid
construction and embeddings restoring. Right: query using open-vocabulary
prompts.

Pipeline of FastLGS

##  3\. Method

###  3.1. Overview

As in Figure 3, the input images are initially put for extracting SAM masks
and CLIP embeddings (Sec 3.2). This information is utilized together for the
building of a semantic feature grid and low-dim feature mapping (Sec 3.3 and
Sec 3.4). Then, the obtained pixel-aligned low-dim features are learned in
parallel with scene reconstruction according to Sec 3.5 where we describe the
training for embedded features through Gaussians. At the inference stage, the
built semantic field can be interactively queried for high quality masks in
novel views using natural language based on restored embeddings. The query
strategy is demonstrated in Sec 3.6.

###  3.2. Initialization

First, we initialize the original semantics. Given input images
$\\{\textbf{I}_{t}|t=0,1,...,T\\}$, for each image $\textbf{I}_{i}$, we
utilize SAM to obtain $whole$ segmentation masks
$\\{\textit{M}_{i,j}|j=0,1,...m_{i}\\}$ with a regular grid of 32 $\times$ 32
point prompts and compute respective CLIP features
$\\{\textbf{L}_{i,j}|j=0,1,...m_{i}\\}$. Masks and CLIP embeddings will be
used for building feature grids along with pixel-aligned low-dim feature maps.
After initialization, no other pre-trained model will be required except for a
CLIP text encoder at the inference stage.

###  3.3. Semantic Feature Grid

By practice, simply integrating CLIP feature with 3DGS can lead to infeasible
demand of memory and limited efficiency of rendering as each Gaussian needs to
record and update a parameter in hundreds of dimensions (Sec 4.3). On the
other hand, the CLIP features of an object could show unexpected variance due
to the view angles, redundant background and complex occlusion, posing a
challenge to maintaining consistency and accuracy of semantics in 3D scenes.
As language features of the same object from multiple views should share
similar semantics, it could be compressed for lower training and query costs.
However, current MLP-based compressed and reconstructed features cannot
provide queries of higher quality as the former can be inconsistent and
inaccurate like the raw features while the latter can be degraded which is not
sufficient for demanding scenes (see relevancy maps in Figure 2 and Figure 6).
Meanwhile, similar methods also can not acquire unseen features in a single
view, restricting the possibility of querying objects with semantics captured
in another view but invisible through the current angle.

To address these issues, we propose semantic feature grid, which stores the
multi-view language features of each object in the scene and is mapped to a
low dimensional feature. Specifically, our goal is to map
$\textbf{L}\in\mathbb{R}^{D}$ to $\textbf{f}\in(0,1)^{d}$ where language
features L of the same object share the same low-dim feature f. Each feature f
is assigned to a grid with the size of $\mathcal{K}^{-\frac{1}{d}}$, where
$\mathcal{K}$ refers to the number of objects in a specific scene computed
during cross view matching (Sec 3.4). The low dimensional features will be
assigned pixel-wise according to corresponding masks and trained efficiently
in 3DGS, while the grid can be used to restore accurate CLIP embeddings based
on rendered low-dim features during inference. In practice, this strategy
enables our method to achieve less consumption and faster speed (Table 4).

###  3.4. Cross View Grid Mapping

To achieve consistent features of objects across views, we sequentially match
the denoised masks in adjacent image set I and assign features. Considering
the possible instability of CLIP features in multi-views mentioned in Sec 3.3,
the matching process consists of both key points correspondence and feature
similarity comparison.

Key points correspondence We prioritize the mask correspondence by key points
obtained using the SIFT and the k-nearest neighbors (KNN) algorithm for
accurate and robust matching. The SAM masks with more than $\tau$
corresponding key point pairs will be assigned to the same low dimensional
feature f and considered as segmentation masks of an object from different
viewpoints.

Feature similarity comparison As some segmentations with smooth pixel value
have few key points, the matching process is employed according to similarity
calculated by hybrid features of CLIP embedding L and color distribution C.
For $\textbf{I}_{i}$ and $\textbf{I}_{j}$ with $m_{i}$ and $m_{j}$ mismatched
masks, we calculate the similarity matrix $\textbf{SIM}_{m_{i}\times m_{j}}$
and choose the mask pairs with highest similarity. A threshold $\theta$ is set
that masks with similarity $sim$ higher than $\theta$ will have the same
feature f and the others will be assigned to a new low dimensional feature.

For two segmentation masks $\textit{M}_{i}$, $\textit{M}_{j}$, the similarity
$sim_{i,j}$is calculated as follow:

(1)
$\mathbf{sim}_{i,j}=\alpha{sim}_{i,j}^{color}+(1-\alpha){sim}_{i,j}^{CLIP},$

where $\alpha$ is the weight of similarity of color distribution features C.
${sim}_{i,j}^{CLIP}$ is calculated based on Bhattacharyya distance of C while
${sim}_{i,j}^{color}$ is calculated by cosine similarity:

(2)
$sim_{i,j}^{CLIP}=\frac{\textbf{L}_{i}\cdot\textbf{L}_{j}}{\left\|\textbf{L}_{i}\right\|\left\|\textbf{L}_{j}\right\|};\hskip
5.69054ptsim_{i,j}^{color}=\sqrt{\textbf{C}_{i}\textbf{C}_{j}}.$

Algorithm 1 displays our procedure of correspondence-based grid feature
mapping. The consistency of objects across views is built and the amount
$\mathcal{K}$ is accessed during the matching process, which helps establish
our semantic feature grids and low-dimensional feature mapping.

Input: Image sequence $\\{\textbf{I}_{t}|t=0,1,...,T\\}$ with segmentation
masks $\\{\textit{M}_{i,j}|i=0,1,...,T;\>j=0,1,...,m_{i}\\}$ CLIP features
$\\{\textbf{L}_{i,j}|i=0,1,...,T;\>j=0,1,...,m_{i}\\}$ and color features
$\\{\textbf{C}_{i,j}|i=0,1,...,T;\>j=0,1,...,m_{i}\\}$, index of each mask
$M_{i,j}$’s low dimensional feature $Idx_{i,j}$, parameters $\alpha$, $\tau$
and $\theta$

Output: low dimensional feature indices $Idx$ of masks M and the number
$\mathcal{K}$ of objects in a scene

for  _each image $\textbf{I}_{i}$_ do

if  _i == 0_ then

$\mathcal{K}$, $Idx_{0}$ = initialise($\textbf{I}_{0}$, $\textit{M}_{0}$,
$\textbf{L}_{0}$);

$\textbf{I}^{prev}$ = $\\{\textbf{I}_{0}\\};\textit{M}^{prev}$ =
$\\{\textit{M}_{0,i}|i=0,1,...,m_{0}\\}$;

else

$corrInfo$ = correspondKp($I_{i}$, $\textbf{I}^{prev}$, $\textit{M}^{prev}$,
);

for  _each mask $\textit{M}_{i,j}$ in image $I_{i}$_ do

$k$, $\textit{M}^{prev}_{k}$, $numKp$ = selMostCorrMask($\textit{M}_{i,j}$,
$corrInfo$);

if  _$numKp$ ¿ $\tau$_ then

$Idx^{prev}_{k}$ = low-dim feature index of $\textit{M}^{prev}_{k}$;

$Idx_{i,k}$ = $Idx^{prev}_{k}$;

else

$sims_{i,j}$ = computeSims($\textit{M}_{i,j}$, $\textit{M}^{prev}$, L, C);

$k$, $\textit{M}^{prev}_{k}$, $sim$ = selMostSimMask($\textit{M}_{i,j}$,
$\textit{M}^{prev}$, $sims_{i,j}$);

if  _$sim$ ¿ $\theta$_ then

$Idx^{prev}_{k}$ = low-dim feature index of $\textit{M}^{prev}_{k}$;

$Idx_{i,j}$ = $Idx^{prev}_{k}$;

else

$\mathcal{K}$ = $\mathcal{K}+1;Idx^{prev}_{k}$ = $\mathcal{K}$;

end if

end if

$\textbf{I}^{prev}$ = union($\textbf{I}^{prev}$, $\\{\textbf{I}_{i}\\}$);

$\textit{M}^{prev}$ = union($\textit{M}^{prev}$,
$\\{\textit{M}_{i,j}|j=0,1,...,m_{i}\\}$);

end for

end if

end for

ALGORITHM 1 Grid Feature Mapping

###  3.5. Training Features for Gaussians

We train a mapped low-dimensional feature $\mathbf{f}_{m}$ for each gaussian
$g$ to build a feature field, which is adaptive for integration.

Rendering Features Given a camera pose $v$, we compute the feature
$\mathbf{F}_{v,p}$ of a pixel by blending a set of ordered Gaussians
$\mathcal{N}$ overlapping the pixel similar to the color computation of 3DGS:

(3)
$\mathbf{F}_{v,p}=\sum_{i\in\mathcal{N}}\mathbf{f}_{i}a_{i}\prod\limits_{j=1}^{i-1}(1-a_{j}),$

where $a_{i}$ is given by evaluating a 2D Gaussian with covariance $\sum$
multiplied with a learned per-Gaussian opacity.

Optimization The low-dim features follows 3DGS optimization pipeline and
especially inherits the fast rasterization for efficient optimization and
rendering. The loss function for features is $\mathcal{L}_{1}$ combined with a
D-SSIM term:

(4) $\mathcal{L}_{f}=(1-\lambda)\mathcal{L}_{1}+\lambda\mathcal{L}_{D-SSIM},$

where $\lambda$ is also fixed to 0.2 in all cases.

###  3.6. Querying Features

Once trained, FastLGS can be interactively queried for different objects in
the scene using open vocabulary text prompts. First, we query the pixel-
aligned semantics $\mathbf{F}_{v}$ by projecting the attached feature to view
the plane using the rendering for features mentioned in Sec. 3.5, which can be
used for all queries within camera pose $v$.

Relevancy Score When provided a prompt, we assign its language relevancy
scores based on grid image features similar to LERF. We compute the cosine
similarity between image embedding $\phi_{img}$ and canonical phrase
embeddings $\phi_{canon}^{i}$, then compute the pairwise softmax between image
embedding and text prompt embedding $\phi_{query}$, so that the relevancy
score is:

(5)
$S_{relev}=\min_{i}\frac{\exp(\phi_{img}\cdot\phi_{query})}{\exp(\phi_{img}\cdot\phi_{canon}^{i})+\exp(\phi_{img}\cdot\phi_{query})}.$

For canonical phrases, we use “object”, “stuff” and “texture” for all queries.

Target Mask We go through feature grids to restore high-level image features
and use them to compute relevancy with text embeddings. The grid with the
highest relevancy score is deemed to be the target grid queried. Because each
grid has multi-view image features of an object, we are even able to locate
targets based on semantics invisible in the current query view (for example,
locating a book named “computer vision” from the back of it). While the
rendered semantics have been smoothed, we locate their corresponding feature
grids by calculating euclidean distance between grids and rendered features:
$Dis_{gf}=\sqrt{\sum_{i=0}^{2}(\mathbf{F}^{v,i}_{p}-\mathbf{p}_{i})^{2}}$
where $\mathbf{p}$ denotes the grid’s corresponding low-dim features. We
compute pixel-wise $Dis_{gf}$ based on target grid, and points with $Dis_{gf}$
lower than threshold $\tau_{ac}$ forms the target mask. By changing the number
of grids used, we can adjustably query multiple targets (as Figure 1).

##  4\. Experiments

In this section, we first show the speed and quality of open-vocabulary object
retrieval in comparison with other state-of-the-art methods through
quantitative experiments in Sec 4.1, then we provide qualitative results of
downstream tasks including 3D segmentation and object deletion in Sec 4.2.
Ablation studies are in Sec 4.3 to demonstrate the rationality of feature grid
based design.

Datasets For quantitative experiments, we train and evaluate the models on
datasets including SPIn-NeRF (Mirzaei et al., 2023), LERF (Kerr et al., 2023)
and 3D-OVS (Liu et al., 2023). For experiments on downstream applications we
also used the MIP-360 datatset (Barron et al., 2022) (see example 4 of Figure
6).

Implementation Details We use the same OpenClip ViT-B/16 model as LERF and SAM
ViT-H model as LangSplat. We train the features and scenes in 3DGS for 30,000
iterations. Activation threshold $\tau_{ac}$, correspondence threshold $\tau$
and $\theta$ are set to $5.0$, 4 and 0.95. Weight $\alpha$ is set to 0.3. f is
normalized to $(0,255)^{3}$. Experiments show robustness of the proposed
method to the aforementioned parameters. While the original time calculation
of LangSplat puts aside feature rendering time and feature reconstructing
time, here we compute the query time of the whole query process as LERF does.
The tested LERF masks are regions with relevancy higher than 20% after
normalization. The normalization for each query is from 50% (less relevant
than canonical phrases) to the maximum relevancy, which is identical to the
visualization strategy of LERF. All results are reported running on a single
TITAN RTX GPU.

Figure 4. Visual results of retrieved objects in different scenes.

Qualitative visual results on different datasets.

###  4.1. Comparison With Other State-of-the-art Methods

For quantitative experiments, we test different methods on the SPIn-NeRF
dataset, the LERF dataset and 3D-OVS dataset. We show the quality of FastLGS-
generated masks by comparing with other state-of-the-art 3D segmentation
methods including the multi-view segmentation of SPIn-NeRF (MVSeg) (Mirzaei et
al., 2023) and SA3D (Cen et al., 2023b), and compare the language retrieval
ability with LERF (Kerr et al., 2023) and LangSplat (Qin et al., 2024).

SPIn-NeRF dataset We evaluate the IoU and pixel-wise accuracy of masks with
provided ground truth (1008 $\times$ 567), and also show the time consumption
for each query, which is omitted in MVSeg and SA3D because they do not support
single view query. For LERF, LangSplat and FastLGS, we use the same text
queries for the target segmentation objects to generate masks. Other methods
all follow their original settings when tested on this dataset. We also
provide the 2D segmentation results generated by SAM based on manual point
prompts of original scene images for comparison. As shown in Table 1, FastLGS
generates masks with competitive quality which have higher mIoU compared with
other methods, and largely outperforms LERF and LangSplat in query speed.

LERF dataset The LERF dataset contains a number of in-the-wild scenes and is
much more challenging, which strongly requires zero-shot abilities. We report
localization accuracy for the 3D object localization task following LERF (Kerr
et al., 2023) with ground truth annotations provided by LangSplat (Qin et al.,
2024) (resolution around 985 $\times$ 725). Results are shown in Table 2 and
visual examples in Figure 4 (line 1&2), which further demonstrates FastLGS’s
advantages in natural language retrieval.

3D-OVS dataset We also compare with 2D-based open-vocabulary segmentation
methods including ODISE (Xu et al., 2023) and OV-Seg (Liang et al., 2023)
along with 3D-based methods including 3D-OVS (Liu et al., 2023), LERF (Kerr et
al., 2023) and LangSplat (Qin et al., 2024). Results are provided in Table 3,
where our method can outperform both 2D and 3D methods. In Figure 4 (line 3)
our method can also better support demanding custom query “keyboard and pocket
tissues” which fails in LangSplat. Meanwhile, Figure 6 provides a
visualization of pixel-aligned relevancy upon different queries, where our
method has more accurate relevancy scores compared with LERF and LangSplat.
Table 4 further shows the query time and model size where our method also
outperforms LERF and LangSplat, proving its advantages in real-world
applications.

Table 1. Quantitative Results on SPIn-NeRF dataset (LERF masks usually cover large surrounding areas and result in high mAcc and low mIoU). Method | mIoU (%) | mAcc (%) | mTime (s)
---|---|---|---
SAM(2D) | 95.7 | 96.1 | 0.05
MVSeg | 89.5 | 97.8 | -
SA3D | 90.9 | 98.3 | -
LERF | 81.0 | 99.5 | 30.2
LangSplat | 92.2 | 94.7 | 1.43
OURS | 93.1 | 95.2 | 0.31

Table 2. Quantitative Results of localization accuracy on LERF dataset. Scene | LERF | LangSplat | OURS
---|---|---|---
ramen | 61.9 | 73.2 | 84.2
figurines | 75.5 | 80.4 | 91.4
teatime | 84.8 | 88.1 | 95.0
waldo_kitchen | 70.2 | 95.5 | 96.2
Overall | 73.1 | 84.3 | 91.7

Table 3. Quantitative Results of mIoU scores (%) on 3D-OVS dataset. Method | bed | bench | room | sofa | lawn | Overall
---|---|---|---|---|---|---
ODISE | 55.6 | 30.1 | 53.5 | 49.3 | 39.1 | 45.5
OV-Seg | 79.8 | 88.9 | 71.4 | 66.1 | 81.2 | 77.5
3D-OVS | 89.5 | 89.3 | 92.8 | 74.1 | 88.2 | 86.8
LERF | 76.2 | 59.1 | 56.4 | 37.6 | 78.2 | 61.5
LangSplat | 92.6 | 93.2 | 94.1 | 89.3 | 94.5 | 92.7
OURS | 94.7 | 95.1 | 95.3 | 90.6 | 96.2 | 94.4

Table 4. Comparison of performance and consumption (time computed on rendering resolution 1440x1080). Scene “sofa” | LERF | LangSplat | OURS
---|---|---|---
mTime (s/q) | 51.2 | 2.14 | 0.52
Model Size | 1.28GB | 565MB | 200MB

###  4.2. Downstream Applications

In this section, we test our method for the ability of applying to downstream
3D object manipulation tasks. Successful visual results are provided in Figure
6.

Language Driven 3D Segmentation We integrate FastLGS with Segment Any 3D
Gaussians (SAGA) (Cen et al., 2023a) to achieve language driven 3D
segmentation. SAGA attaches a low-dimensional 3D feature to each Gaussian in
the trained 3DGS model to conduct mask-based segmentation. We attach our low-
dim features to Gaussians in parallel with SAGA. The original SAGA requires
manual positional prompting for reference mask selection, but after
integration we can directly use our masks as segmentation references and
conduct 3D segmentation based on natural language. The whole query and
segmentation process can be completed around only 1s.

Language Driven Object Inpainting We generate target masks of queried objects
across views and use them as multi-view segmentation masks. We follow the
multi-view segmentation based inpainting pipeline of SPIn-NeRF (Mirzaei et
al., 2023) to conduct 3D object inpainting using the extracted multi-view
segmentation masks. The masks are dilated by default with a 5x5 kernel for 5
iterations to ensure that all of the object is masked as in SPIn-NeRF. Results
show that consistent masks can be directly used for high-quality object
inpainting through 3D inpainting methods.

According to the above experiments in Sec 4.1 and Sec 4.2, our method proves
to be superior in grounding accurate semantics with interactive efficiency and
more affordable consumption, further paving the way for many downstream 3D
manipulation tasks in open-vocabulary interaction development.

###  4.3. Ablations

We conduct ablation to validate the necessity of our semantic feature grid for
query and the improvements for the accuracy of our components in building
cross-view grid mapping.

Querying Feature The performance is reported in Table 5, where SFG represents
the semantic feature grid. Initially, we directly reconstruct the semantic
field with NeRF using extracted CLIP features for SAM-based masks of images.
When replacing NeRF with 3D Gaussian Splatting, the high dimensions of CLIP
feature result in running out of memory. The issue is addressed by combining
it with a semantic feature grid, which also brings improvements in accuracy
and speed.

Cross view  We report results in Table 6 and Figure 5, where KP represents key
points and CD represents color distribution feature C. Key points
correspondence and features similarity with the auxiliary of color
distribution provide more guidance for maintaining consistency of objects
across a series of continuous perspectives, which are dedicated to training
low-dim features of high quality for Gaussians and effectively improving the
performance.

Table 5. Ablation on querying feature. Component | Performance
---|---
SAM | 3D-GS | SFG | mIoU(%) | mTime(s)
✓ |  |  | 83.3 | 20.1
✓ | ✓ |  | OOM | OOM
✓ | ✓ | ✓ | 95.1 | 0.98

Table 6. Ablation on the semantic feature grid. Component | Performance
---|---
CLIP | KP | CD | mIoU(%) | mAcc(%)
✓ |  |  | 75.2 | 83.2
✓ | ✓ |  | 87.5 | 91.8
✓ | ✓ | ✓ | 92.1 | 97.5

Figure 5. Ablation on keypoint and color feature matching.

This is ablation.

Figure 6. More visual results on different datasets. Example 1: Query
relevancy visualization. Example 2&3: Object retrieval. Example 4: Downstream
3D segmentation. Example 5: Downstream 3D inpainting.

More visual results on different datasets. Example 1: Query relevancy
visualization. Example 2&3: Object retrieval. Example 4: Downstream 3D
segmentation. Example 5: Downstream 3D inpainting.

##  5\. Conclusions and future work

In this paper, we propose FastLGS, a method that speeds up to build accurate
3D open-vocabulary semantic fields within 3DGS. By mapping and restoring
multi-view CLIP embeddings through feature grids instead of MLPs, FastLGS not
only has interactive query speed, but also provides more accurate target masks
and semantic relevancy along with lower consumption compared with state-of-
the-art methods. Meanwhile, experiments show FastLGS’s ability to be
integrated with other downstream 3D manipulation tasks. While FastLGS now
follows an object-centric paradigm, challenges occur in querying for parts of
an object, such as “bicycle handlebars” or “chair leg”. We believe this could
possibly be solved by building the feature grid in a multi-granularity way,
where actual performance (time and space) along with cross-view consistency of
both mask extraction and query results remain a problem, which requires
further researches.

## References

  * (1)
  * Barron et al. (2021) Jonathan T. Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P. Srinivasan. 2021.  Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 5855–5864.
  * Barron et al. (2022) Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. 2022.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 5470–5479.
  * Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021.  Emerging properties in self-supervised vision transformers. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 9650–9660.
  * Cen et al. (2023a) Jiazhong Cen, Jiemin Fang, Chen Yang, Lingxi Xie, Xiaopeng Zhang, Wei Shen, and Qi Tian. 2023a.  Segment Any 3D Gaussians.  _arXiv preprint arXiv:2312.00860_ (2023).
  * Cen et al. (2023b) Jiazhong Cen, Zanwei Zhou, Jiemin Fang, Wei Shen, Lingxi Xie, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, et al. 2023b.  Segment Anything in 3D with NeRFs.  _Advances in Neural Information Processing Systems_ 36 (2023), 25971–25990.
  * Chen et al. (2022) Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. 2022.  Tensorf: Tensorial radiance fields. In _European Conference on Computer Vision (ECCV)_. Springer, 333–350.
  * Cheng et al. (2022) Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. 2022.  Masked-Attention Mask Transformer for Universal Image Segmentation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 1290–1299.
  * Cheng et al. (2021) Bowen Cheng, Alex Schwing, and Alexander Kirillov. 2021.  Per-Pixel Classification is Not All You Need for Semantic Segmentation. In _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 17864–17875.
  * Fang et al. (2022) Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. 2022.  Fast Dynamic Radiance Fields with Time-Aware Neural Voxels. In _SIGGRAPH Asia 2022 Conference Papers_. Association for Computing Machinery, New York, NY, USA, Article 11, 9 pages.
  * Goel et al. (2023) Rahul Goel, Dhawal Sirikonda, Saurabh Saini, and P. J. Narayanan. 2023.  Interactive Segmentation of Radiance Fields. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 4201–4211.
  * Gu et al. (2024) Qiao Gu, Zhaoyang Lv, Duncan Frost, Simon Green, Julian Straub, and Chris Sweeney. 2024.  EgoLifter: Open-world 3D Segmentation for Egocentric Perception.  arXiv:2403.18118 [cs.CV]
  * He et al. (2017) Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross Girshick. 2017.  Mask R-CNN. In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_.
  * Huang et al. (2018) Qiangui Huang, Weiyue Wang, and Ulrich Neumann. 2018.  Recurrent Slice Networks for 3D Segmentation of Point Clouds. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Kerbl et al. (2023) Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 2023.  3D Gaussian Splatting for Real-Time Radiance Field Rendering.  _ACM Transactions on Graphics_ 42, 4 (July 2023).
  * Kerr et al. (2023) Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. 2023.  LERF: Language Embedded Radiance Fields. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 19729–19739.
  * Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. 2023.  Segment Anything. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 4015–4026.
  * Kobayashi et al. (2022) Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. 2022.  Decomposing NeRF for Editing via Feature Field Distillation. In _Advances in Neural Information Processing Systems_ , Vol. 35.
  * Liang et al. (2023) Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. 2023.  Open-vocabulary semantic segmentation with mask-adapted clip. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 7061–7070.
  * Liu et al. (2023) Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik, Christian Theobalt, Eric Xing, and Shijian Lu. 2023.  Weakly Supervised 3D Open-vocabulary Segmentation.  _arXiv preprint arXiv:2305.14093_ (2023).
  * Long et al. (2015) Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015.  Fully Convolutional Networks for Semantic Segmentation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Luiten et al. (2024) Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. 2024.  Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis. In _International Conference on 3D Vision (3DV)_.
  * Mildenhall et al. (2021) Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. 2021.  Nerf: Representing scenes as neural radiance fields for view synthesis.  _Commun. ACM_ 65, 1 (2021), 99–106.
  * Mirzaei et al. (2023) Ashkan Mirzaei, Tristan Aumentado-Armstrong, Konstantinos G Derpanis, Jonathan Kelly, Marcus A Brubaker, Igor Gilitschenski, and Alex Levinshtein. 2023.  SPIn-NeRF: Multiview segmentation and perceptual inpainting with neural radiance fields. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 20669–20679.
  * Qi et al. (2017a) Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2017a.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Qi et al. (2017b) Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017b.  PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. In _Advances in Neural Information Processing Systems_ , Vol. 30.
  * Qin et al. (2024) Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. 2024.  LangSplat: 3D Language Gaussian Splatting. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International conference on machine learning_. PMLR, 8748–8763.
  * Siddiqui et al. (2023) Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder. 2023.  Panoptic Lifting for 3D Scene Understanding With Neural Fields. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 9043–9052.
  * Strudel et al. (2021) Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. 2021.  Segmenter: Transformer for Semantic Segmentation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 7262–7272.
  * Tang et al. (2023b) Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. 2023b.  DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation.  arXiv:2309.16653 [cs.CV]
  * Tang et al. (2023a) Songlin Tang, Wenjie Pei, Xin Tao, Tanghui Jia, Guangming Lu, and Yu-Wing Tai. 2023a.  Scene-Generalizable Interactive Segmentation of Radiance Fields. In _Proceedings of the 31st ACM International Conference on Multimedia_. 6744–6755.
  * Tschernezki et al. (2022) Vadim Tschernezki, Iro Laina, Diane Larlus, and Andrea Vedaldi. 2022.  Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations. In _2022 International Conference on 3D Vision (3DV)_. 443–453.
  * Xie et al. (2021) Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. 2021.  SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers. In _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 12077–12090.
  * Xu et al. (2023) Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. 2023.  Open-vocabulary panoptic segmentation with text-to-image diffusion models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 2955–2966.
  * Ye et al. (2023) Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. 2023.  Gaussian Grouping: Segment and Edit Anything in 3D Scenes.  arXiv:2312.00732 [cs.CV]
  * Yen-Chen et al. (2022) Lin Yen-Chen, Pete Florence, Jonathan T. Barron, Tsung-Yi Lin, Alberto Rodriguez, and Phillip Isola. 2022.  NeRF-Supervision: Learning Dense Object Descriptors from Neural Radiance Fields. In _2022 International Conference on Robotics and Automation (ICRA)_. 6496–6503.
  * Zheng et al. (2021) Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip H.S. Torr, and Li Zhang. 2021.  Rethinking Semantic Segmentation From a Sequence-to-Sequence Perspective With Transformers. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_. 6881–6890.
  * Zhi et al. (2021) Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. 2021.  In-Place Scene Labelling and Understanding With Implicit Scene Representation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_. 15838–15847.
  * Zhou et al. (2020) Hui Zhou, Xinge Zhu, Xiao Song, Yuexin Ma, Zhe Wang, Hongsheng Li, and Dahua Lin. 2020.  Cylinder3D: An Effective 3D Framework for Driving-scene LiDAR Semantic Segmentation.  _CoRR_ abs/2008.01550 (2020).
  * Zou et al. (2023) Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. 2023.  Segment Everything Everywhere All at Once. In _Advances in Neural Information Processing Systems_ , A. Oh, T. Neumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (Eds.), Vol. 36. Curran Associates, Inc., 19769–19782.