#  LabelMaker:
Automatic Semantic Label Generation from RGB-D Trajectories

Silvan Weder∗,1 Hermann Blum∗,1 Francis Engelmann1,2,3 Marc Pollefeys1,4

1ETH Zurich 2ETH AI Center 3Google 4Microsoft

###### Abstract

Semantic annotations are indispensable to train or evaluate perception models,
yet very costly to acquire. This work introduces a fully automated 2D/3D
labeling framework that, without any human intervention, can generate labels
for RGB-D scans at equal (or better) level of accuracy than comparable
manually annotated datasets such as ScanNet. Our approach is based on an
ensemble of state-of-the-art segmentation models and 3D lifting through neural
rendering. We demonstrate the effectiveness of our LabelMaker pipeline by
generating significantly better labels for the ScanNet datasets and
automatically labelling the previously unlabeled ARKitScenes dataset. Code and
models are available at labelmaker.org.

Figure 1:  LabelMaker bundles a collection of state-of-the-art segmentation
models with different sets of predicted classes in a neural field. LabelMaker
can refine existing annotations and produce highly accurate 2D as well as 3D
labels on ScanNet _(top-right)_. At the same time, it opens new possibilities
to rapidly label large-scale datasets without human effort such as ARKitScenes
_(bottom-right)_.

##  1 Introduction

Semantic perception of the world around us is of central importance for many
computer vision applications [23, 28, 33]. Without semantic perception,
meaningful interactions with our environment are hardly possible. Thus,
semantic scene perception has been a long-standing problem in computer vision
and robotics [19, 23, 12, 6]. In recent years, most solutions have converged
towards using deep neural networks. However, training and evaluating these
networks is hard. As recent works such as SAM [10], language-based models [16,
20, 27], or InternImage [30] have shown, huge quantities of training data,
orders of magnitude larger than any single existing research dataset, are
necessary to achieve good generalization. On the other hand, generalization is
necessary because the distribution of the deployment environment - _e.g._ , a
particular user’s home, in which a robotic application is to be deployed - is
outside of the distribution of existing annotated training datasets. To
evaluate generalization in or adapt to specific deployment environments,
labeled data of these environments is required. From both training and
deployment perspectives, the availability of labeled data is therefore a key
problem. Unfortunately, the acquisition of this data is usually very expensive
as semantic ground-truth annotation is a time-consuming manual process.

In this work, we particularly focus on 3D semantic segmentation. The available
scale of 3D semantic segmentation data such as ScanNet [8] or Matterport3D [4]
is far below the scale of 2D semantic segmentation datasets like ADE20k [38],
COCO-stuff [3], or others [25, 31, 7]. Even tough tasks such as semantic
segmentation or online semantic reconstruction gain maturity and are crucial
for interactive applications, there is even less semantic data with paired
camera trajectories and corresponding scene reconstructions. ScanNet [8] is by
far the largest in this domain with an abundance of scenes and a well-
established benchmark. However, both camera images and labels are oftentimes
noisy, making it hard to generalize from ScanNet to other datasets.
ARKitScenes [1] shows the growing possibility to capture RGB-D trajectories at
scale, and at the same time illustrates the cost of semantic annotations,
featuring an incomplete list of bounding boxes.

To push the scale and accuracy of 3D semantic segmentation datasets, we
present _LabelMaker_. LabelMaker automatically creates labels that are on the
same level of accuracy as the established ScanNet benchmark, but without any
human annotation. Further, we show that it can produce better labels than the
original ScanNet labels when using the human annotations as an additional
input.

The design of our method is motivated by two observations. The first
observation is on recent advances in 2D semantic segmentation, where a leap in
training data scale through combination of different tasks and datasets [30]
or visual-language models [16] has boosted generalization. The second
observation is in the field of neural radiance fields, where [36, 17, 24] have
shown that NeRFs can be used to denoise semantic input labels and learn a
multi-view consistent semantic label field. We leverage these two observations
and motivate an automatic labelling pipeline with two main components at its
heart. First, we leverage large 2D models, that combine the power of different
tasks and input modalities, in order to predict different hypothesis for
labels in 2D. These labels are aggregated using our consensus voter in order
to obtain a single 2D prediction for every frame. Second, all 2D predictions
are aggregated and made consistent using a neural radiance field. This neural
radiance field can be used to render clean and consistent 2D label maps.
Alternatively, the labels can be aggregated and mapped into 3D to obtain
labeled pointclouds or meshes.

With a comparison to SOTA methods and datasets and an extensive ablation
study, we showcase that our method automatically generates labels of similar
quality than human annotators. We also demonstrate fully automatic labelling
for ARKitScenes, for which no dense labels exist to date.

In summary, our contributions are:

  * •

A curated mapping between the indoor label sets NYU40, ADE20k, ScanNet,
Replica, and into the wordnet graph.

  * •

A pipeline to automatically label RGB-D trajectories, as well as corresponding
3D point clouds, that achieves higher quality than the original labels of
ScanNet.

  * •

Generated labels in 3D meshes and 2D images for ScanNet [8] and ARKitScenes
[1].

##  2 Related Work

Labelling in 2D. Cityscapes [7] is one of the most established 2D semantic
segmentation datasets. The authors report an effort of more than 1.5h to
annotate a single frame. Similar frame-by-frame manual annotations were
provided in NYU Depth [25], ADE20k [38], or COCO-stuff [3]. While frame-by-
frame annotations yield very high quality segmentation masks, they are
expensive to obtain. Although the effort can be reduced through comfortable
annotation tools [13, 2], it cannot be avoided that a human inspects every
image and performs at least a couple of clicks.

Labelling in 3D. If scenes are annotated in 3D, their annotations can easily
be rendered into any localized camera image in the same scene, therefore
potentially reducing labeling effort. This approach was followed in Replica
[26] and ScanNet [8]. iLabel [37] pioneered to use NeRFs for this type of
rendering, additionally showing that NeRFs have an intrinsic capability to
segment whole objects along texture boundaries from a few clicks. Similarly,
[11, 34] also reduce the manual labelling effort to a few positive and
negative clicks per object. Matterport [4] consists of large labeled 3D scans,
but does not have corresponding 2D images and therefore can only be used for
3D methods.

Pretrained Models. It is a well-established approach in labelling to label
parts of a dataset, train a model on that part, and use its predictions to
bootstrap labels for the rest of the data. More recently, models pretrained on
large amounts of data have been introduced to help labelling completely unseen
datasets. SAM [10] showed impressive results of segmenting objects in images
from close to zero clicks where only labels have to be assigned. The seconds
step can even be bootstrapped through CLIP [21]. CLIP2Scene [5] takes a
similar approach in 3D to train a pointcloud classifier on previously
unlabeled data.

##  3 Method

We briefly discuss the relabelling of ScanNet scenes. Then, we discuss the
translation between prediction spaces. Finally, we present our automatic
labelling pipeline.

###  3.1 Relabeling ScanNet Scenes

To be able to evaluate the quality of LabelMaker, we want to compare it
against existing human annotations. We choose the ScanNet dataset because its
scale has a large potential for automatic processing. To be able to evaluate
the quality of the existing labels and compare them with LabelMaker, we create
high-quality annotations for a selection of scenes.

The original ScanNet [8] labels were created using free text user prompts.
They consequently have duplicates or are ill-defined. This reflects the open-
world approach of Dai et al. [8], but contradicts the use as benchmark labels,
for which they map them to other class sets. As a set of annotation classes,
we therefore did not directly annotate with ScanNet classes, but use wordnet
[18] synkeys111Wordnet is a dictionary and synkeys are the names of its
entries. I.e., a set of synonymous words has 1 synkey, but a word with
different meanings as one synkey per definition.. In particular, we start from
the mapping that ScanNet defined between their labels and wordnet and take the
categories that occur at least three times in the dataset. This yields an
initial list of 199 categories, already resolving many ambiguities. We then
check the definitions of all of these categories in the wordnet database and
correct the initial mapping, as well as merged categories that are still too
ambiguous by their definitions in wordnet (e.g. rug.n.01 “rug, carpet,
carpeting; floor covering consisting of a piece of thick heavy fabric (usually
with nap or pile)” and mat.n.01 “a thick flat pad used as a floor covering” ).
The result are 186 categories that come with a text definition, a defined
hierarchy, and all possible synonyms that describe the category.

We then annotate our selected ScanNet scenes with these 186 categories based
on their wordnet definitions. We use [11] to annotate the fine meshes of the
scenes with a minimum number of necessary clicks. Only the authors of this
paper provided annotations, and each annotation was cross-checked by at least
one other author. In case of doubt, individual objects were discussed
together. On average, labeling of a scene took 5 hours.

###  3.2 Translation between Prediction Spaces

We employ different predictors that were trained on different data sets with
different numbers and definitions of classes. This requires translating
between different prediction spaces. We therefore build a mapping between the
class definitions of NYU40, ADE20k, ScanNet20, ScanNet200, Replica, and the
WordNet semantic language graph.

In this effort, we build on top of previous work, as the original ScanNet [8]
already defined a mapping between ScanNet classes, NYU40 classes, Eigen13
classes, and wordnet synkeys [18]. Furthermore, Lambert et al. [14] curated
mappings between the taxonomies of semantic segmentation datasets, out of
which NYU40, SUNRGBD, and ADE20k are most relevant for indoor perception. We
take the union of both works as initial mapping, but find that many
corrections are needed, especially with regard to wordnet synkeys, and many
ADE20k are missing because [14] only considered 20 out of 40 NYU categories.
We then add mappings to the Replica categories for the purpose of evaluation,
since Replica is one of the most accurately annotated indoor semantic
datasets.

When mapping between two class spaces, for any class in the source space there
are three cases in the target space: _a)_ there is no corresponding class in
the target space, _b)_ there is exactly one corresponding class in the target
space. This may be an exact match, or a class to which multiple class ids from
the source space are matched (_e.g._ , the source space may distinguish
between office chair, chair, and stool but the target space just has one
general chair class), _c)_ there are multiple corresponding classes in the
target space because the target space has a higher resolution than the source
space (_e.g._ , a general chair class in the source space can be split up in
the target space to distinguish between office chair, chair, or stool).

For (a) and (b), mappings are straightforward. We resolve (c) dependent on the
use cases:

  * •

_Evaluating a class with multiple correspondences._ A label of any of the
correspondences is treated as a true positive. If none of the correspondences
is the true class, all of them are counted as false positives.

  * •

_Computing model consensus._ Predictions in the source space vote for all
possible correspondences in the target space. The ambiguity between the
possible correspondences is usually resolved through an additional predictor
with a prediction space of higher resolution. If no resolution is achieved, we
pick the first of the possible classes.

Figure 2: Pipeline Overview. The base models predict individual semantic maps
for each 2D frame of the trajecotry. The consensus first maps the label spaces
in our unified label space and then runs our consensus voting mechanism for
every frame. Finally, the 3D lifting aggregates the per-frame predictions in
3D that improves the segment quality due to the additional denoising. The
final 3D annotation can be rendered back into 2D to obtain a multi-view
consistent labelling across the entire trajecotry.

###  3.3 Base Models

We employ an ensemble of strong base models, each state-of-the-art in their
respective task and data characteristic:

InternImage [30] is a supervised 2D RGB-only semantic segmentation model that
at the time of writing has state-of-the-art performance on the Cityscapes and
ADE20k benchmarks. It achieves this by performing large-scale joined
pretraining on most available visual classification datasets. We use the
ADE20k fine-tuned variant.

OVSeg [15] is an open-vocabulary semantic segmentation model based on CLIP
[21], a visual-language representation model. OVSeg segments images by
assigning region proposals to a set of given prompts and is therefore not
limited to a fixed set of classes. In particular, we added such an open-
vocabulary segmentation model not because they achieve the best performance on
a given task but because of their generalization ability. We generate prompts
from our set of wordnet synkeys by averaging over language prompts such as “A
_ in a room.”, but also using all possible synonyms according to wordnet.

CMX [35] is at the time of writing the state-of-the-art 2D semantic
segmentation model for NYU Depth v2, a RGB+Depth indoor dataset. Its
predictions also take the geometric cues from the depth into account.

Mask3D [23] is at the time of writing the state-of-the-art 3D instance
segmentation model on ScanNet200 [22]. This method operates on an accumulated
pointcloud of a scene instead of frames, therefore taking the geometry even
better into account. It is trained on ScanNet. We render the 3D semantic
instance predictions into the 2D training frames to map them into the same
space as all other base models.

The four semantic models produce classifications in four different sets of
classes. InternImage predicts 150 ADE20k classes, CMX predicts 40 NYU classes,
Mask3D predicts 200 ScanNet classes, and our OVSeg prompts cover 186 wordnet
classes. In addition to the semantic models, we use OmniData [9] to complement
the depth sensor.

###  3.4 Model Consensus

As illustrated in Fig. 2, we run all models of Sec. 3.3 individually on every
frame and then, per frame, merge their predictions together using the
translation described in Sec. 3.2. We further use left-right flipping as test
time augmentation, which means that each pixel receives votes for possible
classes from:

  * •

the standard RGB image and it’s flipped version for the 2D segmentation models
InternImage, CMX, and OVSeg

  * •

2 votes (to equalize the test-time augmentation of the RGB frame) from the
Mask3D prediction rendered into the current frame

  * •

in the variant where we additionally use available human annotations, 5 votes
from the original ScanNet labels

For every pixel, we choose the class with the maximum number of votes. If no
class has sufficient votes (parameterized as a threshold), we set the
prediction to “ _unknown_ ” and it will have no loss in the 3D lifting.

###  3.5 3D Lifting

By computing a consensus over a diverse set of 2D predictors, we leverage the
knowledge and scale of 2D semantic segmentation datasets. However, the per-
frame predictions are noisy and often inconsistent, especially around image
boundaries. These inconsistencies can be mitigated and the performance can
even be improved, as previous work has shown [24, 17], by lifting the 2D
predictions into 3D.

Therefore, we leverage the recent progress based on NeRFs to generate multi-
view consistent 2D semantic segmentation labels in all frames. Based on the
observation in previous works [24, 17] that accurate geometry is important to
resolve inconsistencies between predictions of multiple frames instead of
hallucinating geometry that would explain semantic predictions, we train an
implicit surface model from sdfstudio [32] that has a more explicit surface
definition compared to a NeRF yielding improved geometry compared to vanilla
NeRF. Thus, we add a semantic head to the Neus-Acc model, train it on all
views with losses from RGB reconstruction, sensor depth, monocular normal
estimation, and our semantic consensus. Finally, we render the optimized
semantics back into all camera frames.

To generate consistent 3D semantic segmentation labels, we follow an
established and more direct approach. Given a pointcloud of the scene, we
project the pointcloud into each consensus frame to find corresponding pixels
and then take a majority vote over all pixels corresponding to a point.

##  4 Experiments

| 2D | 3D
---|---|---
evaluation class set |  NYU (40 classes) |  wordnet (186 classes) |  NYU (40 classes) |  wordnet (186 classes)
metric | mIoU | mAcc | tAcc | mIoU | mAcc | tAcc | mIoU | mAcc | tAcc | mIoU | mAcc | tAcc
ScanNet labels [8] | 47.7 | 56.2 | 69.2 | 38.1 | 46.3 | 69.7 | 40.1 | 48.2 | 68.6 | 17.7 | 21.3 | 70.6
SemanticNerf* [36] | 45.2 | 56.6 | 69.3 | 32.9 | 43.7 | 71.2 | 36.7 | 47.1 | 68.4 | 14.8 | 19.3 | 71.0
LabelMaker w/o ScanNet (automatic labels) | 50.7 | 64.0 | 75.3 | 33.5 | 43.5 | 72.3 | 41.3 | 47.3 | 71.2 | 15.7 | 18.1 | 71.5
LabelMaker (Ours) | 53.4 | 65.0 | 77.5 | 39.1 | 49.3 | 77.2 | 44.1 | 53.4 | 76.1 | 18.2 | 22.0 | 76.7
Table 1: Comparison of the label quality of the ScanNet labels, LabelMaker
without any human input, and LabelMaker taking the ScanNet annotations as
additional input. The results are measured over 5 scenes from ScanNet against
newly annotated high-quality ground truth. Based on our translation of
prediction spaces, we measure metrics over the medium-tail NYU40 set of
categories and our full long-tail ground truth categories. For NYU40 classes,
LabelMaker is capable of producing labels of higher quality than the ScanNet
human annotations, without any human input. For more long-tail categories, the
automatic mode does not reach the quality of ScanNet, but LabelMaker is able
to considerably improve human annotations.

###  4.1 Implementation Details

For the 2D models, we use the corresponding available open-source code and
adjust it to our pipeline. As described in Sec. 3.2, we generate votes from
each 2D model into a common label space. We choose our defined 186 class
wordnet label space as output. We choose the label with highest votes, but
require a minimum of 3 out of 13 (with ScanNet annotations) resp. 4 out of 8
(automatic pipeline) votes. For 3D optimization, we build on top of SDFStudio
[32], specifically the Neus-Acc [29] model, and add a semantic head and
semantic rendering similar to [36].

###  4.2 Datasets

We run our proposed method on three different datasets to show its performance
and validate our design choices.

ScanNet [8] We randomly select 5 scenes from the ScanNet that cover all
frequent room types. We carefully annotate high-resolution meshes of the
scenes using [11] as described in Sec. 3.1 in order to have a complete and
accurate groundtruth to evaluate against.

Replica [26] We also evaluate our method on the Replica dataset. This is a
semi-synthetic dataset, captured as a high accuracy mesh from real
environments and then rendered into trajectories in [36]. We select the 3
‘room’ scenes and evaluate against the given annotation.

ARKitScenes [1] To showcase the automatic labelling pipeline on an existing
dataset, we run it on selected scenes of the ARKitScenes dataset, where only
sparse bounding box labels are available up to date. ARKit Scenes consists of
trajectories captured with consumer smartphones which are registered to a
professional 3D scanner.

###  4.3 Baselines

We mainly compare LabelMaker to the existing manually created annotations in
ScanNet [8]. As an additional baseline, we report the result of fitting and
rendering the ScanNet annotations with our adapted SemanticNeRF [36].

ScanNet [8]. For this baseline, we measure the quality of the annotations in
ScanNet. To this end, we take the raw ScanNet labels and map them into our
labelspace defined by wordnet. The mapping from ScanNet IDs to wordnet synkeys
is to a large extent already provided in [8].

SemanticNeRF [36]. This baseline is inspired by [36] and adapted to our
pipeline by integrating the semantic head into SDFStudio. Then, we run this
version of SemanticNeRF on the ScanNet 2D semantic labels. Thus, we can
measure the effect of multi-view aggregation and optimization on the
groundtruth ScanNet labels. The hypothesised effect is that through the extra
RGB and geometry information provided to the NeRF, segmentation boundaries may
be smoother than those of the ScanNet ‘supervoxels’.

RGB | ScanNet label | LabelMaker | LabelMaker - NYU40 | Ground-truth
---|---|---|---|---
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
|  |  |  |
Figure 3: LabelMaker generates more accurate and more complete labels compared
to the labels annotated by humans and provided by ScanNet. Particularly,
unlabeled sections in ScanNet are correctly filled in and many wrong
annotations such as missing rogs and pictures are corrected. The output labels
can then be projected into differnet label spaces, such as our wordnet space
or the NYU40 categories.

###  4.4 Comparison to State-of-the-Art

In Tab. 1, we compare LabelMaker to the state-of-the-art baselines ScanNet and
SemanticNeRF. We report mean intersection-over-union (mIoU), mean accuracy
(mAcc), as well as total accuracy (tAcc). We evaluate the methods in 2D by
comparing the renderings or labeled frames with renderings from the ground-
truth 3D mesh and in 3D by mapping the 2D renderings onto the corresponding
vertices in the 3D ground-truth mesh. Further, we measure the metrics over two
different label sets. The NYU40 label set [25] consists of 40 semantic classes
representing the common indoor classes in the short tail of the label
distribution. The wordnet label set consists of 186 classes, therefore
measuring performance also over the long tail of the label distribution.

Scannet |  LabelMaker (Ours) | Groundtruth
---|---|---
|  |
|  |
|  |
Figure 4: Dense 3D labels for ScanNetv2 [8]. We generate more consistent
labels compared to human annotators and preserve rare classes (_e.g._ , swivel
chair in front of the desk). Further, the labels are more complete (_e.g._ ,
wall in bathroom) and we can capture all object in the scene (_e.g._ , dustpan
in bathroom).

We show that our proposed pipeline generates better labels than human-
annotated ScanNet labels and their lifted version through SemanticNeRF [36].
Particularly, on the short tail of the distribution (NYU label set), our
pipeline significantly improves over the human annotated labels. This is due
to more accurate object boundaries as well as more consistent and complete
labels. For the long tail of the label distribution, our method also
outperforms all existing baselines indicating that different 2D expert votes
and 3D aggregation boosts the quality of the annotated labels. Finally, we
show that our fully automatic pipeline outperforms human annotations on NYU40
classes, highlighting the potential of LabelMaker to generate labels at scale.

Qualitative comparison with ScanNet [8] In Fig. 4, we compare qualitative
results for ScanNet [8] with LabelMaker, and our groundtruth. To this end, we
mapped the 2D renderings onto the high-resolution ground-truth mesh by
projecting the mesh vertices into all labels using a visibility check. One can
see that our pipeline produces consistently more complete and correct labels
than the human annotations provided by ScanNet [8]. E.g., our method
consistently labels the kitchen countertop, the mats in the bathroom, and even
the folded chair leaned against the desk.

RGB |  LabelMaker 2D (Ours) | Mask3D |  LabelMaker 3D (Ours)
---|---|---|---
|  |  |
|  |  |
|  |  |
Figure 5: Automatic dense labelling of ARKitScenes. We demonstrate the applicability to label RGB-D datasets that do not have dense labels available. Compared to state-of-the-art Mask3D [23], we generate dense annotations for all classes in the scene. Further, we segment on a higher level of detail (see picture and books in bookshelf, or objects on the cabinet/nightstand). Thus, our labelling pipeline can readily be used on non-label dataset to provide training data for segmentation methods.  |  ScanNet (186 classes) |  Replica (150 classes)
---|---|---
| mIoU | mAcc | tAcc | mIoU | mAcc | tAcc
OVSeg | 15.3 | 24.4 | 43.7 | 20.7 | 26.5 | 69.4
InternImage | 30.8 | 43.5 | 59.4 | 38.3 | 47.7 | 84.6
CMX | 28.2 | 41.0 | 54.2 | 17.0 | 38.0 | 84.6
Mask3D | 33.7 | 40.2 | 38.5 | 22.6 | 27.9 | 30.4
Consensus | 38.9 | 48.3 | 77.0 | 39.1 | 46.2 | 84.3
LabelMaker (ours) | 39.1 | 49.3 | 77.2 | 42.1 | 51.0 | 86.7
Table 2: Ablation of all base models in LabelMaker on our 5 labelled ScanNet
[8] scenes and Replica [26]. InternImage is the strongest single base model,
but the fusion with other predictions and 3D lifting increases the accuracy
considerably beyond any of the state-of-the-art single models.

ScanNet Label Quality Because our experiments require new high-accuracy
annotations of ScanNet scenes, we are able to estimate the quality of the
default ScanNet labels. As Tab. 1 shows, but also any human who inspects the
ScanNet labels knows, these are not perfect. We argue in Sec. 3.1 that this
reflects the open-world approach of the dataset and annotation workflow, where
– exactly as in any real application – semantics are ambiguous and not always
clearly defined. We should also point out that even the detailed annotations
we provide are not fully perfect. However, given the background that the
ScanNet labels are also used as a benchmark to compare accuracy of semantic
classifiers, our results indicate that a perfect prediction would reach
accuracy values much lower than 100%. If two methods achieve higher mIoU on
ScanNet than the ScanNet labels themselves, it is not possible to draw a clear
conclusion about which method is better. This highlights the usefulness of
improving the quality of the labels in datasets where some labels already
exist.

###  4.5 Ablation Study

Does consensus voting make the model better? Tab. 2 shows the evaluation on
the standard metrics (mIoU, mAcc, tAcc) in 2D for the ScanNet and the Replica
datasets. We demonstrate that aggregating individual 2D predictions with our
consensus voting mechanism improves upon the individual 2D models. Further, we
also show that lifting the 2D consensus into 3D using our optimization
pipeline further improves the results compared to the individual 2D models.

Which model is the most important? Tab. 2 shows that the performance of models
differs noticeably. Compared to the others, InternImage and Mask3D have the
strongest positive impact on the segmentation quality. Additionally and
unsurprisingly, Tab. 1 shows that using ScanNet [8] labels as additional votes
further improves performance.

Importance of 3D Lifting? We show in Tab. 2 the effect of 3D lifting to
aggregate semantic labels and make them multi-view consistent. We compare
LabelMaker with the aggregated consensus, as well as with individual
models,and compute the 2D metrics on ScanNet and Replica. One can see that the
3D lifting significantly improves the performance by at least $+1$ mIoU.

###  4.6 Experiments on ARKitScenes

To demonstrate the applicability of our labelling pipeline to new datasets,
for which no dense labels exist, we run our pipeline on a set of scenes from
the ARKitScenes [1] dataset. To this end, we process the smartphone
trajectories using the low resolution depth maps as sensor depth and the
corresponding VGA-resolution images as RGB input. We established these
correspondences by synchronizing the depth and RGB timestamps. In Fig. 5, we
show qualitative results for 2 scenes of the data set. One can see that the
produced labels are more complete and accurate than for Mask3D, a state-of-
the-art 3D instance segmentation method. Thus, we demonstrate the feasibility
of automatically labeling huge datasets with zero human intervention.

##  5 Limitations

LabelMaker is still limited to a fixed set of classes. Extending it to output
language embeddings instead of classes would make it more flexible and
potentially help to resolve ambiguities. The 3D lifting with SDFStudio has
numerous hyper-parameters, and this work possibly did not yet find the optimal
settings. In terms of accuracy, the pipeline can be further profit from newly
developed models as research progresses, which will improve the output
quality. An interesting next step would be to implement a feedback loop where
LabelMaker is used to produce a vast amount of automatically labeled training
data, on which an additional model is trained as a distillation of the model
zoo.

##  6 Conclusion

We present a fully automatic labeling pipeline that generates semantic
annotations of similar quality to human annotations, with zero manual human
labeling effort. The method also improves the accuracy and consistency of
existing annotations. We quantitatively validate the performance of our
pipeline on the ScanNet and Replica datasets. On ScanNet, it outperforms the
existing human annotations, and on Replica it improves over all baseline
methods. Finally, we showcase the applicability to large-scale 3D datasets and
label images and point clouds of ARKitScenes.

Acknowledgments This project is partially funded by the Swiss National Science
Foundation (SNSF) project TMAG-2_216260, by two ETH Career Seed Awards
“ScanNetter” and “Towards Open-World 3D Scene Understanding”. FE is a
postdoctoral research fellow at the ETH AI Center.

## References

  * Baruch et al. [2021] Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman.  ARKitscenes - a diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_ , 2021.
  * Bréhéret [2017] Amaury Bréhéret.  Pixel Annotation Tool.  https://github.com/abreheret/PixelAnnotationTool, 2017.
  * Caesar et al. [2018] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari.  Coco-stuff: Thing and stuff classes in context.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1209–1218, 2018.
  * Chang et al. [2017] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.  Matterport3D: Learning from RGB-D Data in Indoor Environments.  In _International Conference on 3D Vision (3DV)_ , 2017.
  * Chen et al. [2023] Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang.  Clip2scene: Towards label-efficient 3d scene understanding by clip.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Chibane et al. [2022] Julian Chibane, Francis Engelmann, Tuan Anh Tran, and Gerard Pons-Moll.  Box2Mask: Weakly Supervised 3D Semantic Instance Segmentation using Bounding Boxes.  In _European Conference on Computer Vision (ECCV)_ , 2022.
  * Cordts et al. [2016] M Cordts, M Omran, S Ramos, T Rehfeld, M Enzweiler, R Benenson, U Franke, S Roth, and B Schiele.  The Cityscapes Dataset for Semantic Urban Scene Understanding.  pages 3213–3223, 2016.
  * Dai et al. [2017] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner.  ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Eftekhar et al. [2021] Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir.  Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 10786–10796, 2021.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Kontogianni et al. [2023] Theodora Kontogianni, Ekin Celikkan, Siyu Tang, and Konrad Schindler.  Interactive object segmentation in 3d point clouds.  In _International Conference on Robotics and Automation (ICRA)_ , 2023.
  * Kreuzberg et al. [2022] Lars Kreuzberg, Idil Esen Zulfikar, Sabarinath Mahadevan, Francis Engelmann, and Bastian Leibe.  4D-STOP: Panoptic Segmentation of 4D LiDAR using Spatio-temporal Object Proposal Generation and Aggregation.  In _European Conference on Computer Vision (ECCV) Workshops_ , 2022.
  * [13] labelme github contributors.  labelme: Image polygonal annotation with python.
  * Lambert et al. [2020] John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun.  MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation.  pages 2879–2888, 2020.
  * Liang et al. [2023a] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu.  Open-vocabulary semantic segmentation with mask-adapted clip.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7061–7070, 2023a.
  * Liang et al. [2023b] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu.  Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023b.
  * Liu et al. [2023] Zhizheng Liu, Francesco Milano, Jonas Frey, Roland Siegwart, Hermann Blum, and Cesar Cadena.  Unsupervised continual semantic adaptation through neural rendering.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Miller [1995] George A Miller.  Wordnet: a lexical database for english.  _Communications of the ACM_ , 1995.
  * Nekrasov et al. [2021] Alexey Nekrasov, Jonas Schult, Or Litany, Bastian Leibe, and Francis Engelmann.  Mix3d: Out-of-context Data Augmentation for 3D Scenes.  In _International Conference on 3D Vision (3DV)_ , 2021.
  * Peng et al. [2023] Songyou Peng, Kyle Genova, Chiyu ”Max” Jiang, Andrea Tagliasacchi, Marc Pollefeys, and Thomas Funkhouser.  OpenScene: 3D Scene Understanding with Open Vocabularies.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning Transferable Visual Models From Natural Language Supervision.  In _International Conference on Machine Learning (ICML)_ , 2021.
  * Rozenberszki et al. [2022] David Rozenberszki, Or Litany, and Angela Dai.  Language-Grounded Indoor 3D Semantic Segmentation in the Wild.  In _European Conference on Computer Vision (ECCV)_ , 2022.
  * Schult et al. [2023] Jonas Schult, Francis Engelmann, Alexander Hermans, Or Litany, Siyu Tang, and Bastian Leibe.  Mask3D for 3D Semantic Instance Segmentation.  In _International Conference on Robotics and Automation (ICRA)_ , 2023.
  * Siddiqui et al. [2023] Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Norman Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder.  Panoptic lifting for 3d scene understanding with neural fields.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023.
  * Silberman et al. [2012] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus.  Indoor Segmentation and Support Inference from RGBD Images.  pages 746–760. Springer Berlin Heidelberg, 2012.
  * Straub et al. [2019] Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe.  The Replica dataset: A digital replica of indoor spaces.  _arXiv_ , 2019.
  * Takmaz et al. [2023a] Ayça Takmaz, Elisabetta Fedele, Robert W Sumner, Marc Pollefeys, Federico Tombari, and Francis Engelmann.  OpenMask3D: Open-Vocabulary 3D Instance Segmentation.  _Neural Information Processing Systems (NeurIPS)_ , 2023a.
  * Takmaz et al. [2023b] Ayça Takmaz, Jonas Schult, Irem Kaftan, Mertcan Akçay, Bastian Leibe, Robert Sumner, Francis Engelmann, and Siyu Tang.  3D Segmentation of Humans in Point Clouds with Synthetic Data.  In _International Conference on Computer Vision (ICCV)_ , 2023b.
  * Wang et al. [2021] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang.  Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction.  _arXiv preprint arXiv:2106.10689_ , 2021.
  * Wang et al. [2022] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al.  Internimage: Exploring large-scale vision foundation models with deformable convolutions.  _arXiv preprint arXiv:2211.05778_ , 2022.
  * Yu et al. [2020] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell.  Bdd100k: A diverse driving dataset for heterogeneous multitask learning.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2020.
  * Yu et al. [2022] Zehao Yu, Anpei Chen, Bozidar Antic, Songyou Peng Peng, Apratim Bhattacharyya, Michael Niemeyer, Siyu Tang, Torsten Sattler, and Andreas Geiger.  Sdfstudio: A unified framework for surface reconstruction, 2022.
  * Yue et al. [2023a] Yuanwen Yue, Theodora Kontogianni, Konrad Schindler, and Francis Engelmann.  Connecting the Dots: Floorplan Reconstruction Using Two-Level Queries.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2023a.
  * Yue et al. [2023b] Yuanwen Yue, Sabarinath Mahadevan, Jonas Schult, Francis Engelmann, Bastian Leibe, Konrad Schindler, and Theodora Kontogianni.  AGILE3D: Attention Guided Interactive Multi-object 3D Segmentation.  _arXiv preprint arXiv:2306.00977_ , 2023b.
  * Zhang et al. [2023] Jiaming Zhang, Huayao Liu, Kailun Yang, Xinxin Hu, Ruiping Liu, and Rainer Stiefelhagen.  Cmx: Cross-modal fusion for rgb-x semantic segmentation with transformers.  _IEEE Transactions on Intelligent Transportation Systems_ , 2023.
  * Zhi et al. [2021a] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J Davison.  In-place scene labelling and understanding with implicit scene representation.  In _International Conference on Computer Vision (ICCV)_ , 2021a.
  * Zhi et al. [2021b] Shuaifeng Zhi, Edgar Sucar, Andre Mouton, Iain Haughton, Tristan Laidlow, and Andrew J Davison.  ilabel: Interactive neural scene labelling.  _arXiv preprint arXiv:2111.14637_ , 2021b.
  * Zhou et al. [2017] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 633–641, 2017.

## Appendix of ‘LabelMaker’

The paper supplement consists of

  * •

anonymized source code

  * •

video supplement with an additional explanaition of LabelMaker and full
renderings of the LabelMaker output in all scenes

  * •

additional experimental details in the following sections:

    * –

the selected scenes from Replica, ScanNet and ARKitScenes,

    * –

an explanation of our curated label mappings with some examples,

    * –

per-category extension of our results on ScanNet,

    * –

implementation details of the NeRF,

    * –

and our used annotation definitions.

##  Appendix A Full qualitative examples

We present full qualitative examples of all evaluated ScanNet trajectories in
the video supplement.

##  Appendix B Code Supplement

As part of the supplement, we also provide an anonymized version of the code
base. It consists of a small library to match and evaluate different label
spaces, and all code to run the LabelMaker pipeline in scripts/.

##  Appendix C Selected Scenes

We use trajectories scene0000_00, scene0164_02, scene0458_00, scene0474_01,
scene0518_00 from ScanNet and environments room_0, room_1, room_2 from
Replica. In the main paper, we additionaly show qualitative results on
ARKitScenes 42445991 and 42897688.

##  Appendix D Label Mapping Examples

In the following, we give a few examples of our curated label mapping that
enables us to jointly use multiple models that are trained on different
datasets (and label categories):

Simple Example 1: ScanNet category 1 is called ‘wall’. It is mapped on NYU40
category ‘wall’ (id 1), ADE20k category ‘wall’ (id 0), Replica category ‘wall’
(id 93), and wordnet synkey wall.n.01, which we assign to our id 1.

Simple Example 2: ScanNet category 56 is called ‘trash can’. It gets mapped on
NYU40 category ‘otherfurniture’ (id 39), ADE20k category ‘ashcan’ (id 138),
Replica category ‘bin’ (id 10), and wordnet synkey ashcan.n.01 (synonyms
ashcan, trash can, garbage can, wastebin, ash bin, ash-bin, ashbin, dustbin,
trash barrel, trash bin), which we assign to our id 7.

Example of many-to-one mapping: In addition to the above examples, we also map
_e.g._ , ScanNet category ‘recycling bin’ (id 97) to the wordnet synkey
ashcan.n.01 and the listed categories of the other datasets.

Example of many-to-many mapping: We map ScanNet categories ‘pillow’ (id 13),
‘couch cushions’ (id 39), and ‘cushion’ (id 39) all to wordnet synkey
cushion.n.03. This gets mapped to NYU40 category ‘pillow’ (id 18), ade classes
‘cushion’ (id 39) and ‘pillow’ (id 57), and Replica categories ‘cushion’ (id
29) and ‘pillow’ (id 61).

##  Appendix E Results for Individual Categories

We present more detailed per-category data of our comparison to the ScanNet
labels in Table 3.

| LabelMaker | LabelMaker w/o ScanNet (automatic) | SemanticNerf* | ScanNet labels
---|---|---|---|---
mIoU | 0.534 | 0.507 | 0.452 | 0.477
mAcc | 0.650 | 0.640 | 0.566 | 0.562
tAcc | 0.775 | 0.753 | 0.693 | 0.692
wall | 0.831 | 0.768 | 0.743 | 0.752
floor | 0.731 | 0.827 | 0.678 | 0.727
cabinet | 0.626 | 0.613 | 0.608 | 0.692
bed | 0.863 | 0.879 | 0.824 | 0.930
chair | 0.822 | 0.781 | 0.522 | 0.555
sofa | 0.797 | 0.807 | 0.790 | 0.811
table | 0.433 | 0.303 | 0.395 | 0.415
door | 0.782 | 0.671 | 0.658 | 0.520
window | 0.362 | 0.336 | 0.331 | 0.354
bookshelf | 0.235 | 0.442 | 0.500 | 0.528
picture | 0.510 | 0.024 | 0.252 | 0.255
counter | 0.682 | 0.524 | 0.640 | 0.639
blinds | 0.007 | 0.360 | 0.000 | 0.000
desk | 0.170 | 0.172 | 0.144 | 0.178
curtain | 0.884 | 0.809 | 0.871 | 0.884
pillow | 0.348 | 0.416 | 0.337 | 0.350
floormat | 0.072 | 0.541 | 0.000 | 0.000
clothes | 0.005 | 0.000 | 0.000 | 0.000
ceiling | 0.733 | 0.767 | 0.741 | 0.782
books | 0.023 | 0.292 | 0.000 | 0.000
refrigerator | 0.938 | 0.922 | 0.907 | 0.951
television | 0.713 | 0.486 | 0.171 | 0.168
paper | 0.000 | 0.061 | 0.000 | 0.000
towel | 0.648 | 0.580 | 0.504 | 0.554
box | 0.757 | 0.551 | 0.748 | 0.806
nightstand | 0.313 | 0.269 | 0.302 | 0.267
toilet | 0.911 | 0.845 | 0.795 | 0.895
sink | 0.777 | 0.802 | 0.569 | 0.715
lamp | 0.307 | 0.278 | 0.484 | 0.491
bag | 0.630 | 0.509 | 0.743 | 0.764
otherstructure | 0.782 | 0.423 | 0.000 | 0.000
otherfurniture | 0.474 | 0.321 | 0.344 | 0.373
otherprop | 0.460 | 0.344 | 0.309 | 0.374
Table 3: Class-by-class evaluation on ScanNet in the NYU40 label space, on our
annotated ScanNet scenes. Large gains of LabelMaker with respect to the
ScanNet labels can be found, e.g., in chair, door, books, and television
classes.

##  Appendix F SDFStudio Semantic Head Details and Parameters

We implement the semantic head as a small 4 layer MLP in parallel to the RGB
head. While the RGB head takes as input the direction and the field feature at
the rendered location, the semantic head is only dependent on the field
feature to force the semantics to be the same from all viewing directions. To
render semantics, we take a simple weighted sum over the output of the
semantic head along the ray.

In the following command, we report the whole set of parameters we use to run
our adapted SDFStudio models in all scenes:

    ns-train neus-facto \
      --pipeline.model.sdf-field.use-grid-feature True \
      --pipeline.model.sdf-field.hidden-dim 256 \
      --pipeline.model.sdf-field.num-layers 2 \
      --pipeline.model.sdf-field.num-layers-color 2 \
      --pipeline.model.sdf-field.semantic-num-layers 4 \
      --pipeline.model.sdf-field.use-appearance-embedding False \
      --pipeline.model.sdf-field.geometric-init True \
      --pipeline.model.sdf-field.inside-outside True  \
      --pipeline.model.sdf-field.bias 0.8 \
      --pipeline.model.sdf-field.beta-init 0.3 \
      --pipeline.model.sensor-depth-l1-loss-mult 0.3 \
      --pipeline.model.sensor-depth-sdf-loss-mult 0.3 \
      --pipeline.model.sensor-depth-freespace-loss-mult 0.3 \
      --pipeline.model.mono-normal-loss-mult 0.02 \
      --pipeline.model.mono-depth-loss-mult 0.000 \
      --pipeline.model.semantic-loss-mult 0.1 \
      --pipeline.model.semantic-ignore-label 0 \
      --trainer.max-num-iterations 20001 \
      --pipeline.datamanager.train-num-rays-per-batch 2048 \
      --pipeline.model.eikonal-loss-mult 0.1 \
      --pipeline.model.background-model none \
      sdfstudio-data \
      --include-sensor-depth True \
      --include-semantics True \
      --include-mono-prior True

##  Appendix G WordNet Labels

We use the following wordnet synkeys and definitions to annotate ScanNet
scenes:

wall.n.01

an architectural partition with a height and length greater than its
thickness; used to divide or enclose an area or to support another structure

chair.n.01

a seat for one person, with a support for the back

book.n.11

a number of sheets (ticket or stamps etc.) bound together on one edge

cabinet.n.01

a piece of furniture resembling a cupboard with doors and shelves and drawers;
for storage or display

door.n.01

a swinging or sliding barrier that will close the entrance to a room or
building or vehicle

floor.n.01

also flooring; the inside lower horizontal surface (as of a room, hallway,
tent, or other structure)

ashcan.n.01

also trash_can, garbage_can, wastebin, ash_bin, ash-bin, ashbin, dustbin,
trash_barrel, trash_bin; a bin that holds rubbish until it is collected

table.n.02

a piece of furniture having a smooth flat top that is usually supported by one
or more vertical legs

window.n.01

a framework of wood or metal that contains a glass windowpane and is built
into a wall or roof to admit light or air

bookshelf.n.01

a shelf on which to keep books

display.n.06

also video_display; an electronic device that represents information in visual
form

cushion.n.03

a soft bag filled with air or a mass of padding such as feathers or foam
rubber etc.

box.n.01

a (usually rectangular) container; may have a lid

picture.n.01

also image, icon, ikon; a visual representation (of an object or scene or
person or abstraction) produced on a surface

ceiling.n.01

the overhead upper surface of a covered space

doorframe.n.01

also doorcase; the frame that supports a door

desk.n.01

a piece of furniture with a writing surface and usually drawers or other
compartments

swivel_chair.n.01

a chair that swivels on its base

towel.n.01

a rectangular piece of absorbent cloth (or paper) for drying or wiping

sofa.n.01

also couch, lounge; an upholstered seat for more than one person

sink.n.01

plumbing fixture consisting of a water basin fixed to a wall or floor and
having a drainpipe

backpack.n.01

also back_pack, knapsack, packsack, rucksack, haversack; a bag carried by a
strap on your back or shoulder

lamp.n.02

a piece of furniture holding one or more electric light bulbs

chest_of_drawers.n.01

also chest, bureau, dresser; furniture with drawers for keeping clothes

apparel.n.01

also wearing_apparel, dress, clothes; clothing in general

armchair.n.01

chair with a support on each side for arms

bed.n.01

a piece of furniture that provides a place to sleep

curtain.n.01

also drape, drapery, mantle, pall; hanging cloth used as a blind (especially
for a window)

mirror.n.01

polished surface that forms images by reflecting light

plant.n.02

also flora, plant_life; (botany) a living organism lacking the power of
locomotion

radiator.n.02

heater consisting of a series of pipes for circulating steam or hot water to
heat rooms or buildings

toilet_tissue.n.01

also toilet_paper, bathroom_tissue; a soft thin absorbent paper for use in
toilets

shoe.n.01

footwear shaped to fit the foot (below the ankle) with a flexible upper of
leather or plastic and a sole and heel of heavier material

bag.n.01

a flexible container with a single opening

bottle.n.01

a glass or plastic vessel used for storing drinks or other liquids; typically
cylindrical without handles and with a narrow neck that can be plugged or
capped

countertop.n.01

the top side of a counter

coffee_table.n.01

also cocktail_table; low table where magazines can be placed and coffee or
cocktails are served

toilet.n.02

also can, commode, crapper, pot, potty, stool, throne; a plumbing fixture for
defecation and urination

computer_keyboard.n.01

also keypad; a keyboard that is a data input device for computers; arrangement
of keys is modelled after the typewriter keyboard

fridge.n.01

also fridge; a refrigerator in which the coolant is pumped around by an
electric motor

stool.n.01

a simple seat without a back or arms

computer.n.01

also computing_machine, computing_device, data_processor, electronic_computer,
information_processing_system; a machine for performing calculations
automatically

mug.n.04

with handle and usually cylindrical

telephone.n.01

also phone, telephone_set; electronic equipment that converts sound into
electrical signals that can be transmitted over distances and then converts
received signals back into sounds

light.n.02

also light_source; any device serving as a source of illumination

jacket.n.01

a short coat

bathtub.n.01

also bathing_tub, bath, tub; a relatively large open container that you fill
with water and use to wash the body

shower_curtain.n.01

a curtain that keeps water from splashing out of the shower area

microwave.n.02

also microwave_oven; kitchen appliance that cooks food by passing an
electromagnetic wave through it; heat results from the absorption of energy by
the water molecules in the food

footstool.n.01

also footrest, ottoman, tuffet; a low seat or a stool to rest the feet of a
seated person

baggage.n.01

also luggage; cases used to carry belongings when traveling

laptop.n.01

also laptop_computer; a portable computer small enough to use in your lap

printer.n.03

also printing_machine; a machine that prints

shower_stall.n.01

also shower_bath; booth for washing yourself, usually in a bathroom

soap_dispenser.n.01

dispenser of liquid soap

stove.n.01

also kitchen_stove, range, kitchen_range, cooking_stove; a kitchen appliance
used for cooking food

fan.n.01

a device for creating a current of air by movement of a surface or surfaces

paper.n.01

a material made of cellulose pulp derived mainly from wood or rags or certain
grasses

stand.n.04

a small table for holding articles of various kinds

bench.n.01

a long seat for more than one person

wardrobe.n.01

also closet, press; a tall piece of furniture that provides storage space for
clothes; has a door and rails or hooks for hanging clothes

blanket.n.01

also cover; bedding that keeps a person warm in bed

booth.n.02

also cubicle, stall, kiosk; small area set off by walls for special use

duplicator.n.01

also copier; apparatus that makes copies of typed, written or drawn material

bar.n.03

a rigid piece of metal or wood; usually used as a fastening or obstruction or
weapon

soap_dish.n.01

a bathroom or kitchen fixture for holding a bar of soap

switch.n.01

also electric_switch, electrical_switch; control consisting of a mechanical or
electrical or electronic device for making or breaking or changing the
connections in a circuit

coffee_maker.n.01

a kitchen appliance for brewing coffee automatically

decoration.n.01

also ornament, ornamentation; something used to beautify

range_hood.n.01

exhaust hood over a kitchen range

blackboard.n.01

also chalkboard; sheet of slate; for writing with chalk

clock.n.01

a timepiece that shows the time of day

railing.n.01

also rail; a barrier consisting of a horizontal bar and supports

mat.n.01

– merged with rug.n.01 – a thick flat pad used as a floor covering

seat.n.03

furniture that is designed for sitting on

bannister.n.02

also banister, balustrade, balusters, handrail; a railing at the side of a
staircase or balcony to prevent people from falling

container.n.01

any object that can be used to hold things (especially a large metal boxlike
object of standardized dimensions that can be loaded from one form of
transport to another)

mouse.n.04

also computer_mouse; a hand-operated electronic device that controls the
coordinates of a cursor on your computer screen as you move it around on a
pad; on the bottom of the device is a ball that rolls on the surface of the
pad

person.n.02

a human body (usually including the clothing)

stairway.n.01

also staircase; a way of access (upward and downward) consisting of a set of
steps

basket.n.01

also handbasket; a container that is usually woven and has handles

dumbbell.n.01

an exercising weight; two spheres connected by a short bar that serves as a
handle

column.n.07

also pillar; (architecture) a tall vertical cylindrical structure standing
upright and used to support a structure

bucket.n.01

also pail; a roughly cylindrical vessel that is open at the top

windowsill.n.01

the sill of a window; the horizontal member at the bottom of the window frame

signboard.n.01

also sign; structure displaying a board on which advertisements can be posted

dishwasher.n.01

also dish_washer, dishwashing_machine; a machine for washing dishes

loudspeaker.n.01

also speaker, speaker_unit, loudspeaker_system, speaker_system; electro-
acoustic transducer that converts electrical signals into sounds loud enough
to be heard at a distance

washer.n.03

also automatic_washer, washing_machine; a home appliance for washing clothes
and linens automatically

paper_towel.n.01

a disposable towel made of absorbent paper

clothes_hamper.n.01

also laundry_basket, clothes_basket, voider; a hamper that holds dirty clothes
to be washed or wet clothes to be dried

piano.n.01

also pianoforte, forte-piano; a keyboard instrument that is played by
depressing keys that cause hammers to strike tuned strings and produce sounds

sack.n.01

also poke, paper_bag, carrier_bag; a bag made of paper or plastic for holding
customer’s purchases

handcart.n.01

also pushcart, cart, go-cart; wheeled vehicle that can be pushed by a person;
may have one or two or four wheels

blind.n.03

also screen; a protective covering that keeps things out or hinders sight

dish_rack.n.01

a rack for holding dishes as dishwater drains off of them

mailbox.n.01

also letter_box; a private box for delivery of mail

bag.n.04

also handbag, pocketbook, purse; a container used for carrying money and small
personal items or accessories (especially by women)

bicycle.n.01

also bike, wheel, cycle; a wheeled vehicle that has two wheels and is moved by
foot pedals

ladder.n.01

steps consisting of two parallel members connected by rungs; for climbing up
or down

rack.n.05

also stand; a support for displaying various articles

tray.n.01

an open receptacle for holding or displaying or serving articles or food

toaster.n.02

a kitchen appliance (usually electric) for toasting bread

paper_cutter.n.01

a cutting implement for cutting sheets of paper to the desired size

plunger.n.03

also plumber’s_helper; hand tool consisting of a stick with a rubber suction
cup at one end; used to clean clogged drains

dryer.n.01

also drier; an appliance that removes moisture

guitar.n.01

a stringed instrument usually having six strings; played by strumming or
plucking

fire_extinguisher.n.01

also extinguisher, asphyxiator; a manually operated device for extinguishing
small fires

pitcher.n.02

also ewer; an open vessel with a handle and a spout for pouring

pipe.n.02

also pipage, piping; a long tube made of metal or plastic that is used to
carry water or oil or gas etc.

plate.n.04

dish on which food is served or from which food is eaten

vacuum.n.04

also vacuum_cleaner; an electrical home appliance that cleans by suction

bowl.n.03

a dish that is round and open at the top for serving foods

hat.n.01

also chapeau, lid; headdress that protects the head from bad weather; has
shaped crown and usually a brim

rod.n.01

a long thin implement made of metal or wood

water_cooler.n.01

a device for cooling and dispensing drinking water

kettle.n.01

also boiler; a metal pot for stewing or boiling; usually has a lid

oven.n.01

kitchen appliance used for baking or roasting

scale.n.07

also weighing_machine; a measuring instrument for weighing; shows amount of
mass

broom.n.01

a cleaning implement for sweeping; bundle of straws or twigs attached to a
long handle

hand_blower.n.01

also blow_dryer, blow_drier, hair_dryer, hair_drier; a hand-held electric
blower that can blow warm air onto the hair; used for styling hair

coatrack.n.01

also coat_rack, hatrack; a rack with hooks for temporarily holding coats and
hats

teddy.n.01

also teddy_bear; plaything consisting of a child’s toy bear (usually plush and
stuffed with soft materials)

alarm_clock.n.01

also alarm; a clock that wakes a sleeper at some preset time

rug.n.01

–merged with mat.n.01– also carpet, carpeting; floor covering consisting of a
piece of thick heavy fabric (usually with nap or pile)

ironing_board.n.01

narrow padded board on collapsible supports; used for ironing clothes

fire_alarm.n.02

also smoke_alarm; an alarm that is tripped off by fire or smoke

machine.n.01

any mechanical or electrical device that transmits or modifies energy to
perform or assist in the performance of human tasks

music_stand.n.01

also music_rack; a light stand for holding sheets of printed music

fireplace.n.01

also hearth, open_fireplace; an open recess in a wall at the base of a chimney
where a fire can be built

furniture.n.01

also piece_of_furniture, article_of_furniture; furnishings that make a room or
other area ready for occupancy

vase.n.01

an open jar of glass or porcelain used as an ornament or to hold flowers

vent.n.01

also venthole, vent-hole, blowhole; a hole for the escape of gas or air

candle.n.01

also taper, wax_light; stick of wax with a wick in the middle

crate.n.01

a rugged box (usually made of wood); used for shipping

dustpan.n.02

a short-handled receptacle into which dust can be swept

earphone.n.01

also earpiece, headphone, phone; electro-acoustic transducer for converting
electric signals into sounds; it is held over or inserted into the ear

jar.n.01

a vessel (usually cylindrical) with a wide mouth and without handles

projector.n.02

an optical instrument that projects an enlarged image onto a screen

gat.n.01

also rod; a gangster’s pistol

step.n.04

also stair; support consisting of a place to rest the foot while ascending or
descending a stairway

step_stool.n.01

a stool that has one or two steps that fold under the seat

vending_machine.n.01

a slot machine for selling goods

coat.n.01

an outer garment that has sleeves and covers the body from shoulder down; worn
outdoors

coat_hanger.n.01

also clothes_hanger, dress_hanger; a hanger that is shaped like a person’s
shoulders and used to hang garments on

drinking_fountain.n.01

also water_fountain, bubbler; a public fountain to provide a jet of drinking
water

hamper.n.02

a basket usually with a cover

thermostat.n.01

also thermoregulator; a regulator for automatically regulating temperature by
starting or stopping the supply of heat

banner.n.01

also streamer; long strip of cloth or paper used for decoration or advertising

iron.n.04

also smoothing_iron; home appliance consisting of a flat metal base that is
heated and used to smooth cloth

soap.n.01

a cleansing agent made from the salts of vegetable or animal fats

chopping_board.n.01

also cutting_board; a wooden board where meats or vegetables can be cut

hanging.n.01

also wall_hanging; decoration that is hung (as a tapestry) on a wall or over a
window

kitchen_island.n.01

an unattached counter in a kitchen that permits access from all sides

shirt.n.01

a garment worn on the upper half of the body

sleeping_bag.n.01

large padded bag designed to be slept in outdoors; usually rolls up like a
bedroll

tire.n.01

also tyre; hoop that covers a wheel

toothbrush.n.01

small brush; has long handle; used to clean teeth

bathrobe.n.01

a loose-fitting robe of towelling; worn after a bath or swim

faucet.n.01

also spigot; a regulator for controlling the flow of a liquid from a reservoir

slipper.n.01

also carpet_slipper; low footwear that can be slipped on and off easily;
usually worn indoors

thermos.n.01

also thermos_bottle, thermos_flask; vacuum flask that preserves temperature of
hot or cold drinks

tripod.n.01

a three-legged rack used for support

dispenser.n.01

a container so designed that the contents can be used in prescribed amounts

heater.n.01

also warmer; device that heats water or supplies warmth to a room

pool_table.n.01

also billiard_table, snooker_table; game equipment consisting of a heavy table
on which pool is played

remote_control.n.01

also remote; a device that can be used to control a machine or apparatus from
a distance

stapler.n.01

also stapling_machine; a machine that inserts staples into sheets of paper in
order to fasten them together

treadmill.n.01

an exercise device consisting of an endless belt on which a person can walk or
jog without changing place

beanbag.n.01

a small cloth bag filled with dried beans; thrown in games

dartboard.n.01

also dart_board; a circular board of wood or cork used as the target in the
game of darts

metronome.n.01

clicking pendulum indicates the exact tempo of a piece of music

painting.n.01

also picture; graphic art consisting of an artistic composition made by
applying paints to a surface

rope.n.01

a strong line

sewing_machine.n.01

a textile machine used as a home appliance for sewing

shredder.n.01

a device that shreds documents (usually in order to prevent the wrong people
from reading them)

toolbox.n.01

also tool_chest, tool_cabinet, tool_case; a box or chest or cabinet for
holding hand tools

water_heater.n.01

also hot-water_heater, hot-water_tank; a heater and storage tank to supply
heated water

brush.n.02

an implement that has hairs or bristles firmly set into a handle

control.n.09

also controller; a mechanism that controls the operation of a machine

dais.n.01

also podium, pulpit, rostrum, ambo, stump, soapbox; a platform raised above
the surrounding level to give prominence to the person on it

dollhouse.n.01

also doll’s_house; a house so small that it is likened to a child’s plaything

envelope.n.01

a flat (usually rectangular) container for a letter, thin package, etc.

food.n.01

also nutrient; any substance that can be metabolized by an animal to give
energy and build tissue

frying_pan.n.01

also frypan, skillet; a pan used for frying foods

helmet.n.02

a protective headgear made of hard material to resist blows

tennis_racket.n.01

also tennis_racquet; a racket used to play tennis

umbrella.n.01

a lightweight handheld collapsible canopy