# Auto-Vocabulary Segmentation for LiDAR Points

Weijie Wei  Osman Ülger  Fatemeh Karimi Najadasl  Theo Gevers  Martin R.
Oswald
University of Amsterdam

###### Abstract

Existing perception methods for autonomous driving fall short of recognizing
unknown entities not covered in the training data. Open-vocabulary methods
offer promising capabilities in detecting any object but are limited by user-
specified queries representing target classes. We propose AutoVoc3D, a
framework for automatic object class recognition and open-ended segmentation.
Evaluation on nuScenes showcases AutoVoc3D’s ability to generate precise
semantic classes and accurate point-wise segmentation. Moreover, we introduce
Text-Point Semantic Similarity, a new metric to assess the semantic similarity
between text and point cloud without eliminating novel classes.

##  1 Introduction

Front right image Back right image

Human Annotation

AutoVoc3D (Ours)

Figure 1: Comparison between Human Annotation and AutoVoc3D on nuScenes
Dataset [3]. AutoVoc3D segments regions with precise class names, _e.g_.
building and pole (highlighted by green boxes), while the dataset is annotated
with a more general term, namely manmade.

Existing perception methods [17, 2, 10, 20, 22, 8] for autonomous driving
often rely on an inclusiveness assumption that all potential categories of
interest must exist in the training dataset. Nevertheless, public datasets
often annotate instances with pre-defined categories, which can vary from
three (_e.g_. vehicle, cyclist and pedestrian) [16, 11] to several dozen types
[3, 1], and fail to annotate rare objects with correct semantic labels.
Failing to recognize atypical objects or road users poses a significant risk
to the perception model’s adaptability to diverse real-life scenarios.

The development of Vision-Language Models (VLMs) strengthens the connection
between vision and language and promotes progress in multi-modal tasks, such
as image captioning [9] and open-vocabulary learning [18]. Open-vocabulary
learning methods often utilize pre-trained VLMs to find the correspondence
between visual entities and a semantic vocabulary, thereby creating the
potential to detect any category of interest [18, 24]. However, these methods
rely on user-specified queries, and thus can not spontaneously recognize the
object categories in a scene. This shortcoming severely limits the real-life
applicability of such methods, as newly encountered object categories could be
unknown - _i.e_. not yet specified - to the model. In this work, we propose
AutoVoc3D, a framework that automatically recognizes what objects are present
in the scene, generates a vocabulary for them and segments the target
categories using a point cloud segmentor. We evaluate our method on the
nuScenes dataset [3] and introduce a Text-Point Semantic Similarity (TPSS)
metric to assess the model performance based on semantic consistency in CLIP
space. Figure 1 compares the human annotations based on a fixed vocabulary
with the result of AutoVoc3D with an automatically generated vocabulary.
AutoVoc3D generates convincing semantic classes as well as accurate point-wise
segmentations. Moreover, whereas the pre-defined categories are general and
ambiguous, _e.g_. manmade, AutoVoc3D recognizes buildings and poles, which are
semantically more precise. Our contributions can be summarized as follows:

  * •

We extend auto-vocabulary segmentation [15, 25] to the 3D domain, namely auto-
vocabulary LiDAR point segmentation. In this task, images and LiDAR points are
given, but no text queries indicate the target categories. In contrast, the
semantic label of each point is automatically generated.

  * •

We propose AutoVoc3D, a framework to automatically recognize existing objects
in a scene and predict point-wise semantic categories.

  * •

We propose Text-Point Semantic Similarity, a CLIP-based metric that measures
semantic consistency between point features and text embeddings.

  * •

We showcase that AutoVoc3D detects some rare objects in autonomous driving
scenery and has a comparable performance with open-vocabulary segmentation
methods.

##  2 Related Work

###  2.1 Open-Vocabulary Segmentation

Open-Vocabulary Segmentation (OVS) aims to perform segmentation based on a
list of arbitrary text queries. CLIP [14] achieves this in 2D domain by
aligning vision and text in a latent space using natural language as
supervision on 400 million text-image pairs. However, there is no point cloud
dataset of the same nature that allows for training at such scale.
Additionally, captions in point cloud datasets are usually much sparser.
Therefore, a common solution is to freeze the text encoder and image encoder,
and push point features to vision-language feature space [19, 13, 5, 23, 12].
ULIP [19] conducts contrastive learning between text-image-point triplets,
which distills vision and language knowledge to a point encoder. CLIP2Scene
[5] trains the point encoder in a self-supervised learning manner.
Specifically, it contrasts features between point-text pairs and pushes point
features to vision-language feature space by considering spatial-temporal
consistency. OpenScene [13] projects LiDAR points into images to find point-
pixel correspondences and supervises the point encoder by pixel features
generated by the CLIP-based image encoder. Although OVS shows encouraging
performance in both 2D and 3D domains, it still relies on user-specified
categories as prompts to guide segmentation. Conversely, our approach
automatically generates categories that potentially appear in the scene
without any human in the loop, as shown in Fig. 2.

###  2.2 Auto-Vocabulary Segmentation

In Open-Vocabulary Segmentation, images or point clouds are grouped into
coherent semantic regions and classified with user-provided target categories.
Auto-Vocabulary Segmentation (AVS), however, performs segmentation without the
need to specify target categories. Relevant target categories are directly
inferred from the image - usually without any additional training, finetuning,
data sourcing or annotation effort - instead. The Zero-Guidance Segmentation
paradigm was the first to achieve this [15]. The authors used clustered DINO
[4] embeddings to obtain binary object masks. These masks were subsequently
utilized to adjust the attention of CLIP, resulting in embeddings that are
more accurately targeted to individual segments. Finally, a trained large
language model is tasked to output texts closest to these embeddings. While
this required switching between three different latent representations,
AutoSeg [25] proposed a more direct approach based on BLIP [9] embeddings
only. They introduced the BLIP-Cluster-Caption procedure, in which multi-scale
BLIP embeddings are enhanced through clustering, alignment and denoising. The
enhanced embeddings are captioned using the BLIP decoder and parsed into
nouns. Finally, the nouns are passed to an OVS model for segmentation.

Figure 2: Open-Vocabulary vs. Auto-Vocabulary Segmentation. The main
difference is that auto-vocabulary methods generate text queries automatically
from the input, while they need to be provided by a user in the open-
vocabulary case.

##  3 Method

###  3.1 Problem Definition

Given a point cloud $\mathbf{P}=\\{(p_{n})\\}_{n=1}^{N}\in\mathbb{R}^{N\times
3}$ with $N$ being the number of points, the aim is to compute the
corresponding semantic labels for every point. Note that the set of candidate
labels differs among different task settings. In conventional LiDAR
segmentation, the label set is predetermined according to the dataset. Open-
vocabulary segmentation allows users to specify the label candidates. In auto-
vocabulary segmentation, the label set is scene-specific and automatically
generated. In other words, the label set is neither pre-defined nor user-
specified.

###  3.2 AutoVoc3D

In this work, we adapt and tailor 2D auto-vocabulary segmentation, as
mentioned in Sec. 2.2, to the 3D domain. LiDAR point cloud datasets consist of
both images and LiDAR scans, which enables the automatic creation of a
vocabulary with an image-based captioning model. To this end, we employ
AutoSeg [25], a training-free method based on BLIP [9], to generate high-
quality nominal queries. As shown in Fig. 2 (b), given a LiDAR point cloud and
its corresponding images, we feed the images into AutoSeg. Since a point cloud
often corresponds to several images, we obtain several sets of queries. These
queries are merged, deduplicated and then input to an open-vocabulary LiDAR
point segmentor. We employ OpenScene [13] as the segmentor. OpenScene is an
open-vocabulary segmentor for 3D point clouds, where the point encoder is
aligned to CLIP vision-language latent space by point-wise feature
distillation from the CLIP image features. We utilize the distilled weights of
OpenScene, input generated queries and LiDAR points to OpenScene and obtain a
point-wise segmentation.

##  4 Evaluation

###  4.1 Challenges

Since auto-vocabulary segmentation is a novel task, there is no established
benchmark to compare different methods. Open-vocabulary segmentation, which is
the most similar but easier task, can be evaluated on conventional
segmentation datasets by using pre-defined categories as queries. However,
auto-vocabulary segmentation lacks access to these categories, which poses a
hurdle when evaluating its performance. On the other hand, as pointed out in
AutoSeg [25], natural language is ambiguous, leading to complex relations
between two classes, such as synonymity, hyponymy, hypernymy, meronymy or
holonymy. Therefore, it is challenging to assert that an instance is
accurately allocated an appropriate semantic label. For instance, a road can
be identified as a drivable surface, street, or roadway. In addition, a tyre
can be classified as either an independent entity or as a constituent part of
a vehicle. Therefore, we explore two quantitative metrics to assess our method
and present qualitative results as Fig. 1 and Fig 3.

###  4.2 LLM-based Auto-Vocabulary Evaluator

In order to enable the evaluation of segmentation with automatically generated
categories, [25] proposed an LLM-based Auto-Vocabulary Evaluator (LAVE)
employing the Llama 2 7B Large Language model111www.huggingface.co/meta-
llama/Llama-2-7b. LAVE is designed to map each unique auto-vocabulary category
(_e.g_. sedan and air) to a category in the list of fixed ground truth classes
(_e.g_. car and sky) in the dataset. Hence, after the segmentation of the
LiDAR point cloud with auto-vocabulary categories, each classification is
updated according to the mapping generated by LAVE. For instance, points
segmented as a sedan become points belonging to the car category. This
approach allows the quality of the segmentation to be evaluated with well-
known evaluation metrics based on the fixed-vocabulary categories, as well as
for us to compare with previous methods.

###  4.3 Text-Point Semantic Similarity

In this section, we elaborate on the intuition and the definition of the
proposed Text-Point Semantic Similarity (TPSS). The CLIP model [14] comprises
an image encoder and a text encoder. The optimization objective is to maximize
the similarity between the features of a paired text and image. CLIP is
believed to properly align visual and text features due to its superior
performance on vision-language tasks. In other words, the outputs of its image
encoder and text encoder reside in a vision-language latent space. At test
time, the inference relies on the assumption that a text matches an image
better if its text feature is more similar to the image feature than other
texts. We design the TPSS based on a similar assumption that a text matches a
point better if its text feature is more similar to the point feature than
other texts. Formally, let $\mathbf{P}=\\{(p_{n})\\}_{n=1}^{N}$ be a point
cloud with $N$ points and $\mathbf{L}=\\{(l_{m})\\}_{m=1}^{M}$ be a set of $M$
unique semantic labels generated for this point cloud. The text embeddings
$\mathbf{E}=\\{(e_{m})\\}_{m=1}^{M}$ and the point feature embeddings
$\mathbf{F}=\\{(f_{n})\\}_{n=1}^{N}$ are obtained as follows:

$\displaystyle\mathbf{E}$ $\displaystyle=h_{t}(\mathbf{L})$ (1)
$\displaystyle\mathbf{F}$ $\displaystyle=h_{p}(\mathbf{P})$ (2)

where $h_{t}(\cdot)$ and $h_{p}(\cdot)$ are the frozen CLIP text encoder [14]
and the trained point encoder, respectively. At test time, we assign a label
to a certain point $n$ based on a similarity metric $\mathrm{SIM}$, which is
an inner product specifically:

$\displaystyle\hat{l}_{n}$
$\displaystyle=\underset{m}{\mathrm{argmax}}\left(\mathrm{SIM}\left(f_{n},e_{m}\right)\right)$
(3)

where $\hat{l}_{n}$ denotes the predicted label for point $n$. This formula
indicates that the point $n$ is more correlated with label $\hat{l}_{n}$ than
other labels in the label set $\mathbf{L}$. Inspired by this process, the TPSS
is calculated as follows:

$\displaystyle S_{n}$
$\displaystyle=\max_{m}\left(\mathrm{SIM}(f_{n},e_{m})\right)$ (4)
$\displaystyle\mathrm{TPSS}(\mathbf{P},\mathbf{L},h_{p},h_{t})$
$\displaystyle=\underset{n}{\mathrm{mean}}(S_{n})$ (5)

where $S_{n}$ is a point-wise similarity score for the point $n$.
$\mathrm{TPSS}(\mathbf{P},\mathbf{L},h_{p},h_{t})$ measures the text-point
semantic similarity between the point cloud $\mathbf{P}$ and the label set
$\mathbf{L}$ given two encoders. Therefore, if both the point cloud and the
point encoder remain unchanged, the TPSS can compare which label set matches
the point cloud better. Note that the TPSS only applies if the point encoder
is aligned with the CLIP encoder, which is where our framework fits.

##  5 Experiments

Method | ZS | UF | nuScenes [3]
---|---|---|---
mIoU | mAcc
Fully-Supervised Methods
|  LidarMultiNet [21] | ✗ | ✗ | 82.0 | -
|  MinkowskiNet [6] | ✗ | ✗ | 78.0 | 83.7
Open-Vocabulary Methods
|  MSeg [7] Voting | ✓ | ✗ | 31.0 | 36.9
|  OpenScene [13] | ✓ | ✗ | 42.1 | 61.8
Auto-Vocabulary Methods
| AutoVoc3D (Ours) w. LAVE | ✓ | ✓ | 30.6 | 44.1

Table 1: Comparisons on nuScenes [3] Validation Set. ZS indicates a zero-shot
method without any annotations as supervision during training. UF indicates a
user-free method where no user-specified queries are needed during inference.
LAVE [25] is utilized to map generated categories to pre-defined categories.
The results of other methods are taken from OpenScene [13].

Method | TPSS
---|---
OpenScene [13] | 8.71
AutoVoc3D (Ours) | 9.26

Table 2: Test-Point Semantic Similarity on nuScenes [3] validation set.
AutoVoc3D outperforms OpenScene by 6% on TPSS, showcasing that our method is
able to generate class names which are more semantically aligned in the
vision-language space than the ground truth classes.

Qualitative comparisons are presented in Fig. 1 and Fig. 3. Because both human
annotations and OpenScene [13] take pre-defined categories as references, they
are unable to investigate novel categories proactively. In contrast, AutoVoc3D
yields more precise categories, _e.g_. building, pole and sign.

To make a quantitative comparison, we employ LAVE [25] to map all generated
novel categories back to pre-defined categories. Next, we calculate
segmentation metrics, namely mean IoU (mIoU) and mean accuracy (mAcc), on the
nuScenes validation set [3] (see Tab. 1). Note that the three types of methods
are solving different tasks. Compared to open-vocabulary methods, AutoVoc3D
does not require user-specified class names at test time, yet achieves
comparable performance.

However, using LAVE [25] makes a concession, as it discards novel categories
after the mapping. The proposed TPSS metric does not suffer from this issue.
Moreover, it provides a more accurate assessment of the semantic similarity
between the text and point cloud. AutoVoc3D outperforms OpenScene [13] (see
Tab. 2), indicating the generated label set is a better match with the point
clouds than pre-defined categories.

Front right image Back right image

OpenScene [13]

AutoVoc3D (Ours)

Figure 3: Comparison between OpenScene [13] and AutoVoc3D. AutoVoc3D
recognizes a sign (highlighted in red boxes), while OpenScene segments it as
manmade. AutoVoc3D identifies a tree as a cone and a tree (green box), while
OpenScene segments its lower part as manmade.

##  6 Conclusion

In this work, we presented AutoVoc3D, the first method for auto-vocabulary
LiDAR point segmentation. Unlike OVS methods which require the user to specify
target classes, our method is capable of automatically generating and
segmenting them. Our experiments demonstrate that our model is able to segment
regions with more precise object namings, as well as competitive accuracy,
without the need for any additional training, finetuning or additional data.

## References

  * Behley et al. [2019] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall.  SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences.  In _ICCV_ , pages 9296–9306. IEEE, 2019.
  * Borse et al. [2021] Shubhankar Borse, Ying Wang, Yizhe Zhang, and Fatih Porikli.  Inverseform: A loss function for structured boundary-aware segmentation.  In _CVPR_ , 2021.
  * Caesar et al. [2020] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuScenes: A multimodal dataset for autonomous driving.  In _CVPR_ , 2020.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  In _ICCV_ , 2021.
  * Chen et al. [2023] Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang.  Clip2scene: Towards label-efficient 3d scene understanding by clip.  In _CVPR_ , 2023.
  * Choy et al. [2019] Christopher Choy, JunYoung Gwak, and Silvio Savarese.  4d spatio-temporal convnets: Minkowski convolutional neural networks.  In _CVPR_ , 2019.
  * Lambert et al. [2020] John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun.  Mseg: A composite dataset for multi-domain semantic segmentation.  In _CVPR_ , 2020.
  * Lang et al. [2019] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  PointPillars: Fast encoders for object detection from point clouds.  In _CVPR_ , pages 12697–12705, 2019.
  * Li et al. [2022a] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _ICML_ , 2022a.
  * Li et al. [2022b] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai.  Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.  In _ECCV_ , 2022b.
  * Mao et al. [2021] Jiageng Mao, Minzhe Niu, Chenhan Jiang, Hanxue Liang, Jingheng Chen, Xiaodan Liang, Yamin Li, Chaoqiang Ye, Wei Zhang, Zhenguo Li, Jie Yu, Hang Xu, and Chunjing Xu.  One million scenes for autonomous driving: Once dataset.  In _NeurIPS_ , 2021.
  * Najibi et al. [2023] Mahyar Najibi, Jingwei Ji, Yin Zhou, Charles R. Qi, Xinchen Yan, Scott Ettinger, and Dragomir Anguelov.  Unsupervised 3d perception with 2d vision-language distillation for autonomous driving.  In _ICCV_ , 2023.
  * Peng et al. [2023-04-06] Songyou Peng, Kyle Genova, Chiyu ”Max” Jiang, Andrea Tagliasacchi, Marc Pollefeys, and Thomas Funkhouser.  Openscene: 3d scene understanding with open vocabularies.  In _CVPR_ , 2023-04-06.
  * Radford et al. [2022] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In _ICML_ , 2022.
  * Rewatbowornwong et al. [2023] Pitchaporn Rewatbowornwong, Nattanat Chatthee, Ekapol Chuangsuwanich, and Supasorn Suwajanakorn.  Zero-guidance segmentation using zero segment labels.  In _ICCV_ , 2023.
  * Sun et al. [2020] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov.  Scalability in perception for autonomous driving: Waymo open dataset.  In _CVPR_ , pages 2443–2451, 2020.
  * Tao et al. [2020] Andrew Tao, Karan Sapra, and Bryan Catanzaro.  Hierarchical multi-scale attention for semantic segmentation.  _arXiv preprint arXiv:2005.10821_ , 2020.
  * Wu et al. [2024] Jianzong Wu, Xiangtai Li, Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, Bernard Ghanem, and Dacheng Tao.  Towards open vocabulary learning: A survey.  _TPAMI_ , 2024.
  * Xue et al. [2023] Le Xue, Mingfei Gao, Chen Xing, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese.  Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding.  In _CVPR_ , 2023.
  * Yan et al. [2018] Yan Yan, Yuxing Mao, and Bo Li.  Second: Sparsely embedded convolutional detection.  _Sensors_ , 18(10):3337, 2018.
  * Ye et al. [2023] Dongqiangzi Ye, Zixiang Zhou, Weijia Chen, Yufei Xie, Yu Wang, Panqu Wang, and Hassan Foroosh.  Lidarmultinet: Towards a unified multi-task network for lidar perception.  In _AAAI_ , 2023.
  * Yin et al. [2021] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl.  Center-based 3d object detection and tracking.  In _CVPR_ , 2021.
  * Zeng et al. [2023] Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung, Zhen Yang, Xiaodan Liang, and Hang Xu.  Clip$^2$: Contrastive language-image-point pretraining from real-world point cloud data.  In _CVPR_ , 2023.
  * Zhou et al. [2023] Xingcheng Zhou, Mingyu Liu, Bare Luka Zagar, Ekim Yurtsever, and Alois C. Knoll.  Vision language models in autonomous driving and intelligent transportation systems.  _arXiv preprint arXiv:2310.14414_ , 2023.
  * Ülger et al. [2024] Osman Ülger, Maksymilian Kulicki, Yuki Asano, and Martin R. Oswald.  Auto-vocabulary semantic segmentation.  _arXiv preprint arXiv:2312.04539_ , 2024.


# Anticipatory Thinking Challenges in Open Worlds: Risk Management

Adam Amos-Binks, 1 Dustin Dannenhauer, 2 Leilani H. Gilpin 3

## Introduction

Anticipatory thinking (Geden et al. 2019) drives our ability to manage risk –
identification and mitigation – in everyday life, from bringing an umbrella
when it might rain to buying car insurance. As AI systems become part of
everyday life, they too have begun to manage risk. Autonomous vehicles log
millions of miles (Kalra and Paddock 2016), StarCraft and Go agents have
similar capabilities to humans, implicitly managing risks presented by their
opponents. To further increase performance in these tasks, out-of-distribution
evaluation has emerged as a way to characterize a model’s bias, what we view
as a type of risk management.

However, learning to identify and mitigate low-frequency, high-impact risks is
at odds with the observational bias required to train machine learning models.
StarCraft and Go are closed-world domains whose risks are known and
mitigations well documented, ideal for learning through repetition.
Adversarial filtering datasets provide difficult examples but are laborious to
curate and static (Hendrycks et al. 2021), both barriers to real-world risk
management. Adversarial robustness focuses on model poisoning under the
assumption there is an adversary with malicious intent, without considering
naturally occurring adversarial examples. Adversarial generation focuses at
the object level (e.g. Zhao, Dua, and Singh 2018) without an open-world
context where adversarial scenes occur. These methods are all important steps
towards improving risk management but they do so without considering open-
worlds, where new risks and new mitigations require dynamic risk management.

We unify these open-world risk management challenges with two contributions.
The first is our perception challenges, designed for agents with imperfect
perceptions of their environment whose consequences have a high impact (e.g.
autonomous vehicles and public safety). Our second contribution are cognition
challenges, designed for agents that must dynamically adjust their risk
exposure as they identify new risks and learn new mitigations. Our goal with
these challenges is to spur research into solutions that assess and improve
the anticipatory thinking required by AI agents to manage risk in open-worlds
and ultimately the real-world.

## Open World Risk Management

Figure 1: Managing risk in open-worlds is more complex than in closed-worlds.
Agents must be resilient to unknown risks and once observed, new risks need to
be balanced with existing risks by applying new mitigations. Figure 2: Flow
diagram of the Perception challenges. Perception challenge 1: Object bias adds
a content generation step at the perception system level. The inputs to the
content generation step are generated from deployment (real, rare objects that
were previously incorrectly labelled). Perception challenge 2: Rare Scenes
adds a content generation step at the system level. The inputs to this content
generation step are generated in development; inspired by “real” rare scenes;
e.g., a truck carrying traffic lights.

We view current AI systems like insurance agents that manage risks in a
closed-world. Within an insurance domain (e.g. home) there are few unknown
risks to insure against and the known risks are static (e.g. flood, fire,
etc), see top Figure 1. Agents calculate mitigations (premiums) from an ample
supply of historical likelihood and impact data. An individual manages their
risk by purchasing policy premiums depends on their risk tolerance and budget.
In contrast, the open-world perception challenge captures how well a
perception system adapts to the disproportional (and growing) number of
unknown risks in relation to known risks, see bottom Figure 1. For example,
perception systems in autonomous vehicles have caused accidents by mis-
classifying concrete barriers as the horizon. This epitomizes the often
referred ‘long-tail’ of errors, the infrequent but highly impactful situations
that open-world AI systems must manage the risk of. Solutions to the
perception challenge will provide accurate assessments of a model’s ability to
identify and manage unknown risks both at the object and scene level. Our
cognition challenge assumes high quality perceptions (e.g. identifying new
known risks) and focuses on balancing newly identified risks and learning new
mitigations to maintain a desired risk profile. Open-world games (those with
large, complex action models and catastrophic consequences such as permanent
death) are ideal domains as long-range credit assignment is difficult to model
and new adversaries and capabilities are constantly being discovered. The
following sections detail these challenges in concrete domains and propose
example high-level solutions that combine both symbolic reasoning and machine
learning to operationalize risk management in open-worlds.

## Perception Challenges

Autonomous vehicles are constantly perceiving their surrounding environments
with perception systems, e.g., vision, LiDAR, and radar, to resolve the
difference between reality and their representation of it. The vision system,
which is opaque to humans, is not prepared for rare but highly impactful
perception errors; making autonomous vehicles an open-world domain (Langley
2020).

One of the challenges of quantifying the risk of autonomous vehicles in the
real world is that they are deployed in an open-world but tested in closed-
world simulations. This juxtaposition highlights how unprepared autonomous
vehicles are for naturally occurring adversarial examples. In this challenge,
we suggest an alternative: generating naturally occurring adversarial examples
from “real” rare cases to supplement the closed-world test simulations,
mimicking the open-worlds where they are deployed. We further refine the
perception challenge at the object and scene level, summarizing them in Figure
2.

### Perception Challenge 1: Object Bias

The current state-of-the-art vision systems are easily fooled (Nguyen,
Yosinski, and Clune 2015) by out-of-distribution inputs that exploit a model’s
observational bias. Some solutions include training on adversarial examples
(Ilyas et al. 2019), but they only lead to improvements on this data set.
Instead, we propose iterative testing where objects mislabelled in deployment
are used as input to a content generation model for testing and evaluation of
a perception system’s observational biases. The generative model creates sets
of naturally adversarial objects that can be tested during development (e.g.
faded stop signs, shadow patterns).

Figure 3: We operationalized cognition challenges in the Dungeon Crawl Stone
Soup environment that include long-term strategy (top) and short-term tactics
(bottom) risk management tasks.

### Perception Challenge 2: Rare Scenes

A second type of imperfect perception occurs when perceiving rare scenes.
While there is still a potentially problematic observational bias at the
object level, we assume that the object detection system is mostly accurate. A
vision system can still be “fooled” by rare scenes such as traffic lights on a
truck111https://futurism.com/the-byte/tesla-autopilot-bamboozled-truck-
traffic-lights. In these cases, the perception is correct: the objects are
correctly identified, but the scene is so rare and disruptive that it should
be immediately recognized so the system can avoid catastrophic failures (e.g.
immediately stopping on a freeway). These “naturally-occurring adversarial
scenes” exemplify the problem with the long tail of errors in autonomous
vehicles. An additional layer of reasoning is required to test and monitor for
these types of rare scenes.

To address rare scene limitations, we propose a similar iterative process: a
generative model that constructs out-of-distribution scenes with commonly
occurring objects. The generative model uses objects that are regularly seen
in the data (e.g., road signs, objects, etc.) and changes their orientation
(e.g., fallen signs on the ground) or environment (e.g., objects in the same
scene that usually are not together).

### Example Solution

Autonomy requires adapting to operational circumstances that cannot be
enumerated pre-hoc. However, assessing this ability is static and contrived,
relying on benchmark tasks (e.g. avoid collisions in a benchmark scenario)
that systems could be gaming with non-generalizable rules. Methods such as
variational auto-encoders (VAEs) and generative adversarial networks (GANs)
can generate a whole family of tasks and push the boundaries of contrived
task-oriented evaluations to rich attribute-oriented evaluations. We envision
integrating physical world knowledge into content generation methods to ensure
real-world properties are preserved and natural adversarial instances are
generated. For example, static traffic indicators (stop signs, traffic lights,
etc) that are now dynamic (e.g. loaded onto a flatbed truck, advertisement
painted on a large truck). Performance against a family of out-of-distribution
tasks would indicate a system’s competency in a risk management and provide a
level of insight into future performance not currently available.

### Impact

Autonomous vehicles in new, open environments are susceptible to causing harm
(even death). Despite high accuracy in testing, object recognition systems
cannot account for rare but consequential out-of-distribution inputs. Testing
and improving a perceptions system’s ability to manage the risk presented by
these new circumstances would make autonomous systems safer and cover a larger
number of error and failure cases. This type of anticipatory, introspective,
solution impacts public safety, manufacturers, and regulators.

## Cognition Challenges

Our cognition challenges for anticipatory thinking in open worlds involve AI
agents acting in unexplored, hostile, and dynamic environments. The first
challenge: long-term strategy concerns issues of risk as agents move between
situations and micro environments where resource availability changes. The
second challenge: short-term tactics concerns more immediate risks that
require agents to make use of the right resources at the right time and choice
of enemy engagements. We use the rogue-like video game Dungeon Crawl Stone
Soup (DCSS) to operationalize this challenge problem. In DCSS a player moves
through a procedurally generated, partially observable, and stochastic
environment to retrieve the ‘Orb of Zot’ while managing the risks (permanent
death) of encountering thousands of monsters. DCSS is one of two rogue-like
games with increased interest in recent years and remains an unsolved domain
for AI (Dannenhauer et al. 2021; Küttler et al. 2020).

We consider DCSS an open-world for both human and AI agents owing to over 600
unique monsters and 100 item types that a player comes across as they delve
deeper into the 100 levels of the dungeon. As the player enters new areas of
the game, the chances of encountering each type of monster and item change,
such that certain items and monsters cannot be found until reaching a certain
depth. Therefore as a player who begins after only playing the short tutorial,
there are many surprises and unknowns encountered. For this reason, DCSS can
be considered an open-world for any learning agent who does not start with a
complete model of the environment (such a model likely only exists in the
minds of the lead developers). For use in AI research, dcss-ai-wrapper
(Dannenhauer et al. 2021) provides an API for both symbolic and vector AI
agents to interact with DCSS.

### Cognition Challenge 1: Long-term Strategy

To win a game of DCSS, AI agents must mitigate risk over longer time horizons
as they visit different regions of a world. Humans perform this type of
reasoning frequently, such as going from home to work, going from the grocery
store to a camping trip, or going from the airport lobby through security to
their gate. In all cases, agents are able to take actions in the source region
that may no longer be available in the destination region. Such actions could
include whether to pack an umbrella and lunch box, or taking a water filter.
These same challenges are present in the DCSS domain and require decision
making to decide where in the dungeon to go next and choices in enhancing
character capabilities. The effects of these decisions are often noticed only
after significant time delays as shown in the top portion of Figure 3. We now
highlight two aspects of the long-term strategy challenge in DCSS:

Capability Enhancement via Skill Point Allocation: The player earns experience
when killing monsters which is permanently allocated to one or more of 33
skills. These skills impact the success of a variety of player-actions and
defensive capabilities. Failure in the mid to late game often results from
poor risk management of skill points.

Region Transitions via Level Choices: There are 24 themed branches (regions)
of the dungeon, each with their own special characteristics. Some branches are
known for an overwhelming majority of poison-based monsters (spider lair and
snake pit), corrosive monsters (slime pit), monsters that will cause unhelpful
mutations to your character (the abyss), or monsters with special abilities,
such as locking stairs preventing escape (the vaults). Further increasing
complexity, monsters move in levels independent of your movement, it is likely
that as you explore a level, monsters will move in between your planned escape
route from the location at which you entered the level. Entering a branch
without dynamically adjusting risks and mitigations (poison resist gear,
corrosion resist gear, potions of cure mutation, etc.) will likely lead to
catastrophic failure.

### Cognition Challenge 2: Short-term Tactics

The limited visibility and highly dynamic, stochastic nature of item and
monster interactions means that there are many possible futures over short
horizons consisting of a small ($<5$) possible actions. Additionally, agents
may never encounter uncommon situations more than once because of the
procedurally generated nature of the game. Therefore short-term anticipatory
thinking should seek to mitigate the absolute worst outcomes even if unable to
accurately predict a single outcome. These types of decisions are shown in the
bottom portion of Figure 3. We now highlight two aspects of the short-term
tactics challenge in DCSS:

Engagements via Direct Combat: It is impossible to progress into the mid to
late game by continuously engaging in combat with little thought to tactical
reasoning. Retreating, hiding from monsters, using special abilities, and
using other terrain features such as hallways, are needed to gain advantages
(i.e. in a hallway you can fight monsters in smaller batches).

Resource Management via Item Inventory: The player will encounter tens of
thousands of item instances from 100 item types and the player has a limited
inventory space. Sometimes items may need to be left and returned to later,
other items dropped to make sure the most important life saving items are kept
for dire situations. Additionally, the choice to use or save a life-saving
consumable item is often the difference between success and failure. These
types of decisions must be made many times during gameplay.

### Example Solutions

Simply learning a complete action model of an open-world is intractable and so
is encoding all knowledge an agent needs for managing risks. A hybrid or 3rd
wave approach would reason over the semantics of what new capabilities or
assets would produce a desired risk profile (e.g. goal reasoning, meta-
cognition), taking into account the small samples of newly observed risks
(e.g. meta-learning, few-shot learning). This approach is just an illustrative
example of the complexities required to go beyond single task performance and
manage risk in open-worlds.

### Impact

Developing AI agents that can solve these two challenges in DCSS will lead to
approaches that can lead to better AT in a wide variety of autonomous systems.
An autonomy approach that can solve the right choice of entering which branch
of the dungeon could be the same approach that helps an autonomous drone
decide whether it can follow a target into a broken down urban building, a
forest, or a cave. An AI that is tasked with it’s mission and can solve the
pre-deployment problem could alert it’s operating before it leaves the forward
operating base that it’s missing a needed component to mitigate a possible
risk from adversaries that the operator may have forgotten. An AI agent that
can manage risks during operation will be able to make the call that it cannot
outrun a new enemy and must seek to hide instead.

## Conclusion

Real-world deployments of AI systems require a level of robustness and safety
best tested in open-worlds. We use a risk management approach to contribute
two challenges and illustrate them in different domains. Our perception
challenges characterize a systems ability to recognizing both rare objects and
configurations of them. Solutions with this capability are essential as the
real-world is constantly producing new permutations of objects and contexts
they appear in. Our cognition challenges characterize the relationship between
new risks and new capabilities to mitigate them. Dynamically balancing a risk
profile with new information enables AI agents to adapt to situations like
humans do and avoid needless catastrophic failures. Our example solutions
focus on integrating symbolic and learning approaches together to manage risks
in perception and cognition.

## References

  * Dannenhauer et al. (2021) Dannenhauer, D.; Dannenhauer, Z. A.; Decker, J.; Amos-Binks, A.; Floyd, M. W.; and and, D. W. A. 2021.  dcss-ai-wrapper: An API for Dungeon Crawl Stone Soup providing both Vector and Symbolic State Representations.  In _Planning and Reinforcement Learning Workshop. International Conference on Automated Planning and Scheduling (ICAPS)_.
  * Geden et al. (2019) Geden, M.; Smith, A.; Campbell, J.; Spain, R.; Amos-Binks, A.; Mott, B.; Feng, J.; and Lester, J. 2019.  Construction and Validation of an Anticipatory Thinking Assessment.  _Frontiers in Psychology_ 10(December): 1–10.  ISSN 16641078.  doi:10.3389/fpsyg.2019.02749.
  * Hendrycks et al. (2021) Hendrycks, D.; Zhao, K.; Basart, S.; Steinhardt, J.; and Song, D. 2021.  Natural Adversarial Examples.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 15257–15266.  doi:10.1109/CVPR46437.2021.01501.
  * Ilyas et al. (2019) Ilyas, A.; Santurkar, S.; Tsipras, D.; Engstrom, L.; Tran, B.; and Madry, A. 2019\.  Adversarial examples are not bugs, they are features.  _arXiv preprint arXiv:1905.02175_ .
  * Kalra and Paddock (2016) Kalra, N.; and Paddock, S. M. 2016.  _Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?_ Santa Monica, CA: RAND Corporation.  doi:10.7249/RR1478.
  * Küttler et al. (2020) Küttler, H.; Nardelli, N.; Miller, A. H.; Raileanu, R.; Selvatici, M.; Grefenstette, E.; and Rocktäschel, T. 2020.  The NetHack Learning Environment.  In _Proceedings of the Conference on Neural Information Processing Systems (NeurIPS)_.
  * Langley (2020) Langley, P. 2020.  Open-World Learning for Radically Autonomous Agents.  _Proceedings of the AAAI Conference on Artificial Intelligence_ 34(09): 13539–13543.  doi:10.1609/aaai.v34i09.7078.  URL https://ojs.aaai.org/index.php/AAAI/article/view/7078.
  * Nguyen, Yosinski, and Clune (2015) Nguyen, A.; Yosinski, J.; and Clune, J. 2015.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 427–436.
  * Zhao, Dua, and Singh (2018) Zhao, Z.; Dua, D.; and Singh, S. 2018.  Generating natural adversarial examples.  _6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings_ (2016): 1–15.