# Few-Shot Classification & Segmentation Using Large Language Models Agent

Tian Meng 1† Yang Tao 2† Wuliang Yin 1∗
1University of Manchester 2Mettler Toledo Safeline
†Joint First Author ∗Project Lead

###### Abstract

The task of few-shot image classification and segmentation (FS-CS) requires
the classification and segmentation of target objects in a query image, given
only a few examples of the target classes. We introduce a method that utilises
large language models (LLM) as an agent to address the FS-CS problem in a
training-free manner. By making the LLM the task planner and off-the-shelf
vision models the tools, the proposed method is capable of classifying and
segmenting target objects using only image-level labels. Specifically, chain-
of-thought prompting and in-context learning guide the LLM to observe support
images like human; vision models such as Segment Anything Model (SAM) and
GPT-4Vision assist LLM understand spatial and semantic information at the same
time. Ultimately, the LLM uses its summarizing and reasoning capabilities to
classify and segment the query image. The proposed method’s modular framework
makes it easily extendable. Our approach achieves state-of-the-art performance
on the Pascal-5i dataset.

##  1 Introduction

Few shot learning [7, 35, 30, 29, 22] is a machine learning paradigm where
only a small number of examples are provided, and requires the model to
generalize well to new, unseen data. Compared with the conventional heavy data
paradigm, it relieves the burden of data labeling and annotation, and it also
extend the application scenarios, where large amount of data cannot be
collected easily. In computer vision, few shot classification [37, 19] and few
shot segmentation [33, 24] has been actively studied. Although the setups of
these two task are similar, only a few works [14, 13] related them together.
In this paper, we aim to solve the few shot image classification and
segmentation (FS-CS), which requires to provide the presence of each support
class in the query image as well as their pixel-level segmentation.

Despite the rapid advancements in few-shot learning approaches, traditional
models heavily rely on meta-learning strategies [11, 27, 32, 21] or transfer
learning paradigms [41, 36, 42] that typically inherit information from a base
dataset with abundant labeled data. These models are then fine-tuned or
adapted using a small number of examples from novel classes. However, such
approaches often suffer from overfitting, especially in the context of few-
shot segmentation, which demands a high level of generalization from limited
annotations. The transformer-based models [31, 9, 10], particularly vision
transformers [6, 20, 16], have recently shifted the paradigm by modeling image
regions through self-attention mechanisms, providing a flexible architecture
capable of capturing long-range dependencies. Although these models have the
potential for few-shot tasks, they are still bottlenecked by the requirement
of substantial computation and fine-tuning on novel datasets.

Figure 1: Few-Shot Classification & Segmentation Using Large Language Models
Agent. By providing vision tools to LLM like GPT-4, it can solve the task of
Few-Shot Classification & Segmentation with only image-level label in a
training-free manner.

Furthermore, the integration of language and vision to create multimodal
models, such as CLIP [23] and DALL-E [25], has opened new research avenues for
leveraging descriptive language to guide visual understanding. While these
advancements are encouraging, they have not fully utilized the potential of
large language models (LLMs) in directly guiding few-shot image classification
and segmentation (FS-CS) tasks. LLMs like GPT-4, with their in-context
learning capabilities, can process complex instructions and understand
abstract concepts, making them prime candidates to address FS-CS through a
training-free approach. In this paper, we leverage the in-context learning and
reasoning capabilities of a LLM to make sense of the visual world through
textual descriptions and then guide pre-trained vision models, such as SAM
[18] and GPT-4Vsion in performing FS-CS tasks. This strategy of LLM as an
agent eliminates the need for extensive training or fine-tuning on novel
datasets.

To summarize, our contributions are the following:

  * •

We present a novel, modular framework that harnesses the power of LLMs for the
FS-CS problem without additional training, enabling rapid adaptation to new
tasks and domains.

  * •

We exploit text prompting to guide the LLM in parsing few-shot tasks and
generating action sequences for vision tools, and integrate text and visual
prompting for VLM to perform visual tasks. This approach effectively
transforms the LLM into an intelligent task planner that vision model tools
for computer vision applications.

  * •

We demonstrate state-of-the-art results on standard few-shot benchmark
Pascal-5i [26], showcasing the efficiency and effectiveness of our approach
for both classification and segmentation tasks.

##  2 Related work

###  2.1 Few shot classification and segmentation

Few-shot image classification and segmentation (FS-CS) [14, 13] aims to
generalize an algorithm to new classes not seen during training, given only a
small sample of images. Few-shot classification [15, 3, 4] has been
extensively studied, with approaches ranging from metric-based learning such
as Siamese Networks [40] and Matching Networks [1], to model-based methods [8]
that meta-learn an internal model that can quickly adapt to new tasks. Few-
shot segmentation [5, 34], on the other hand, has gained attention more
recently. It expands the challenge by requiring pixel-level annotations of
novel classes from a few instances. Approaches like prototypical networks
[28], which learn a metric space where segmentation can be performed as a form
of nearest neighbor classification, and gradient-based meta-learning
techniques [17], which aim at rapid adaptation of model parameters, have shown
promising results. However, the generalization capability to new classes
without further fine-tuning remains a substantial hurdle to overcome.

###  2.2 Visual prompting

Visual prompting is an innovative strategy that steers pretrained vision
models’ behavior using input modifications or augmentations [12, 2, 38, 39].
It has roots in the success of textual prompts in LLMs and aims to achieve
similar flexibility in vision tasks. Methods like CLIP [23] have pioneered the
field by using text-image pairs to learn general visual features that can be
queried using textual prompts. Visual prompting can be as simple as applying
visual transformations that cue the model towards certain responses or as
complex as synthesizing images. Our approach explores the efficacy of coupling
textual and visual prompting for vision language models to enhance the
problem-solving strategy delineated by an LLM. By conditionally adapting
prompts based on the reasoning supplied by the LLM, we can direct the
computational attention of vision models toward relevant aspects of the FS-CS
tasks.

##  3 Problem formulation

Figure 2: Overview of LLM agent using visual tools to solve the task of few-
shot classification and segmentation. A TODO instruction list with a N-way
K-shot task sampled from the database is provided to the LLM agent. Chain of
thought is utilised to help LLM generating plan and sampling corresponded
tools in the tool space. Self-reflection helps LLM improve the final
segmentation result progressively.

The core problem addressed in this paper is the Few-Shot Image Classification
and Segmentation (FS-CS), which involves both identifying the category of
target objects in a query image and delineating their precise pixel-level
segmentation, given only a handful of labeled examples. Formally, we consider
the few-shot setting with a support set $S$ and a query set $Q$. For a $N$-way
$K$-shot classification and segmentation task, the support set $S$ consists of
$N$ distinct object classes, each with $K$ labeled examples:

$S=\\{(x_{n,k},m_{n,k})\\}_{n=1,k=1}^{N,K}$ (1)

where $x_{n,k}$ is the $k$-th image instance of the $n$-th class, and
$m_{n,k}$ is the binary mask for image $x_{n,k}$, indicating the pixel-level
presence of the $n$-th class object.
The query set $Q$ is defined as:

$Q=\\{(x^{\prime},y^{\prime},m^{\prime})\\}$ (2)

where $x^{\prime}$ is a new query image potentially containing instances of
object classes from the support set $S$, $y^{\prime}$ denotes the set of class
labels present in $x^{\prime}$, and $m^{\prime}$ is the ground truth
segmentation mask for the query image that needs to be predicted. The
objective is to learn a model that uses the support set $S$ to classify and
segment the query image $x^{\prime}$ correctly.

##  4 Method

To tackle this problem, we propose a modular framework that divides the task
into a sequence of sub-problems formalized through the use of Large Language
Models (LLMs) as task planners, and then solve each sub-problem using
corresponding tools.

###  4.1 Task Planning

Using the chain-of-thought approach, the LLM generates an explicit reasoning
sequence that maps the support set $S$ to a strategy that can be executed on
any query image $x^{\prime}$ by vision models. The output is a structured
action plan $P$, containing a sequence of vision model tasks $T$:

$P(T|S)=\text{LLM}_{\text{planner}}(S)$ (3)

This step effectively translates the problem into a set of instructions which
can be understood and executed by vision models. Based on the generated plan
$P$, vision models are prompted to perform the task $T$, which involves
classifying or segmenting the objects in the query image $x^{\prime}$. This
step leverages pre-trained vision models that can understand spatial and
semantic information, guided by the language-based plan:

$(\hat{y^{\prime}},\hat{m^{\prime}})=\text{execute}(T,x^{\prime},\text{Vision
Models})$ (4)

where $\hat{y^{\prime}}$ is the predicted class label, and $\hat{m^{\prime}}$
is the predicted segmentation mask for the query image. The effectiveness of
the proposed framework is measured based on the accuracy of the classification
and the quality of the segmentation for the query image $x^{\prime}$, with
standard metrics such as Intersection over Union (IoU) for segmentation and
accuracy for classification. The novel contribution of this formulation is the
decoupling of image understanding, task planning, and execution, allowing the
LLM to use its reasoning capabilities without necessitating any end-to-end
training or fine-tuning for novel FS-CS tasks. This approach aims at achieving
rapid adaptation to new classes and tasks in a training-free manner, bridging
the gap between natural language understanding and visual perception.

###  4.2 Cognition

Cognition task processes the support set $S$, recognizing the classes and the
associated examples. It encompasses both identifying the target classes and
extracting semantic insights from the support images. The cognitive task is
formalized as follows:

$C=\text{execute}(\text{cognize},S,\text{GPT-4Vision})$ (5)

where $C$ represents the cognitive output, containing a mapping between
language-based descriptions and visual features that other tasks can use.
The GPT-4Vision’s comprehension of natural language descriptions is harnessed
to establish a cognitive connection between the textual class and their visual
counterparts. These insights inform the construction of a task plan, which
outlines how the vision models should interpret the query image $x^{\prime}$.
A hybrid prompting, including both text and visual prompting, is utilised to
allow GPT-4Vision to understand the target object in the support image and
provide accurate language-based descriptions. The image-level labels and other
possible metadata associated with the support set are provided to GPT-4Vision
as shown in Fig. 3. On the other hand, the coupled text prompting, including
the descriptive information such as ”The target object is in the RED bounding
box and covered by LIGHT BLUE mask.”, is also used to guide GPT-4Vision to
complete the task.

Figure 3: GPT-4Vision visual prompt for cognition

###  4.3 Questing

With cognitive outputs in hand, the LLM could employs the ”questing” task
where it utilises GPT-4Vision to perform specific tasks on the query image.
Here, ”questing” refers to the journey of seeking out and localizing target
objects within the image. For each class in the support set, the LLM instructs
GPT-4Vision to: (i) Determine the presence of the same object within the query
image $x^{\prime}$, giving rise to a binary decision for classification. (ii)
Provide the bounding box coordinates of the object assuming its presence,
which serves as a precursor for the segmentation task. This is described
through the questing function:

$R=\text{execute}(\text{quest},\\{C,x^{\prime}\\},\text{GPT-4Vision})$ (6)

where $R$ represents questing response, including class presence indicators
and bounding box coordinates. GPT-4Vision processes the cognitive insights $C$
and the query image $x^{\prime}$, executing the dual objectives of
identification and localization. GPT-4Vision’s capability to interpret and
adhere to the guidance of LLM-generated plans is crucial to the success of
this step. Similarly, related visual and text prompting are provided to guide
GPT-4Vision to output the bounding box accurately. We used a new style of
visual prompting, as shown in Fig. 4, by plotting the coordinate ticks or grid
directly on the image to aid object localization.

Figure 4: GPT-4Vision visual prompt for questing

###  4.4 Segmentation

LLM provides the foundational input for the ”Segment Anything model” (SAM),
which specializes in creating pixel-level representations of objects within
the given bounding boxes. SAM leverages the bounding box coordinates to mask
the pertinent region of the query image, distinguishing between the target
object and its background. The segmentation process is specced out thus:

$\hat{m^{\prime}}=\text{execute}(\text{segment},\\{R,x^{\prime}\\},\text{SAM})$
(7)

where $\hat{m^{\prime}}$ symbolizes the output segmentation mask for each
class depicted in the query image $x^{\prime}$. SAM’s performance is
instrumental, as it pushes the frontiers of few-shot segmentation by
establishing class-specific segmentation without needing class-wise annotated
examples, which is different from traditional methods relying on extensive
fine-tuning with detailed pixel annotations.

###  4.5 Quality Judgement and Self-Reflection

The final stage in our framework involves a self-reflection task, designed to
critically evaluate and improve the segmentation quality. The LLM re-engages
post-segmentation to perform a binary assessment of $\hat{m^{\prime}}$. The
GPT-4Vision reviews the segmentation quality of $\hat{m^{\prime}}$ with
codified qualitative metrics, such as shape conformity, coverage, and class
confidence, to form a judgement on segmentation quality:

$J=\text{execute}(\text{judge},\\{C,\hat{m^{\prime}}\\},\text{GPT-4Vision})$
(8)

where $J$ denotes the judgement output, which is a binary approval and a set
of refinement suggestions. Should the quality not meet predefined standards,
the self-reflection loop triggers iterative adjustments via the LLM. The LLM
refines the plan $P$, leading to a recalibration of both the questing $R$ and
segmentation $\hat{m^{\prime}}$. This iterative mechanism ensures continuous
improvement and learning, essentially embodying a vigilant quality control
overseer within the FS-CS framework. Through this self-aware process, the
proposed method not only sustains high accuracy but also fosters an internal
validation mechanism, promoting reliability and trustworthiness in autonomous
FS-CS systems.
In addition, in context learning is utilised to guide GPT-4Vision gives
objective assessment and constructive refinement suggestions. By providing
examples of good and poor segmentation outcomes along with their critiques,
which enables GPT-4Vision to adaptively learn the judgement standard. And LLM
could refine the segmentation mask over successive iterations.

##  5 Experiment

In order to validate the efficacy of our proposed framework, we conducted
extensive experiments on the widely used few-shot learning benchmark
Pascal-5i. We compared our model’s performance against several state-of-the-
art few-shot classification and segmentation approaches. This section details
the experimental setup, datasets, evaluation metrics, baselines for
comparison, and the results obtained.

###  5.1 Datasets

Pascal-5i derived from PASCAL VOC Challenge contains 20 object classes and is
split into 4 different sets, with each set treated as one cross-validation
fold. We follow the standard protocol for few-shot segmentation where each
class is evaluated under the one-shot scenarios.

Figure 5: GPT-4 prompt for the task of Few-Shot Classification & Segmentation
Figure 6: GPT-4Vision prompt for cognition object in support image Figure 7:
GPT-4Vision prompt for identifying the query image object and localisation
Figure 8: GPT-4Vision prompt for judging the segmentation mask quality Figure
9: A example showing the workflow of solving the task of classicication &
segmentation

###  5.2 Implementation Details

Our Large Language Model (LLM) used for task planning and reasoning is based
on GPT-4, while the vision models involved are pre-trained versions of Segment
Anything Model Huge (SAM-H) and GPT-4Vision. We apply a chain-of-thought
prompting method to guide the LLM through the reasoning process while
employing a hybrid form of visual and textual prompting for GPT-4Vision.
Detailed prompt designs are described as the following.
LLM task planning. The purpose of the LLM task planning prompt, as shown in
Fig. 5, is to articulate and organize the approach to the few-shot
classification and segmentation task. By using a chain-of-thought style of
prompting, we encourage GPT-4 to (i) Generate a step-by-step reasoning
sequence reflecting the LLM’s understanding of the task requirements. (ii)
Formulate an action plan that indicates how vision models should be used to
classify and segment objects in the query image, given the information about
support classes. (iii) Reflect the obtained information and continue to
improve segmentation mask quality.
Cognition. In the cognition prompt, as shown in Fig. 6, the main aim is to
leverage GPT-4Vision to interpret the visual data provided in the support set
and establish cognitive links between the textual labels and their visual
representations. The prompt guides GPT-4Vision to: (i) Recognize the target
classes within the support images. (ii) Describe the target object accurately
leveraging any metadata provided. The cognitive prompt could be structured as
part narrative and part directive, telling a story of the image and guiding
GPT-4Vision to focus on certain attributes relevant to the task, such as
distinguishing features of classes or peculiarities in the images that might
aid classification and segmentation.
Questing. The questing prompt, as shown in Fig. 7, serves to direct
GPT-4Vision to identify and localize objects in a new query image as specified
by the LLM-generated plan. This prompt: (i) Instructs GPT-4Vision to search
for instances of the support set classes within the query image. (ii) Guides
GPT-4Vision to provide a binary classification decision on whether each class
is present or not in the query image. (iii) Directs GPT-4Vision to discern and
output bounding box coordinates for objects, serving as input for the
segmentation task if a class is present. The questing prompt could utilize
cues on how to perform visual search within the image, alongside the cognitive
insights generated earlier, to improve GPT-4Vision’s accuracy in object
localization and identification.
Judgement. Following the output generation by the vision model, the judgement
prompt, as shown in Fig. 8, involves self-assessment by GPT-4Vision on the
quality of the generated segmentation masks. The judgement prompt encourages
GPT-4Vision to: (i) Critically review the output masks with respect to the
input query image and cognitive output. (ii) Assess the segmentation quality
using pre-defined criteria such as conformity to the object shape, coverage of
the object area, and confidence of the classified segments. (iii) Offer binary
approval of the segmentation quality and provide refinement suggestions if the
quality does not meet the criteria. This prompt would be used to trigger a
refinement loop, if necessary, based on the self-review, thus embodying a sort
of internal quality control to ensure the reliability of the FS-CS system.

| 1-way 1-shot
---|---
| classification 0/1 exact ratio (%) | segmentation mIoU (%)
Method | $5^{0}$ | $5^{1}$ | $5^{2}$ | $5^{3}$ | avg. | $5^{0}$ | $5^{1}$ | $5^{2}$ | $5^{3}$ | avg.
HSNet | 84.5 | 84.8 | 60.8 | 85.3 | 78.9 | 20.0 | 23.5 | 16.2 | 16.6 | 19.1
ASNet | 80.2 | 84.0 | 66.2 | 82.7 | 78.3 | 11.7 | 21.1 | 13.4 | 16.2 | 15.6
DINO | - | - | - | - | - | 20.0 | 23.4 | 16.2 | 16.6 | 19.1
CST | 84.0 | 82.2 | 70.8 | 82.6 | 79.9 | 35.8 | 38.9 | 28.9 | 29.2 | 33.2
Ours | 93.5 | 80.3 | 84.4 | 87.3 | 86.4 | 37.3 | 45.5 | 34.2 | 35.6 | 38.2
Table 1: Comparing model performance on FS-CS of th 1-way 1-shot setting.

###  5.3 Results

For classification, we report the average classification accuracy across all
classes and folds. For segmentation, Intersection over Union (IoU) is employed
as the key metric. We calculate the mean IoU across all classes and query
images to obtain the overall performance evaluation. The result is shown in
Tab. 1. Our model achieved remarkable results, surpassing all baselines by a
significant margin on Pascal-5i dataset. In one-shot classification, our model
achieved an average accuracy of 86.4%, outperforming the best baseline by
6.5%. In the segmentation task, our model demonstrated superior performance
with an average IoU of 38.2% in the one-shot scenario, surpassing the
strongest baseline by 5.0%.
Detailed Example Analysis. In this subsection, we zoom in on a specific one-
shot instance from the Pascal-5i dataset to elucidate how our model functions
in a practical setting. We selected an example, as shown in Fig. 9, that is
challenging due to the presence of multiple object instances within the scene.
The chosen query image depicts an scene with a cat and a dog. The support
image provided was that of a dog. This example tests the model’s capacity to
recognize and segment the dog instances while discriminating against other
objects. Here is how our model processed this example:

  1. 1.

Cognitive Observation. Upon reviewing the support image, GPT-4Vision provided
a detailed description, recognizing the target object (a dog) in the bounding
box and covered by the light blue mask. This cognitive output includes a
description of the size, shape, spatial relationships between different parts.

  2. 2.

Questing and Localization. Equipped with the cognitive insights, GPT-4Vision
commenced the questing task on the query image. The model successfully
identified the dog from the cat and background. It determined the presence of
the dog class, issued positive classification decisions, and provided bounding
box coordinates at the first time.

  3. 3.

Segmentation. The SAM acted upon the bounding box coordinates provided by
GPT-4Vision and yielded segmentation masks that delineated the dog from the
rest of the image content. However, due to the bounding box size is too large,
a part of cat is also segmented out.

  4. 4.

Quality Judgement. Following segmentation, GPT-4Vision engaged in self-
reflection and quality judgement. On evaluating the segmentation masks against
the established qualitative metrics, it noted some regions of imprecision.
Utilizing the judgement prompt, GPT-4Vision offered a BAD assessment result
and constructive feedback for mask refinement.

  5. 5.

Refinement. GPT-4 continue to invoke questing task and provide the feedback to
GPT-4Vision. GPT-4Vision refined the bounding box based on the feedback.
Finally, the new mask passed the by the quality judgement and the task
stopped.

Through this detailed instance analysis, we highlighted the model’s integrated
intelligence - its capability to undertake complex visual reasoning,
classification, and segmentation without the need for any additional fine-
tuning. Additionally, the model displayed a robust understanding of object
features and contexts, facilitated effective communication between language
and vision components, and upheld high-quality standards through self-
assessment and refinement loops. Overall, this case serves as a testament to
the dynamism and adaptability of our framework, confirming its potential to
deliver state-of-the-art performance on few-shot classification and
segmentation tasks.

##  6 Conclusion

In conclusion, our framework demonstrates a novel and effective approach to
few-shot classification and segmentation by bridging the high-level reasoning
capabilities of Large Language Models with the precise image interpretation of
vision models. It achieves state-of-the-art performance without the reliance
on extensive dataset-specific training, highlighting the power of chain-of-
thought prompting, in-context learning, and self-reflection for rapid task
adaptation and quality execution. It shows a potential that language models
act as meta-learners, capable of directing and refining visual processing
tasks.

## References

  * Chang and Chen [2018] Jia-Ren Chang and Yong-Sheng Chen.  Pyramid stereo matching network.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Chen et al. [2023] Aochuan Chen, Yuguang Yao, Pin-Yu Chen, Yihua Zhang, and Sijia Liu.  Understanding and improving visual prompting: A label-mapping perspective.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 19133–19143, 2023.
  * Chen et al. [2021] Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang.  Meta-baseline: Exploring simple metalearning for few-shot learning.  _In Proc. IEEE International Conference on Computer Vision (ICCV)_ , 2021.
  * Dhillon et al. [2019] Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto.  A baseline for few-shot image classification.  _In International Conference on Learning Representations_ , 2019.
  * Dong and Xing [2018] Nanqing Dong and Eric P Xing.  Few-shot semantic segmentation with prototype learning.  In _BMVC_ , 2018.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Finn et al. [2017a] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  _In Proc. International Conference on Machine Learning (ICML)_ , 2017a.
  * Finn et al. [2017b] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International conference on machine learning_ , pages 1126–1135. PMLR, 2017b.
  * Ganesh et al. [2021] Prakhar Ganesh, Yao Chen, Xin Lou, Mohammad Ali Khan, Yin Yang, Hassan Sajjad, Preslav Nakov, Deming Chen, and Marianne Winslett.  Compressing large-scale transformer-based models: A case study on bert.  _Transactions of the Association for Computational Linguistics_ , 9:1061–1080, 2021.
  * Gillioz et al. [2020] Anthony Gillioz, Jacky Casas, Elena Mugellini, and Omar Abou Khaled.  Overview of the transformer-based models for nlp tasks.  In _2020 15th Conference on Computer Science and Information Systems (FedCSIS)_ , pages 179–183, 2020.
  * Hospedales et al. [2022] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey.  Meta-learning in neural networks: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(9):5149–5169, 2022.
  * Jia et al. [2022] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In _European Conference on Computer Vision_ , pages 709–727. Springer, 2022.
  * Kang and Cho [2022] Dahyun Kang and Minsu Cho.  Integrative few-shot learning for classification and segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 9979–9990, 2022.
  * [14] Dahyun Kang, Piotr Koniusz, Minsu Cho, and Naila Murray.  Distilling self-supervised vision transformers for weakly-supervised few-shot classification & segmentation–supplementary material–.  _image_ , 79:6–9.
  * Kang et al. [2021] Dahyun Kang, Heeseung Kwon, Juhong Min, and Minsu Cho.  Relational embedding for few-shot classification.  _IEEE International Conference on Computer Vision (ICCV)_ , 2021.
  * Khan et al. [2022] Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah.  Transformers in vision: A survey.  _ACM Computing Surveys._ , 54(10s):1–41, 2022.
  * Khodak et al. [2019] Mikhail Khodak, Maria-Florina F Balcan, and Ameet S Talwalkar.  Adaptive gradient-based meta-learning methods.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2019.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Liu et al. [2020] Bin Liu, Yue Cao, Yutong Lin, Qi Li, Zheng Zhang, Mingsheng Long, and Han Hu.  Negative margin matters: Understanding margin in few-shot classification.  In _Computer Vision – ECCV 2020_ , pages 438–455, Cham, 2020. Springer International Publishing.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10012–10022, 2021.
  * Nichol et al. [2018] Alex Nichol, Joshua Achiam, and John Schulman.  On first-order meta-learning algorithms.  _arXiv preprint arXiv:1803.02999_ , 2018.
  * Oreshkin et al. [2018] Boris N Oreshkin, Pau Rodriguez, and Alexandre Lacoste.  TADAM: Task dependent adaptive metric for improved few-shot learning.  _Conference on Neural Information Processing Systems (NeurIPS 2018_ , 2018.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Rakelly et al. [2018] Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alexei A. Efros, and Sergey Levine.  Conditional networks for few-shot semantic segmentation.  In _International Conference on Learning Representations_ , 2018.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Shaban et al. [2017] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots.  One-shot learning for semantic segmentation.  _arXiv preprint arXiv:1709.03410_ , 2017.
  * Snell et al. [2017a] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  _In Advances in Neural Information Processing Systems (NeurIPS)_ , 2017a.
  * Snell et al. [2017b] Jake Snell, Kevin Swersky, and Richard Zemel.  Prototypical networks for few-shot learning.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2017b.
  * Sun et al. [2019] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele.  Meta-transfer learning for few-shot learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Sung et al. [2018] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip H.S. Torr, and Timothy M. Hospedales.  Learning to compare: Relation network for few-shot learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2018.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang [2021] Jane X Wang.  Meta-learning in natural and artificial intelligence.  _Current Opinion in Behavioral Sciences_ , 38:90–95, 2021.
  * Wang et al. [2019a] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng.  Panet: Few-shot image semantic segmentation with prototype alignment.  _In Proc. IEEE International Conference on Computer Vision (ICCV)_ , 2019a.
  * Wang et al. [2019b] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng.  Panet: Few-shot image semantic segmentation with prototype alignment.  In _proceedings of the IEEE/CVF international conference on computer vision_ , pages 9197–9206, 2019b.
  * Wang et al. [2021] Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni.  Generalizing from a few examples: A survey on few-shot learning.  _Association for Computing Machinery (ACM)_ , 53(3), 2021.
  * Weiss et al. [2016] Karl Weiss, Taghi M Khoshgoftaar, and DingDing Wang.  A survey of transfer learning.  _Journal of Big data_ , 3(1):1–40, 2016.
  * Wertheimer et al. [2021] Davis Wertheimer, Luming Tang, and Bharath Hariharan.  Few-shot classification with feature map reconstruction networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 8012–8021, 2021.
  * Wu et al. [2022] Junyang Wu, Xianhang Li, Chen Wei, Huiyu Wang, Alan Yuille, Yuyin Zhou, and Cihang Xie.  Unleashing the power of visual prompting at the pixel level.  _arXiv preprint arXiv:2212.10556_ , 2022.
  * Zhang et al. [2023] Yimeng Zhang, Xin Chen, Jinghan Jia, Sijia Liu, and Ke Ding.  Text-visual prompting for efficient 2d temporal video grounding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14794–14804, 2023.
  * Zhang and Peng [2019] Zhipeng Zhang and Houwen Peng.  Deeper and wider siamese networks for real-time visual tracking.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Zhu et al. [2023] Zhuangdi Zhu, Kaixiang Lin, Anil K. Jain, and Jiayu Zhou.  Transfer learning in deep reinforcement learning: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(11):13344–13362, 2023.
  * Zhuang et al. [2020] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He.  A comprehensive survey on transfer learning.  _Proceedings of the IEEE_ , 109(1):43–76, 2020.


# One Stack to Rule them All:
To Drive Automated Vehicles, and Reach for the 4th Level

Sven Ochs1, Jens Doll1, Daniel Grimm1, Tobias Fleck1, Marc Heinrich1, Stefan
Orf1,
Albert Schotschneider1, Helen Gremmelmaier1, Rupert Polley1, Svetlana
Pavlitska1, Maximilian Zipfl1,
Helen Schneider2, Ferdinand Mütsch2, Daniel Bogdoll1, Florian Kuhnt1,
Philip Schörner1, Marc René Zofka1 and J. Marius Zöllner1,2 1 Department of
Technical Cognitive Systems, FZI Research Center for Information Technology,
Germany<EMAIL_ADDRESS>Karlsruhe Institute of Technology (KIT), Germany.

###### Abstract

Most automated driving functions are designed for a specific task or vehicle.
Most often, the underlying architecture is fixed to specific algorithms to
increase performance. Therefore, it is not possible to deploy new modules and
algorithms easily. In this paper, we present our automated driving stack which
combines both scalability and adaptability. Due to the modular design, our
stack allows for a fast integration and testing of novel and state-of-the-art
research approaches. Furthermore, it is flexible to be used for our different
testing vehicles, including modified EasyMile EZ10 shuttles and different
passenger cars. These vehicles differ in multiple ways, e.g. sensor setups,
control systems, maximum speed, or steering angle limitations. Finally, our
stack is deployed in real world environments, including passenger transport in
urban areas. Our stack includes all components needed for operating an
autonomous vehicle, including localization, perception, planning, controller,
and additional safety modules. Our stack is developed, tested, and evaluated
in real world traffic in multiple test sites, including the Test Area
Autonomous Driving Baden-Württemberg.

##  I INTRODUCTION

Automated driving introduces high requirements to the underlying system
architecture. Given the non-linear and often unpredictable nature of research,
it is critical that such an architecture is both adaptable and modular,
allowing it to accommodate a wide range of potential requirements and
developments. Such flexibility is essential in facilitating the development
and deployment of state-of-the-art algorithms. In addition to the
aforementioned demands, automated driving functions must also strive for
reproducible results that are widely generalizable and can be effectively
applied to a variety of vehicles and sensor setups.

The most prominent open source systems are Autoware [1, 2, 3] and Apollo [4].
Autoware was the first framework that included components for perception,
prediction, planning, and control. It is based on ROS 2 and was demonstrated
in a wide variety of scenarios [5]. Autoware is subdivided into Core modules,
managed by a foundation, and Universe modules, which come on top and are
community driven. Multiple projects are based on Autoware [6, 7]. Apollo has a
similar scope as Autoware. It is based on its own middleware, called CyberRT.
Some functionalities are only provided by closed-source cloud services. Apollo
comes with over 25 pre-trained neural networks (NN) for traffic light and lane
perception, obstacle detection, and trajectory prediction. The underlying
training data are not necessarily provided, which makes it hard to reuse the
models for other vehicle setups [8, 9]. Compared to Autoware, Apollo appears
to be more robust but suffers from a smaller ecosystem [10]. Further open
source projects, such as Pylot [11] or AVStack [12], did not gain much
attraction so far.

Figure 1:  The TCS-AD stack provides all the capabilities that are needed for
driving on public roads. This ranges from localization, perception, and
planning to controller. The performance of the software stack has been
demonstrated on various types of vehicles and across multiple test
environments. III-B which do not have a steering wheel.

A multitude of closed source driving stacks have been developed in the past.
Taş et al. [13] presented an overview of platforms developed between the early
days of autonomous driving up until 2015. In 2016, the ROS-based BerthaOne
platform was presented [14]. While the focus was on V2X enabled cooperative
driving behaviors within the Grand Cooperative Driving Challenge, the stack is
also able to handle urban road traffic. As part of the SAE AutoDrive
Challenge, the Zeus driving stack was introduced in 2021 [15]. They
demonstrated a variety of defined traffic scenarios, with all algorithms
running on CPUs as part of challenge restrictions. The TiEV stack competed in
the Intelligent Vehicle Future Challenge 2017, which includes driving on
express roads and urban roads [16]. Similar to Apollo, they have developed
their software stack from scratch. Marin-Plaza et al. presented a ROS-based
stack, which was deployed on two different vehicle types for primarily grid-
based off-road environments [17]. Their stack relies extensively on off-the-
shelf ROS packages and uses a simple state machine for decision-making.

From the presented driving stacks, patterns emerge. Often, the
interchangeability of modules is not considered, which directly hinders
collaborative projects. Most industrial stacks are designed with a single
vehicle in mind, which introduces scalability issues. Additionally, particular
competition requirements are often times at the heart of architectural
decisions, which hinders deployment in real-world scenarios. Finally, maturity
is also key, enabling the usage by nonexpert people, which expands the testing
possibilities in real-world traffic.

During the development of the software stack, we placed a particular emphasis
on well-defined interfaces. These interfaces have been incorporated,
particularly in publicly-funded projects, to ensure smooth collaboration with
partners. The ability to replace components within the software stack and
observe their impact on different vehicles and real-world driving conditions
has enhanced our research capabilities. By swapping out individual components,
we can isolate and identify the effect of each component on the overall
driving performance. This interchangeability has enabled us to conduct
controlled experiments to determine the optimal configuration of the software
stack in various driving scenarios and to easily include and evaluate newly
developed components. Furthermore, we present a baseline implementation for
each component, which constitutes a complete stack and reaches a level of
maturity such that expert knowledge is not required to operate the test
vehicles.

In Sec. II, we, the group for Technical Cognitive Systems (TCS) at the
Research Center for Information Technology (FZI) in Karlsruhe, Germany present
our TCS-AD stack and its architecture. In Sec. III we present our research
vehicles, followed by demonstrations in Sec. IV. We conclude our work in Sec.
V, where we present open research questions.

##  II ARCHITECTURE

In this section, the architecture of our automated driving (AD) function and
the baseline implementation of each component is presented. As visualized in
Fig. 2, the architecture is decomposed into different modules, such as
localization, perception and planning. Since we focus on interchangeability
and modularity for easy testing of different components, the interfaces and
the entire data flow are fixed, while the corresponding components can be
easily exchanged.

The localization module is responsible to provide global position and current
velocity information to all other modules where needed, e.g. the perception
module and the planning modules. The information is provided in a broadcast
fashion as a rigid body transform between the world frame and the vehicle
reference frame.

The main design decision in the perception module is the separation of dynamic
and static obstacles. Due to the modular system design, different detection
and tracking modules have to be used, depending on the sensor setup of the
vehicle platform. The different perception algorithms therefore have to
deliver a 2D occupancy grid and a list of predicted obstacles. An obstacle is
defined with a XYZ position, extends, classification and the predicted
positions.

The motion pipeline provides a trajectory in the form of Curvepoints.
Curvepoints are spatially equidistant sampled trajectory points in Cartesian
world coordinates. Additionally, the single points contain a heading angle, a
curvature value and a scalar velocity and acceleration. On the top level, the
trajectory also contains a timestamp, a strictly monotonically increasing
generation number, the number of valid points in the array for support of
fixed-size array interfaces and the reference point to convert the Cartesian
coordinates into spherical coordinates. Additionally, a car-index adds the
possibility to treat the part till the car index either as a prefix or
history. This prefix can be used to form higher-order derivatives, which can
support the controllers.

All components are reporting their current diagnostic status. The status is
one of the following: OK, WARN, ERROR and STALE. Additional information can be
provided over the message field. This information is processed by the
diagnostics module, which is described in Sec. II-H.

Figure 2: Architecture overview of the TCS-AD function.

###  II-A High Definition Maps

High Definition (HD)-Maps are a crucial part of most environment
representations for AD functions. The presented driving stack utilizes the
Lanelet format [18, 19]. This representation of the static environment
considers geometric as well as topological and semantic aspects. The key
aspects of the Lanelet format are the precise geometric descriptions in
geodetic coordinates. These are based on primitives like points, ways, areas
and relations. This allows the definition and annotation of semantic
information like traffic rules, the definition of relations between lanes and
the annotation with arbitrary attributes.

The map is used by the route planner to determine the optimal route to the
destination. Furthermore, it provides the boundaries of the area, where the
vehicle is allowed to drive. This is not necessarily only the actual lane, but
can also cover the adjacent lane, if the traffic rules permit a transition.
This is the so-called driving area. It is derived in the form of 2D polygons
from the course of the lanes and the related traffic rules. The left and right
side edges limit the driving area in which trajectories may potentially be
calculated. This generalization for the trajectory planning allows driving in
right-hand as well as left-hand traffic.

###  II-B Localization

In order to use the information from the HD-map, a precise global localization
of the vehicle is needed. The two localization approaches utilized by our
software are _Simultaneous Localization and Mapping (SLAM)_ and the _Global
Navigation Satellite Systems with real-time kinematic (GNSS-RTK)_. Our SLAM
algorithm is based on Google cartographer [20] with upcoming additions of
utilization of semantic information. Since its publication in 2016,
Cartographer has been improved over the last years and the community continues
to actively improve and expand the method. Cartographer utilizes a two-stages
model: Local scan matching and a global optimization. The global optimizaton
searches for the best fit for the local scan matching result

The selection of the actual localization is determined based on the target
scenario. For narrow streets or dense environments as encountered in Karlsruhe
Weiherfeld-Dammerstock, see upcoming Sec. IV, the SLAM approach is used, but
for areas with less dense scenarios GNSS-RTK is utilized.

###  II-C Perception

In the following section, we present our environment model used in our driving
stack. In CoCar, see Sec. III-A, a commercial LiDAR tracking system consisting
out of five IbeoLux LiDAR scanners is available that provides tracked,
classified bounding boxes and contour tracks for the most relevant object
classes. In our FZI-Shuttles, see Sec. III-B, a multi LiDAR sensor setup is
used with a custom detection and tracking pipeline that consists of the
following components: preprocessing, including ground classification (per
sensor), pointcloud segmentation (per sensor) and cluster transform, following
cluster merging, feature computation, feature tracking and static object
removal.

####  II-C1 Preprocessing

The preprocessing steps for the pointcloud data is utilized by static and
dynamic object detection. _Ground classification_ decides per point if the
point results from a ground reflection or not. For this, an iterative ground
plane estimation technique highly inspired by the paradigm in [21] is used.
_Pointcloud segmentation_ clusters points using a range-image based approach
inspired by [22]. As the approach is range-image based and operates on
spherical point coordinates instead of Cartesian XYZ coordinates, a clustering
per sensor is computed independently. _Cluster transform_ performs a
coordinate transformation from sensor coordinate systems to the odometry
coordinate frame, which is given by the localization system and the local ego
motion estimation.

####  II-C2 Static Obstacles

The static environment is represented in a 2D occupancy grid. Therefore, an
extended version of the base implementation of [23], a so called cost-map, is
used. The enhancement, for example, comprises the clearing of unknown regions
behind obstacles. The costmap is built from the ground segmented and
transformed pointcloud.

####  II-C3 Detection & Tracking of Dynamic Objects

The goal of the Detection & Tracking components is to provide the locations,
extents, classification and kinematic states of relevant objects (pedestrians,
bicycles, cars, trucks, etc.) in the surroundings of the vehicle for the
prediction, maneuver and motion planning components. Additional to the
preprocessing, the following steps are conducted:

_Cluster merging_ merges overlapping clusters that relate to point clusters
originating from the same objects based on a point cluster distance metric.
_Feature computation_ selects features of the object that are used to track
the object and estimate the object’s kinematics over time. This includes
bounding box fitting and L-shape computation based on the closeness-metric
based algorithm in [24]. _Feature tracking_ associates the computed shape
features over time, estimates the vehicle’s kinematics using a labeled
Gaussian Mixture Probability Hypothesis Density Filter (GMPHD) [25, 26]
_Static object removal_ removes all objects from the resulting object list
based on a threshold on the estimated velocity of the objects as static
objects are already considered in the costmap.

####  II-C4 Prediction

The prediction of the environment is done in a hierarchical order based on
[27]. First, the intentions of the traffic participants are inferred. The
route influences the long term motion of a traffic participant the most. To
obtain estimates about their route, the road users are continuously associated
with their corresponding lane using a heuristic consisting of their position
and orientation. The likelihood of the routes is then weighted accordingly to
the confidence of the lane matching. The possible routes are determined
considering turn-options up to a certain look-ahead. Bicycles are assumed to
follow along the side of the road whereas vehicles tend to stay in the center.
The longitudinal motion is determined using the Intelligent Driver Model [28].
The lateral controls are calculated using a Stanley Controller. These control
inputs are then used in an Extended Kalman Filter to predict the next state of
the traffic participant.

Traffic participants that do not follow a certain route are predicted using
constant velocity or constant acceleration assumptions. This applies for
example to pedestrians or objects of unknown classification.

An orthogonal approach is used by a graph neural network (GNN), see [29].
There, the environment is modelled as a heterogeneous graph which consists of
the two node types, namely an agent node and map node. An agent-node
represents a state at a certain time of a traffic participant, e.g., position
and velocity. A map-node represents a segment of a centerline of the HD map.
Context between different nodes is modelled with edges of a certain type.
Edges are also used to model the historic context of a tracked traffic
participants, social interaction with others and the connection to HD-map. The
model is able to predict multiple scored trajectories for all traffic
participants at the same time.

###  II-D Mission

To guarantee the acceptance of only compliant missions into the motion
pipeline, a mission control is employed. The mission control checks the
incoming requests to avoid situations that are infeasible or could cause the
motion pipeline to fail. Most importantly, it guarantees a continuity in local
routing. To maximize flexibility, the mission control supports both micro-
missions with waypoints and the possibility of appending goals to running
missions on the fly. To the outside, the mission control provides an
interface, which can then be used with various mission providers, such as a
backend of a booking system. This has been shown with a proprietary MQTT-based
backend connection as well as the booking backend system. A behavior tree is
used to coordinate all preparation steps and the invocation of the motion
pipeline. Additional features that are required for specific use cases can be
added to the mission as needed, such as the starting and stopping of a trip
recorder that persists relevant sensor data during the autonomous driving
mode.

###  II-E Maneuver

To safely and effectively navigate in complex situations involving dynamic
obstacles, we decompose the situation into simpler decision-making units,
which we denote as Traits. For example, a trait may involve a decision to make
the driving area narrower due to oncoming traffic. Another trait wants to
extend the driving area due to a parked vehicle on our lane. To find a
solution for these contradicting statements, a resolver is implemented. The
visualization in Fig. 3 shows the algorithm used for resolving the problem.
Each trait is represented by a maximum and minimum value, represented as an
interval. The interval is also associated with a weight, which is determined
trough expert knowledge. The first step of the resolver intersects pairwise
all intervals and creates thereby smaller disjunct intervals. We then sum up
all weights in the disjunct interval. The final consent is found by linear
search in the disjunct intervals for the sign change. The change of sign
represents the equilibrium. The result of this example is the extension of the
driving area by $2\text{\,}\mathrm{m}$.

Figure 3: Exemplary visualization for the resolving process for computing the
maneuver. In the first step, all intervals are intersected, and the weights
are added. The second step includes the search for the equilibrium of the
forces. The green interval sets a limit on the maximum extension of the
driving area, while the remaining weights vote for an extension, represented
in the blue intervals. The consent is found by searching for the change of
sign.

###  II-F Planning

The goal of path planning is to create a safe, collision free trajectory that
maximizes passenger comfort. To achieve this goal, we rely on Particle-Swarm-
Optimization (PSO). PSO has been shown to be a superior optimization technique
compared to other planning algorithms, such as tree-based planning algorithms
[30] that require post-processing smoothing. Additionally, PSO offers
advantages over Sequential Quadratic Programming (SQP) [31], which requires a
continuously differentiable and convex problem formulation for effective
optimization.

PSO is able to handle arbitrary optimization topologies without any gradient
at all. Consequently, this enables the planning algorithm to be used with
arbitrary constraints and cost terms, further enhancing the flexibility of the
planning process. The trajectory produced by the Particle Swarm Optimization
(PSO) algorithm can be described as an equitemporal sequence of poses. The PSO
generates these poses by utilizing two constraint concepts: internal and
external. Internal constraints refer to the properties that govern the
transition between two different poses, such as velocity and curvature. These
internal values provide the foundation upon which external constraints are
derived. However, due to the inherent complexity of these formulations, they
may not always be invertible, which is resolved through the use of discrete
differentiation. With these constraints, intelligent sampling strategies are
implemented to make the sampling of valid particles feasible. The initial
solution is provided by a tree-based pre-planner to find valid paths. The
flexibility of this planning architecture has been shown in various
applications, such as curvature-rate-limited vehicles as well as multi-axis
steering platforms.

###  II-G Execution

####  II-G1 Guards

Since the nature of the planning process is an optimization problem, there is
the possibility that the trajectory violates certain constraints. These
constraints are continuously checked by our different guards, which are based
on the same architectural structure. As input, there is the Curvepoints from
the planning process and the current environment from the latest available
timestamp.By default, the guards are in the triggered state. The triggered
state is initially entered and when a violation of the specific constraints is
detected. During this time, the desired velocity of the incoming Curvepoints
are replaced by a braking profile. When the violation is no longer detected,
the guard transitions into a state, that applies an acceleration profile from
the current velocity of the vehicle to the desired velocity of the planner.

####  II-G2 Controller

The trajectory is then executed by a trajectory controller. Given the global
position estimate by the localization, the controller projects the vehicle
onto the trajectory. Depending on the vehicle architecture, the vehicle
reference point is either the center of the rear axle, for default Ackermann
vehicles, or the center between both axles, for dual steering vehicles. From
that projection, it calculates the lateral error $e_{lat}$ and angular error
$e_{phi}$. Moreover, the curvature, speed and acceleration values are
interpolated using temporal look-aheads to compensate for the delay in the
actuators. The curvature $\kappa$ is used for a feed-forward control using
inverse kinematics of a simple bicycle model with the wheelbase $l$. For
vehicles with steering rate limits, such as the FZI-Shuttles, see Sec. III-B,
the controller also monitors the steering velocity. To avoid exceeding the
limits, the controller proactively reduces the velocity for sections with high
steering velocities, taking the deceleration limits into account.

###  II-H Diagnostics

Figure 4: User interface of the diagnostic module. It enables fast debugging
of the current state of the AD function. This is especially useful for the
safety operator, which do not have deep insight into the driving function.

In complex systems, such as autonomous vehicles, the ability to identify and
track failures during operation is of utmost importance. To accomplish this,
various aspects of the system are monitored, typically through observation of
the components’ behavior or the use of self-states. These monitoring methods
provide insights into the inner workings of the system and allow operators to
pinpoint any anomalies that may arise. One way to observe the system is to
measure the absence of data, as well as their frequency and delay. The data
and corresponding meta-data are then analyzed to detect any abnormal behavior.
The detection can be data based or model based driven. The used models can be
constructed using either manual techniques or machine learning approaches. For
example, the localization component is diagnosed by statistically modelling
its behavior based on odometry [32].

It is inevitable that failures occur during operation for example due to
hardware degradation. In such cases, it is crucial to provide timely and
accurate information to operators about the nature of the anomaly, using
abstracted error descriptions that allow for appropriate countermeasures to be
taken. The TCS-AD stack is suitable to both applications by recognizing and
addressing failures on a technical level, and then aggregating the data to
provide a summary that is abstracted and easier to comprehend for operators.

###  II-I Vehicle2X

Vehicle2X (V2X) communication enables real-time data sharing, cooperative
driving and promises to increase safety. MAP Data Messages (MAP) and Signal
Phase and Timing Messages (SPaT) are used to receive intersection and traffic
light information. This information is used in combination with HD maps for
anticipatory driving and braking. Sensed obstacles and traffic participants
are broadcast between the AD stack and smart infrastructure both ways via
Collective Perception Messages (CPM) to enrich the environment model of both
systems. Automated vehicles can utilize these as extensions of the vehicle’s
sensors for safer driving, especially in intersections with limited
visibility. These use-cases are already presented with the support of the Test
Area Autonomous Driving Baden-Württemberg (TAF-BW). V2X also supports to
achieve automation levels that do not require safety drivers through online
monitoring and controlling of the vehicle from remote. I.e. the national legal
regulation in Germany [33] requires a technical supervision which can also be
remote.

##  III TESTING VEHICLES

((a)) CoCar is based on an Audi Q5 and was build up 2011.

((b)) FZI-Shuttles are based on EasyMile-EZ10 Gen2. and used for on-demand
passenger transport since 2019.

((c)) Simulator CARLA with CoCar and pedestrians on an intersection in
Karlsruhe.

Figure 5: All test vehicles of the FZI are presented. They were all equipped
at the time of construction with state-of-the-art LiDAR system and D-GPS
sensors.

The TCS-AD Stack was deployed in several public funded projects. Especially on
our own FZI vehicles, the stack is constantly tested and improved. In the
following, we want to give a brief overview over the vehicles.

###  III-A CoCar

Our testing vehicle CoCar (Cognitive Car) is based on an Audi Q5 and is
equipped with LiDAR, GPS and Camera systems. The sensor setup consists of a
high-precision GNSS/INS provided by Oxford Technical Solutions (OxTS RT3000),
five Ibeo Lux sensors with 4 Layers to provide a 360 degree surround view. The
road approval of CoCar allows us to test our autonomous driving function in
all areas in Germany.

###  III-B FZI-Shuttles

The FZI-Shuttles ELLA and ANNA are based on EasyMile EZ10 Gen2 platforms. The
shuttles are able to steer with both axes and therefore have a high lateral
mobility. Since the driving stack is not bound to virtual rails but to move
freely in everyday traffic, the platforms were modified to fit our needs. This
includes the mounting of six LiDAR sensors.

###  III-C Simulation

In addition to real road testing, we use simulations to develop and validate
our driving stack and for testing individual components. The simulation data
such as environmental information, sensor data and positional data of other
road users is used as input for the AD-Stack and processed as on the real
vehicle to plan the trajectory and control the vehicle. For validation of
interactions with other road users, we use different methods. For instance, we
use SUMO [34] in co-simulation with CARLA [35] to simulate high traffic
volume. We also define scenarios in the ASAM OpenSCENARIO format[36] to
achieve very specific behavior of other road users. Due to the repetitive
behavior, we are able to analyze, fine-tune and evaluate our AD stack.
Additionally, using digital twins of out testing areas and vehicles, as shown
in Fig. 5(c), we are able to early integrate and test the AD stack before
going into the site itself. This also enables endurance tests to be carried
out. Road users such as pedestrians can be immersed in the simulation using
virtual reality. In this way, it is possible to familiarize people with the
AD-Stack and to research interactions in a very realistic way.

##  IV DEMONSTRATIONS AND APPLICATIONS

((a)) The KIT Campus East is a restricted area with about
$2\text{\,}\mathrm{km}$ of road and $5000\text{\,}{\mathrm{m}}^{2}$ free
space.

((b)) Weiherfeld-Dammerstock with a street network of about
$6.4\text{\,}\mathrm{km}$

Figure 6: Both main test sites, KIT Campus East and Weiherfeld-Dammerstock
(WD), are presented. The KIT Campus East is used for the integration of
automated driving functions and initial tests. Afterward, the tests are
conducted in WD, where we are interacting with mixed traffic-flows and VRUs.
Background for images provided by [37].

The automated driving stack is mainly evaluated in and around Karlsruhe,
Germany in urban and suburban areas. Two selected locations are presented in
the following. The first one is a restricted area that is part of the
Karlsruhe Institute of Technology (KIT), the so-called Campus East. It is part
of the TAF-BW, dedicated to early rapid prototyping vehicle tests. The 2nd
test site is Weiherfeld-Dammerstock, an urban area with various types of
public roads. The challenges faced there are narrow streets, parked vehicles
along the road, intersections with poor visibility, and interaction with VRUs.
We autonomously drove over $3000\text{\,}\mathrm{km}$ with our testing
vehicles in these areas. Also, the FZI-Shuttles were employed in regular
passenger transport, operated by safety drivers from the local public service
provider. This required a very high quality and safety standard, as these
safety operators do not have the same insights as developers have and
passenger transport is only possible when achieving high comfort and safety
standards. Also, the stack and its modules are used in multiple projects with
different partners. Because of this reason, the stack also shows its
adaptability and flexibility. It follows a small excerpt of the projects:
RobustSense [38], SmartLoad [39], EVA-Shuttle [40], SafeADArchitect [41], VVM
[42], SmartEPark [43], Shuttle2X [44], ÖV-Leitmotiv-KI [45], KIsSME [46],
KIGLIS [47].

##  V CONCLUSION

We presented a modular architecture for automated driving in this paper, which
can be adapted to various vehicles and simulation environments. With this
modular design, we can deploy new components on the vehicles easily and
evaluate them in restricted and public areas with mixed traffic. Our test
vehicles drove autonomously for over $3000\text{\,}\mathrm{km}$ in these
environments. We are able to incorporate machine learning-based algorithms in
various parts of the software stack, because our model based trajectory
planner outputs only drive-able and compliant trajectories. Improvements in
our algorithms will also enable us to advance in more complex traffic
scenarios. We aim to develop algorithms that can eliminate the need for safety
operators at all, or at least from the inside of the vehicle. This will
involve designing and testing advanced autonomous systems that can ensure safe
and reliable operations without human intervention.

## References

  * [1] Autoware Foundation  “Autoware” Accessed: 2022-12-19, https://github.com/autowarefoundation/autoware, 2022
  * [2] Shinpei Kato et al.  “An Open Approach to Autonomous Vehicles”  In _Micro_ 35, 2015
  * [3] Shinpei Kato et al.  “Autoware on Board: Enabling Autonomous Vehicles with Embedded Systems”  In _International Conference on Cyber-Physical Systems (ICCPS)_ , 2018
  * [4] Apollo Auto  “Apollo” Accessed: 2022-12-19, https://github.com/ApolloAuto/apollo, 2022
  * [5] Zirui Zang et al.  “Winning the 3rd Japan Automotive AI Challenge - Autonomous Racing with the Autoware.Auto Open Source Software Stack”  In _Intelligent Vehicles Symposium (IV)_ , 2022
  * [6] Project Aslan  “ASLAN Autonomy” Accessed: 2023-03-15, https://www.project-aslan.org/, 2023
  * [7] Goodarz Mehr et al.  “X-CAR: An Experimental Vehicle Platform for Connected Autonomy Research Powered by CARMA”  In _Intelligent Transportation Systems Magazine_ , 2022
  * [8] Tobias Kessler et al.  “Bridging the Gap between Open Source Software and Vehicle Hardware for Autonomous Driving”  In _Intelligent Vehicles Symposium (IV)_ , 2019
  * [9] Tobias Kessler, Klemens Esterle and Alois Knoll  “Mixed-Integer Motion Planning on German Roads Within the Apollo Driving Stack”  In _Transactions on Intelligent Vehicles_ 8, 2023
  * [10] Vysyaraju Manikanta Raju, Vrinda Gupta and Shailesh Lomate  “Performance of Open Autonomous Vehicle Platforms: Autoware and Apollo”  In _International Conference for Convergence in Technology (I2CT)_ , 2019
  * [11] Ionel Gog et al.  “Pylot: A Modular Platform for Exploring Latency-Accuracy Tradeoffs in Autonomous Vehicles”  In _International Conference on Robotics and Automation (ICRA)_ , 2021
  * [12] R. Spencer Hallyburton, Shucheng Zhang and Miroslav Pajic  “AVstack: An Open-Source, Reconfigurable Platform for Autonomous Vehicle Development”  In _arXiv:2212.13857_ , 2022
  * [13] Ömer Şahin Taş et al.  “Functional system architectures towards fully automated driving”  In _Intelligent Vehicles Symposium (IV)_ , 2016
  * [14] Ömer Şahin Taş et al.  “Making Bertha Cooperate–Team AnnieWAY’s Entry to the 2016 Grand Cooperative Driving Challenge”  In _Transactions on Intelligent Transportation Systems_ 19, 2018
  * [15] Keenan Burnett et al.  “Zeus: A System Description of the Two-Time Winner of the Collegiate SAE Autodrive Competition”  In _Journal of Field Robotics_ 38, 2021
  * [16] Junqiao Zhao et al.  “TiEV: The Tongji Intelligent Electric Vehicle in the Intelligent Vehicle Future Challenge of China”  In _International Conference on Intelligent Transportation Systems (ITSC)_ , 2018
  * [17] Pablo Marin-Plaza et al.  “Complete ROS-based Architecture for Intelligent Vehicles”  In _Iberian Robotics Conference (ROBOT)_ , 2018
  * [18] Philipp Bender, Julius Ziegler and Christoph Stiller  “Lanelets: Efficient map representation for autonomous driving”  In _Intelligent Vehicles Symposium (IV)_ , 2014
  * [19] Fabian Poggenhans et al.  “Lanelet2: A high-definition map framework for the future of automated driving”  In _International Conference on Intelligent Transportation Systems (ITSC)_ , 2018
  * [20] Wolfgang Hess et al.  “Real-Time Loop Closure in 2D LIDAR SLAM”  In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2016
  * [21] Dimitris Zermas, Izzat Izzat and Nikolaos Papanikolopoulos  “Fast segmentation of 3d point clouds: A paradigm on lidar data for autonomous vehicle applications”  In _IEEE International Conference on Robotics and Automation (ICRA)_ , 2017
  * [22] Frederik Hasecke, Lukas Hahn and Anton Kummert  “Flic: Fast lidar image clustering”  In _arXiv:2003.00575_ , 2020
  * [23] ROS  “Costmap 2D” Accessed: 2023-04-20, https://github.com/ros-planning/navigation/tree/noetic-devel/costmap_2d
  * [24] Xiao Zhang et al.  “Efficient L-shape fitting for vehicle detection using laser scanners”  In _IEEE Intelligent Vehicles Symposium (IV)_ , 2017
  * [25] B-N Vo and W-K Ma  “The Gaussian mixture probability hypothesis density filter”  In _IEEE Transactions on signal processing_ 54, 2006
  * [26] Kusha Panta, Daniel E Clark and Ba-Ngu Vo  “Data association and track management for the Gaussian mixture probability hypothesis density filter”  In _IEEE Transactions on Aerospace and Electronic Systems_ 45, 2009
  * [27] Philip Schörner, Tobias Fleck and J Marius Zöllner  “What if? Behavior Estimation by Predicting Traffic Scenes from State Histories”  In _IEEE International Intelligent Transportation Systems Conference (ITSC)_ , 2021
  * [28] Arne Kesting, Martin Treiber and Dirk Helbing  “Enhanced intelligent driver model to access the impact of driving strategies on traffic capacity.”  In _Philosophical transactions. Series A, Mathematical, physical, and engineering sciences_ 368, 2010
  * [29] Daniel Grimm et al.  “Holistic Graph-based Motion Prediction”  In _arXiv:2301.13545_ , 2023
  * [30] Sören Kammel et al.  “Team AnnieWAY’s autonomous system for the 2007 DARPA Urban Challenge”  In _Journal of Field Robotics_ 25, 2008
  * [31] Julius Ziegler et al.  “Trajectory planning for Bertha—A local, continuous method”  In _IEEE intelligent vehicles symposium proceedings_ , 2014
  * [32] Stefan Orf et al.  “Modeling Localization Uncertainty for Enhanced Robustness of Automated Vehicles”  In _2022 IEEE 18th International Conference on Intelligent Computer Communication and Processing (ICCP)_ , 2022, pp. 175–182  DOI: 10.1109/ICCP56966.2022.10053961
  * [33] Bundesministerium für Digitales und Verkehr  “Gesetz zum autonomen Fahren tritt in Kraft” Accessed: 2022-10-06, https://www.bmvi.de/SharedDocs/DE/Artikel/DG/gesetz-zum-autonomen-fahren.html, 2021
  * [34] Pablo Alvarez Lopez et al.  “Microscopic Traffic Simulation using SUMO”  In _IEEE International Conference on Intelligent Transportation Systems (ITSC)_ , 2018
  * [35] Alexey Dosovitskiy et al.  “CARLA: An Open Urban Driving Simulator”  In _Conference on Robot Learning (CoRL)_ , 2017
  * [36] Association for Standardisation of Automation and MeasuringΩSystems (ASAM) e. V.  “ASAM SIM:GUIDE” https://www.asam.net/index.php?eID=dumpFile&t=f&f=4297&token=cbeec18c4770d0ff88e3d3c539c13a32abed6300, accessed: 2023-04-27, https://www.asam.net/index.php?eID=dumpFile&t=f&f=4297&token=cbeec18c4770d0ff88e3d3c539c13a32abed6300 URL: \url{https://www.asam.net/index.php?eID=dumpFile&t=f&f=4297&token=cbeec18c4770d0ff88e3d3c539c13a32abed6300}
  * [37] OpenStreetMap contributors  “Planet dump retrieved from https://planet.osm.org ”, https://www.openstreetmap.org, 2017
  * [38] European Center for Information and CommunicationΩTechnologies – EICT GmbH  “RobustSENSE-Homepage” Accessed: 2023-09-02, http://www.robustsense.eu/
  * [39] AVL DEUTSCHLAND GMBH  “Smartload-Homepage” Accessed: 2023-09-02, http://www.smartload-project.de/, 2021
  * [40] FZI Research Center for Information Technology  “EVA-Shuttle-Homepage” Accessed: 2023-09-02, http://www.eva-shuttle.de/, 2019
  * [41] Intel Labs Germany  “SafeADArchitect-Homepage” Accessed: 2023-09-02, http://www.safeadarchitect.de/, 2021
  * [42] European Center for Information and CommunicationΩTechnologies – EICT GmbH  “Verification Validation Methods-Homepage” Accessed: 2023-09-02, http://www.vvm-projekt.de/, 2021
  * [43] e-mobil BW GmbH  “SmartEPark-Homepage” Accessed: 2023-09-04, https://www.e-mobilbw.de/ueber-uns/projektaktivitaeten/smart-mobility, 2020
  * [44] European Center for Information and CommunicationΩTechnologies – EICT GmbH  “RobustSENSE-Homepage” Accessed: 2023-09-02, https://www.fzi.de/project/shuttle2x/, 2023
  * [45] Bundesministerium für Digitales und Verkehr  “ÖV-Leitmotif-KI-Homepage” Accessed: 2023-09-02, https://bmdv.bund.de/SharedDocs/DE/Artikel/DG/AVF-projekte/oev-leitmotif-ki.html, 2022
  * [46] AVL DEUTSCHLAND GMBH  “KIsSME-Homepage” Accessed: 2023-09-02, https://www.kissme-projekt.de/, 2021
  * [47] TelemaxX Telekommunikation GmbH  “KIGLIS-Homepage” Accessed: 2023-09-04, https://www.kiglis.de/, 2020