# Interpretable agent communication from scratch
(with a generic visual processor emerging on the side)

Roberto Dessí
Facebook AI Research
Universitat Pompeu Fabra
<EMAIL_ADDRESS>
Eugene Kharitonov
Facebook AI Research
<EMAIL_ADDRESS>
Marco Baroni
Facebook AI Research
Universitat Pompeu Fabra
ICREA
<EMAIL_ADDRESS>

###### Abstract

As deep networks begin to be deployed as autonomous agents, the issue of how
they can communicate with each other becomes important. Here, we train two
deep nets from scratch to perform realistic referent identification through
unsupervised emergent communication. We show that the largely interpretable
emergent protocol allows the nets to successfully communicate even about
object types they did not see at training time. The visual representations
induced as a by-product of our training regime, moreover, show comparable
quality, when re-used as generic visual features, to a recent self-supervised
learning model. Our results provide concrete evidence of the viability of
(interpretable) emergent deep net communication in a more realistic scenario
than previously considered, as well as establishing an intriguing link between
this field and self-supervised visual learning.111Code available at
https://github.com/facebookresearch/EGG/tree/master/egg/zoo/emcom_as_ssl.

##  1 Introduction

As deep networks become more effective at solving specialized tasks, there has
been interest in letting them develop a language-like communication protocol
so that they can flexibly interact to address joint tasks [1]. One line of
work within this tradition has focused on what is arguably the most basic
function of language, namely to point out, or refer to, objects through
discrete symbols. Such ability would for example allow deep-net-controlled
agents, such as self-driving cars, to inform each other about the presence and
nature of potentially dangerous objects, besides being a basic requirement to
support more advanced capabilities (e.g., denoting relations between objects).

While discreteness is not a necessary prerequisite for agent communication [2,
3], practical and ethical problems might arise if communication is
incomprehensible to humans. A discrete code analogous to language is certainly
easier to decode for us, helping us to understand the agents’ decisions, and
ultimately contributing to the larger goal of explainable AI [4].

In this paper, we study emergent discrete referential communication between
two deep network agents that are trained from scratch on the task. We observe
that the referential discrimination task played by the networks is closely
related to pretext contrastive objectives used in self-supervised visual
representation learning [5, 6, 7, 8]. We exploit this insight to develop a
robust end-to-end variant of a communication game. Our experiments confirm
that, in our setup:  i) the nets develop a set of discrete symbols allowing
them to successfully discriminate objects in natural images, including novel
ones that were not shown during training;  ii) these symbols denote
interpretable categories, so that their emergence constitutes a form of fully
unsupervised image annotation;  iii) the visual features induced as a by-
product can be used as high-quality general-purpose representations, whose
performance in various object classification tasks is not lagging much behind
that of features induced by a popular self-supervised representation method
specifically designed for this task.

##  2 Background

###  2.1 Deep net emergent communication

There has recently been interest in letting deep nets communicate through
learned protocols. This line of work has addressed various challenges, such as
communication in a dynamic environment or how to interface the emergent
protocol with natural language (see [1] for a recent survey). Probably the
most widely studied aspect of emergent communication is the ability of deep
net agents to use the protocol to refer to objects in their environment [e.g.,
9, 10, 11, 12, 13, 14, 15, 16].

The typical setup is that of a referential, or discriminative, communication
game. In the simplest scenario, which we adopt here, an agent, the Sender,
sees one input (the target) and it sends a discrete symbol to another agent,
the Receiver, that sees an array of items including the target, and has to
point to the latter for communication to be deemed successful. Importantly,
task success is the only training objective; the communication protocol
emerges purely as a by-product of game-playing, without any direct supervision
on the symbol-transmission channel.

In one of the earliest papers in this line of research, Lazaridou et al. [9]
used images from ImageNet [17] as input to the discrimination game; Havrylov
and Titov [11] used MSCOCO [18]; and Evtimova et al. [12] used animal images
from Flickr. While they used natural images, all these studies were limited to
small sets of carefully selected object categories. Moreover, in all these
works, the agents processed images with convolutional networks pretrained on
supervised object recognition. While this sped up learning, it also meant that
all the proposed systems de facto relied on the large amount of human
annotated data used for CNN training. Lazaridou et al. [13] and Choi et al.
[14] dispensed with pre-trained CNNs, but they used synthetically generated
geometric shapes as inputs.

Results on the interpretability of symbols in games with realistic inputs have
generally been mixed. Indeed, Bouchacourt and Baroni [19] showed that, after
training Lazaridou et al. [9]’s networks on real pictures, the networks could
use the learned protocol to successfully communicate about blobs of Gaussian
noise, suggesting that their code (also) denoted low-level image features,
differently from the general semantic categories that words in human language
refer to. In part for this reason, recent work tends to focus on controlled
symbolic inputs, where it is easier to detect degenerate communication
strategies, rather than attempting to learn communication “in the wild” [e.g.,
10, 15, 16].

###  2.2 Self-supervised representation learning

Self-supervised learning of general-purpose visual features has received much
attention in recent years. The main idea is to train a visual network on a
pretext task that does not require manual annotation. After convergence, the
net is used to extract high-quality features from images, to be applied to
various “downstream” tasks of interest. This is often done by training a
simple classifier on top of the frozen trained architecture [20, 21, 22, 23].

Early models used image patch prediction as the proxy task [8, 24]. Recent
work has instead focused on an instance-level contrastive discrimination
objective [5, 6, 7, 25]. Two symmetric networks encode different views of the
same input images obtained through a stochastic data augmentation pipeline.
Optimization is done with variants of the InfoNCE loss [24, 26], that tries to
maximize similarity among representations of the same image while minimizing
similarity of different ones.

Interestingly, the contrastive pretext task is very close to the one of
identifying a target image among distractors, as in the standard referent
discrimination game of emergent communication. The influential SimCLR model
proposed by Chen et al. [5] is particularly similar to our setup. It uses two
twin networks with a shared convolutional module optimizing the
(dis)similarity of sets of target/distractor images. The main conceptual
differences are that there is no discrete bottleneck imposed on
“communication” between the networks, and there is no asymmetry, so that both
networks act simultaneously as Sender and Receiver (both networks produce a
continuous “message” that must be as discriminative for the other network as
possible).

We rely here on the connection with self-supervised learning in two ways.
First, we import the idea of data augmentation from this literature into the
communication game, showing how it helps in evolving a more semantically
interpretable protocol. Second, we evaluate the discrimination game as a self-
supervised feature extraction method. We find that the visual features induced
by the CNNs embedded in our agents are virtually as good as those induced by
SimCLR, while the emergent protocol is better for communication than the one
obtained by adapting SimCLR to the discrete communication setup.

##  3 Setup

###  3.1 The discrimination game

A Sender network receives as input a target picture, and it produces as output
one of $|V|$ symbols. A Receiver network receives in input this symbol, as
well as a list of $n$ pictures, one of them (randomly placed in the $i$th
position of the list) being the same target presented to the Sender. Receiver
produces a probability distribution of cardinality $n$, interpreted as its
guess over the position of the target. The guess is correct iff Sender
concentrates the largest probability mass on the $i$th position, corresponding
to the target slot.

Figure 1: Game setup and agent architectures. Image sources:
https://unsplash.com

#### Agent architecture

Agent architecture and game flow are schematically shown in Fig. 1. Sender
reads the target image through a convolutional module, followed by a one-layer
network mapping the output of the CNN onto $|V|$ dimensions and applying batch
normalization [27], to obtain vector $\mathbf{v}$. Following common practice
when optimizing through discrete bottlenecks, we then compute the Gumbel-
Softmax continuous relaxation [28, 29], which was shown to also be effective
in the emergent communication setup [11]. At train time, Sender produces an
approximation to a one-hot symbol vector with each component given by
$m_{i}=\frac{\exp{[(s_{i}+v_{i}})/\tau]}{\sum_{j}{\exp{[(s_{j}+v_{j}}})/\tau]}$,
where $s_{i}$ is a random sample from Gumbel(0,1) and ${v_{i}}$ a dimension of
$\mathbf{v}$. The approximation is controlled by temperature parameter $\tau$:
as $\tau$ approaches 0, the approximation approaches a one-hot vector, and as
$\tau$ approaches $+\infty$, the relaxation becomes closer to uniform.
Importantly, at test time the Sender’s output is generated by directly argmax-
ing v, so that it is a discrete one-hot vector indexing one of $|V|$ possible
symbols.

Receiver passes each input image through its visual module (a CNN
architecture), followed by a two-layer MLP with batch normalization and ReLU
after the first layer [30]. It then computes temperature-weighted cosine
scores for the linearly embedded symbol compared to each image representation.
The resulting vector of cross-modal (symbol-image) similarities is transformed
into a probability distribution over which image is the likely target by
applying the softmax operation.

For both Sender and Receiver, we use ResNet-50 [31] as visual module. As they
are different agents, that could (in future experiments) have very different
architectures and interact with further agents, the most natural assumption is
that each of them does visual processing with its own CNN. We consider however
also a setup in which the CNN module is shared (closer to earlier emergent-
communication work, where the agents relied on the same pre-trained CNN).

#### Optimization

Optimization is performed end-to-end and the error signal, backpropagated
through Receiver and Sender, is computed using the cross-entropy cost function
by comparing the Receiver’s output with a one-hot vector representing the
position of the target in the image list.

#### SimCLR as a comparison model

Given the similarity between the referential communication game and
contrastive self-supervised learning in SimCLR [5], we use the latter as a
comparison point for our approach. Fig. 2 schematically shows the SimCLR
architecture. The crucial differences between SimCLR and our communication
game are the following.  i) In SimCLR, the agents are parameterized by the
same network, that is, the visual encoder and transformation modules in the
two branches of Fig. 2 are instances of the same net.  ii) The setup is fully
symmetric. Like our Receiver, both agents get a set of images in input, and,
like our Sender, both agents can be seen as producing “messages” representing
each input image.  iii) Instead of (a probability distribution over) symbols,
the exchanged information takes the form of continuous vectors ($s$ in the
figure).  iv) The loss is based on directly comparing embeddings of these
continuous vectors ($z$ in the figure), maximizing the similarity between
pairs representing the same images (positive examples in contrastive-loss
terminology) and minimizing that of pairs representing different images
(negative examples). This differs from our loss, that maximizes the similarity
of the Receiver embedding of the Sender-produced discrete symbol with its own
representation of the target image, while minimizing the similarity of the
symbol embedding with its representation of the distractors.

Figure 2: The SimCLR architecture. aug_p and aug_q are outputs of the
stochastic augmentation pipeline used to generate two views of the same image.

#### Data augmentation

In the original discrimination game proposed in [9], the agents are shown
exactly the same target image. In self-supervised learning, on the other hand,
it is common practice to “augment” images in different ways, e.g., by applying
different croppings or color perturbations [5, 32, 33]. In standard
contrastive learning frameworks, where all the weights are shared and there is
no communication bottleneck, it is necessary to create these different views,
or else the system would trivially succeed at the pretext contrastive task
without any actual feature learning. We conjecture that data augmentation,
while not strictly needed, might also be beneficial in the communication game
setup: presenting different views of the target to Sender and Receiver should
make it harder for them to adopt degenerate strategies based on low-level
image information [19].222Lazaridou and colleagues [9] also considered a
variant of the game in which the agents see different pictures of the same
category (e.g., the shared target is dog, but the agents get different dog
pictures). This version of the game is however severely limited by the
requirement of manual category annotation. Lazaridou et al. [13] also provide
different images to Sender and Receiver, by feeding them different viewpoints
of the same synthetically generated objects: again, a strategy that will not
scale up to natural images. We follow the same data augmentation pipeline as
[5], stochastically applying crop-and-resize, color perturbation, and random
Gaussian blurring to every image.

#### Implementation details

All hidden and output layers are set to dimensionality 2048.333This is the
same size used in the original SimCLR paper, except for the nonlinear
projection head. For the latter, a number of sizes were tested, and the
authors report that they do not impact final performance. We use $2048$ for
direct comparability with our setting. Note that this implies $|V|=2048$, more
than double the categories in the dataset we use to train the model (see
Section 3.2 below), to avoid implicit supervision on optimal symbol count. We
fix Gumbel-Softmax temperature at $5.0$, and Receiver cosine temperature at
0.1. The latter value is also used for the equivalent $\tau$ parameter in the
NTXent-loss of our SimCLR implementation.

We train with mixed precision [34] for 100 epochs, with a batch of size
$16\times 128=2048$, divided across 16 GPUs. Rather than sampling distractors
from the entire dataset, we take them from the current device’s batch, thus
playing the communication game with 127 distractor images in all reported
experiments. We do not share distractors (negative samples) across devices. As
in SimCLR, we use the LARS optimizer [35] with linear scaling [36], resulting
in an initial learning rate of $2.4$. We apply a cosine decay schedule without
warmup nor restart [37]. Compute requirements are reported in Appendix A.1.
All models are implemented with the EGG toolkit [38].

###  3.2 Data

#### Data for model training

Targets and distractors are sampled from the ILSVRC-2012 training set [39],
containing 1.3M natural images from 1K distinct categories.

#### Referential game testing

We use two image sources for testing. First, we use the ILSVRC-2012 validation
set, containing around 50K images from the same categories as the training
data. Second, in order to probe if the emergent protocol possesses a language-
like ability to seamlessly refer to new things, we introduce a new “out-of-
distribution” dataset (henceforth, the OOD set). To build the latter, we
relied on the whole ImageNet database [17], exploiting its WordNet-derived
hierarchy [40]. In particular, we randomly picked (and manually sanity-
checked) 80 categories that were neither in ILSVRC-2012 nor hypernyms or
hyponyms of ILSVR-2012 categories (e.g., since hamster is in ILSVR-2012, we
avoided both rodent and golden hamster). We also attempted to sample
categories of comparable degree of generality to those in ILSVRC-2012. For
each of the categories chosen according to these criteria, we randomly sampled
128 images from ImageNet. Examples of included categories are eucalyptus,
amoeba, and drawer.444Paths to the ImageNet images in the OOD set and the
corresponding categories are available at
https://github.com/facebookresearch/EGG/blob/master/egg/zoo/emcom_as_ssl/OOD_set.txt.

#### Linear evaluation of visual features on downstream tasks

Following standard practice in self-supervised learning [e.g., 5, 20, 41], we
evaluate the visual features induced by the CNN components of our models by
training a linear object classifier on top of them. We use four common data-
sets: ILSVRC-2012, Places205 [42], iNaturalist2018 and
VOC07.555http://places.csail.mit.edu/index.html,
https://www.kaggle.com/c/inaturalist-2018,
http://host.robots.ox.ac.uk/pascal/VOC/voc2007/ Evaluation is carried out with
the VISSL toolkit [43],666https://vissl.ai/ adopting the hyperparameters in
its configuration files without changes.

##  4 Experiments

###  4.1 Referential communication accuracy

We start by analyzing how well our models learn to refer to object-depicting
images through a learned protocol.777Appendix A.2 reports this and all
following experiments repeated with 5 distinct initializations of our most
representative model (+augmentations -shared). It shows that variance across
runs is negligible. We did not repeat the check for the remaining models due
to time and resource constraints (see Appendix A.1). While some models use
data augmentation at training time, we do not apply this transformation when
testing the learned communication pipeline.

As a strong baseline, we let the trained SimCLR model play the referential
game by argmax-ing its $s$ layer (see Fig. 2 above) into a discrete “symbol”
(SimCLRdisc).888When looking at SimCLR as a communication game, $s$
constitutes the discrete symbol layer, with $z$ functioning as symbol
embedding layer. Discretizing $h$ produced worse game-playing performance in
any case.

Accuracy is given by the proportion of times in which a system assigns the
largest symbol-embedding/image-representation similarity to the target
compared to 127 distractors (chance $\approx{}0.8\%$).

Results are in Table 1. The ILSVRC-val column shows that all models can play
the game well above chance, when tested on new images of the same categories
encountered during training. The next column (OOD set) shows that the models
also play the game well above chance with input images from new categories,
although mostly with a drop in performance. All variants of our model are
considerably more robust than the SimCLR baseline (which, however, does
remarkably well, out-of-the-box, at this discrete communication game).

| ILSVRC-val | OOD set | Gaussian Blobs
---|---|---|---
SimCLRdisc | 56.9% | 47.4% | 0.8%
Communication Game |  |  |
-augmentations -shared | 91.2% | 90.8% | 43.4%
-augmentations +shared | 92.8% | 92.7% | 84.7%
+augmentations -shared | 81.5% | 72.0% | 0.8%
+augmentations +shared | 82.2% | 73.7% | 0.8%
Table 1: Game-playing accuracy. $\pm$ augmentations marks whether the game was
trained with data augmentations or not. $\pm$ shared indicates whether there
was CNN weight sharing or not between Sender and Receiver.

In summary, neural networks trained from scratch are able to communicate quite
accurately through a discrete channel even in the challenging setup in which
the target referent is mixed with more than one hundred distractors, and when
it belongs to new categories not seen at training time.

Looking at model variants, sharing CNN weights or not makes little difference
(an encouraging first step towards communication between widely differing
agents, that will obviously not be able to share weights). On the other hand,
data augmentations harm performance.

However, it turns out that the better performance of the non-augmented models
is due to an opaque communication strategy in which the agents are evidently
referring to low-level aspects of images (perhaps, specific pixel intensity
levels?), and not to the high-level semantic information they contain
(ideally, object categories). To show this, we replicated the sanity check
from [19]. We freeze the trained models and let them play the communication
game with blobs of Gaussian noise as targets and distractors. We use 384
batches of 128 $224$x$224$-sized random images whose pixels are drawn from the
standard Gaussian distribution $\mathcal{N}(0,1)$, for a total of $49152$
items (a size comparable to that of ILSVRC-val). Results are in the last
column of Table 1.

The game-based models with augmentations and SimCLR fully pass the sanity
check, with performance exactly at the $0.8\%$ chance level. The -augmentation
models, on the other hand, are able to use the symbols they learned from sane
input to communicate about the Gaussian blobs, showing that they developed a
deeply opaque protocol. The -augmentations +shared model, in particular, is
hardly affected by the switch to noise data. This is the closest setup to
previous emergent communication studies, in which no data augmentation is
applied and the agents share the same pre-trained CNN. It is no surprise,
then, that protocol interpretability results are often mixed.

###  4.2 Protocol analysis: emergent communication as unsupervised image
annotation

The Gaussian blob test suggests that the +augmentations models do not fall
into the trap of a degenerate low-level protocol. However, it is not
sufficient to conclude that they learned to associate symbols with human-
meaningful referents. To test whether this is the case, we exploit the fact
that, as we are working with ImageNet data, we have labels denoting the
objects depicted in the images. We use this information in two ways. We
compute the normalized mutual information (nMI) between the ground-truth
labels of target images and the symbols produced for the same images by the
trained Sender. The nMI of two variables is obtained dividing their MI by
their average entropy, and it ranges between 0 and 1. We also compute a
normalized similarity measure based on the shortest path between two
categories in the WordNet is-a taxonomy. Our WNsim score is the average
shortest-path similarity of the ground-truth categories of all target pairs
that share the same Sender symbol, and it also ranges between 0 and 1. WNSim
is more nuanced than nMI, as it will penalize less a Sender using the same
symbol for similar categories, such as cats and dogs, than one using the same
symbol for dissimilar ones, such as cats and skyscrapers. WNsim is computed
with NLTK [44].

We again take SimCLRdisc as a comparison point, where a “symbol” is simply the
dimension with the largest value on a certain layer. To give this approach its
best chance, we evaluated its $h$, $s$ and $z$ layers (see Fig. 2), and report
statistics for $h$ (CNN output), as it produced the best overall scores across
data sets. We also run $k$-means clustering on the $h$ layer (SimCLRkmeans).
We tried clustering with $k=1000$ (ground-truth class cardinality) and
$k=2048$ (same as vocabulary size of our models).999Cluster centroids were
estimated on a random 10% of the training set. Cluster assignments of the test
data were used as symbols in the protocol analysis for SimCLRkmeans. We report
the significantly better results we obtained with the second choice.

| ILSVRC-val | OOD set
---|---|---
| | P | | nMI | WNsim | | P | | nMI | WNsim
SimCLRdisc | 1489 | 0.49 | 0.11 | 1069 | 0.46 | 0.19
SimCLRkmeans | 2035 | 0.59 | 0.18 | 1519 | 0.54 | 0.30
Communication Game |  |  |  |  |  |
-augmentations -shared | 2044 | 0.50 | 0.08 | 1921 | 0.45 | 0.11
-augmentations +shared | 2048 | NS | NS | 2025 | NS | NS
+augmentations -shared | 2042 | 0.58 | 0.18 | 1752 | 0.53 | 0.32
+augmentations +shared | 2046 | 0.56 | 0.15 | 1765 | 0.51 | 0.25
Table 2: Protocol analysis. |P| is the observed protocol size, that is, the
number of distinct symbols actually used at test time. We mark as NS the cases
where the obtained scores were not significantly different from chance
according to a permutation test with $\alpha=0.01$.

Looking at Table 2, we first observe that, consistent with the Gaussian blob
sanity check, there is no significant sign of symbol-category association for
the -augmentations +shared protocol. All other models show some degree of
symbol interpretability (with significantly above-chance nMI and WNsim
scores). Even when there is no data augmentation during training, using
different visual modules (-augmentations -shared) leads to some protocol
interpretability, coherently with the fact that this configuration was less
able than its +shared counterpart to communicate about noise. We were moreover
surprised to find that simply argmaxing the SimCLR visual feature layer
produces meaningful “symbols”, which suggests that information might be more
sparsely encoded by this model than one could naively assume.

Importantly, the game protocols derived with data augmentation have
particularly high nMI and WNsim scores. Impressively, the scores achieved by
our models, and the +augmentations -shared setup in particular, are very close
to those obtained by clustering SimCLR visual features. Recall that, unlike
our models, whose protocol independently emerges during discriminative game
training, SimCLRkmeans runs a clustering algorithm on top of the
representations produced by the SimCLR visual encoder with the express goal to
discretize them into coherent sets, thus constituting a hard competitor to
reach.

Even more than the quantitative results, a sense of how good our symbols are
as unsupervised image labels is given by qualitative inspection of images
sharing the same assigned symbol. Fig. 3 shows a random set of such images for
the 9 symbols most frequently produced by the +augmentations -shared Sender in
ILSVRC-val, without any hand-picking.101010To preserve privacy, we excluded
25% of symbol-5 images before sampling, as they depicted people. Consistent
with this symbol’s “theme”, the latter mostly show people in dark backgrounds.
Some symbols denote intuitive categories, although, interestingly, ones that
do not correspond to specific English words (birds on branches, dogs
indoors…). Other sets are harder to characterize, but they still share a clear
high-level “family resemblance” (Symbol 2: objects that glow in the dark;
Symbol 3: human artifacts with simple flat shapes, etc). Frequency imbalance
in input categories, together with the fact that the agents are allowed to use
a large number of symbols, leads to partially overlapping categories (Symbol 9
might denote living things in the grass, whereas Symbol 4 seems to
specifically refer to mammals in the grass).

Figure 3: Randomly selected ILSVRC-val images triggering the +augmentations
-shared Sender to produce its 9 most frequent symbols. Figure 4: Randomly
selected OOD set images triggering the +augmentations -shared Sender to
produce the 3 most frequent ILSVRC-val symbols (cf. Fig. 3).

The fact that symbols do not exactly denote ILSVRC categories plays to the
agents’ advantage when they must communicate about OOD set images. While this
set, by construction, does not contain ILSVRC categories, as Figure 4 shows,
it still contains birds on tree, glowing objects and artifacts with flat
shapes.

###  4.3 Downstream object classification: emergent communication as self-
supervised visual feature learning

Finally, we evaluate how the features produced by the Sender CNN trained on
the communication game work as out-of-the-box visual representations. We
follow the standard protocol for training a linear classifier on the output of
the frozen CNN trunk with various object classification datasets (see 3.2
above). We focus on Sender because the features produced by the two agent
networks are always highly correlated.111111Across setups and data sets, the
Sender/Receiver correlation between all pairwise visual representation
similarities was never below $0.96$. We further exclude the -augmentations
models, given that, having learned a degenerated strategy, they extremely poor
performance on ILSVRC-val (below 5% accuracy). We thus decided not to test
them on other downstream tasks, as this is a lengthy and resource-intensive
procedure (see Appendix A.1).

| ILSVRC-val | Places205 | iNaturalist2018 | VOC07
---|---|---|---|---
Supervised | 76.5% | 53.2% | 46.7% | 87.5%
SimCLR | 60.6% | 49.0% | 31.8% | 78.7%
Communication Game |  |  |  |
+augmentations -shared | 59.0% | 47.9% | 30.8% | 77.0%
+augmentations +shared | 60.2% | 49.1% | 31.3% | 78.8%
Table 3: Linear evaluation on object classification. Reported scores are mAP
for VOC07, top-1 accuracy elsewhere. Supervised results are from [21].

As a reasonable upper bound, Table 3 reports the fully-supervised object
classification results from [21]. As a more direct point of comparison, we
also report the performance of our SimCLR implementation.121212It is difficult
to compare our SimCLR ILSVRC-val performance precisely to that reported in the
original paper since, coherently with the communication game setup, we use a
per-GPU batch size of 128 without sharing negatives across GPUs. By looking at
the leftmost bars of Fig. 9 in [5], we note that the performance we report is
within the range of their results for the same number of training epochs
(100). As the table shows, the features developed as by-product of
communication game playing are consistently of comparable quality to those of
SimCLR, a method developed specifically for visual feature learning. This is
an extremely promising first step towards employing emergent communication as
a form of self supervision. Many ideas from the self-supervised literature
(e.g., new data augmentation pipelines, the use of memory banks for distractor
sampling or variants of the similarity-based pretext task) could
straightforwardly be integrated into our setup, hopefully leading to the
emergence of even better visual features and, perhaps, an even more
transparent protocol.

##  5 Conclusion

Deep agent coordination through communication has recently attracted
considerable interest. Referential games are a natural environment to test the
agents’ emergent communication strategies. Past approaches, however, relied on
relatively small image pools processed with pretrained visual networks, or on
artificial input. We showed instead that deep agents can learn to refer to a
high number of categories depicted in large-scale image datasets, while
communicating through a discrete channel and developing their visual
processing modules from scratch. Performance on referential games with two
distinct test sets (one with categories not presented at training), along with
protocol analysis, shows that the agents’ protocol is effective and
interpretable.

A key ingredient to success was input data augmentation. We borrowed this idea
from recent approaches to self-supervised visual learning. We believe that the
connection between emergent communication and this field can be mutually
beneficial. In this work we showed how the agents’ visual networks emerging
from discriminative game playing already produce high-quality visual features.
On the other hand, imposing a discrete bottleneck on network communication can
be seen as an augmentation operation in the latent space of an existing
contrastive learning system. Further integration with self-supervised learning
methods should be explored in the future.

In our experiment, agents communicate through a single symbol, but the true
expressive power of human language comes from the infinite combinatorial
possibilities offered by composing sequences of discrete units [45]. Allowing
messages of greater length and probing whether this results in the development
of a compositional code should be a priority for future work. Additionally,
while in our experiments distractors are selected at random, this is obviously
not the case in real-life referential settings (dogs will tend to occur near
other dogs or humans, rather than between a whale and a space shuttle).
Dealing with realistic category co-occurence is thus another important future
direction. Finally, although we provided quantitative and qualitative evidence
that the agents’ protocol is reasonably transparent, the extent to which the
achieved degree of interpretability is good enough for human-in-the-loop
scenarios remains to be experimentally investigated.

Much recent work in the field has moved towards a theoretically-oriented
understanding of deep agent communication in symbolic and artificial setups.
We went back instead to the original motivation behind the study of emergent
language, as a path towards the development of autonomous AIs that can
interact with each other in a realistic environment. While we are still far
from real-life-deployable and fully interpretable machine-machine interaction,
we believe that our work represents an important early step towards this goal.

### Acknowledgments

We would like to thank Gemma Boleda, Rahma Chaabouni, Emmanuel Chemla, Simone
Conia and Lucas Weber for feedback on an earlier version of this manuscript,
Priya Goyal for technical support on VISSL, Mathilde Caron and the
participants of the EViL meeting, the FAIR EMEA-NLP meetup and the TAB meeting
for fruitful discussions. We also want to thank Jade Copet for sharing an
early version of the code used in this work.

## References

  * [1] Angeliki Lazaridou and Marco Baroni.  Emergent multi-agent communication in the deep learning era.  https://arxiv.org/abs/2006.02419, 2020.
  * [2] Sainbayar Sukhbaatar, Arthur Szlam, and Rob Fergus.  Learning multiagent communication with backpropagation.  In Proceedings of NIPS, pages 2244–2252, Barcelona, Spain, 2016\.
  * [3] Nur Geffen Lan, Emmanuel Chemla, and Shane Steinert-Threlkeld.  On the Spontaneous Emergence of Discrete and Compositional Signals.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4794–4800, Online, July 2020. Association for Computational Linguistics.
  * [4] Ning Xie, Gabrielle Ras, Marcel van Gerven, and Derek Doran.  Explainable deep learning: A field guide for the uninitiated.  https://arxiv.org/abs/2004.14545, 2020.
  * [5] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597–1607. PMLR, 13–18 Jul 2020\.
  * [6] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [7] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko.  Bootstrap your own latent - a new approach to self-supervised learning.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21271–21284. Curran Associates, Inc., 2020.
  * [8] Carl Doersch, Abhinav Gupta, and Alexei A. Efros.  Unsupervised visual representation learning by context prediction.  In 2015 IEEE International Conference on Computer Vision (ICCV), pages 1422–1430, 2015.
  * [9] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni.  Multi-agent cooperation and the emergence of (natural) language.  In Proceedings of ICLR Conference Track, Toulon, France, 2017.  Published online: https://openreview.net/group?id=ICLR.cc/2017/conference.
  * [10] Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra.  Natural language does not emerge ‘naturally’ in multi-agent dialog.  In Proceedings of EMNLP, pages 2962–2967, Copenhagen, Denmark, 2017\.
  * [11] Serhii Havrylov and Ivan Titov.  Emergence of language with multi-agent games: Learning to communicate with sequences of symbols.  In Proceedings of NIPS, pages 2149–2159, Long Beach, CA, 2017.
  * [12] Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho.  Emergent communication in a multi-modal, multi-step referential game.  In Proceedings of ICLR Conference Track, Vancouver, Canada, 2018\.  Published online: https://openreview.net/group?id=ICLR.cc/2018/Conference.
  * [13] Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark.  Emergence of linguistic communication from referential games with symbolic and pixel input.  In Proceedings of ICLR Conference Track, Vancouver, Canada, 2018\.  Published online: https://openreview.net/group?id=ICLR.cc/2018/Conference.
  * [14] Edward Choi, Angeliki Lazaridou, and Nando de Freitas.  Compositional obverter communication learning from raw visual input.  In Proceedings of ICLR Conference Track, Vancouver, Canada, 2018\.  Published online: https://openreview.net/group?id=ICLR.cc/2018/Conference.
  * [15] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni.  Anti-efficient encoding in emergent communication.  In Proceedings of NeurIPS, Vancouver, Canada, 2019.  Published online: https://papers.nips.cc/paper/2019.
  * [16] Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni.  Compositionality and generalization in emergent languages.  In Proceedings of ACL, pages 4427–4442, virtual conference, 2020\.
  * [17] Jia Deng, Wei Dong, Richard Socher, Lia-Ji Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In Proceedings of CVPR, pages 248–255, Miami Beach, FL, 2009.
  * [18] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision – ECCV 2014, pages 740–755, Cham, 2014. Springer International Publishing.
  * [19] Diane Bouchacourt and Marco Baroni.  How agents see things: On visual representations in an emergent language game.  In Proceedings of EMNLP, pages 981–985, Brussels, Belgium, 2018\.
  * [20] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  arXiv preprint arXiv:2104.14294, 2021.
  * [21] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 9912–9924. Curran Associates, Inc., 2020.
  * [22] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman.  With a little help from my friends: Nearest-neighbor contrastive learning of visual representations.  CoRR, abs/2104.14548, 2021.
  * [23] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  CoRR, abs/2103.03230, 2021.
  * [24] Aäron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  CoRR, abs/1807.03748, 2018.
  * [25] Xinlei Chen and Kaiming He.  Exploring simple Siamese representation learning.  CoRR, abs/2011.10566, 2020.
  * [26] Michael Gutmann and Aapo Hyvärinen.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 297–304, Chia Laguna Resort, Sardinia, Italy, 13–15 May 2010. PMLR.
  * [27] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 448–456, Lille, France, 07–09 Jul 2015\. PMLR.
  * [28] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with Gumbel-Softmax.  In Proceedings of ICLR Conference Track, Toulon, France, 2017.  Published online: https://openreview.net/group?id=ICLR.cc/2017/conference.
  * [29] Chris J Maddison, Andriy Mnih, and Yee Whye Teh.  The concrete distribution: A continuous relaxation of discrete random variables.  arXiv preprint arXiv:1611.00712, 2016.
  * [30] Xavier Glorot, Antoine Bordes, and Yoshua Bengio.  Deep sparse rectifier neural networks.  In Proceedings of AISTATS, pages 315–323, Fort Lauderdale, FL, 2011\.
  * [31] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [32] Philip Bachman, Devon Hjelm, and William Buchwalter.  Learning representations by maximizing mutual information across views.  In Proceedings of NeurIPS, Vancouver, Canada, 2019.  Published online: https://papers.nips.cc/paper/2019.
  * [33] Mang Ye, Xu Zhang, Pong Yuen, and Shih-Fu Chang.  Unsupervised embedding learning via invariant and spreading instance feature.  In Proceedings of CVPR, pages 6210–6219, Long Beach, CA, 2019.
  * [34] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu.  Mixed precision training.  CoRR, abs/1710.03740, 2017.
  * [35] Yang You, Igor Gitman, and Boris Ginsburg.  Scaling SGD batch size to 32k for imagenet training.  CoRR, abs/1708.03888, 2017.
  * [36] Priya Goyal, Piotr Dollár, Ross B. Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Accurate, large minibatch SGD: training imagenet in 1 hour.  CoRR, abs/1706.02677, 2017.
  * [37] Ilya Loshchilov and Frank Hutter.  SGDR: stochastic gradient descent with restarts.  CoRR, abs/1608.03983, 2016.
  * [38] Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni.  EGG: a toolkit for research on Emergence of lanGuage in Games.  In EMNLP-IJCNLP: System Demonstrations, 2019.
  * [39] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition challenge.  International Journal of Computer Vision, 115(3):211–252, 2015\.
  * [40] Christiane Fellbaum, editor.  WordNet: An Electronic Lexical Database.  MIT Press, Cambridge, MA, 1998.
  * [41] Junnan Li, Pan Zhou, Caiming Xiong, and Steven Hoi.  Prototypical contrastive learning of unsupervised representations.  In Proceedings of ICLR, Online conference., 2021.  Proceedings at: https://openreview.net/group?id=ICLR.cc/2021/Conference.
  * [42] Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva.  Learning deep features for scene recognition using Places database.  In Proceedings of NIPS, Montreal, Canada, 2014.  Published online: https://papers.nips.cc/paper/2014.
  * [43] Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Ishan Misra.  VISSL.  https://github.com/facebookresearch/vissl, 2021.
  * [44] Steven Bird, Ewan Klein, and Edward Loper.  Natural language processing with Python: analyzing text with the natural language toolkit.  " O’Reilly Media, Inc.", 2009.
  * [45] Robert Berwick and Noam Chomsky.  Why Only Us: Language and Evolution.  MIT Press, Cambridge, MA, 2016.

##  Appendix A Appendix

###  A.1 Compute details

All experiments were run using Tesla V100 GPUs on an internal SLURM-based
cluster, except where indicated. Training a communication game takes
approximately 16 hours on 16 GPUs. Testing on the referential game takes less
than 5 minutes on a single NVIDIA Quadro GP100. The permutation procedure used
to establish statistical significance for purposes of protocol analysis takes
up to about 24 hours, and do not requires GPUs. The downstream object
classification experiments take up to about 16 hours on 8 GPUs.

###  A.2 Impact of random seeds on +augmentation -shared model performance

To gauge the robustness of our results to model initialization variance, we
repeated all experiments after training our most representative model
(+augmentation -shared) with 5 different random seeds (including the randomly
picked seed consistently used for the results reported in the main text). The
outcomes, summarized in Tables 4, 5 and 6, show that the effect of this source
of variation on model performance is negligible.

task | avg | sd | min | max
---|---|---|---|---
ILSVRC-val | 81.4% | 0.2% | 81.1% | 81.6%
OOD set | 71.7% | 0.6% | 71.0% | 72.4%
Gaussian Blobs | 0.8% | 0.1% | 0.8% | 1.0%
Table 4: Game playing accuracy of +augmentation -shared model across 5 seeds. task | avg | sd | min | max
---|---|---|---|---
ILSVRC-val
|P| | 2040.4 | 2.1 | 2037 | 2042
nMI | 0.58 | 0.00 | 0.58 | 0.58
WNsim | 0.18 | 0.00 | 0.17 | 0.18
OOD set
|P| | 1749.8 | 15.0 | 1723 | 1767
nMI | 0.53 | 0.00 | 0.52 | 0.53
WNsim | 0.29 | 0.02 | 0.27 | 0.32
Table 5: Protocol analysis statistics of +augmentation -shared model across 5 seeds. task | avg | sd | min | max
---|---|---|---|---
ILSVRC-val | 59.10 | 0.10 | 58.98 | 59.20
Places205 | 48.2 | 0.17 | 47.9 | 48.3
iNaturalist2018 | 31.1 | 0.16 | 30.8 | 31.3
VOC07 | 77.1 | 0.06 | 77.0 | 77.2
Table 6: Linear evaluation accuracy on object classification for +augmentation
-shared model across 5 seeds. Reported scores are mAP for VOC07, top-1
accuracy elsewhere.


# A9 Intersection Dataset:
All You Need for Urban 3D Camera-LiDAR Roadside Perception

Walter Zimmer* ID Christian Creß* ID Huu Tung Nguyen* ID Alois C. Knoll ID The
authors are with the Chair of Robotics, Artificial Intelligence and Real-time
Systems, TUM School of Computation, Information and Technology (CIT),
Technical University of Munich, Munich, Germany. Contact:
<EMAIL_ADDRESS><EMAIL_ADDRESS>authors contributed
equally.

###### Abstract

Intelligent Transportation Systems (ITS) allow a drastic expansion of the
visibility range and decrease occlusions for autonomous driving. To obtain
accurate detections, detailed labeled sensor data for training is required.
Unfortunately, high-quality 3D labels of LiDAR point clouds from the
infrastructure perspective of an intersection are still rare. Therefore, we
provide the A9 Intersection Dataset, which consists of labeled LiDAR point
clouds and synchronized camera images. Here, we recorded the sensor output
from two roadside cameras and LiDARs mounted on intersection gantry bridges.
The point clouds were labeled in 3D by experienced annotators. Furthermore, we
provide calibration data between all sensors, which allow the projection of
the 3D labels into the camera images and an accurate data fusion. Our dataset
consists of 4.8k images and point clouds with more than 57.4k manually labeled
3D boxes. With ten object classes, it has a high diversity of road users in
complex driving maneuvers, such as left and right turns, overtaking, and
U-turns. In experiments, we provided multiple baselines for the perception
tasks. Overall, our dataset is a valuable contribution to the scientific
community to perform complex 3D camera-LiDAR roadside perception tasks. Find
data, code, and more information at https://a9-dataset.com.

###### Index Terms:

Dataset, 3D Perception, Camera, LiDAR, Intelligent Transportation Systems,
Autonomous Driving

TABLE I: Comparison of popular autonomous driving datasets. Here we compare the perspective, the number of frames, the number of classes, the number of labeled objects, the number of tracks, and the license terms. The datasets cover different view perspectives: the vehicle view (V), from the steep elevated view (SE), and from the roadside view (R).  Name | Year | Perspective | # Point Clouds | # Images | # Classes | # 3D Labels | # Tracks | License
---|---|---|---|---|---|---|---|---
KITTI [1] | 2013 | V | 15.4k | 15k | 8 | 80k | - | CC BY-NC-SA 3.0
Cityscapes [2] | 2016 | V | - | 25k | 30 | - | - | Non-Commercial Use
highD [3] | 2018 | SE | - | 1.4M | 2 | - | 20k | Custom
nuScenes[4] | 2020 | V | 400k | 1.4M | 23 | 1.4M | - | CC BY-NC-SA 4.0
Waymo [5] | 2020 | V | 200 | 1M | 4 | 12.6M | 7.6M | Custom
inD [6] | 2020 | SE | - | 0.9M | 5 | - | 11.5k | Custom
rounD [7] | 2020 | SE | - | 0.5M | 8 | - | 13.7k | Custom
exiD [8] | 2022 | SE | - | 1.4M | 3 | - | 69k | Custom
MONA [9] | 2022 | SE | - | 11.7M | 2 | - | 702k | Custom
DAIR-V2X∗ [10] | 2022 | V,R | 71k | 71k | 10 | 1.2M | - | Non-Commercial Use
IPS300+∗∗ [11] | 2022 | R | 14.1k | 14.1k | 7 | 4.5M | - | CC BY-NC-SA 4.0
Rope3D [12] | 2022 | R | - | 50k | 13 | 1.5M | - | Non-Commercial Use
LUMPI [13] | 2022 | R | 90k | 200k | 6 | - | - | CC BY-NC 3.0
A9 | 2022 | R | 5.3k | 5.4k | 10 | 71.9k | 506 | CC BY-NC-ND 4.0
\- A9 [14] | 2022 | R | 0.5k | 0.6k | 9 | 14.5k | - | CC BY-NC-ND 4.0
\- A9-I (Ours) | 2023 | R | 4.8k | 4.8k | 10 | 57.4k | 506 | CC BY-NC-ND 4.0
∗ 40% of data from roadside perspective, 60% from vehicle perspective.
∗∗Trucks and buses are sparsely represented which can lead to a limited
perception performance.

##  I INTRODUCTION

The roadside deployment of high-tech sensors to detect road traffic
participants offers significant added value for intelligent and autonomous
driving. This technology allows the vehicle to react to events and situations
that are not covered by the vehicle’s internal sensor range. Thus, the
advantage is the drastic expansion of the field of view and the reduction of
occlusions. For this reason, we can observe a continuous increase in
Intelligent Transportation Systems (ITS) world-wide. It is noticeable that
cameras and increasingly LiDARS are used to create a live digital twin of road
traffic [15]. To obtain accurate detections with such sensor systems, labeled
sensor data is required for training.

Numerous datasets in the field of intelligent and autonomous driving have
already been created. Datasets like [1, 4, 2, 5] are taken from the vehicle
perspective. In contrast, [3, 16, 7, 8, 9] are recorded from a very steep
elevated view from a drone or a high building, so they are more suitable for
trajectory prediction and tracking tasks. They are less suitable for 3D object
detection because vehicles are far away and are only observed from above.
Recently, a few datasets [10, 11, 12, 13, 14] have been acquired from an
roadside perspective and are thus suitable for improving perception algorithms
for ITS. However, some datasets have deficiencies in their labeling quality,
which harm the training of the algorithms (e.g., censored image areas with
filled rectangles), or they lack certain vehicle classes (e.g., missing trucks
and buses), or the datasets are too small in terms of 3D box labels and
attributes.

According to the work mentioned, it can be recognized that high-quality 3D box
labels of LiDAR point clouds from the roadside perspective with a wide
diversity of traffic participants and scenarios are still rare. Therefore, our
A9 Intersection (A9-I) Dataset provides LiDAR point clouds and camera images
from a road intersection. The $4.8\text{k}$ labeled point cloud frames, which
were labeled by experts, contain complex driving maneuvers such as left and
right turns, overtaking maneuvers, and U-turns. With its ten object classes,
our dataset has a high variety of road users, including vulnerable road users.
Furthermore, we provide synchronized camera images and the extrinsic
calibration data between LiDARs and the cameras. These matrices allow the
projection of the 3D box labels to the camera images. All in all, our A9-I
offers synchronized $4.8\text{k}$ images and $4.8\text{k}$ point clouds with
$57.4\text{k}$ 3D box labels with track IDs that were manually labeled. In
this work, we show additional comprehensive statistics and the effectiveness
of our dataset. Over and beyond, we would like to emphasize that A9-I is an
extension of our previous debut the A9 Dataset [14], which covers highway
traffic scenarios. Thus, we extend the existing A9 Dataset with additional
traffic scenarios on a crowdy intersection and scale it up from $15\text{k}$
labeled 3D box labels to $57.4\text{k}$ including vulnerable road users. In
evaluation experiments, we provide multiple baselines for the 3D perception
task of 3D object detection with a monocular camera, a LiDAR sensor, and a
multi-modal camera-LiDAR setup. Last but not least, we offer our dataset in
OpenLABEL format under the Creative Commons License CC BY-NC-ND 4.0 so that it
can be widely used by the scientific research community.

In summary, our contributions are:

  * •

A detailed and diverse dataset of $4.8\text{k}$ camera images as well as
$4.8\text{k}$ labeled LiDAR point cloud frames. Thereby, we used two
synchronized cameras and LiDARS, which cover an intersection from an elevated
view of an ITS.

  * •

Extrinsic calibration data between cameras and LiDARs allow an early and late
fusion of objects.

  * •

We provide an extensive A9-Devkit to load, transform, split, evaluate and
visualize the data.

  * •

$57.4\text{k}$ high-quality manually labeled 3D boxes with $273\text{k}$
attributes for both LiDARS resulting in $38\text{k}$ 3D box labels after data
fusion.

  * •

Comprehensive statistics and analysis of the labels, number of points,
occlusions, and tracks on the dataset, and the distribution of ten different
object classes of road traffic.

  * •

Multiple baselines for the 3D perception task of 3D object detection with a
monocular camera, a LiDAR sensor, and a multi-modal camera-LiDAR setup.

##  II RELATED WORK

As part of the development in the field of autonomous driving and intelligent
vehicles, the number of datasets is increasing rapidly. The most popular
datasets in this field are KITTI [1], nuScenes [4], Cityscapes [2], and Waymo
Open dataset [5]. Except for the Cityscapes, the datasets provide labeled
camera images and LiDAR point clouds. These datasets are used to train
perception algorithms. Unfortunately, these valuable datasets only contain
data from a vehicle’s perspective. Therefore, this ego perspective is
suboptimal for transfer learning. Networks trained on a dataset from the
vehicle’s perspective do not perform well on data obtained, e.g. from a
roadside perspective.

Another sensor perspective is, for example, the elevated view. With this, the
scene can ideally be viewed without occlusions. To achieve a high level of
perception for this elevated view, training with appropriate datasets is
necessary. The focus of the drone dataset family highD [3], inD [16], rounD
[7], and exiD [8] is the trajectory of road users in the city as well as in
the freeway area. The datasets were recorded by a drone and provide a vast
top-down view of the scene. The main limitation is the limited recording time
in challenging weather conditions. To overcome this drone-related issue, the
MONA [9] dataset provides data that was created with a camera mounted on a
building. On the one hand, these datasets are ideal for trajectory research,
because they were recorded from a very steep angle to the road. On the other
hand, they are less suitable for 3D object detection, because of the missing
3D dimensions.

A dataset, which contains data from an elevated view of an ITS with an angle
that is not too steep, is the DAIR-V2X [10]. The main focus of DAIR-V2X is the
support of of 3D object detection tasks. It consists of $71\text{k}$ labeled
camera images and LiDAR point clouds, $40\%$ of which are from an roadside
infrastructure. For this purpose, the dataset covers city roads, highways, and
intersections in different weather and lighting conditions. Unfortunately, no
exact statistics for this variation or exact sensor specifications are
available. As a last point, the quality of the data is further compromised by
filled rectangles over privacy-sensitive image areas (e.g., license plates),
which can lead to problems during training for object detection. Another
dataset from the roadside infrastructure perspective with a camera and LiDAR
combination is the IPS300+ [11]. The dataset includes $14\text{k}$ data
frames, with an average of $319$ labels per frame. They used $1$ LiDAR and $2$
cameras as a stereo setup with a lens focal length of $4.57~{}\text{mm}$. The
dataset was recorded several times a day at one intersection and provides
seven different object categories: car, cyclist, pedestrian, tricycle, bus,
truck, and engineer car. According to the statistics, unfortunately, there is
less representation in the classes of trucks and buses, so that the
recognition of these classes will probably be poor. The Roadside Perception 3D
dataset (Rope3D) [12] provides $50\text{k}$ images including 3D box labels
from an monocular infrastructure camera at an intersection. The missing 3D
information of the detected objects in the 2D camera image was added with a
LiDAR, which was mounted on a vehicle. In total, the images contain over
$1.5\text{M}$ labeled 3D boxes, $670\text{k}$ 2D bounding boxes, in various
scenes at different times (daytime, night, dawn/dusk), different weather
conditions (sunny, cloudy, rainy), and different traffic densities.
Furthermore, the objects are divided into $13$ classes with several
attributes. Another roadside infrastructure dataset is LUMPI [13], which was
recorded at an intersection in Hanover, Germany. For this purpose, a total of
$200\text{k}$ images as well as $90\text{k}$ point clouds were acquired. Three
different cameras and five different LiDARs provide several field of views on
the scene. Here, different sensor configurations were used for the recordings.
The sensor perspective is from a vehicle as well as from the roadside
infrastructure. Unfortunately, the number of labels and other detailed
information about the labeled objects was not provided. A further contribution
in the field of roadside infrastructure data for training perception algorithm
is the A9-Dataset [14]. It is our preliminary work and includes $642$ camera
images and $456$ LiDAR point clouds. In total, this dataset consists of 1k 3D
box labels. The charm is that most camera images contain the same traffic
scene from four different viewpoints. Here, we labeled 14k 3D boxes. Moreover,
the frames contain $13.17$ 3D box labels in average. For this purpose, we
supported the common classes of car, trailer, truck, van, pedestrian, bicycle,
bus, and motorcycle in the domain of a highway. The main limitations in our
previous work were firstly the small number of labeled LiDAR point clouds and
secondly that we only had a simple highway scenario. For this reason, we
present an extension to our dataset that addresses these weaknesses.

Figure 1: Two cameras and two LiDARs are used to create the A9 Intersection
Dataset. The sensors are mounted on a sign gantry and thus record the central
intersection area. Then, the Data Fusion Unit (DFU) process the data streams,
which results in a fused digital twin of the road traffic. Furthermore, the
coordinate systems of the individual sensors and the road coordinate system,
which was defined at the northern stem of the bridge, can be taken from the
figure.

##  III A9 INTERSECTION DATASET

In this section, we present the A9 Intersection Dataset. It is an extension of
our previous work, the A9-Dataset [14], which covers the highway domain. We
describe the sensor setup at our intersection, the data selection and
annotation process, and the data structure used. Last, this section contains
comprehensive statistics and an introduction to our A9-Devkit.

###  III-A Sensor Setup

The A9-I Dataset is recorded on the ITS testbed, which was established as part
of the Providentia++ project [17, 18]. Here, roadside sensors are set up on a
gantry located at the intersection of Schleißheimer Straße (B471) and
Zeppelinstraße in Garching near Munich, Germany. For this dataset, we use two
cameras and two LiDARs with the following specifications:

  * •

Camera: Basler ace acA1920-50gc, $1920\times 1200$, Sony IMX174, glo. shutter,
color, GigE with 8 mm lenses.

  * •

LiDAR: Ouster OS1-64 (gen. 2), 64 vert. layers, $360\text{\textdegree FOV}$,
below horizon configuration, $120~{}\text{m}$ range, $1.5-10~{}\text{cm}$
accuracy.

The sensors are mounted side by side on the gantry, as shown in Figure 1.
Here, the sensors detect the traffic in the center of the intersection from a
height of $7~{}\text{m}$. It is worth mentioning that the cameras and LiDARs
are spatiotemporally calibrated. For the temporal calibration, we synchronized
the sensors with a Network Time Protocol (NTP) time server, for the extrinsic
calibration between the cameras and the LiDARs, we used a targetless extrinsic
calibration method, which was inspired by [19].

###  III-B Data Selection and Annotation

We select the data based on interesting and challenging traffic scenarios like
left, right, and U-turns, overtaking maneuvers, tail-gate events, and lane
merge scenarios. Furthermore, we take highly diverse and dense traffic
situations into account, so that we get an average of over $15$ road users per
frame. To cover diverse weather and light conditions in our A9-I Dataset, it
consists of $25\%$ nighttime data including heavy rain, and $75\%$ daytime
data with sunny and cloudy weather conditions. This enables a good performance
of the detector even in challenging weather conditions.

We record camera data at $25\text{ Hz}$ and LiDAR data at $10\text{ Hz}$ into
rosbag files. Then we extract the raw data and synchronize the camera and
LiDAR frames at $10\text{Hz}$ based on timestamps. Based on the raw data of
the LiDAR point clouds, 3D box labels were created by experts. As all four
sensors are cross-calibrated, we can also use these 3D box labels from the
point cloud to evaluate monocular 3D object detection algorithms. Since the
labeling quality of the test sequence is very important, it was reviewed
multiple times by us. Here, we improve the labeling quality by using our
preliminary proAnno labeling toolbox [20].

###  III-C Data Structure

Our dataset is divided into subsets S1 through S4, which contain continuous
camera and labeled LiDAR recordings. Set S1 and S2 are each 30 seconds long
and demonstrate a daytime scenario at dusk. A 120-second long sequence during
daytime and sunshine can be found in sequence S3. Sequence S4 contains
30-second data recording at night and in heavy rain. The file structure is
given below:

.1 a9-intersection-dataset. .2 a9_dataset_r02_s01. .3 point_clouds. .4
s110_lidar_ouster_north. .5 timestamp_sensor_id.pcd. .4
s110_lidar_ouster_south. .5 timestamp_sensor_id.pcd. .3 images. .4
s110_camera_basler_south1_8mm. .5 timestamp_sensor_id.jpg. .4
s110_camera_basler_south2_8mm. .5 timestamp_sensor_id.jpg. .3 labels. .4
s110_lidar_ouster_north. .5 timestamp_sensor_id.json. .4
s110_lidar_ouster_south. .5 timestamp_sensor_id.json. .2 a9_dataset_r02_s02.
.2 a9_dataset_r02_s03. .2 a9_dataset_r02_s04.

All labeled data is in OpenLABEL format [21]. OpenLABEL files are stored in
.json format. One file contains all labeled objects of a single frame with
32-bit long unique identifiers (UUIDs), the position, dimensions, rotation,
and the attributes like the occlusion level, the body color, the number of
trailers, the specific object type, and the number of 3D points. Furthermore,
a frame contains properties like the exact epoch timestamp, the weather type,
the time of day, and the corresponding image and point cloud file names. In
OpenLABEL the label files also contain the calibration data – intrinsic and
extrinsic information.

[Object classes including occlusion level. ] [Labeled 3D box labels per frame.
] [Rotations of 3D box labels. ]

Figure 2: The A9-I Dataset consists of $38,045$ 3D box labels after data
fusion, where the CAR class is dominant and $78.2\%$ of the objects are
occlusion-free. The 3D box labels show different rotations, which is due to
the road traffic in an intersection.

[Average number of points per class. ] [Number of points grouped by distance.
] [Number of points in a 3D box label. ]

Figure 3: As expected, there is a causality between the dimensions of road
users and the number of points. Most classes have the highest number of points
at a distance between $10$ to $30$ meters. On average, each 3D box label
contains $73.52$ points.

[Average track length per class. ] [Distribution of the several track lengths.
] [Visualization of tracks in BEV. ]

Figure 4: The 3D box label of each traffic participant has always received the
same tracking ID in successive frames in a sequence. The average track length
is $24.18~{}\text{m}$. In total, our 3D box labels have a track length of
$12.23~{}\text{km}$.

We suggest a split into training ($80\%$), validation ($10\%$), and test set
($10\%$). The test set is made up of a continuous sequence with track IDs, as
well as randomly sampled frames from four different scenarios and daytimes. We
sample frames using stratified sampling to create a balanced dataset among
sensor types, weather scenarios, and day times. To prevent overfitting, we do
not publish our test set labels.

###  III-D Data Statistics

In total, we provide $4,800$ labeled LiDAR point cloud frames sampled from
four different sequences. Here, $57,406$ 3D objects ($506$ unique objects)
were annotated with $273,861$ object attributes. After fusing the labels from
both LiDARs we get $38,045$ registered 3D objects ($482$ unique objects) with
$171,045$ attributes. The following statistics refer to the fusion result with
the complete dataset inclusive of training, validation, and test set. In Table
II, we can see an overview of the registered 3D box labels.

TABLE II: The total number of labels, average dimensions in meters, and the average number of 3D LiDAR points among all classes. Class | #Labels |  $\diameter$Length |  $\diameter$Width |  $\diameter$Height |  $\diameter$Points
---|---|---|---|---|---
Car | 22,773 | 4.27 | 1.91 | 1.59 | 34.03
Truck | 2,704 | 3.11 | 2.90 | 3.43 | 116.87
Trailer | 3,177 | 10.19 | 3.12 | 3.65 | 328.36
Van | 4,353 | 6.35 | 2.52 | 2.47 | 86.11
Motorcycle | 734 | 1.90 | 0.83 | 1.60 | 21.23
Bus | 908 | 12.65 | 2.95 | 3.27 | 222.36
Pedestrian | 2,507 | 0.80 | 0.73 | 1.72 | 14.98
Bicycle | 663 | 1.57 | 0.74 | 1.72 | 20.95
Emergency Vehicle | 142 | 6.72 | 2.35 | 2.35 | 58.95
Other | 84 | 5.28 | 1.92 | 1.90 | 128.17
Total | 38,045 | - | - | - | 103.20

A deep dive into distribution of the labels of our A9-I Dataset is provided in
Figure 2. Here, the distribution of the ten object classes is shown. The
vehicle class CAR is dominant, followed by the classes TRUCK, TRAILER, VAN,
and PEDESTRIAN, which occur in roughly the same order of magnitude. The
classes MOTORCYCLE, BUS, BICYCLE, EMERGENCY VEHICLES, and OTHER are present in
a slightly smaller number. Since we have annotated the occlusion level for
each 3D box label, we come to the result that $78.2\%$ were classified as
NOT_OCCLUDED, $16.1\%$ as PARTIALLY_OCCLUDED, $0.8\%$ as MOSTLY_OCCLUDED, and
$4.9\%$ were classified as UNKNOWN (not labeled). It can also be seen that
most of the labeled frames contain between $15$ and $20$ labeled 3D boxes. In
100 frames, there are even between $45$ and $50$ labeled 3D objects.
Furthermore, the A9-I includes significantly more variations in the maneuvers
of road users at the intersection, as compared to our previous work [14]. We
can see three peaks where vehicles are moving to the south, north and east
direction of the intersection. Vehicles moving between south and north are
indicated by the peaks around $90$ and $270$ degrees. The smaller peaks
adjacent to the main peaks correspond to turning maneuvers, such as right or
left turns.

The labels are based on the LiDAR point clouds. In Figure 3, we performed a
detailed analysis of the points concerning the labeled classes, of the
individual distances of the points concerning the labeled classes, and of the
distribution of the points. Firstly, as expected, the correlation between the
average number of points and the average size of the class can be observed.
Here, the TRAILER class, which has the highest height, also has the highest
average number of points, followed by the BUS class, which is the longest.
Conversely, the PEDESTRIAN class, which has the smallest size, has the lowest
average number of points. Second, in general, due to the elevated position of
the LiDARs, the field of view only starts to have an effect from about
$10~{}\text{m}$ onwards. Most classes have the highest number of points at a
distance between $10~{}\text{m}$ to $30~{}\text{m}$. Interestingly, the class
TRAILER has the highest average number of points at a distance between
$15~{}\text{m}$ and $20~{}\text{m}$. With increasing distance, the average
number of points is naturally declining. Lastly, all 3D box labels have a
total of $2,797,112$ points. According to the distribution of the number of
points per 3D box label, most of the boxes have a maximum of about $50$
points. However, the 3D box labels have on average $73.52$ points per object.

In addition to the statistics about the labels and the underlying point
clouds, we also analysed the calculated tracks, see Figure 4. We were able to
determine these trivially since the same tracking ID was selected for each
consecutive frame when marking the 3D box labels. The average track length in
our A9-I Dataset is $24.18~{}\text{m}$. Here, the class BUS is very dominant
with an average track length of $75~{}\text{m}$. The reason for this is
because, firstly, the buses are very visible, and secondly, completely cross
the intersection. All in all, the full dataset contains $506$ unique objects
(3D box labels) with a total track length of $12.23~{}\text{km}$ with a
maximum track length of $162.87~{}\text{m}$. Thus, our A9 Intersection Dataset
can also be used to handle issues regarding tracking that are addressed by
[22].

###  III-E A9-Devkit

To work with our A9-I Dataset, we provide the A9 Development Kit:
https://github.com/providentia-project/a9-dev-kit. It contains a dataset
loader to load images, point clouds, labels, and calibration data.
Furthermore, we provide a converter from OpenLABEL to multiple different
dataset formats like KITTI, COCO, YOLO, and the other way round. We follow the
.json-based OpenLABEL standard [21] from the ASAM organization for the label
structure. Some pre-processing scripts transform and filter the raw point
cloud .pcd ASCII data into binary data to reduce the file size and make it
compatible with point cloud loaders. In addition, a point cloud registration
module can merge multiple point clouds to increase the point density. Finally,
we provide a data visualization module to project the point clouds and labels
into the camera images.

##  IV EVALUATION

TABLE III: Evaluation results on the A9-I Dataset test set (N=North, S=South,
EF=Early Fusion, LF=Late Fusion).
We report the<EMAIL_ADDRESS>results for the following six classes: Car, Truck, Bus, Motorcycle, Pedestrian, Bicycle. According to [23], we crop the dataset into three subsets: A9-I-south1 (Camera_South1), A9-I-south2 (Camera_South2), and A9-I-full (Camera_full).  FOV | Model | Sensor Modality | $\boldsymbol{mAP_{3D}}$
---|---|---|---
|  |  | Easy | Mod. | Hard | Overall
Camera_S1 | MonoDet3D[23] | Camera_S1 | 43.27 | 13.28 | 2.16 | 19.57
Camera_S1 | PointPillars⋆[24] | LiDAR_N | 76.19 | 34.58 | 30.00 | 46.93
Camera_S1 | PointPillars⋆[24] | LiDAR_S | 46.35 | 41.05 | 24.16 | 37.18
Camera_S1 | PointPillars⋆[24] | EF(LiDAR_N + LiDAR_S) | 75.81 | 47.66 | 42.16 | 55.21
Camera_S1 | InfraDet3D[23] | LF(Camera_S1 + EF(LiDAR_N + LiDAR_S)) | 67.08 | 31.38 | 35.17 | 44.55
Camera_S2 | MonoDet3D[23] | Camera_S2 | 16.82 | 27.87 | 26.67 | 23.78
Camera_S2 | PointPillars⋆[24] | LiDAR_N | 45.26 | 27.26 | 26.24 | 32.92
Camera_S2 | PointPillars⋆[24] | LiDAR_S | 26.27 | 38.24 | 13.16 | 25.89
Camera_S2 | PointPillars⋆[24] | EF(LiDAR_N + LiDAR_S) | 38.92 | 46.60 | 43.86 | 43.13
Camera_S2 | InfraDet3D[23] | LF(Camera_S2 + EF(LiDAR_N + LiDAR_S)) | 58.38 | 19.73 | 33.08 | 37.06
Camera_full | MonoDet3D[23] | LF(Camera_S1 + Camera_S2) | 19.05 | 24.12 | 25.55 | 22.91
Camera_full | PointPillars⋆[24] | LiDAR_N | 76.04 | 26.03 | 20.60 | 40.89
Camera_full | PointPillars⋆[24] | LiDAR_S | 38.82 | 32.83 | 10.93 | 27.53
Camera_full | PointPillars⋆[24] | EF(LiDAR_N + LiDAR_S) | 70.53 | 44.20 | 39.04 | 51.25
Camera_full | InfraDet3D[23] | LF(LF(Camera_S1+Camera_S2) + LF(LiDAR_N+LiDAR_S)) | 47.27 | 26.15 | 19.71 | 31.04
Camera_full | InfraDet3D[23] | LF(LF(Camera_S1+Camera_S2) + EF(LiDAR_N+LiDAR_S)) | 64.30 | 23.83 | 26.05 | 38.06
⋆PointPillars inference score threshold is set to 0.3. |

(a) MonoDet3D

(b) PointPillars

(c) InfraDet3D

Figure 5: Qualitative results on the test sequence of the three baselines:
MonoDet3D (camera-only), PointPillars (LiDAR-only) and InfraDet3D (fusion)
during the day (top row) and the night time (bottom row). Detections for
MonoDet3D and PointPillars are colored by their class color. The fusion
results of the InfraDet3D model are shown in red (fused detections), green
(unmatched camera detections), and blue (unmatched LiDAR detections).

In our study, we conducted a comparative analysis of monocular camera and
LiDAR 3D object detection with early and late fusion. In our first evaluation
experiment, we used our MonoDet3D [23] 3D object detector that takes camera
images as input. It transforms the 2D instance masks into 3D bottom contours
by using extrinsic calibration data. Our augmented L-Shape-Fitting algorithm
extracts the dimensions and calculates the rotation for each object. In our
second experiment, we used PointPillars [24] and trained the model from
scratch on all classes of our camera field of views Camera_south1,
Camera_south2, and full. In the last experiment, we evaluate our multi-modal
InfraDet3D [23] detector, which incorporates a late fusion approach,
leveraging the Hungarian algorithm to establish correspondences between
detections obtained from the MonoDet3D and PointPillars baselines. For all
these experiments, we provide post-processing scripts in our A9-Devkit for
early data fusion, and cropping the point cloud labels to fit the mentioned
field of view.

We evaluated each detector on three difficulty levels Easy, Moderate, and
Hard, see Table III. The Hard category contains objects with a distance over
$50~{}\text{m}$, objects that are mostly occluded, or objects that have less
than 20 points within the 3D box. Partially occluded objects with a distance
of $40~{}\text{to}~{}50~{}\text{m}$, and 20 to 50 points are part of the
Moderate category. Lastly, the Easy category contains objects that are not
occluded, less than $40~{}\text{m}$ away, and contain more than 50 points. As
a quantitative metric, we used the mean Average Precision (mAP) to evaluate
the performance. The overall mAP is the average of Easy, Moderate, and Hard.

The advantage of using a monocular setup is a better detection of small
objects such as pedestrians. On the other side, a LiDAR detector can detect
objects during nighttime. The combination of LiDAR and the camera through late
fusion techniques can significantly enhance the overall performance. In this
work, we were able to confirm this assumption in our evaluation. We achieved
the best detection results with the LiDAR_North modality and the InfraDet3D
model in the Easy difficulty level. Interestingly, the early fusion approach
with PointPillars consistently achieves the best performance in all subsets at
Moderate difficulty level. The better performance of PointPillars and
InfraDet3D over the MonoDet3D shows the strengths of the LiDAR sensor in
comparison to a camera. Mostly, the late fusion of LiDAR and the camera
provided better overall results than a single LiDAR detector. Moreover, the
combination of early fusion between LiDAR sensors with camera sensors via late
fusion, which combines the advantages of both sensor modalities, gives
consistently robust results. A visual representation of the qualitative
results is provided in Figure 5.

##  V CONCLUSIONS

In this work we extended the A9 Dataset with labeled data of an intersection.
We provided 3D box labels from elevated road side sensors. Two synchronized
cameras and LiDARs were used to record challenging traffic scenarios. Our data
was labeled by experienced experts. As all sensors were calibrated to each
other, we can use the 3D bounding box point cloud labels to perform Monocular
3D object detection. In total, our dataset contains $4.8\text{k}$ RGB images
and $4.8\text{k}$ LiDAR point cloud frames with $57.4\text{k}$ high-quality
labeled 3D boxes, partitioned into ten object classes of traffic participants.
We offered a comprehensive statistics of the labels including their occlusion
levels, the number of points grouped by class category and distance, and an
extensive analysis of the labeled tracks. In our evaluation experiments, we
provided three baselines for the perception task of 3D object detection: A
camera, a LiDAR and a multi-modal camera-LiDAR combination. With these
experiments, we were able to show the potential of our dataset for your 3D
perception tasks.

For future work, we plan to create and publish more ground truth labels based
on the presented camera images which can support more evaluation methods for
our data fusion algorithm. Furthermore, the publication of further labeled
sensor data with specific traffic scenarios, e.g. accidents, as well as the
usage of other sensor modalities is also on our agenda.

## ACKNOWLEDGMENT

This research was supported by the Federal Ministry of Education and Research
in Germany within the project AUTOtech.agil, Grant Number: 01IS22088U. We
thank Venkatnarayanan Lakshminarasimhan and Leah Strand for the collective
work on the A9 Intersection (A9-I) Dataset.

## References

  * [1] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” _The International Journal of Robotics Research_ , vol. 32, no. 11, pp. 1231–1237, 2013.
  * [2] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele, “The cityscapes dataset for semantic urban scene understanding.” [Online]. Available: https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Cordts_The_Cityscapes_Dataset_CVPR_2016_paper.pdf
  * [3] R. Krajewski, J. Bock, L. Kloeker, and L. Eckstein, “The highd dataset: A drone dataset of naturalistic vehicle trajectories on german highways for validation of highly automated driving systems,” in _2018 IEEE Intelligent Transportation Systems Conference_. Piscataway, NJ: IEEE, 2018, pp. 2118–2125.
  * [4] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 11 621–11 631.
  * [5] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine _et al._ , “Scalability in perception for autonomous driving: Waymo open dataset,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 2446–2454.
  * [6] J. Bock, R. Krajewski, T. Moers, S. Runde, L. Vater, and L. Eckstein, “The ind dataset: A drone dataset of naturalistic road user trajectories at german intersections,” in _2020 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2020, pp. 1929–1934.
  * [7] R. Krajewski, T. Moers, J. Bock, L. Vater, and L. Eckstein, “The round dataset: A drone dataset of road user trajectories at roundabouts in germany,” in _2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)_. Piscataway, NJ: IEEE, 2020, pp. 1–6.
  * [8] T. Moers, L. Vater, R. Krajewski, J. Bock, A. Zlocki, and L. Eckstein, “The exid dataset: A real-world trajectory dataset of highly interactive highway scenarios in germany,” in _2022 IEEE Intelligent Vehicles Symposium (IV)_. Piscataway, NJ: IEEE, 2022, pp. 958–964.
  * [9] L. Gressenbuch, K. Esterle, T. Kessler, and M. Althoff, “Mona: The munich motion dataset of natural driving,” in _2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2022, pp. 2093–2100.
  * [10] H. Yu, Y. Luo, M. Shu, Y. Huo, Z. Yang, Y. Shi, Z. Guo, H. Li, X. Hu, J. Yuan _et al._ , “Dair-v2x: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 21 361–21 370.
  * [11] H. Wang, X. Zhang, Z. Li, J. Li, K. Wang, Z. Lei, and R. Haibing, “Ips300+: a challenging multi-modal data sets for intersection perception system,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 2539–2545.
  * [12] X. Ye, M. Shu, H. Li, Y. Shi, Y. Li, G. Wang, X. Tan, and E. Ding, “Rope3d: The roadside perception dataset for autonomous driving and monocular 3d object detection task,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 21 341–21 350.
  * [13] S. Busch, C. Koetsier, J. Axmann, and C. Brenner, “Lumpi: The leibniz university multi-perspective intersection dataset,” in _2022 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2022, pp. 1127–1134.
  * [14] C. Creß, W. Zimmer, L. Strand, M. Fortkord, S. Dai, V. Lakshminarasimhan, and A. Knoll, “A9-dataset: Multi-sensor infrastructure-based dataset for mobility research,” in _2022 IEEE Intelligent Vehicles Symposium (IV)_ , 2022, pp. 965–970.
  * [15] C. Creß, Z. Bing, and A. C. Knoll, “Intelligent transportation systems using external infrastructure: A literature survey.” [Online]. Available: https://arxiv.org/pdf/2112.05615
  * [16] J. Bock, R. Krajewski, T. Moers, S. Runde, L. Vater, and L. Eckstein, “The ind dataset: A drone dataset of naturalistic road user trajectories at german intersections,” in _2020 IEEE Intelligent Vehicles Symposium (IV)_. Piscataway, NJ: IEEE, 2020, pp. 1929–1934.
  * [17] A. Krämmer, C. Schöller, D. Gulati, V. Lakshminarasimhan, F. Kurz, D. Rosenbaum, C. Lenz, and A. Knoll, “Providentia-a large-scale sensor system for the assistance of autonomous vehicles and its evaluation,” _Journal of Field Robotics_ , pp. 1156–1176, 2022.
  * [18] “Projekt providentia++,” 2022. [Online]. Available: https://innovation-mobility.com/projekt-providentia/
  * [19] Y. Chongjian, X. Liu, X. Hong, and F. Zhang, “Pixel-level extrinsic self calibration of high resolution lidar and camera in targetless environments,” _IEEE Robotics and Automation Letters_ , vol. PP, pp. 1–1, 07 2021.
  * [20] W. Zimmer, A. Rangesh, and M. Trivedi, “3d bat: A semi-automatic, web-based 3d annotation toolbox for full-surround, multi-modal data streams,” in _2019 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2019, pp. 1816–1821.
  * [21] N. Hagedorn, “OpenLABEL Concept Paper.” [Online]. Available: https://www.asam.net/project-detail/asam-openlabel-v100/
  * [22] L. Strand, J. Honer, and A. Knoll, “Systematic error source analysis of a real-world multi-camera traffic surveillance system,” in _2022 25th International Conference on Information Fusion (FUSION)_. IEEE, 7/4/2022 - 7/7/2022, pp. 1–8.
  * [23] W. Zimmer, J. Birkner, M. Brucker, H. T. Nguyen, S. Petrovski, B. Wang, and A. C. Knoll, “Infradet3d: Multi-modal 3d object detection based on roadside infrastructure camera and lidar sensors,” _arXiv preprint arXiv:2305.00314_ , 2023.
  * [24] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom, “Pointpillars: Fast encoders for object detection from point clouds,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 12 697–12 705.