# Restricted Hidden Cardinality Constraints in Causal Models

Beata Zjawin Perimeter Institute for Theoretical Physics 31 Caroline St. N,
Waterloo, Ontario, Canada, N2L 2Y5 International Centre for Theory of Quantum
Technologies, University of Gdańsk, 80-308 Gdańsk, Poland
<EMAIL_ADDRESS>Perimeter Institute for Theoretical Physics 31
Caroline St. N, Waterloo, Ontario, Canada, N2L 2Y5 Elie Wolfe Robert W.
Spekkens Perimeter Institute for Theoretical Physics 31 Caroline St. N,
Waterloo, Ontario, Canada, N2L 2Y5

###### Abstract

Causal models with unobserved variables impose nontrivial constraints on the
distributions over the observed variables. When a common cause of two
variables is unobserved, it is impossible to uncover the causal relation
between them without making additional assumptions about the model. In this
work, we consider causal models with a promise that unobserved variables have
known cardinalities. We derive inequality constraints implied by
$d$-separation in such models. Moreover, we explore the possibility of
leveraging this result to study causal influence in models that involve
quantum systems.

##  1 Introduction

In causal studies, systems of variables are described by _causal models_ [21,
25], which are composed of two elements: (i) the graphical representation of
relationships between variables in a model, encoded in a directed acyclic
graph, and (ii) the mathematical description of conditional probability
distribution of each variable given its causal parents. When a causal model
involves _hidden_ (i.e., unobserved) variables, any characterization of the
model verifiable by observations should only include observed variables.
Therefore, one of the objectives of causal inference is to eliminate all
hidden variables from inequalities and equalities that describe the model. In
principle, this can be achieved using the Tarski–Seidenberg quantifier
elimination method [14]. However, its complexity is such that only models with
few variables can be solved using this technique, hence the reason for the
many attempts to simplify the problem. For example, one can derive a set of
incomplete constraints on probability distributions in latent causal models
[10, 28, 17, 23, 27]. As these models impose highly nontrivial constraints on
the probability distributions, Chaves et al. [7] derive constraints expressed
using _entropies_. Interestingly, the entropic framework originates from the
foundations of quantum mechanics [6, 13]. Furthermore, causal effects in
latent causal structures can be estimated by limiting the causal influence
between a confounder and its children [24, 26]. Unfortunately, these methods
only provide the necessary, but not sufficient, constraints for complete
characterization of a causal model.

The field of causal inference is relevant to a wide range of disciplines.
Curiously, quantum correlations seem to have no satisfactory explanation via
classical causal models. The most famous example is Bell’s theorem [3]. The
underlying causal structure of a Bell experiment was investigated in [29]. It
was shown that it is impossible to explain Bell-inequality violations using a
classical causal model without fine-tuning. As this result only holds for the
framework of classical causal inference, it appears that quantum theory calls
for a quantum generalization of this formalism. Many works attempt to describe
causality in quantum systems by generalizing classical concepts to the quantum
formalism, for example: quantum Reichenbach’s principle [2], quantum Bayes’
theorem [19] and quantum randomized trial [22]. We suggest that studying
classical causal models with hidden variables can provide a valuable insight
into quantum causality.

In this work, we analyze causal models with hidden variables that exhibit
$d$-separation relations, under the assumption that the cardinality of the
unobserved variables is known. We derive inequality constraints on the
probability distribution over observed variables in such scenarios. We discuss
possible applications of these constraints, both in classical causal inference
and in quantum causality.

### Outline

Section 2 reviews the framework of graphical causal models and the definition
of nonnegative rank. Section 3 contains the main results. It begins with
description of nonnegative rank as a measure of dependence. Afterwards, it
provides constraints implied by $d$-separation in causal models with hidden
variables. Section 4 describes applications of constraints derived in Section
3. Section 5 shows that our constraints are sufficient, but not necessary for
identifying direct causal influence. Finally, Section 6 discusses possible
generalisations of our constraints to the quantum formalism. Lastly, Section 7
concludes the previous sections.

##  2 Preliminaries

### Graphical Causal Models

In this section, we briefly recall basic notions and important results from
graph theory and causal inference.

Consider a system of _causally_ related events. We represent each event by a
random variable and we denote the set of variables that corresponds to the
system of events as $\mathbf{V}$. In general, $\mathbf{V}$ can contain both
observed and unobserved variables.

The variables in $\mathbf{V}$, and relations between them, can be represented
by a graph. Graphs consist of vertices and edges. We call a graph a _causal
graph_ , if the edges that connect the vertices are directed and correspond to
causal relations. For example, a causal graph $X\rightarrow Y$ indicates that
the variable $X$ has a direct causal influence on $Y$. Here, $X$ is said to be
a parent of $Y$. We denote all parents of a random variable $V$ in a graph
$\mathscr{G}$ by $pa_{\mathscr{G}}(V)$.

In causal inference studies, relationships between variables are illustrated
by _Directed Acyclic Graphs_ (DAGs) 111A graph is directed, if all its edges
are directed (they point in a certain direction which is marked with a single
arrow). A directed graph is acyclic if there is no directed path from any
variable back to itself.. A _causal model_ consists of a DAG $\mathscr{G}$
over a set of variables $\mathbf{V}$ and a joint probability distribution
$P_{\mathbf{V}}$. Causal models provide not only a probabilistic, but also a
causal interpretation of the relationships between variables in a system.

Properties of a DAG that represents variables in $\mathbf{V}$ imply properties
of the distribution $P_{\mathbf{V}}$. The essential one is the Markov
condition which states that $P_{\mathbf{V}}$ can be decomposed as

$\displaystyle P_{\mathbf{V}}=\prod_{V\in{\mathbf{V}}}P(V\mid
pa_{\mathscr{G}}(V)).$ (1)

Naturally, this condition can be tested only if all variables in $\mathbf{V}$
are observed. It means that given a DAG $\mathscr{G}$ and a probability
distribution $P_{\mathbf{V}}$, with all variables in $\mathbf{V}$ being
observed, the Markov condition enables us to test the _compatibility_ of
$\mathscr{G}$ and $P_{\mathbf{V}}$. However, when some variables in
$\mathbf{V}$ are unobserved, testing compatibility is a hard task, as we
already mentioned in the introduction.

A different property of DAGs that is important in causal inference studies is
$d$-separation. Before we formally define it, let us recall the notion of
blocking a path.

Let $p$ be a path in a DAG $\mathscr{G}$. We say that $p$ is blocked by a set
of vertices $\mathbf{Z}$ if and only if

  * •

$p$ contains a fork structure ($X\leftarrow Z\rightarrow Y$) or a chain
structure ($X\rightarrow Z\rightarrow Y$) in which the middle vertex $Z$ is in
$\mathbf{Z}$, or

  * •

$p$ contains a collider structure ($X\rightarrow Z\leftarrow Y$) in which
neither the middle vertex $Z$ nor any of its descendants is in $\mathbf{Z}$.

######  Definition 1 ($d$-separation)

$\mathbf{X}$ and $\mathbf{Y}$ are said to be _d-separated_ by a set
$\mathbf{Z}$, $(\mathbf{X}\perp_{d}~{}\mathbf{Y}|\mathbf{Z})$, if and only if
$\mathbf{Z}$ blocks every path from a vertex in $\mathbf{X}$ to a vertex in
$\mathbf{Y}$.

Any $d$-separation relation $(\mathbf{X}\perp_{d}~{}\mathbf{Y}|\mathbf{Z})$ in
$\mathscr{G}$ implies that $\mathbf{X}$ and $\mathbf{Y}$ are conditionally
independent given $\mathbf{Z}$, ($\mathbf{X}\perp\mathbf{Y}|\mathbf{Z}$), in
the corresponding distribution $P_{\mathbf{X}\mathbf{Y}\mathbf{Z}}$.

Notice that testing whether conditional independence relations implied by
$\mathscr{G}$ hold is only possible when all variables in $\mathbf{X}$,
$\mathbf{Y}$ and $\mathbf{Z}$ are observed. When dealing with unobserved
variables, conditional independence relations are sometimes impossible to
evaluate.

### Nonnegative factorization

We now review the definitions of nonnegative matrix factorization and
nonnegative rank as they will play an important role in our results.

Nonnegative matrix factorization is a group of algorithms used for factorizing
a nonnegative matrix (matrix with no negative entries) into two or more
nonnegative matrices. The problem of performing a nonnegative matrix
factorization is not always tractable, nevertheless numerical approximations
of this task exist. This method is used in many fields of science, such as
astronomy, bioinformatics and machine learning, as they study data with
positive entries [4, 5, 20]. Furthermore, the nonnegative rank, which we
define below, plays a critical role in communication theory as it
characterizes randomized communication complexity and randomized correlation
complexity [16, 30].

Formally, given a nonnegative matrix $\mathbf{M}\in\mathbb{R}_{+}^{n\times
m}$, a nonnegative factorization of $\mathbf{M}$ means factorizing it into

$\mathbf{M}=\mathbf{U}\mathbf{V},$

where $\mathbf{U}\in\mathbb{R}_{+}^{n\times r}$ and
$\mathbf{V}\in\mathbb{R}_{+}^{r\times m}$ are nonnegative. The nonnegative
rank of $\mathbf{M}$, $rank_{+}[\mathbf{M}]$, is the smallest number $r$ for
which such a factorization can be found. Equivalently, the nonnegative rank of
$\mathbf{M}$ is the smallest number for which $\mathbf{M}$ can be decomposed
into a positive sum over $r$ nonnegative rank-1 matrices [8]. For the purpose
of this paper, we formalize the definition of the nonnegative rank in a
following way.

######  Definition 2 (nonnegative rank)

Let $\mathbf{P}_{XY}$ be a ($|X|\times|Y|$)-dimensional nonnegative matrix.
The nonnegative rank of $\mathbf{P}_{XY}$ is defined as

$\displaystyle
rank_{+}[\mathbf{P}_{XY}]\equiv\min\\{r\mid\exists(\mathbf{P}_{X}^{(1)},\mathbf{P}_{Y}^{(1)}),\dots,(\mathbf{P}_{X}^{(r)},\mathbf{P}_{Y}^{(r)})~{}~{}\mathbf{P}_{XY}=\sum_{z=1}^{r}\mathbf{P}_{X}^{(z)}\mathbf{P}_{Y}^{(z)}\\},$
(2)

where $\mathbf{P}_{X}^{(z)}$ is a nonnegative $|X|$-dimensional row vector and
each $\mathbf{P}_{Y}^{(z)}$ is a nonnegative $|Y|$-dimensional column vector.

The matrices in the set $\\{\mathbf{P}_{X}^{(z)}\mathbf{P}_{Y}^{(z)}\\}_{z}$
have rank 1, hence this definition coincides with the standard definition of
the nonnegative rank.

##  3 Rank constraints for causal models

In this section, we explore a relation between the nonnegative rank of a
probability distribution over two variables, and the causal relation between
these variables.

### Nonnegative rank as a measure of dependence

We start by making a trivial observation that any joint probability
distribution over two variables can be represented as a nonnegative matrix
$\mathbf{P}_{XY}$, with entries
$\mathbf{P}_{xy}=\mathbf{P}_{X=x,Y=y}=P(X=x,Y=y)$. Notice that although we
donote $\mathbf{P}_{xy}$ using the bold font, it is a single element of the
matrix $\mathbf{P}_{XY}$. If $X$ and $Y$ are independent, $\mathbf{P}_{XY}$
can be factorized as $\mathbf{P}_{XY}=\mathbf{P}_{X}\mathbf{P}_{Y}$. Then, it
follows from Definition 2 that $\mathbf{P}_{XY}$ has nonnegative rank $1$.
Conversely, as the correlation between $X$ and $Y$ increases, so does the
nonnegative rank of their distribution. It means that the nonnegative rank of
a probability distribution can be interpreted as a measure of dependence
between two variables.

Consider an example in which the dependence between $X$ and $Y$ is only due to
a third variable, denoted by $Z$. Let $Z$ take $r$ different values, i.e., the
cardinality of $Z$ is $r$, which we denote $|Z|=r$. Throughout this paper we
take $r$ to be the minimum cardinality of $Z$ for which $X$ and $Y$ are
conditionally independent. If $X$ and $Y$ are conditionally independent given
$Z$, the probability distribution must factorize as:

$\displaystyle\mathbf{P}_{XY}=\sum_{z=1}^{r}\mathbf{P}_{X|Z=z}\mathbf{P}_{Y|Z=z}\mathbf{P}_{Z=z}.$
(3)

Here, $\mathbf{P}_{X|Z=z}$ ($\mathbf{P}_{Y|Z=z}$) is a conditional probability
distribution over $X$ ($Y$) given that $Z$ takes a particular value $Z=z$.
This factorization is equivalent to the one given in Eq. (2), with
$rank_{+}[\mathbf{P}_{XY}]=r$. We summarize this result in the following
Proposition.

###### Proposition 1

Let $\mathbf{P}_{XY}\in\mathbb{R}_{+}^{n\times m}$ represent a joint
probability distribution on random variables $X$ and $Y$ that have finite
cardinalities, $|X|=n$ and $|Y|=m$. The following are equivalent:

  1. 1.

There exists a nonnegative factorization of $\mathbf{P}_{XY}$ with
$rank_{+}[\mathbf{P}_{XY}]=r$.

  2. 2.

$\mathbf{P}_{XY}$ can be decomposed as in Eq. (3) with $|Z|=r$.

###### Proof 1

1 $\implies$ 2: Suppose $\mathbf{P}_{XY}$ factorizes as in Eq. (2), with
$\mathbf{P}_{X}^{(z)}$ and $\mathbf{P}_{Y}^{(z)}$ being unnormalized
probability distributions. Define the sums

$Q^{(z)}_{X}=\sum_{x}\mathbf{P}_{X=x}^{(z)}\,,\;\;\;\;\;\;\;\;\;Q^{(z)}_{Y}=\sum_{y}\mathbf{P}_{Y=y}^{(z)}\,.$

Notice that the set of products $\\{Q^{(z)}_{X}Q^{(z)}_{Y}\\}_{z}\,$ form a
valid probability distribution. We denote it as

$\displaystyle\begin{split}\mathbf{P}_{Z=z}&=Q^{(z)}_{X}Q^{(z)}_{Y},\\\
where\,\,\,\,\,\sum_{z}\mathbf{P}_{Z=z}&=\sum_{x,\;y,\;z}\mathbf{P}_{X=x}^{(z)}\mathbf{P}_{Y=y}^{(z)}=\sum_{x,\;y}\mathbf{P}_{xy}=1.\end{split}$

Further, define the following vectors:

$\displaystyle\begin{split}\mathbf{C}_{X}^{(z)}=\frac{\mathbf{P}_{X}^{(z)}}{Q^{(z)}_{X}}\,,\;\;\;\;\;\;\;\;\;\mathbf{C}_{Y}^{(z)}=\frac{\mathbf{P}_{Y}^{(z)}}{Q^{(z)}_{Y}}.\end{split}$

We can now decompose $\mathbf{P}_{XY}$ as

$\mathbf{P}_{XY}=\sum_{z}\mathbf{P}_{X}^{(z)}\mathbf{P}_{Y}^{(z)}=\sum_{z}\mathbf{\mathbf{missing}}C_{X}^{(z)}\,\mathbf{C}_{Y}^{(z)}\,(\,Q^{(z)}_{X}Q^{(z)}_{Y}\,).$

It is now clear that one can define the following conditional probabilities

$\displaystyle\begin{split}\mathbf{P}_{X|Z=z}\equiv\mathbf{C}_{X}^{(z)}\,,\;\;\;\;\;\;\;\;\;\mathbf{P}_{Y|Z=z}\equiv\mathbf{C}_{Y}^{(z)},\end{split}$

and decompose $\mathbf{P}_{XY}$ as

$\mathbf{P}_{XY}=\sum_{z}\mathbf{P}_{X|Z=z}\mathbf{P}_{Y|Z=z}\mathbf{P}_{Z=z},$

which is the factorization from Eq. (3).
2 $\implies$ 1: Assume $\mathbf{P}_{XY}$ can be factorized as in Eq. (3).
Then, since $\mathbf{P}_{Z=z}$ is just a positive number, it follows trivially
that $rank_{+}[\mathbf{P}_{XY}]=r$.

This proposition was also stated in Refs. [15], [18, Appendix 7.5] and [9,
Proposition 2].

Recall that we assumed that $r$ is the minimum cardinality of $Z$ for which
$X$ and $Y$ are conditionally independent. The direction (2 $\implies$ 1) of
Proposition 1 is true due to this assumption. In general, the cardinality of
$Z$ can be greater than $rank_{+}[\mathbf{P}_{XY}]$, which we exploit in the
next Subsection.

### Restricted cardinality constraints

Consider the example from the previous Subsection, where $Z$ is assumed to be
the only source of dependence between $X$ and $Y$. If $Z$ is an observed
variable, it is straightforward to verify whether this hypothesis holds.
However, if $Z$ is unobserved, constraints implied by conditional independence
relations are impossible to directly evaluate. We now show that knowledge
about the cardinality of $Z$ is enough to test conditional independence
relations in some cases.

Let us interpret Proposition 1 from the perspective of causal models. If
$\mathbf{P}_{XY}$ can be factorized as in Eq. (3), then there exist a
hypothetical variable $Z$, with cardinality $|Z|=r$, and probability
distribution $\mathbf{P}_{Z}$ that makes $X$ and $Y$ conditionally
independent. In graphical causal models, the notion of conditional
independence is captured by $d$-separation. If two variables $X$ and $Y$ are
conditionally independent given $Z$, a DAG that illustrates the causal
structure of $X$, $Y$ and $Z$ must satisfy the following two properties.
First, there must be no direct causal influence between $X$ and $Y$ (no direct
arrow form $X$ to $Y$, or from $Y$ to $X$). Second, $Z$ must $d$-separate $X$
and $Y$. Three examples of causal structures that satisfy these properties are
depicted in Figure 1. Circles correspond to observed variables, and dashed
circles correspond to unobserved variables. In Figure 1 (a), $Z$ is a common
cause of $X$ and $Y$. In Figure 1 (b), $Z$ is a causal mediary between $X$ and
$Y$. Finally, in Figure 1 (c), a $d$-separating set
$\mathbf{Z}=\\{Z_{1},Z_{2}\\}$ consists of both a common cause and a causal
mediary. In the first two examples, the cardinalities of the $d$-separating
variables are given by $|Z|$. In the third example, the cardinality of the
$d$-separating set is equal to $|Z_{1}|~{}\times~{}|Z_{2}|$.

$X$$Y$$Z$$(a)$$X$$Z$$Y$$(b)$$X$$Z_{2}$$Y$$Z_{1}$$(c)$ Figure 1:  Examples of
$d$-separating variables.

From Proposition 1, the variable that $d$-separates $X$ and $Y$ must have at
least cardinality $r$, which is given by $rank_{+}[\mathbf{P}_{XY}]$. This
result is summarized in the corollary below.

###### Corollary 1

The nonnegative rank of the matrix $\mathbf{P}_{XY}$ whose components are
$\mathbf{P}_{xy}=P(X=x,Y=y)$ constitutes a lower bound for the cardinality of
a variable $\mathbf{Z}$ that d-separates $\mathbf{X}$ and $\mathbf{Y}$,

$\displaystyle rank_{+}[\mathbf{P}_{XY}]\leq|\mathbf{Z}|.$ (4)

The constraint given in Eq. (4) enables to verify whether conditional
independence relations implied by a DAG hold, even when the $d$-separating
variable is unobserved. The knowledge about $Z$ required to evaluate the bound
is only its cardinality. In the next section, we give examples that use
Corollary 1 to test conditional independence relations in DAGs with latent
variables.

##  4 Applications

In this section, we give two applications of Corollary 1. First, we describe
its potential to improve causal discovery algorithms. In particular, we
consider a scenario in which our constraint witnesses direct causal influence
between two variables in a presence of a confounder. Second, we reprove two
results from communication theory: characterization of randomized correlation
complexity and randomized communication complexity.

### Witnessing Causal Influence

Imagine that two ternary events, $X$ and $Y$ ($|X|=|Y|=3$), are observed to
exhibit some statistical dependence. Additionally, assume they share an
unobserved common cause $U$ which is known to have cardinality $|U|=2$. Two
possible hypotheses regarding the causal relations between $X$, $Y$ and $U$
are illustrated in Fig. 2.

$X$$Y$$U$$(a)$$X$$Y$$U$$(b)$$X$$Y$$U$$Z$$(c)$$X$$Y$$U$$Z$$(d)$ Figure 2:
Witnessing direct causal influence in a presence of a confounder.

In Fig. 2 (a), one believes the statistical dependence is purely due to a
common cause, whereas in Fig. 2 (b), $X$ additionally has a direct causal
influence on $Y$.

In Fig. 2 (a), $U$ $d$-separates $X$ and $Y$. From Corollary 1, we know that

$rank_{+}[\mathbf{P}_{XY}]\leq|U|.$

Therefore, if $rank_{+}[\mathbf{P}_{XY}]>2$, one can discard the hypothesis of
Fig. 2 (a) and conclude that there must be a direct causal influence of $X$ on
$Y$ as in Fig. 2 (b).

As an example, suppose $\mathbf{P}_{XY}$ is given by the probability
distribution from Table 1. In this case, the nonnegative rank of
$\mathbf{P}_{XY}$ is equal to 3. One can certify that
$rank_{+}[\mathbf{P}_{XY}]$ is greater than $|U|$, and only the causal
hypothesis of Fig. 2 (b) is compatible with this probability distribution. In
this example, observing $rank_{+}[\mathbf{P}_{XY}]=3$ _witnesses_ direct
influence between $X$ and $Y$.

|  |  | Y |
---|---|---|---|---
|  | 0 | 1 | 2
| 0 | 0 | $\nicefrac{{1}}{{6}}$ | $\nicefrac{{1}}{{6}}$
X | 1 | $\nicefrac{{1}}{{6}}$ | 0 | $\nicefrac{{1}}{{6}}$
| 2 | $\nicefrac{{1}}{{6}}$ | $\nicefrac{{1}}{{6}}$ | 0
Table 1: An example of a probability distribution $\mathbf{P}_{XY}$, with
$|X|=|Y|=3$.

The example above shows the essence of Corollary 1. It captures the fact that
strong correlation between high cardinality variables, when no direct
influence between them exists, must be explained by a high cardinality common
cause. This dependence was also noticed in Ref. [18, Corollary 1], where the
authors show that the graphs illustrated in Fig. 2 (a) and Fig. 2 (b) may
sometimes be distinguished under the presumption that the confounder has small
cardinality, i.e., $|U|<\,\textrm{min}\,\\{|X|,|Y|\\}.$

Corollary 1 can also be applied to scenarios where the common cause is
_partially_ observed. We define a partially observed set as a set in which
only some variables are observed. An example of a partially observed common
cause $\mathbf{C}=\\{Z,U\\}$ is illustrated in Fig. 2. In Fig. 2 (c), $X$ and
$Y$ share two common causes: an unobserved one, denoted by $U$, and an
observed one, denoted by $Z$. In Fig. 2 (d), $X$ additionally influences $Y$.

In Fig. 2 (c), $Z$ is an observed variable, and we can condition the values of
$X$ and $Y$ on its particular value, $Z=z$. Let $\mathbf{P}^{(z)}_{XY}$ denote
the probability distribution over $X$ and $Y$ conditioned on a particular
value $z$ of $Z$, $\mathbf{P}^{z}_{xy}~{}=~{}P(X=x,Y=~{}y|Z=z)$. After
conditioning, $U$ $d$-separates $X$ and $Y$ (given $Z=z$) in Fig. 2 (c). The
constraint given by Corollary 1 in this case is

$\forall z:\,\,\,rank_{+}[\mathbf{P}^{(z)}_{XY}]\leq|U|.$

Given $\mathbf{P}^{(z)}_{XY}$ for all $z$, any violation of this condition
witnesses direct causal influence between $X$ and $Y$, just like in the
example with an unobserved common cause.

Corollary 1 can also be used to solve a converse problem. Suppose a
distribution $\mathbf{P}_{XY}$ is observed, where variables $X$ and $Y$ are
known to share an unobserved common cause $Z$ (or more generally, are
$d$-separated by $Z$). If nothing is known about $Z$, constraint from Eq. (4)
can be used to lower bound its cardinality.

### Distribution generation

Recall a distribution generation protocol that is often discussed in
communication theory [30, 16]. Suppose two parties, Alice and Bob, are
responsible for generating random variables $X$ and $Y$ respectively, such
that $X$ and $Y$ have a prespecified joint distribution $\mathbf{P}_{XY}$. Our
goal is to quantify how much information must be communicated or shared to
generate $\mathbf{P}_{XY}$.

In the literature about this problem two scenarios have been developed and
studied separately. In the first scenario, Alice and Bob each receive
information from a third party before they generate $X$ and $Y$. We denote the
information they share by $Z$. In this protocol, no communication between
Alice and Bob is allowed. The information $Z$ distributed this way is often
referred to as a random seed. The minimum size of a seed that must be given to
Alice and Bob in order to produce the required distribution $\mathbf{P}_{XY}$
is called the randomized correlation complexity of the joint distribution, and
is denoted $RCorr[\mathbf{P}_{XY}]$. Here, a size of a seed is the number of
bits that is distributed to each party, and is given by $\text{log}_{2}(|Z|)$.
In the second scenario, one-way communication between Alice and Bob is
allowed. After generating $X$, Alice sends a message to Bob through an
intermediary, which we denote $M$. The minimum number of bits Alice needs to
communicate to Bob such that they can produce the target distribution is
called the randomized communication complexity, and is denoted
$RComm[\mathbf{P}_{XY}]$. The number of bits communicated between the parties
is given by $\text{log}_{2}(|M|)$.

It has been shown that $RCorr[\mathbf{P}_{XY}]$ and $RComm[\mathbf{P}_{XY}]$
are equal, and are fully characterized by
$\text{log}_{2}(rank_{+}[\mathbf{P}_{XY}])$ [30]. We now show that given
Corollary 1, this result follows trivially from a causal analysis of the
problem. To begin, we note that in the first scenario, the seed $Z$ is
distributed to both Alice and Bob, hence it can affect both $X$ and $Y$.
Graphically, this scenario corresponds to Fig. 1 (a), where $Z$ is a common
cause of $X$ and $Y$. In the second scenario, the message $M$ can depend on
the random variable $X$, which in turn affects how Bob generates $Y$. This
scenario is illustrated by a causal structure in Fig. 1 (b), with $M$ playing
the role of $Z$ in the figure.

We now observe that $X$ and $Y$ are d-separated by $Z$ and $M$ respectively in
the two scenarios. Corollary 1 implies that

$|Z|\geq rank_{+}[\mathbf{P}_{XY}]\qquad\text{and}\qquad|M|\geq
rank_{+}[\mathbf{P}_{XY}].$

It follows that

$\text{log}_{2}(|Z|)\geq\text{log}_{2}(rank_{+}[\mathbf{P}_{XY}])\qquad\text{and}\qquad\text{log}_{2}(|M|)\geq\text{log}_{2}(rank_{+}[\mathbf{P}_{XY}]).$

Note that $RCorr[\mathbf{P}_{XY}]$ and $RComm[\mathbf{P}_{XY}]$ correspond to
the _minimum_ size of a seed and a message, respectively, needed to generate
$\mathbf{P}_{XY}$. Therefore, they are given by the lower bound of the above
inequalities. Moreover, Corollary 1 does not distinguish a common cause $Z$
from a causal mediary $M$, as they both $d$-separate $X$ and $Y$ in this
example. This result yields both the equivalence of $RCorr[\mathbf{P}_{XY}]$
and $RComm[\mathbf{P}_{XY}]$, and the fact that they are fully characterized
by $\text{log}_{2}(rank_{+}[\mathbf{P}_{XY}])$. This example shows that
adopting a causal perspective can simplify results in communication theory.

Finally, we analyze a more involved protocol, in which Alice and Bob share a
seed $Z_{1}$, and Alice can communicate to Bob through a mediary $Z_{2}$. This
scenario is illustrated in Fig. 1 (c). The cardinality of the d-separating set
$\mathbf{Z}=\\{Z_{1},Z_{2}\\}$ is equal to $|Z_{1}|\times|Z_{2}|$. It follows
from Corollary 1 that

$|Z_{1}|\times|Z_{2}|\geq
rank_{+}[\mathbf{P}_{XY}]\qquad\text{and}\qquad\text{log}_{2}(|Z_{1}|\times|Z_{2}|)\geq\text{log}_{2}(rank_{+}[\mathbf{P}_{XY}]).$

We can intuitively view the constraint on $|Z_{1}|\times|Z_{2}|$ as defining a
trade-off between the amount of information shared through a seed and the
amount transferred using communication. This perspective enables studying
relatively complex problems in which the size of the seed or the capacity of
the communication channel is constrained. Similarly, if one of these
cardinalities is unobserved, information about the other can be used to bound
it, using Corollary 1.

##  5 A Specific Case with Additional Constraints

The inequality given in Corollary 1 gives a sufficient, but not a necessary
condition for identifying causal relations between variables. In this Section,
we give an example of a correlation that satisfies the constraints implied by
the $d$-separation relations in a model, but nevertheless imposes new
constraints that require additional causal mechanisms.

Let us revisit the causal hypotheses illustrated in Fig. 2 (c) and Fig. 2 (d).
Assume $X$, $Y$ and $Z$ are binary observed variables, and $U$ is a binary
unobserved variable. In this scenario, $\mathbf{P}^{(z)}_{XY}$ is a $2\times
2$ matrix and the constraint given in Corollary 1 is _always_ satisfied. This
is the case because $rank_{+}[\mathbf{P}^{(z)}_{XY}]\leq 2$ and $|U|=2$.
However, this is not enough to conclude that the causal relations between $X$,
$Y$, $Z$ and $U$ are those illustrated by Fig. 2 (c). In this Section, we give
an example of a probability distribution $\mathbf{P}^{(z)}_{XY}$ that
satisfies the above conditions, and requires a direct causal influence as in
Fig. 2 (d). First, we consider a specific probability distribution
$\mathbf{P}^{(z)}_{XY}$, with $|X|=|Y|=|Z|=|U|=2$, under the causal hypothesis
of Fig. 2 (c). Then, we show that the causal structure of Fig. 2 (c) implies
some nontrivial constraints that must be satisfied in order to explain the
specific form of $\mathbf{P}^{(z)}_{XY}$.

We start with a reminder that an intuitive way of describing an outcome of a
random variable is with its expectation value. Let
$\mathbf{V}=\\{V_{1},...,V_{n}\\}$ be a set of $n$ binary variables that each
take a value in $\\{-1,1\\}$. The conversion formula between the probability
distribution $P(\mathbf{V})=P(V_{1},...V_{n})$ and the expectation values of
variables in $\mathbf{V}$ can be written as

$\displaystyle
P(V_{1},...V_{n})=\frac{1}{2^{n}}\left(1+\sum_{k=1}^{n}\;\sum_{1\leq
i_{1}<...<i_{k}\leq n}v_{i_{1}}...v_{i_{k}}\langle
V_{i_{1}}...V_{i_{k}}\rangle\right).$ (5)

Let us consider a special case when $X$ and $Y$ are observed to be perfectly
positively correlated for both possible values of $Z$,

$\displaystyle\langle XY|Z=z\rangle=1.$ (6)

Assume that this perfect correlation arises due to the causal mechanism
encoded in Fig. 2 (c). Eq. (6) implies that

$\displaystyle\begin{split}&\text{(a)}\;\;\;\;\langle X|Z,U\rangle=\pm
1;\text{and}\\\ &\text{(b)}\;\;\;\;\langle Y|Z,U\rangle=~{}\langle
X|Z,U\rangle,\end{split}$ (7)

for both values of $Z$ and $U$, that is, $X$ and $Y$ need to depend
deterministically on $Z$ and $U$ (Eq. (7a)) in the same way (Eq. (7b)).

Our goal is to derive constraints on $\langle X|Z=z\rangle$ that arise due to
this condition. If any nontrivial constraints exist, the causal hypothesis of
Fig. 2 (c) explains the perfect correlation between $X$ and $Y$ _only if_ they
are satisfied. Violation of these constraints calls for an additional causal
mechanism (additional to ones in Fig. 2 (c)) that explains the dependence
between $X$ and $Y$.

In order for the constraints on $\langle X|Z=z\rangle$ to be verifiable, they
can only depend on the observed variables. We perform a quantifier elimination
using the known constraints to eliminate the unobserved variable from the set
of inequalities and equalities. We find that either

$\displaystyle\begin{split}&{\langle X|z=1\rangle=\pm\langle X|z=-1\rangle}\\\
&\text{or Min}[\langle X|z=1\rangle,\langle X|z=-1\rangle]=-1\\\ &\text{or
Max}[\langle X|z=1\rangle,\langle X|z=-1\rangle]=1.\end{split}$

It then follows from $\langle XY|Z=z\rangle=1$ that

$\displaystyle\begin{split}&{\langle Y|z=1\rangle=\pm\langle Y|z=-1\rangle}\\\
&\text{or Min}[\langle Y|z=1\rangle,\langle Y|z=-1\rangle]=-1\\\ &\text{or
Max}[\langle Y|z=1\rangle,\langle Y|z=-1\rangle]=1.\end{split}$

These constraints show that for the special case being considered, one of the
following conditions must be true for the marginal expectation values of $X$
conditional on $Z$:

  1. 1.

$\langle X|Z=z\rangle$ has the same magnitude for $z=1$ and $z=-1$ (it is
equal up to a sign); or

  2. 2.

$\langle X|Z=z\rangle$ is deterministic and does not depend on $U$ at all for
one value of $Z=z$.

The same holds for $\langle Y|Z=z\rangle$. If any of these conditions on
$\langle X|Z=z\rangle$ and $\langle Y|Z=z\rangle$ are violated, then we
conclude that perfect positive correlation between $X$ and $Y$ cannot be
explained by the causal structure encoded in Fig. 2 (c). An additional causal
influence between $X$ and $Y$ is needed to explain the considered correlation,
for example the one in Fig. 2 (d).

This is an example of a correlation between $X$ and $Y$ wherein the constraint
induced by Corollary 1 is satisfied for the $d$-separation relation pertinent
to the causal hypothesis of Fig. 2 (c), but nonetheless this correlation is
incompatible with this causal hypothesis.

##  6 Towards Quantum Causality

We now give a different perspective on the motivation behind studying causal
models with hidden common causes. In this Section, we show that DAGs with
unobserved common causes can act as classical analogues of models in which a
_quantum_ common cause is shared.

Consider a scenario in which two parties share a source of entangled quantum
systems and use it to output a classical correlation $\mathbf{P}_{AB}$. The
structure of this protocol is illustrated in Fig. 3. The quantum seed that is
distributed to Alice and Bob is denoted by $\rho_{C}$, quantum systems are
represented by double wires and classical variables are represented by single
wires. Under the assumption that only one copy of $\rho_{C}$ is available in
this protocol, some form of a quantum copy operation must be performed on it
before the distribution. As quantum systems cannot be cloned, the quantum
decomposition operation introduced in [2] is applied. It is represented with
the white dot symbol and implies the decomposition of the Hilbert space
$\mathscr{H}_{C}$ into a direct sum of tensor products. This operation
decomposes $\rho_{C}$ into factors on $\mathscr{H}_{C_{B}}$ carried to Bob and
$\mathscr{H}_{C_{A}}$ carried to Alice. The outputs of this operation are
denoted by $\rho_{A}$ and $\rho_{B}$. Assume $\mathbf{P}_{AB}$ is generated by
one of the two protocols. The first protocol is illustrated in Fig. 3 (a).
Here, Bob and Alice perform measurements $M_{A}$ and $M_{B}$ on their share of
the system $\rho_{C}$ to generate classical variables $A$ and $B$,
respectively. The second protocol is encoded in Fig. 3 (b). It allows for
classical communication from Bob to Alice before Alice outputs $A$. In this
protocol, the value of $A$ is an output of a gate $U_{AB}$ that depends on the
value of $B$.

Figure 3:  Circuit representation of correlation generation protocols.

The main challenge of analyzing these protocols is the restriction on
acquiring information about quantum systems relative to classical systems.
Once a quantum state is sharply measured, it is projected to a post-
measurement state that corresponds to the choice of measurement. This feature
of quantum theory prevents the observer from making passive observations of
quantum common causes. However, the observer can possess some limited
information about the quantum system, for example, its dimension. This
situation is analogous to a classical causal model involving an unobserved
common cause with restricted cardinality. In such models, the observer has
limited knowledge about the classical common cause, namely, its cardinality.
This perspective on the problem allows to see the similarities between
classical causal models illustrated in Figs. 2 (a) and (b), and circuits in
Figs. 3 (a) and (b).

Although Corollary 1 only concerns classical causal models, a similar concept
is used in communication theory [30, 12], where models with a quantum seed can
be characterized using positive semidefinite rank [16]. Therefore, Corollary 1
has the potential to be further extended to models that include quantum
systems. Specifically, knowledge about the dimension of a quantum common cause
could play a role similar to that of the cardinality of a classical common
cause. Formally, the minimum dimension of the local Hilbert space of a shared
entangled state needed to generate $\mathbf{P}_{AB}$ is defined as quantum
correlation complexity. It is characterized by the positive semidefinite rank
of $\mathbf{P}_{AB}$ [16], where positive semidefinite rank is defined as
follows. Let $\text{dim}(\rho_{A})$ and $\text{dim}(\rho_{B})$ be the
dimensions of the local Hilbert space of Alice and Bob, respectively.

######  Definition 3 (positive semidefinite rank)

Let $\mathbf{P}_{AB}$ be a
$\text{dim}(\rho_{A})\times\text{dim}(\rho_{B})$-dimensional nonnegative
matrix. The positive semidefinite rank of $\mathbf{P}_{AB}$ is defined as

$\displaystyle\begin{split}rank_{PSD}[\mathbf{P}_{AB}]\equiv&\min\\{r\mid\exists(\mathbf{E}_{1},\mathbf{F}_{1}),...,(\mathbf{E}_{\text{dim}(\rho_{A})},\mathbf{F}_{\text{dim}(\rho_{B})})\\\
&P_{ab}=\mathrm{tr}\\{E_{a}F_{b}\\}~{}~{}\forall
a\in\\{1...\text{dim}(\rho_{A})\\},b\in\\{1...\text{dim}(\rho_{B})\\},\end{split}$
(8)

where $\mathbf{E}_{a}$ and $\mathbf{F}_{b}$ are $r\times r$ positive
semidefinite matrices.

Assume $\text{dim}(\rho_{A})=\text{dim}(\rho_{B})=d$. It follows that for the
hypothesis illustrated in Fig. 3 (a) to hold, the relation

$d\geq rank_{PSD}[\mathbf{P}_{AB}]$

must be satisfied. This relation enables us to analyse Figs. 3 (a) and (b) in
a similar manner to the analysis of Figs. 2 (a) and (b) we conducted in the
subsection _Witnessing Causal Influence_.

One interesting property of the positive semidefinite rank is that for any
matrix $\mathbf{M}$

$rank_{PSD}[\mathbf{M}]\leq rank_{+}[\mathbf{M}].$

For example, for the probability distribution $\mathbf{P}_{XY}$ defined in
Table 1, $rank_{+}[\mathbf{P}_{XY}]=3$, while $rank_{PSD}[\mathbf{P}_{XY}]=2$
(see Example 2.6 in [11]). It means that the dimension of a quantum resource
needed to generate $\mathbf{P}_{XY}$ is smaller than the cardinality of a
classical resource needed for this task. Examples of this form have been
studied before [30], and they show an advantage of quantum resources over
classical ones.

We finish by noting that a similar approach can be used to analyse a protocol
in which a quantum message is sent from Bob to Alice (the quantum analogue of
Fig. 1 (b)). Similarities between classical and quantum protocols for
correlation generation suggest that classical results, like the one given in
Corollary 1, can be used to set the stage for the quantum equivalent of causal
analysis.

##  7 Conclusions

We derived universal constraints for causal models in which the cardinality of
an unobserved $d$-separating variable is known. While the majority of existing
methods of constraining distributions compatible with latent causal structures
do not use any information about unobserved variables, our result relies on
having knowledge of the cardinality of an unobserved variable. We showed that
our result has applications in causal analysis. Furthermore, we recognized the
analogy between classical causal models with hidden common causes and models
that involve quantum systems. As we identify a new way of bounding causal
effects in models with hidden variables, we set the stage for quantum
extensions of this formulation.

## Acknowledgments

B.Z. acknowledges useful discussions with Noam Finkelstein. This research was
supported by Perimeter Institute for Theoretical Physics. Research at
Perimeter Institute is supported in part by the Government of Canada through
the Department of Innovation, Science and Economic Development Canada and by
the Province of Ontario through the Ministry of Colleges and Universities.

## References

  * [1]
  * [2] John-Mark A Allen, Jonathan Barrett, Dominic C Horsman, Ciarán M Lee & Robert W Spekkens (2017): _Quantum common causes and quantum causal models_.  Physical Review X 7(3), p. 031021, 10.1103/PhysRevX.7.031021.
  * [3] John S Bell (1964): _On the Einstein Podolsky Rosen paradox_.  Physics Physique Fizika 1(3), p. 195, 10.1103/PhysicsPhysiqueFizika.1.195.
  * [4] Olivier Berne, C Joblin, Y Deville, JD Smith, M Rapacioli, JP Bernard, J Thomas, W Reach & A Abergel (2007): _Analysis of the emission of very small dust particles from Spitzer spectro-imagery data using blind signal separation methods_.  Astronomy & Astrophysics 469(2), p. 575, 10.1051/0004-6361:20066282.
  * [5] Michael W Berry, Murray Browne, Amy N Langville, V Paul Pauca & Robert J Plemmons (2007): _Algorithms and applications for approximate nonnegative matrix factorization_.  Computational statistics & data analysis 52(1), p. 155, 10.1016/j.csda.2006.11.006.
  * [6] Rafael Chaves (2013): _Entropic inequalities as a necessary and sufficient condition to noncontextuality and locality_.  Physical Review A 87(2), p. 022102, 10.1103/PhysRevA.87.022102.
  * [7] Rafael Chaves, Lukas Luft, Thiago O Maciel, David Gross, Dominik Janzing & Bernhard Schölkopf (2014): _Inferring latent structures via information inequalities_.  ArXiv e-prints.  arXiv:1407.2256.
  * [8] Joel E Cohen & Uriel G Rothblum (1993): _Nonnegative ranks, decompositions, and factorizations of nonnegative matrices_.  Linear Algebra and its Applications 190, pp. 149–168, 10.1016/0024-3795(93)90224-C.
  * [9] Chris Ding, Tao Li & Wei Peng (2008): _On the equivalence between non-negative matrix factorization and probabilistic latent semantic indexing_.  Computational Statistics & Data Analysis 52(8), pp. 3913–3927, 10.1016/j.csda.2008.01.011.
  * [10] Robin J Evans (2012): _Graphical methods for inequality constraints in marginalized DAGs_.  In: 2012 IEEE International Workshop on Machine Learning for Signal Processing, IEEE, p. 1, 10.1109/MLSP.2012.6349796.
  * [11] Hamza Fawzi, João Gouveia, Pablo A Parrilo, Richard Z Robinson & Rekha R Thomas (2015): _Positive semidefinite rank_.  Mathematical Programming 153(1), pp. 133–177, 10.1007/s10107-015-0922-1.
  * [12] Samuel Fiorini, Serge Massar, Sebastian Pokutta, Hans Raj Tiwary & Ronald De Wolf (2012): _Linear vs. semidefinite extended formulations: exponential separation and strong lower bounds_.  In: Proceedings of the forty-fourth annual ACM symposium on Theory of computing, p. 95, 10.1145/2213977.2213988.
  * [13] Tobias Fritz (2012): _Beyond Bell’s theorem: correlation scenarios_.  New Journal of Physics 14(10), p. 103001, 10.1088/1367-2630/14/10/103001.
  * [14] Dan Geiger & Christopher Meek (2013): _Quantifier elimination for statistical problems_.  ArXiv e-prints.  arXiv:1301.6698.
  * [15] Mena Gonzalo (2017): _Conditional Independence as Non-Negative Matrix Factorization_.  Note.  Available at http://stat.columbia.edu/~gonzalo/pubs/CondIndep.pdf.
  * [16] Rahul Jain, Yaoyun Shi, Zhaohui Wei & Shengyu Zhang (2013): _Efficient protocols for generating bipartite classical distributions and quantum states_.  IEEE Transactions on Information Theory 59(8), p. 5171, 10.1109/TIT.2013.2258372.
  * [17] Changsung Kang & Jin Tian (2012): _Polynomial constraints in causal Bayesian networks_.  ArXiv e-prints.  arXiv:1206.5275.
  * [18] Murat Kocaoglu, Sanjay Shakkottai, Alexandros G Dimakis, Constantine Caramanis & Sriram Vishwanath (2020): _Applications of common entropy for causal inference_.  Advances in Neural Information Processing Systems.  Available at https://papers.nips.cc/paper/2020/file/cae7115f44837c806c9b23ed00a1a28a-Paper.pdf.
  * [19] Matthew S Leifer & Robert W Spekkens (2013): _Towards a formulation of quantum theory as a causally neutral theory of Bayesian inference_.  Physical Review A 88(5), p. 052130, 10.1103/PhysRevA.88.052130.
  * [20] Ben Murrell, Thomas Weighill, Jan Buys, Robert Ketteringham, Sasha Moola, Gerdus Benade, Lise Du Buisson, Daniel Kaliski, Tristan Hands & Konrad Scheffler (2011): _Non-negative matrix factorization for learning alignment-specific models of protein evolution_.  PLOS one 6(12), 10.1371/journal.pone.0028898.
  * [21] Judea Pearl (2009): _Causality_.  Cambridge University Press, 10.1017/CBO9780511803161.
  * [22] Katja Ried, Megan Agnew, Lydia Vermeyden, Dominik Janzing, Robert W Spekkens & Kevin J Resch (2015): _A quantum advantage for inferring causal structure_.  Nature Physics 11(5), p. 414, 10.1038/nphys3266.
  * [23] Denis Rosset, Cyril Branciard, Tomer Jack Barnea, Gilles Pütz, Nicolas Brunner & Nicolas Gisin (2016): _Nonlinear Bell inequalities tailored for quantum networks_.  Physical Review Letters 116(1), p. 010403, 10.1103/PhysRevLett.116.010403.
  * [24] Denis Rosset, Nicolas Gisin & Elie Wolfe (2018): _Universal bound on the cardinality of local hidden variables in networks_.  Quantum Information and Computation 18(11-12), p. 0910, 10.26421/QIC18.11-12.
  * [25] Peter Spirtes, Clark N Glymour, Richard Scheines & David Heckerman (2000): _Causation, prediction, and search_.  MIT press, 10.1007/978-1-4612-2748-9.
  * [26] Bastian Steudel & Nihat Ay (2015): _Information-theoretic inference of common ancestors_.  Entropy 17(4), p. 2304, 10.3390/e17042304.
  * [27] Armin Tavakoli (2016): _Bell-type inequalities for arbitrary noncyclic networks_.  Physical Review A 93(3), p. 030101, 10.1103/PhysRevA.93.030101.
  * [28] Jin Tian & Judea Pearl (2012): _On the testable implications of causal models with hidden variables_.  ArXiv e-prints.  arXiv:1301.0608.
  * [29] Christopher J Wood & Robert W Spekkens (2015): _The lesson of causal discovery algorithms for quantum correlations: Causal explanations of Bell-inequality violations require fine-tuning_.  New Journal of Physics 17(3), p. 033002, 10.1088/1367-2630/17/3/033002.
  * [30] Shengyu Zhang (2012): _Quantum strategic game theory_.  In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference, p. 39, 10.1145/2090236.2090241.


# ad-datasets: a meta-collection of data sets for autonomous driving

Daniel Bogdoll12*, Felix Schreyer2*, and J. Marius Zöllner12
1FZI Research Center for Information Technology, Germany
2Karlsruhe Institute of Technology, Germany
<EMAIL_ADDRESS>

###### Abstract

Autonomous driving is among the largest domains in which deep learning has
been fundamental for progress within the last years. The rise of datasets went
hand in hand with this development. All the more striking is the fact that
researchers do not have a tool available that provides a quick, comprehensive
and up-to-date overview of data sets and their features in the domain of
autonomous driving. In this paper, we present _ad-datasets_ , an online tool
that provides such an overview for more than 150 data sets. The tool enables
users to sort and filter the data sets according to currently 16 different
categories. _ad-datasets_ is an open-source project with community
contributions. It is in constant development, ensuring that the content stays
up-to-date.

##  1 Introduction

††* These authors contributed equally

One of the core building blocks on the way to fully autonomous vehicles are
data sets. Of particular interest are those that contain data on all aspects
of traffic. Their area of application in the research area related to
autonomous driving is diverse. Hence, their number has multiplied
significantly over the years. They have proven to be a necessary tool on the
way to achieving the goal of autonomous vehicles.

Given this increase in importance, it seems all the more surprising that
researchers still do not have a tool at hand, that provides them an overview
of existing data sets and their characteristics. Even today, the search for
fitting data sets is a tedious and cumbersome task. Existing overviews are
typically either incomplete and miss relevant data sets or come in the form of
scientific papers, therefore slowly but steadily becoming outdated. This is an
especially undesirable condition in such a rapidly evolving field.

Researchers are therefore regularly reliant on finding suitable data sets on
their own. However, this task is not only extremely time-consuming, as it
involves studying numerous websites and papers. It also in no way guarantees
that researchers will indeed find a suitable, perhaps even optimal, data set.

In this publication, we present our attempt to address this unanswered
challenge. With _ad-datasets_ 111https://ad-datasets.com/ we have developed a
tool that provides users with a comprehensive, up-to-date overview of existing
data sets in the field of autonomous driving, as shown in Figure 1. In
addition, the properties of the data sets are broken down into different
categories. Users are given the opportunity to individually filter and sort
the data sets according to certain criteria or rather, to their individual
needs. Currently222as of Nov 25, 2021, _ad-datasets_ comprises 158 data sets.

Furthermore, _ad-datasets_ is an open-source project. The community is
encouraged to contribute, such as missing data sets or metadata. We are proud
to have already received several community contributions only a few weeks
after the initial release, including ones from ARGO AI, a well-known company
within the domain. _ad-datasets_ , hosted via Github Pages, enables this
particularly easily via pull requests and automated deployments.

The further structure of this paper is organized as follows: In section 2,
relevant related work is presented. Section 3 introduces the tool _ad-
datasets_ and its technical implementation. Section 4 analyzes the information
obtained. The final section 5 provides an outlook into future extensions.

##  2 Related Work

Dataset Overview | Entries | Filterable | Sortable |  Community Contribution |  Number of Itemized Properties | Last Update
---|---|---|---|---|---|---
ad-datasets | 158 | Yes | Yes | Yes | 16 | 2021
Bifrost | 50 | Yes | Yes | No | - | 2020
Dataset list | 25 | Yes | No | Yes | 4 | 2021
Kaggle | 31 | Yes | Yes | Yes | - | 2021
RList | 9 | Yes | Yes | No | 5 | 2021
Scale | 50 | Yes | No | No | 4 | 2019
YonoStore | 10 | No | Yes | No | - | 2021
Table 1: Comparison between _ad-datasets_ and other online tools. We compare
the number of entries in each tool, the features the tools provide, the number
of properties broken down in detail and when the tools were last updated.

The research area around autonomous driving shines with great progress and a
high rate of development. Numerous advances are made every year and the amount
of literature continues to grow. This is also the case in the area of data
collection, respectively the area of data sets in autonomous driving. Over the
years, however, the sheer amount of data sets has become increasingly
complicated. For this reason, there have been attempts in the past to provide
a structure to these advances. In general, these attempts can be divided into
two separate categories.

###  2.1 Scientific Papers

First, there are studies that aim to create a general, comprehensive overview
of existing data sets from the area of research. These include [Yin and
Berger, 2017], where 27 data sets prevalent at the time were presented. Around
three times that amount was summarized in [Laflamme et al., 2019].

In addition to these studies, which aim directly at creating an overview of
data sets, there are also studies that focus primarily on different research
questions, but also contain such an overview. [Feng et al., 2020] presented
their work, which also comes in an online version briefly mentioned in the
following subsection, that investigated the research question of Deep Multi-
modal Object Detection and Semantic Segmentation for Autonomous Driving. Yet,
the authors also examined multi-modal data sets, with the multimodality
referring to the sensors used. Compared to the works of Yin and Berger and
Laflamme et al. however, this collection is much smaller in scope. [Heidecker
et al., 2021] takes on the topic of corner cases in highly automated driving.
Here, too, a section revolves solely around suitable data sets for corner case
detectors. Finally, in the same year [Kim and Hwang, 2021] was published that
contains a survey addressing data sets for monocular 3D detection. Yet, these
papers also lag behind in scope. It is important to note, that publications
which do not focus on data sets but provide overviews of them as side effect
typically focus strongly on their area of research. Thus, they rarely provide
a broad picture of data sets and tend to focus either on popular and well
known data sets or their specific niche.

What all these publications share is that they appeared in the format of
scientific works. Thus, they have some weaknesses in common when it comes to
searching for data sets. To start with, they share the problem that they
become out-of-date relatively quickly. This is an undesirable characteristic,
especially in a research area that is developing as quickly as the one of
autonomous driving. In addition, these overviews lack a convenient format.
Naturally, the publications offer neither a filtering nor a sorting function,
therefore not being as effortless and time saving as one would desire.

###  2.2 Online Sources

Further, there exist various sources in online formats. These can in turn be
broken down into read-only textual sources, mostly in the form of blogs
[Choudhury, 2020][Cambridge Spark, 2018][Nguyen’s, 2021], wiki entries
[Wikipedia, 2021][FOT-Net, 2020], git repositories [Heyuan, 2019][Diaz, 2021]
or miscellaneous entries [DIY Robocars, 2017][Krunal, 2018][Feng et al.,
2021], and into interactive tools.

The textual sources are usually kept very compact and often contain fewer than
ten data sets, which are typically among the better known ones. Additionally,
many times the summaries are primarily aimed at machine learning related data
sets in general. It is then left to the users to filter out the relevant
autonomous driving related data sets. Since all of these sources, much like
scientific papers, do neither allow any filtering nor sorting functions and
are typically not kept up-to-date, they are poorly suited as sources for
extensive data set searches.

A completely different picture emerges when looking at online tools. Their
format is much more suitable for searches of any kind.

_RList_ [RList, 2021] provides such a tool. Both sorting and search functions
are available to users. Further, entries are broken down into categories by
release date, organization, frames and location. However, with only nine data
set entries it remains rather small. _Scale_ [Scale, 2019] provides another,
much more extensive tool. It comprises 50 data sets from the areas of
autonomous driving and natural language processing. Further, the tool
highlights a variety of categories, namely sensor types, annotations,
diversity and recording location. Filter functions are also available to the
user. Unfortunately, however, the tool does not contain any data set entries
after 2019 and can therefore not be considered as up-to-date. This is where
the _Dataset list_ [Plesa, 2021] tool stands out. Up to the time of
publication of this work, the tool was regularly updated, and can therefore be
regarded as up-to-date. The tool comes with four categories of which the
highlighting of the licenses differs from the ones previously described. Yet,
with a volume of 25 data sets from the field of autonomous driving, it is too
small to be able to claim a complete overview of existing data sets.

Also _Bifrost_ with 50 data sets [Bifrost, 2020], _Kaggle_ with 31 data sets
[Kaggle, 2021] and _YonoStore_ with ten data sets [YonoStore, 2021] cannot be
seen as complete, either. Further, unlike the aforementioned online tools,
they do not provide the user with an overview of data set properties at first
glance.

More general offers such as _Google Dataset Search_ [Google, 2018], _DeepAI
Datasets_ [DeepAI, 2017] or _Papers with Code Datasets_ [Facebook AI, 2018]
have the issue that no domain-specific overview is possible. Therefore,
although they list numerous data sets, they are not suitable.

Finally, it should be noted that the majority of the tools examined here do
not allow the user to contribute content. Only _Kaggle_ and _Dataset list_
provide the community with an opportunity to independently add missing data
sets.

##  3 Selection Process

Figure 1: Screenshot of the ad-datasets web application

Name | Size [h] | Frames | N° Scenes | Sensortypes | Publish Date | Sources
---|---|---|---|---|---|---
| camera | lidar | radar | other |
ApolloScapei | 100 | 143,906 |  | ✓ | ✓ |  | ✓ | 2018.03 |  [Wang et al., 2019][ApolloScape, 2018]
Argoverse Motion Forecastingi | 320 |  | 324,557 | ✓ | ✓ |  | ✓ | 2019.06 |  [Chang et al., 2019][ARGO AI, 2019]
Argoverse 3D Trackingi |  |  | 113 | ✓ | ✓ |  | ✓ | 2019.06 |  [Chang et al., 2019][ARGO AI, 2019]
A2D2i |  | 433,833 | 3 | ✓ | ✓ |  | ✓ | 2020.04 |  [Geyer et al., 2020][A2D2, 2020]
BDD100ki | 1,111 | 120,000,000 | 100,000 | ✓ |  |  | ✓ | 2020.04 |  [Yu et al., 2020][ETH VIS Group, 2018]
Bosch Small Traffic Lightsj |  | 13,427 |  | ✓ |  |  |  | 2017.05 |  [Behrendt and Novak, 2017][for Image Processing, 2017]
Caltech Pedestriani | 10 | 1,000,000 | 137 | ✓ |  |  |  | 2010.03 |  [Dollár et al., 2009][California Institute of Technology, 2009]
Cityscapes 3Di |  |  |  | ✓ |  |  | ✓ | 2016.02 |  [Cordts et al., 2016][Gählert et al., 2020][Cityscapes Dataset, 2016]
Comma2k19i | 33.65 |  | 2,019 | ✓ |  | ✓ | ✓ | 2018.12 |  [Schafer et al., 2018][comma.ai, 2019]
DDD 20j | 51 |  | 216 | ✓ |  |  | ✓ | 2020.02 |  [Hu et al., 2020][Inst. of Neuroinformatics, Univ. of Zurich and ETH Zurich, 2020]
Fishyscapesi |  |  |  | ✓ |  |  |  | 2019.09 |  [Blum et al., 2019][ETH Zürich, 2019]
Ford Autonomous Vehiclei |  |  |  | ✓ | ✓ |  | ✓ | 2020.03 |  [Agarwal et al., 2020][Ford, 2020]
H3Dj | 0.77 | 27,721 | 160 | ✓ | ✓ |  | ✓ | 2019.03 | [Patil et al., 2019]
India Drivingj |  | 10,004 | 182 | ✓ |  |  |  | 2018.11 |  [Varma et al., 2019][INSAAN, 2018]
INTERACTIONi | 16.5 | 594,588 |  | ✓ |  |  |  | 2019.09 |  [Zhan et al., 2019][INTERACTION Dataset Consortium, 2019]
KAIST Multi-Spectral Day/Nighti |  |  |  | ✓ | ✓ |  | ✓ | 2017.12 |  [Choi et al., 2018][KAIST, 2017]
KAIST Urbani |  |  | 18 | ✓ | ✓ |  | ✓ | 2017.09 |  [Jeong et al., 2019][Autonomy and Lab, 2021]
KITTIi | 6 |  | 50 | ✓ | ✓ |  | ✓ | 2012.03 |  [Geiger et al., 2013][cvlibs, 2012]
KITTI-360i |  | 400,000 |  | ✓ | ✓ |  | ✓ | 2015.11 |  [Xie et al., 2016][cvlibs, 2021]
LostAndFoundi |  | 21,040 | 112 | ✓ |  |  |  | 2016.09 |  [Pinggera et al., 2016][6D-Vision, 2016]
Lyft Level5 Perceptioni | 2.5 |  | 366 | ✓ | ✓ |  |  | 2019.07 |  [Houston et al., 2020][Level 5, 2020a]
Lyft Level5 Predictioni | 1,118 | 42,500,000 | 170,000 | ✓ | ✓ | ✓ |  | 2020.06 |  [Houston et al., 2020][Level 5, 2020b]
MCity Data Collectioni | 50 |  | 255 | ✓ | ✓ | ✓ | ✓ | 2019.12 | [Dong et al., 2019]
NightOwlsj | 5.17 | 279,000 | 40 | ✓ |  |  |  | 2018.12 |  [Neumann et al., 2018][NightOwls dataset, 2018]
nuImagesi | 150 | 1,200,000 | 93,000 | ✓ | ✓ | ✓ | ✓ | 2020.07 | [Motional, 2019a]
nuScenesi | 15 | 1,400,000 | 1,000 | ✓ | ✓ | ✓ | ✓ | 2019.03 |  [Caesar et al., 2019][Motional, 2019b]
openDDi | 62.7 | 6,771,600 | 501 | ✓ |  |  |  | 2020.09 |  [Breuer et al., 2020][L3Pilot, 2019]
Oxford Radar Robot Cari |  |  | 32 | ✓ | ✓ | ✓ | ✓ | 2020.02 |  [Barnes et al., 2020][Oxford Robotics Institute, 2020]
Oxford Robot Cari | 210 |  | 100 | ✓ | ✓ |  | ✓ | 2016.11 |  [Maddern et al., 2017][Oxford Robotics Institute, 2017]
PandaSeti | 0.23 | 48,000 | 103 | ✓ | ✓ |  | ✓ | 2020.04 | [Hesai, Scale AI, 2020]
RadarScenesj | 4 |  | 158 | ✓ |  | ✓ | ✓ | 2021.03 |  [Schumann et al., 2021][RadarScenes, 2021]
RADIATEj | 5 |  |  | ✓ | ✓ | ✓ | ✓ | 2020.10 |  [Sheeny et al., 2020][Vision Lab, Perception and Robotics Group, 2020]
RoadAnomaly21i |  | 100 | 100 | ✓ |  |  |  | 2021.04 |  [Chan et al., 2021][Segment Me If You Can, 2021]
Semantic KITTIi |  | 43,552 | 21 |  | ✓ |  |  | 2019.07 |  [Behley et al., 2019][University of Bonn, 2019]
Synscapesj |  | 25,000 | 25,000 | ✓ |  |  |  | 2018.10 |  [Wrenninge and Unger, 2018][7D Labs Inc., 2018]
Udacityi | 10 |  |  | ✓ | ✓ |  | ✓ | 2016.09 | [Udacity, 2020]
Waymo Open Motioni | 574 | 20,670,800 | 103,354 | ✓ | ✓ |  |  | 2021.03 |  [Ettinger et al., 2021][Waymo LLC, 2021]
Waymo Open Perceptioni | 10.83 | 390,000 | 1,950 | ✓ | ✓ |  |  | 2019.08 |  [Sun et al., 2020][Waymo LLC, 2019]
WildDashi |  |  | 156 | ✓ |  |  |  | 2018.02 |  [Zendel et al., 2018][AIT, 2020]
4Seasonsj |  |  | 30 | ✓ |  |  | ✓ | 2020.10 |  [Wenzel et al., 2020][Artisense, 2020]

Table 2: Overview of the data sets analyzed in detail. Data sets marked with j
belong to those which were chosen randomly, data sets marked with i have been
selected deterministically by the authors

_ad-datasets_ is an online tool designed as the central point of contact for
data sets in the field of autonomous driving. It includes a detailed
representation of the data sets according to 16 different property categories
and enables users to interact via filter and sorting functions. As of writing,
_ad-datasets_ comprises 158 data sets, 40 of which were examined in detail
according to the 16 categories.

###  3.1 Content

The search for data sets poses a major challenge. In this work, large and
well-known data sets were easily found, both via the numerous online sources
and the numerous papers which refer to them. Less known, older data sets were
collected through an extensive literature research. An approach that has
proven to be well-suited for their finding has been the snowballing principle.
However, newer, lesser-known publications cannot be found this way. In fact,
finding them has turned out to be the greatest difficulty. Since they are
rarely mentioned in literature or online sources, their search had to be
designed differently. In this work, these data sets were identified through
the use of search engines and via communities such as LinkedIn. It should be
noted, that this procedure is associated with a high level of effort, so being
well-connected in the relevant communities is of great benefit. Needless to
say, this practice is not very scientific, but extremely effective, as new
data sets in the community are often shared through social media.

In this paper, autonomous driving related data sets are defined as data sets
that contain data on all aspects of traffic. They can both consist of scenes
and scenarios333Following the definitions of [Ulbrich et al., 2015] of road
traffic or its participants. _ad-datasets_ includes, for example, data sets
with video sequences of intersections from a bird’s eye view, but also
recordings from a vehicle directly participating in traffic.

The primary focus of _ad-datasets_ are those data sets which were published
after the famous KITTI data set, which serves, so to speak, as a time
benchmark. Regarding the selection of the data sets which were analyzed in
detail, the selection can be separated into two parts. 31 of the data sets
were selected manually by the authors, focusing on the most popular ones. The
remaining nine data sets were selected at random. The breakdown can be seen in
table 2.

Property | Included
---|---
Annotations | ✓
Benchmark | ✓
Frames | ✓
Last Update | ✓
Licensing | ✓
Location | ✓
N° Scenes | ✓
Publish Date | ✓
Related Datasets | ✓
Related Paper | ✓
Sampling Rate [Hz] | ✓
Scene Length [s] | ✓
Sensor Types | ✓
Sensors - Details | ✓
Size [GB] | ✓
Size [h] | ✓
Data Format | ✗
Main Focus | ✗
Recording Perspective | ✗
Statistics | ✗
Tooling | ✗
Table 3: Property categories resulting from an expert survey with indication
whether they have been included

###  3.2 Structure

The selected property categories are in turn the result of an expert survey
conducted of the research group for technical cognitive systems at the FZI
Research Center for Information Technology. Over 20 different categories were
suggested in the survey (Table 3). Ultimately, 16 categories found their way
into the initial version due to time constraints. The selection of these
properties that were included in the tool was made based on an examination of
ten exemplary data sets. In this examination, the time required to collect the
data for each property was investigated. The data was collected via the web
presences of the data sets and their corresponding papers. The decision on
whether a category was included was made based upon the author’s perceived
importance of the property and the associated time required to include the
category in the selection. The ten exemplary data sets were Cityscapes 3D,
ApolloScape, Lyft Level5 Prediction, Oxford Robot Car, nuScenes, PandaSet,
Waymo Open Motion, KITTI, BDD100k and openDD. The resulting 16 categories are
presented in detail at this point.
Annotations. This property describes the types of annotations with which the
data sets have been provided.
Benchmark. If benchmark challenges are explicitly listed with the data sets,
they are specified here.
Frames. Frames states the number of frames in the data set. This includes
training, test and validation data.
Last Update. If information has been provided on updates and their dates, they
can be found in this category.
Licensing. In order to give the users an impression of the licenses of the
data sets, information on them is already included in the tool.
Location. This category lists the areas where the data sets have been
recorded.
N° Scenes. N° Scenes shows the number of scenes contained in the data set and
includes the training, testing and validation segments. In the case of video
recordings, one recording corresponds to one scene. For data sets consisting
of photos, a photo is the equivalent to a scene.
Publish Date. The initial publication date of the data set can be found under
this category. If no explicit information on the date of publication of the
data set could be found, the submission date of the paper related to the set
was used at this point.
Related Data Sets. If data sets are related, the names of the related sets can
be examined as well. Related data sets are, for example, those published by
the same authors and building on one another.
Related Paper. This property solely consists of a link to the paper related to
the data set.
Sampling Rate [Hz]. The Sampling Rate [Hz] property specifies the sampling
rate in Hertz at which the sensors in the data set work. However, this
declaration is only made if all sensors are working at the same rate or,
alternatively, if the sensors are being synchronized. Otherwise, the field
remains empty.
Scene Length [s]. This property describes the length of the scenes in seconds
in the data set, provided all scenes have the same length. Otherwise, no
information is given. For example, if a data set has scenes with lengths
between 30 and 60 seconds, no entry can be made. The background to this
procedure is to maintain comparability and sortability.
Sensor Types. This category contains a rough description of the sensor types
used. Sensor types are, for example, lidar or radar.
Sensors - Details. The Sensors - Detail category is an extension of the Sensor
Types category. It includes a more detailed description of the sensors. The
sensors are described in detail in terms of type and number, the frame rates
they work with, the resolutions which sensors have and the horizontal field of
view.
Size [GB]. The category Size [GB] describes the storage size of the data set
in gigabytes.
Size [h]. The Size [h] property is the equivalent of the Size [GB] described
above, but provides information on the size of the data set in hours.

It should be noted that, the name of the data set is naturally listed, too. It
further acts as a link to the respective website of the data set. Further, it
is worth mentioning that the general aim is to state the properties as
precisely as possible. Yet, this also depends on how accurate the
documentation of the data set is. For example, a size specification of 100+
hours is less precise than a specification of 103 hours. In the case of the
former, the tool would only state a size of 100 hours.

###  3.3 Technical Implementation

_ad-datasets_ is hosted on GitHub Pages. This allows for a seamless
integration of the research community. Not only can the community influence
the development process, but also actively expand its content. This is
especially valuable as, no matter how extensive a search, it cannot be
guaranteed that the tool is indeed complete. Missing entries or metadata can
be added via pull requests, which automatically trigger the build system to
publish the changes.

The implementation of the tool itself was done using the frameworks React
[Facebook, 2013] and Material-UI [Material-UI, 2014]. The latter enables a
quick and uncomplicated creation of filterable as well as sortable tables. In
this work, the data grid component [Material-UI, 2020] of the Material-UI
framework was used.

##  4 Property Examination

This section takes a closer look at the 40 data sets from _ad-datasets_ that
have been analyzed in detail.

###  4.1 Data sets over time

In Figure 2, the observed data sets are shown on a timeline. While keeping in
mind that the majority of the data sets were not chosen randomly, a clear
tendency can be identified that both the amount of data sets and the amount of
data set publications per year are steadily increasing.

For a start, this finding emphasizes once again the dynamics in the research
field of autonomous driving. More to the point, it demonstrates the importance
of a well-maintained and up-to-date tool in order to tackle the currently
prevailing inconvenience that come with data set searches. The accelerating
pace of data set publications makes it even clearer, that unsupervised
overviews age at an equally increasing pace.

Figure 2: Timeline when data sets have been published

###  4.2 Use of Sensor Types

When looking at the sensor types used in the various data sets (Figure 3), it
is first noticeable that the sensor type used most frequently is the camera.
In fact, the only data set analyzed that does not utilize camera sensors is
the Semantic KITTI data set. The remaining 39 out of 40 data sets make use of
the camera sensor type.

The Semantic KITTI data set in return deploys lidar sensors. 22 other data
sets do the same, so that more than 50% of the data sets include lidar data.
Radar data is used much less frequently. Only eight data sets make use of the
sensors. When additionally considering publication dates, it shows that radar
data sets were added much later. None of the eight radar data sets have been
published before the end of 2018.
Yet, it seems that the importance of data sets containing radar data is
increasing. Of the ten data sets published in 2020, four contained radar data.

The outsiders among the sensor types include thermal cameras and thermometers.
Each appear in only one data set, the thermal camera in Cityscapes 3D and the
thermometer in the Multi-Spectral Day / Night data set.

Figure 3: Frequency of use of sensor types in the data sets

###  4.3 Size

Upon analyzing the size of data sets, one can distinguish between storage
size, the time span and the number of scenes of the data sets.

The inspection of storage size (Figure 5) reveals that of the eleven data sets
which provide information on their size, five data sets are smaller than 1,000
GB with the median being 1,300 GB. At the same time, however, there coexist
significantly larger data sets. The MCity Data Collection data set is 11,000
GB large. The even larger Oxford Robot Car set is in fact with 23,150 GB more
than 17 times as large as the median. It can therefore be seen that data sets
are mainly of a similar order of magnitude in terms of storage size. However,
there are also sets of much larger sizes. When looking at the scope of time
(Table 4), an even more pronounced picture emerges. The median of the 23 data
sets, for which the corresponding information could be obtained, is 16.5
hours. There are more data sets smaller than ten hours than there are data
sets larger than 100 hours. But here, too, there are examples of very large
data sets. The Lyft Level5 Prediction data set is 1,118 hours large, the
BDD100k data set 1,111 hours. These two data sets are over 60 times as large
as the median.

On closer inspection, however, one can spot a difference between the storage
wise large data sets and the time wise large data set. Some of those data sets
that are rather large in terms of storage space have been existing for
multiple years. For example, the Oxford Robot Car data set was published back
in 2016. Data sets which feature large scopes of time have been published
rather recently. Lyft Level5 Prediction and BDD100k had not been published
until 2020.

These impressions can also be transferred to the analysis of the number of
scenes in data sets (Table 6). Once again, the majority revolves around a
similar range regarding the number of scenes. However, again, there are
outliers that are significantly larger. The median over all 30 data sets, for
which information could be gathered, is 158 scenes. Opposite to that, the Lyft
Level5 Prediction (170,000 scenes) and the Argoverse Motion Forecasting
(324,557 scenes) are over 1,000 respectively over 2,000 times as large.

What’s more, data sets that come with large number of scenes have been
published relatively recently as well.

Figure 4: Histogram depicting the distribution of the data sets over their
scope of time

Figure 5: Histogram depicting the distribution of the data sets over their
storage size

Figure 6: Histogram depicting the distribution of the data sets over their
number of scenes

##  5 Conclusion and Outlook

With _ad-datasets_ , we have presented a tool that aims to simplify the
previously complex and time-consuming search for suitable data sets related to
the research area of autonomous driving. _ad-datasets_ offers users an
overview of over 150 data sets, which are further broken down into 16
different properties. Finally, users can interact with the tool using filter
and sorting functions. The timeliness of _ad-datasets_ is maintained through
further maintenance of the tool and can be supported by the community, e.g.,
via pull requests.

However, the full potential of the tool has not yet been realized. To fully do
so, a couple of aspects can be addressed. First, it is obviously necessary to
complete the detailed analysis of the remaining data sets. This is already an
ongoing work in progress.

In addition, it is crucial to obtain feedback from the research community.
After all, they are the target audience of _ad-datasets_. Hence, their
feedback is essential in developing a truly value-adding tool. Initial
feedback from the community has been very positive and has already led to
contributions.

Finally, it must be borne in mind that some of the properties suggested in the
expert survey did not find their way into the initial version of _ad-
datasets_. Therefore, they are subject to future work. It was proposed to
include information on the statistical distribution of classes, labels etc.
For the time being, this proposal remains subject to future work, as there
were concerns regarding copyright. Likewise, the suggested property categories
data format and tooling options remain subject to future work. Important
categories, which for the moment were associated with too much effort as well,
are the recording perspective and the key aspects addressed by the data sets.
In a later version of the tool, these will be included.

##  6 Acknowledgment

This work results partly from the KIGLIS project supported by the German
Federal Ministry of Education and Research (BMBF), grant number 16KIS1231.

## REFERENCES

  * [6D-Vision, 2016] 6D-Vision (2016).  LostAndFoundDataset.  https://www.6d-vision.com/current-research/lostandfounddataset.  Accessed 2022-02-03.
  * [7D Labs Inc., 2018] 7D Labs Inc. (2018).  Synscapes.  https://7dlabs.com/synscapes-overview.  Accessed 2021-08-12.
  * [A2D2, 2020] A2D2 (2020).  Driving dataset.  . Accessed 2021-08-22.
  * [Agarwal et al., 2020] Agarwal, S., Vora, A., Pandey, G., Williams, W., Kourous, H., and McBride, J. (2020).  Ford multi-AV seasonal dataset.
  * [AIT, 2020] AIT (2020).  Wilddash 2 benchmark.  https://wilddash.cc/.  Accessed 2021-08-09.
  * [ApolloScape, 2018] ApolloScape (2018).  ApolloScape.  http://apolloscape.auto/.  Accessed 2021-08-04.
  * [ARGO AI, 2019] ARGO AI (2019).  Argoverse.  https://www.argoverse.org/.  Accessed 2021-08-22.
  * [Artisense, 2020] Artisense (2020).  4seasons dataset.  https://www.4seasons-dataset.com/.  Accessed 2021-08-16.
  * [Autonomy and Lab, 2021] Autonomy, I. R. and Lab, P. I. (2021).  Complex urban dataset.  https://sites.google.com/view/complex-urban-dataset.
  * [Barnes et al., 2020] Barnes, D., Gadd, M., Murcutt, P., Newman, P., and Posner, I. (2020).  The oxford radar RobotCar dataset: A radar extension to the oxford RobotCar dataset.  In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA).
  * [Behley et al., 2019] Behley, J., Garbade, M., Milioto, A., Quenzel, J., Behnke, S., Stachniss, C., and Gall, J. (2019).  SemanticKITTI: A dataset for semantic scene understanding of LiDAR sequences.  In Proc. of the IEEE/CVF International Conf. on Computer Vision (ICCV).
  * [Behrendt and Novak, 2017] Behrendt, K. and Novak, L. (2017).  A deep learning approach to traffic lights: Detection, tracking, and classification.  In Robotics and Automation (ICRA), 2017 IEEE International Conference on.
  * [Bifrost, 2020] Bifrost (2020).  Search for visual datasets.  https://datasets.bifrost.ai/.  Accessed 2021-08-16.
  * [Blum et al., 2019] Blum, H., Sarlin, P., Nieto, J. I., Siegwart, R., and Cadena, C. (2019).  The fishyscapes benchmark: Measuring blind spots in semantic segmentation.  arXiv preprint:1904.03215.
  * [Breuer et al., 2020] Breuer, A., Termöhlen, J., Homoceanu, S., and Fingscheidt, T. (2020).  opendd: A large-scale roundabout drone dataset.  arXiv preprint:2007.08463.
  * [Caesar et al., 2019] Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. (2019).  nuScenes: A multimodal dataset for autonomous driving.  arXiv preprint:1903.11027.
  * [California Institute of Technology, 2009] California Institute of Technology (2009).  Caltech Pedestrian Detection Benchmark.  http://www.vision.caltech.edu/Image˙Datasets/CaltechPedestrians/.  Accessed 2021-08-22.
  * [Cambridge Spark, 2018] Cambridge Spark (2018).  50 free machine learning datasets: Self-driving cars.  https://blog.cambridgespark.com/50-free-machine-learning-datasets-self-driving-cars-d37be5a96b28.  Accessed 2021-08-16.
  * [Chan et al., 2021] Chan, R., Lis, K., Uhlemeyer, S., Blum, H., Honari, S., Siegwart, R., Salzmann, M., Fua, P., and Rottmann, M. (2021).  SegmentMeIfYouCan: A benchmark for anomaly segmentation.  arXiv preprint:2104.14812.
  * [Chang et al., 2019] Chang, M.-F., Lambert, J., Sangkloy, P., Singh, J., Bak, S., Hartnett, A., Wang, D., Carr, P., Lucey, S., Ramanan, D., and Hays, J. (2019).  Argoverse: 3d tracking and forecasting with rich maps.  arXiv preprint:1911.02620.
  * [Choi et al., 2018] Choi, Y., Kim, N., Hwang, S., Park, K., Yoon, J. S., An, K., and Kweon, I. S. (2018).  Kaist multi-spectral day/night data set for autonomous and assisted driving.  IEEE Transactions on Intelligent Transportation Systems, 19(3).
  * [Choudhury, 2020] Choudhury, A. (2020).  Top 10 popular datasets for autonomous driving projects.  https://analyticsindiamag.com/top-10-popular-datasets-for-autonomous-driving-projects/.  Accessed 2021-08-18.
  * [Cityscapes Dataset, 2016] Cityscapes Dataset (2016).  Cityscapes dataset.  https://www.cityscapes-dataset.com/.  Accessed 2021-08-20.
  * [comma.ai, 2019] comma.ai (2019).  commai/comma2k19.  https://github.com/commaai/comma2k19.  Accessed 2021-08-10.
  * [Cordts et al., 2016] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., and Schiele, B. (2016).  The cityscapes dataset for semantic urban scene understanding.  arXiv preprint:1604.01685.
  * [cvlibs, 2012] cvlibs (2012).  The kitti vision benchmark suite.  http://www.cvlibs.net/datasets/kitti/.  Accessed 2021-08-07.
  * [cvlibs, 2021] cvlibs (2021).  Kitti-360: A large-scale dataset with 3d&2d annotations.  http://www.cvlibs.net/datasets/kitti-360/.  Accessed 2021-08-21.
  * [DeepAI, 2017] DeepAI (2017).  Discover datasets for machine learning and a.i.  https://deepai.org/datasets.  Accessed 2021-08-20.
  * [Diaz, 2021] Diaz, M. (2021).  manfreddiaz/awesome-autonomous-vehicles.  https://github.com/manfreddiaz/awesome-autonomous-vehicles.
  * [DIY Robocars, 2017] DIY Robocars (2017).  Open datasets.  https://diyrobocars.com/open-datasets/.  Accessed 2021-08-20.
  * [Dollár et al., 2009] Dollár, P., Wojek, C., Schiele, B., and Perona, P. (2009).  Pedestrian detection: A benchmark.  In IEEE Conference on Computer Vision and Pattern Recognition.
  * [Dong et al., 2019] Dong, Y., Zhong, Y., Yu, W., Zhu, M., Lu, P., Fang, Y., Hong, J., and Peng, H. (2019).  Mcity data collection for automated vehicles study.  arXiv preprint:1912.06258.
  * [ETH VIS Group, 2018] ETH VIS Group (2018).  Bdd100k.  https://www.bdd100k.com/.  Accessed 2021-08-07.
  * [ETH Zürich, 2019] ETH Zürich (2019).  The fishyscapes benchmark.  https://fishyscapes.com/.  Accessed 2021-08-21.
  * [Ettinger et al., 2021] Ettinger, S., Cheng, S., Caine, B., Liu, C., Zhao, H., Pradhan, S., Chai, Y., Sapp, B., Qi, C. R., Zhou, Y., Yang, Z., Chouard, A., Sun, P., Ngiam, J., Vasudevan, V., McCauley, A., Shlens, J., and Anguelov, D. (2021).  Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
  * [Facebook, 2013] Facebook (2013).  React.  https://reactjs.org/.  Accessed 2021-08-22.
  * [Facebook AI, 2018] Facebook AI (2018).  Datasets.  https://paperswithcode.com/datasets.  Accessed 2021-08-16.
  * [Feng et al., 2020] Feng, D., Haase-Schütz, C., Rosenbaum, L., Hertlein, H., Glaeser, C., Timm, F., Wiesbeck, W., and Dietmayer, K. (2020).  Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges.  IEEE Transactions on Intelligent Transportation Systems, 22(3).
  * [Feng et al., 2021] Feng, D., Haase-Schütz, C., Rosenbaum, L., Hertlein, H., Glaeser, C., Timm, F., Wiesbeck, W., and Dietmayer, K. (2021).  Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges.  https://boschresearch.github.io/multimodalperception/dataset.html.  Accessed 2021-08-22.
  * [for Image Processing, 2017] for Image Processing, H. C. (2017).  Bosch Small Traffic Lights Dataset.  https://hci.iwr.uni-heidelberg.de/content/bosch-small-traffic-lights-dataset.  Accessed 2021-08-12.
  * [Ford, 2020] Ford (2020).  Ford Autonomous Vehicle Dataset.  https://avdata.ford.com/.  Accessed 2021-08-20.
  * [FOT-Net, 2020] FOT-Net (2020).  Automated driving datasets.  https://wiki.fot-net.eu/index.php/Automated˙Driving˙Datasets.  Accessed 2021-08-20.
  * [Geiger et al., 2013] Geiger, A., Lenz, P., Stiller, C., and Urtasun, R. (2013).  Vision meets robotics: The KITTI dataset.  International Journal of Robotics Research (IJRR).
  * [Geyer et al., 2020] Geyer, J., Kassahun, Y., Mahmudi, M., Ricou, X., Durgesh, R., Chung, A. S., Hauswald, L., Pham, V. H., Mühlegg, M., Dorn, S., Fernandez, T., Jänicke, M., Mirashi, S., Savani, C., Sturm, M., Vorobiov, O., Oelker, M., Garreis, S., and Schuberth, P. (2020).  A2d2: Audi autonomous driving dataset.  arXiv preprint:2004.06320.
  * [Google, 2018] Google (2018).  Datasetsearch - autonomous driving.  https://datasetsearch.research.google.com/search?query=Autonomous%20driving&docid=L2cvMTFwd2Y0amZ0Yw%3D%3D.  Accessed 2021-08-12.
  * [Gählert et al., 2020] Gählert, N., Jourdan, N., Cordts, M., Franke, U., and Denzler, J. (2020).  Cityscapes 3d: Dataset and benchmark for 9 DoF vehicle detection.  arXiv preprint:2006.07864.
  * [Heidecker et al., 2021] Heidecker, F., Breitenstein, J., Rösch, K., Löhdefink, J., Bieshaar, M., Stiller, C., Fingscheidt, T., and Sick, B. (2021).  An application-driven conceptualization of corner cases for perception in highly automated driving.  arXiv preprin:2103.03678.
  * [Hesai, Scale AI, 2020] Hesai, Scale AI (2020).  PandaSet by Hesai and Scale AI.  https://pandaset.org/.  Accessed 2021-08-05.
  * [Heyuan, 2019] Heyuan, L. (2019).  lhyfst/awesome-autonomous-driving-datasets.  https://github.com/lhyfst/awesome-autonomous-driving-datasets.  Accessed 2021-08-10.
  * [Houston et al., 2020] Houston, J., Zuidhof, G., Bergamini, L., Ye, Y., Jain, A., Omari, S., Iglovikov, V., and Ondruska, P. (2020).  One thousand and one hours: Self-driving motion prediction dataset.  arXiv preprint:2006.14480.
  * [Hu et al., 2020] Hu, Y., Binas, J., Neil, D., Liu, S.-C., and Delbrück, T. (2020).  DDD20 end-to-end event camera driving dataset: Fusing frames and events with deep learning for improved steering prediction.  arXiv preprint:2005.08605.
  * [INSAAN, 2018] INSAAN (2018).  India driving dataset.  https://idd.insaan.iiit.ac.in/.  Accessed 2021-08-12.
  * [Inst. of Neuroinformatics, Univ. of Zurich and ETH Zurich, 2020] Inst. of Neuroinformatics, Univ. of Zurich and ETH Zurich (2020).  DDD20: end-to-end DAVIS driving dataset.  https://sites.google.com/view/davis-driving-dataset-2020/home.
  * [INTERACTION Dataset Consortium, 2019] INTERACTION Dataset Consortium (2019).  INTERACTION Dataset.  https://interaction-dataset.com/.  Accessed 2021-08-18.
  * [Jeong et al., 2019] Jeong, J., Cho, Y., Shin, Y.-S., Roh, H., and Kim, A. (2019).  Complex urban dataset with multi-level sensors from highly diverse urban environments.  The International Journal of Robotics Research, 38(6).
  * [Kaggle, 2021] Kaggle (2021).  Datasets - autonomous driving.  https://www.kaggle.com/datasets?search=Autonomous+Driving&sort=updated.  Accessed 2021-08-15.
  * [KAIST, 2017] KAIST (2017).  Visual Perception for Autonomous Driving.  https://sites.google.com/view/multispectral.  Accessed 2021-08-22.
  * [Kim and Hwang, 2021] Kim, S.-h. and Hwang, Y. (2021).  A survey on deep learning based methods and datasets for monocular 3d object detection.  Electronics, 10(4).
  * [Krunal, 2018] Krunal (2018).  Semantic segmentation datasets for urban driving scenes.  https://autonomous-driving.org/2018/07/15/semantic-segmentation-datasets-for-urban-driving-scenes/.  Accessed 2021-08-22.
  * [L3Pilot, 2019] L3Pilot (2019).  Opendd.  https://l3pilot.eu/data/opendd.  Accessed 2021-08-09.
  * [Laflamme et al., 2019] Laflamme, C.-É. N., Pomerleau, F., and Giguere, P. (2019).  Driving datasets literature review.  arXiv preprint:1910.11968.
  * [Level 5, 2020a] Level 5 (2020a).  Perception dataset.  https://level-5.global/data/perception/.  Accessed 2021-08-20.
  * [Level 5, 2020b] Level 5 (2020b).  Prediction dataset.  https://level-5.global/data/prediction/.  Accessed 2021-08-20.
  * [Maddern et al., 2017] Maddern, W., Pascoe, G., Linegar, C., and Newman, P. (2017).  1 year, 1000km: The oxford RobotCar dataset.  The International Journal of Robotics Research (IJRR), 36(1).
  * [Material-UI, 2014] Material-UI (2014).  The React UI library you always wanted.  https://material-ui.com/.  Accessed 2021-08-22.
  * [Material-UI, 2020] Material-UI (2020).  Data grid.  https://material-ui.com/components/data-grid/.  Accessed 2021-08-22.
  * [Motional, 2019a] Motional (2019a).  nuImages.  https://www.nuscenes.org/nuimages.  Accessed 2021-08-20.
  * [Motional, 2019b] Motional (2019b).  nuScenes.  https://www.nuscenes.org/.  Accessed 2021-08-05.
  * [Neumann et al., 2018] Neumann, L., Karg, M., Zhang, S., Scharfenberger, C., Piegert, E., Mistr, S., Prokofyeva, O., Thiel, R., Vedaldi, A., Zisserman, A., and Schiele, B. (2018).  NightOwls: A pedestrians at night dataset.  In Asian Conference on Computer Vision.
  * [Nguyen’s, 2021] Nguyen’s, T. (2021).  List of public large-scale datasets for autonomous driving research.  https://tin.ng/public-datasets-for-autonomous-driving-research/.  Accessed 2021-08-22.
  * [NightOwls dataset, 2018] NightOwls dataset (2018).  NightOwls dataset.  https://www.nightowls-dataset.org/.  Accessed 2021-08-18.
  * [Oxford Robotics Institute, 2017] Oxford Robotics Institute (2017).  Oxford robotcar dataset.  https://robotcar-dataset.robots.ox.ac.uk/.  Accessed 2021-08-05.
  * [Oxford Robotics Institute, 2020] Oxford Robotics Institute (2020).  Oxford Radar RobotCar Dataset.  https://oxford-robotics-institute.github.io/radar-robotcar-dataset/.  Accessed 2021-08-18.
  * [Patil et al., 2019] Patil, A., Malla, S., Gang, H., and Chen, Y.-T. (2019).  The h3d dataset for full-surround 3d multi-object detection and tracking in crowded urban scenes.  arXiv preprint:1903.01568.
  * [Pinggera et al., 2016] Pinggera, P., Ramos, S., Gehrig, S., Franke, U., Rother, C., and Mester, R. (2016).  Lost and found: Detecting small road hazards for self-driving vehicles.  arXiv preprint:1609.04653.
  * [Plesa, 2021] Plesa, N. (2021).  Machine learning datasets.  https://www.datasetlist.com/.  Accessed 2021-08-14.
  * [RadarScenes, 2021] RadarScenes (2021).  RadarScenes.  https://radar-scenes.com/.  Accessed 2021-08-16.
  * [RList, 2021] RList (2021).  List of autonomous driving open datasets.  https://rlist.io/l/list-of-autonomous-driving-open-datasets?utm˙source=insights.rlist.io&utm˙medium=referral.  Accessed 2021-08-16.
  * [Scale, 2019] Scale (2019).  Open datasets.  https://scale.com/open-datasets.  Accessed 2021-08-12.
  * [Schafer et al., 2018] Schafer, H., Santana, E., Haden, A., and Biasini, R. (2018).  A commute in data: The comma2k19 dataset.
  * [Schumann et al., 2021] Schumann, O., Hahn, M., Scheiner, N., Weishaupt, F., Tilly, J. F., Dickmann, J., and Wöhler, C. (2021).  RadarScenes: A real-world radar point cloud data set for automotive applications.  arXiv preprint:2104.02493.
  * [Segment Me If You Can, 2021] Segment Me If You Can (2021).  Datasets.  https://segmentmeifyoucan.com/datasets.  Accessed 2021-08-09.
  * [Sheeny et al., 2020] Sheeny, M., De Pellegrin, E., Mukherjee, S., Ahrabian, A., Wang, S., and Wallace, A. (2020).  RADIATE: A radar dataset for automotive perception.  arXiv preprint:2010.09076.
  * [Sun et al., 2020] Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., et al. (2020).  Scalability in perception for autonomous driving: Waymo open dataset.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
  * [Udacity, 2020] Udacity (2020).  udacity/self-driving-car.  https://github.com/udacity/self-driving-car/.  Accessed 2021-08-22.
  * [Ulbrich et al., 2015] Ulbrich, S., Menzel, T., Reschka, A., Schuldt, F., and Maurer, M. (2015).  Defining and substantiating the terms scene, situation, and scenario for automated driving.  In 2015 IEEE 18th International Conference on Intelligent Transportation Systems.
  * [University of Bonn, 2019] University of Bonn (2019).  SemanticKITTI.  http://www.semantic-kitti.org/.  Accessed 2021-08-22.
  * [Varma et al., 2019] Varma, G., Subramanian, A., Namboodiri, A., Chandraker, M., and Jawahar, C. (2019).  Idd: A dataset for exploring problems of autonomous navigation in unconstrained environments.  In IEEE Winter Conference on Applications of Computer Vision (WACV).
  * [Vision Lab, Perception and Robotics Group, 2020] Vision Lab, Perception and Robotics Group (2020).  Heriot-Watt RADIATE Dataset.  http://pro.hw.ac.uk/radiate/.  Accessed 2021-08-12.
  * [Wang et al., 2019] Wang, P., Huang, X., Cheng, X., Zhou, D., Geng, Q., and Yang, R. (2019).  The apolloscape open dataset for autonomous driving and its application.  IEEE transactions on pattern analysis and machine intelligence.
  * [Waymo LLC, 2019] Waymo LLC (2019).  WAYMO Open Dataset - Perception.  https://waymo.com/open/data/perception/.  Accessed 2021-08-07.
  * [Waymo LLC, 2021] Waymo LLC (2021).  WAYMO Open Dataset - Motion.  https://waymo.com/open/data/motion/.  Accessed 2021-08-07.
  * [Wenzel et al., 2020] Wenzel, P., Wang, R., Yang, N., Cheng, Q., Khan, Q., Stumberg, L. v., Zeller, N., and Cremers, D. (2020).  4seasons: A cross-season dataset for multi-weather SLAM in autonomous driving.  arXiv preprint:2009.06364.
  * [Wikipedia, 2021] Wikipedia (2021).  List of datasets for machine-learning research.  https://en.wikipedia.org/wiki/List˙of˙datasets˙for˙machine-learning˙research#Object˙detection˙and˙recognition.  Accessed 2021-08-15.
  * [Wrenninge and Unger, 2018] Wrenninge, M. and Unger, J. (2018).  Synscapes: A photorealistic synthetic dataset for street scene parsing.  arXiv preprint:1810.08705.
  * [Xie et al., 2016] Xie, J., Kiefel, M., Sun, M.-T., and Geiger, A. (2016).  Semantic instance annotation of street scenes by 3d to 2d label transfer.  In Conference on Computer Vision and Pattern Recognition (CVPR).
  * [Yin and Berger, 2017] Yin, H. and Berger, C. (2017).  When to use what data set for your self-driving car algorithm: An overview of publicly available driving datasets.  In IEEE 20th International Conference on Intelligent Transportation Systems (ITSC).
  * [YonoStore, 2021] YonoStore (2021).  Datasets.  https://store.yonohub.com/product-category/datasets/?filters=product˙cat[19].  Accessed 2021-08-12.
  * [Yu et al., 2020] Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., and Darrell, T. (2020).  Bdd100k: A diverse driving dataset for heterogeneous multitask learning.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  * [Zendel et al., 2018] Zendel, O., Honauer, K., Murschitz, M., Steininger, D., and Dominguez, G. F. (2018).  WildDash - creating hazard-aware benchmarks.  In Proceedings of the European Conference on Computer Vision (ECCV).
  * [Zhan et al., 2019] Zhan, W., Sun, L., Wang, D., Shi, H., Clausse, A., Naumann, M., Kümmerle, J., Königshof, H., Stiller, C., Fortelle, A. d. L., and Tomizuka, M. (2019).  INTERACTION dataset: An INTERnational, adversarial and cooperative moTION dataset in interactive driving scenarios with semantic maps.  arXiv preprint:1910.03088.