# Efficient Social Choice via NLP and Sampling

Lior Ashkenazy and Nimrod Talmon
Ben-Gurion University

###### Abstract

Attention-Aware Social Choice tackles the fundamental conflict faced by some
agent communities between their desire to include all members in the decision
making processes and the limited time and attention that are at the disposal
of the community members. Here, we investigate a combination of two techniques
for attention-aware social choice, namely Natural Language Processing (NLP)
and Sampling. Essentially, we propose a system in which each governance
proposal to change the status quo is first sent to a trained NLP model that
estimates the probability that the proposal would pass if all community
members directly vote on it; then, based on such an estimation, a population
sample of a certain size is being selected and the proposal is decided upon by
taking the sample majority. We develop several concrete algorithms following
the scheme described above and evaluate them using various data, including
such from several Decentralized Autonomous Organizations (DAOs).

##  1 Introduction

We consider the problem of Attention-Aware Social Choice, in which the desire
of a digital community to govern itself through the participation of its
community members conflicts with the limited resources of time and attention
that are at the disposal of the community members. Indeed, direct democracy
has a scalability limitation as community members cannot be subjected to a
huge amount of decisions, which would reduce the quality of the individual
decision making and thus of the collective decisions that are being made.
Thus, there is a need for a decentralized governance system that is efficient,
reliable, and resilient, at scale [11]. Below we list several natural
approaches for attention-aware social choice.

Sampling-based solutions

These are based on randomly choosing subsets of the community as ad-hoc
committees. E.g., a simple sampling-based solution operating on a community of
$n$ agents proceeds by assigning a number of agents that are chosen uniformly
at random for each proposal to change the status quo, allows these agents to
vote on whether to accept or reject the proposal, and decides using majority
vote.

Delegation-based solutions

These are based on allowing voters to delegate certain decisions (possibly
transitively as in liquid democracy [3]).

Cryptoeconomic solutions

Here, economic incentives are aligned to cause certain desired behavioral
properties of the system follow from rational agents’ actions (e.g.,
holographic consensus [10]).

Solutions based on machine learning (ML)

ML can be used to learn agent and community preferences regarding certain
proposals and draw conclusions about future proposals. Concretely, a trained
model can suggest governance actions for community members, or, in an extreme
(and possibly dangerous) setting, a system can completely replace agent
participation in the community decision-making process.

The focus of this paper is on the evaluation of a combination of machine
learning techniques and sampling-based solutions. Our approach is essentially
as follows: we use techniques from machine learning (ML) and natural language
processing (NLP) to develop a trained model that is able to take a textual
governance proposal to change the status quo and estimate the probability that
the community would accept the proposal if directly voted on it; then, based
on the estimation of such a trained model, we select a numerical value
representing the fraction of the community that we then ask to actively vote
on the proposal. We then take the sampled votes and use majority voting to
decide on the proposal. At a glimpse, our solution is composed of an
estimation module – that predicts an acceptance probability for a given
proposal; a sampling module – that, based on the prediction of the estimation
module, selects a vote sample; and a decision module that decides on the fate
of the proposal based on the vote sample. (A more detailed description of the
architecture is given in Section 3.)

Our aim is to evaluate the above-mentioned architecture as a solution approach
to attention-aware social choice. To this end, we develop several variants of
the architecture and evaluate their performance with respect to two data
sources: data obtained from Kaggle111https://www.kaggle.com/ (a popular venue
for ML-related data) as well as data from Snapshot222https://snapshot.org/ (a
popular voting application for digital communities).

##  2 Preliminaries

We provide preliminaries regarding sampling-based techniques in computational
social choice as well as some technical background to NLP.

###  2.1 Sampling-Based Techniques in Social Choice

There are some works that consider sampling-based techniques for decision
making processes. For example, Bhattacharyya and Dey [8] study sampling-based
algorithms for computing election winners when votes arrive in a stream. Dey
et al. [9] consider similar algorithms but for multiwinner voting rules that
aim at proportional representation.

Our use of sampling is as one module in our overall architecture that aims at
solving attention-aware social choice.

###  2.2 Natural Language Processing

Natural Language Processing (NLP) – a sub-field of artificial intelligence and
linguistics – deals with problems that relate to the processing, manipulation,
and understanding of natural language to allow for computer reasoning on text
[12]. Deep learning models are very popular in NLP, although they require
massive amounts of labeled data; indeed, assembling this sort of big data set
is one of the main challenges of NLP [2, 7]. Some of the main functions of NLP
algorithms are: text classification – involving categorization of text by
tagging; text extraction – involving text summarization; machine translation;
and more. NLP employs both _syntactic analysis_ – based on the arrangement of
words in a sentence; and _semantic analysis_ – based on analyzing the use of
and the meaning behind words.

#### Words as Features

In NLP, it is likely that some words in a large text corpus are very prevalent
(e.g. ”the”, ”a”, and ”is” in English), thus conveying very little information
about the actual content of a document. By feeding the count data directly to
a classifier, those very frequent terms may ”obscure” the frequencies of
rarer, yet more interesting terms. To re-weight the count features into values
that are suitable for usage by a classifier it is very common to use the TF-
IDF transform [18], a numerical statistic that reflects how important a word
is to a document in the collection or corpus [20]. The TF-IDF value grows
proportionally to how often a word appears in a document, but is offset by the
word’s frequency (the raw frequency of a term in a document) in the corpus,
which helps in reducing the effect of some words being more common than
others.

#### Words as Vector Spaces

A more in-depth approach to NLP involves capturing the meaning of words in
vector spaces, along with modeling their compositionality, hierarchy, and
recursion. Given a vocabulary of words $W$, a classical NLP approach is to
define a $|W|$-dimensional vector with all entries set to $0$ except one entry
that identifies a word $w_{t}\in W$ (a fixed vocabulary of words); this is
called _one-hot encoding_ [15]. The disadvantage of such a representation is
that it does not consider the semantic features of words, and it is rather
voluminous and redundant as each word is represented by a vector [14]. A
related approach is the _bag of words_ model. Here, the entire text is
represented by a vector of size $|W|$, where each component represents the
number of times that each word occurs in the text. Note that also this model
does not take into account word semantics [14]. A more involved solution
consists of the encoding of words into dense vectors that capture the
syntactic and semantic properties of words, allowing related words to be close
in a corresponding metric space. Such representation, in a space whose
dimension is low, compared to the size $|W|$ of the vocabulary, is called a
_word embedding_ [15, 1].

#### Language Models

More advanced text models use _transformers_ to represent text [16]: these are
neural networks that allow the representation of the token (word) to be
influenced by the context [5]. In particular, _Bidirectional Encoder
Representations from Transformers_ (BERT) is a transformer-based ML technique
for NLP [14]. It provides powerful solutions for contextualized word
representations and can create a context-sensitive embedding for each word in
a sentence. Contextual word embedding models such as BERT have dramatically
improved performance for many NLP tasks [19].

##  3 The Overall Architecture

We discuss our overall architecture, which consists of an estimation module, a
sampling module, and a decision module. A graphical representation of our
solution architecture is given in Figure 1.

Figure 1: The Overall Architecture: when a proposal to change the status quo
arrives, it goes to the trained ML model that outputs its estimation on the
probability that the proposal would have passed if voted on by the community
at large. Based on this prediction, a predefined function selects a fraction
of the community to then actively vote on the proposal. The fate of the
proposal is decided based on the majority of the sampled votes.

#### The prediction module

The prediction module gets as input a proposal and outputs an estimation
regarding whether the proposal would have passed if all community members
directly vote on it. This estimated probability is used as the input for the
sampling module.

In our realizations of the architecture, the predition module is implemented
as an ML model that is pre-trained on a corpus of textual proposals that are
labeled by whether the proposal was accepted or rejected.

#### The sampling module

The sampling module gets the estimated probability of the estimation module as
input and, based on it, selects a subset of the votes to the directly vote on
that proposal. The essence and purpose of the sampling module is to reduce the
number of community members that actively need to vote on a given proposal,
thereby limiting the required community attention.

In our realizations of the general architecture, we use different functions
that take as input the probability that the estimation module outputs, and
return a fraction of the population that shall be sampled; then, such a
population fraction is sampled uniformly at random. Intuitively, the larger
the sample size, the more accurate and quality the decision will be. However,
the larger the sample size, the higher the community attention used. As our
goal is to reduce the attention and effort needed for the decision making
process, we explicitly consider the trade-off between the usage of community
attention (i.e., the average sample size) and the quality of the decision-
making process.

#### The decision module

The decision module collects the votes of the vote sample that is selected by
the sampling module and decides o the fate of the proposal.

In our realizations we decide according to the vote majority of the sample.

##  4 The Estimation Module

Next we discuss our specific realizations of the estimation module. Generally
speaking, we use ML models that we train on a corpus of labeled proposals,
based on NLP techniques. First, given the raw textual data, we have chosen two
approaches, one simple approach that is based on TF-IDF, and another that is
based on BERT.

###  4.1 SimpleNLP: An Initial NLP Algorithm

As a starting point, we implemented a simple ML model that converts raw data
into a matrix of TF-IDF features, indicating how important a word is to a
corpus, and incorporates them into a classification algorithm. Moreover, in
order to improve those models, we performed a tuning process for some of the
hyper-parameters of each model using grid search. The following four classic
algorithms have been chosen for the classification task:

Random Forest Classifier: This classifier consists of a combination of tree
classifiers where each classifier is generated using a random vector sampled
independently from the input vector, and each tree casts a unit vote for the
most popular class to classify an input vector [17]. The tuning was made on
the hyper-parameters: n estimators, which is the number of trees in the
forest; and max depth i.e. the maximum depth of the tree.

Multinomial Naive Bayes Classifier: This classifier belongs to a family of
simple probability analyzers based on the Bayes theorem and is based o the
assumption that the value of specific features is always different from the
value of other features. There are several notable advantages to this
classifier – efficiency in time, memory, and CPU usage, as well as higher
quality for small training sets [21]. Technically, this is a generative model:
it assumes that each document (i.e., proposal) is generated by selecting some
class for it and then generating each word of that document independently
according to a class-specific distribution [22].

The tuning of this classifier was made on the hyper-parameters: alpha, which
is the additive smoothing parameter; and fit prior, which decides whether to
learn prior probabilities or not.

Logistic Regression: This model is a general linear regression model. Its
technique allows for the examination of the impact of numerical factors on
binary responses. The logistical function, also known as the sigmoid function,
is used to calculate the logistic model in which the output is in the range
between $0$ and $1$. Due to its capability to understand vector variables and
evaluate the coefficients or weights of each input variable, this model can be
used in ML applications [21].

Its tuning was made on the hyper-parameters: C, which is the inverse of
regularization strength (smaller values specify stronger regularization); and
class weight, which means the weights associated with classes.

###  4.2 AdvancedNLP: An Improved NLP Algorithm

For improving SimpleNLP, we wanted to represent the data (in particular, the
words) in a better way; specifically, in a way that takes into account both
the meaning of the words as well as their context. This was done using BERT,
which provides strong solutions for representing contextualized words. Next we
explain the steps we worked on when implementing the BERT model.

First, we processed our text by removing special characters and entity
mentions; then we used the BERT tokenizer, which prepares the inputs for the
model and as part of that has a particular way of handling words that are not
in the vocabulary. In addition, we added special tokens to the start and to
the end of each sentence, padded and truncated all sentences to a single
constant length, and explicitly specified what are those padding tokens, using
the ”attention mask”.333The attention mask is a binary tensor indicating the
position of the padded indices so that the model does not attend to them. We
created an iterator for our dataset that helps saving memory during training
and, by doing so, boost the training speed; then, we produced the BERT
classifier and created an _Adaptive Moment Estimation_ (ADAM) optimizer, an
efficient optimization algorithm based on gradient descent, with batch size
$32$, learning rate $5e^{-5}$, and $2$ epochs.

##  5 The Sampling Module

The goal of the sampling module is to maximize the quality of the decision
made on each proposal while minimizing the amount of needed community
attention. In our design of the sampling module we aim at explicitly
considering this trade-off between the attention used from the community and
the quality of the decision making.

Figure 2: A triangle function as a sampling module: the $x$-axis represents
the estimation of the probability of accepting a proposal if a regular vote
were to be held (obtained from the estimation module), while the $y$-axis
represents the sample size to be taken from the community for the proposal.
For the function shown here, if a proposal receives an estimated acceptance
probability that is less than $a$ or greater than $b$, then the proposal is
automatically rejected or accepted (respectively), and, in particular, no
attention is needed from the community (as the value of the function at these
regions is $0$); intuitively, this is so as, in such cases, the proposal is
estimated to be almost in consensus. Between those points (i.e., for values
between $a$ and $b$), the less estimated agreement there is on a proposal the
larger sample size is selected to decide on it; hence the triangular shape.

Intuitively, the more in consensus a proposal is – according to the estimation
module – the smaller the sample size can be; while, correspondingly, the less
in consensus a proposal is – according to the estimation module – the larger
the sample size can be: e.g., if the estimation module estimates the success
probability of a proposal by almost $100\%$, then it is unlikely – given that
the estimation module is realized reasonably well – that even a small sample
size will be incorrect on the proposal; while if the estimation module
estimates the success probability of a proposal to be, say, $51\%$, then even
a high quality realized estimation module may be wrong, and, crucially, a
small vote sample may also be wrong.

Corresponding to the intuition explained above, we consider a family of
functions – _triangle functions_ – as shown in Figure 2, that take as input a
probability and outputs a sample size (as a fraction).

To completely realize the sampling module it is not enough to select the
family of such triangle functions, but a specific triangle function has to be
selected (consider Figure 2). According to the above intuition, as a larger
sample size leads to better decision quality – but also uses more of the
community attention – it would be unfair to compare different triangle
functions that use different amounts of community attention, as the one that
allows for comparatively larger sample sizes would outperform the other, less
attention-demanding one. Thus, to allow for an upright comparison of functions
it is first necessary to divide the family of triangular functions into
subfamilies, one subfamily for each expected use of community attention.

In our experiments (to be described in details in Section 6), we have several
values of required average community attention, specifically: $0.1$, $0.2$,
$0.3$, $0.4$, $0.5$; each such value in effect defines a subfamily of triangle
functions that, given specific data – and hence, given specific output values
from the estimation module – use this value-worth of average community
attention. Equivalently, to determine the relevant functions for a specific,
given required average community attention (i.e., average fractional sample
size), i.e., specific values of the points $a$, $b$, $c$, and $d$, we proceed
as follows. First, for simplicity, we refer only to the case where $a=0$ and
$b=1$, i.e., there is no probability value $z_{i}$ that below it a proposal is
to be automatically rejected or above it a proposal is to be automatically
accepted (equivalently, with a sample fraction of $0$). This means that a
specific triangle function can be represented by a tuple of $(c,d)$. Second,
for a given average community attention value $q$ from the ones mentioned
above, we repeat the following process $100$ times, to get $100$ different
triangle functions whose average community attention equals to $q$: we sample
a value for $c$ uniformly at random between $0$ and $1$ (i.e., $c\sim
U(0,1)$); then, we do a binary search to find a value for $0\leq d\leq 1$ such
that the average community attention of the triangle function represented by
$(c,d)$ corresponds to $q$, with respect to the distribution of the outputs of
the estimation module (for a specific test data, as described in Section 6).
As a result, we have $100$ triangle functions whose expected average community
attention equals $q$, for each $q\in\\{0.1,0.2,0.3,0.4,0.5\\}$.

##  6 Experimental Design

We describe the datasets used for the computer-based simulations as well as
the specific computations and evaluation metrics that were used.

###  6.1 Datasets

In order to examine our architecture, we used real data collected both from
Kaggle, a popular machine learning site, and Snapshot, a popular DAO voting
platform.

#### Kaggle data

Kaggle is a platform that hosts data science competitions for business
problems, recruitment, and academic research purposes [4]. As part of that,
Kaggle find and publish datasets.

Kaggle was used to find similar data that could be treated as proposals and
decision-making. Our goal was to find a dataset contains text-type data that
we could use to run the NLP methods on. Moreover, we searched for data that
has binary tags.

The data we have chosen is the _Research Articles_
dataset.444https://www.kaggle.com/datasets/vetrirah/janatahack-independence-
day-2020-ml-hackathon It consists of an abstract and a title for a set of
research articles and the purpose is to use it to predict the topic of each
article. The research articles are sourced from the following 6 topics:
Computer Science, Physics, Mathematics, Statistics, Quantitative Biology and
Quantitative Finance.

In order to use the above dataset, we had to choose a topic that would serve
as a label; furthermore, ideally, that topic should be relatively balanced. We
chose the topic “Computer Science” to represent our label: the label is $1$ if
the article is associated with this topic and $0$ otherwise.

There are $20,972$ samples in the dataset, but due to computing power
constraints, our dataset contained only $1000$ samples. The training set
contained $85\%$ of the dataset (i.e. $850$ samples) while the test set
comprised of $15\%$ (i.e. $150$ samples). Almost $41\%$ of the training set
and $46\%$ of the test set has the tag $1$; i.e., the data is relatively
balanced. To use the data for our needs, we consider $1$ as an expected
approval of a proposal and $0$ as its rejection.

#### Snapshot data

Snapshot is a popular tool used by certain Decentralized Autonomous
Organizations (DAOs) that allows for community using tokens. The platform
allows for multiple voting systems - Single choice, Approval voting, Quadratic
voting, and more. Considering the platform is an open source project, it
provides access to information of participating digital communities, including
published proposals and votes.

To create a relatively large data set, we combined data from four
organizations: Balancer – an automated portfolio manager and trading platform;
YAM Finance – engaged in DeFi projects; Aavegotchi – a DeFi-enabled crypto
collectibles game; Aave – a decentralized finance protocol. Filtering has been
done so that only binary proposals, with the option of accepting or rejecting
them, were considered.

There are $499$ samples (after the filtering process) in the dataset. The
training set contains $85\%$ of the dataset (i.e., $424$ samples) while the
test set comprises of $15\%$ (i.e., $75$ samples). Almost $86\%$ of the
training set and $83\%$ of the test set is with tag $1$; i.e., the data is
relatively unbalanced. (We will address this issue later by selecting certain
metrics for testing the algorithms.)

###  6.2 Evaluation Metrics

We describe the evaluation metrics used for the estimation module, for the
sampling module, and for the overall realizations of the architecture.

#### The Estimation Module

In the estimation module, each classifier produced a set of probabilistic
estimations for whether each of the proposals would have accepted. For
evaluating and comparing the different algorithmic realizations of the
estimation module, we have chosen the following metrics:

  * •

accuracy: This metric is a classic metric that is equivalent to the proportion
of the number of signals that were predicted correctly to the total of the
input samples [13]. This metric provides an overoptimistic estimate of the
classifier ability on the majority class and therefore suitable only to
balance data [6].

  * •

The F1 score: This metric consists of the _precision_ , which is the
proportion of the true positive samples of the overall predicted positive
observations; and _recall_ , which is the portion of the true positive samples
from the overall predicted negative observations. F1 score is especially
useful for imbalanced data [13], and is formally as follows:
$\textrm{F1-score}=\frac{2\cdot(Recall\cdot Precision)}{Recall+Precision}$.

To evaluate our models we used the accuracy for the Kaggle dataset – which is
balanced – and the F1-score for the Snapshot dataset – which is imbalanced.

#### The Functions Quality

As described earlier, we evaluated the sampling module for five subfamilies of
triangle functions, with a size of $100$ each, where each subfamily
corresponds to one of the five values of the average community attention
$q\in\\{0.1,0.2,0.3,0.4,0.5\\}$. To determine which function is best for each
subfamily, we define the quality of a solution, as follow.

Consider $n$ proposals and a list of estimated probabilities (obtained from
the estimation module), $z_{j}$, $j\in[n]$. Consider a triangle function $f$
whose outputs on those $z_{j}$ values is $t_{j}$, $j\in[n]$ (so
$t_{j}=f(z_{j})$, $j\in[n]$). Recall that the decisions $\widehat{y}_{j}$,
$j\in[n]$ will eventually be made by majority vote of a randomly selected
subset of $t_{j}$, for proposal $j$. Consider the correct decisions $y_{j}$,
$j\in[n]$, which correspond to a full vote on proposal $j$. Then, the quality
of a solution (represented by $\widehat{y}_{j}$, $j\in[n]$ is defined to be
the fraction of computed decisions that are consistent with the correct
decisions; formally:

$I_{i}=\cases{1},\textrm{if }\widehat{y}_{i}=y_{i}\\\ 0,\textrm{otherwise}$
$quality=\frac{\sum_{i=1}^{n}I_{i}}{{n}}$

Due to the random selection of votes (based on the sample size), it is
important to evaluate the average quality. Functions with the highest value
are the best for each subfamily.

##  7 Results

We describe and discuss our results for the estimation module, for the
sampling module, and for the architecture as a whole.

###  7.1 The Estimation Module

Figure 3: Performance For the Kaggle Data Set according to the Accuracy
metric: the $x$-axis represents the four models we have used - the three on
the left are classic classifiers and the fourth is a language model. In the
$y$-axis, the accuracy index values for each model are presented. Figure 4:
Models Performance For Snapshot Data Set According to the F1 Score Metric: the
x-axis represents the four models we used - the three on the left are classic
classifiers and the fourth is a language model. The y-axis represent the F1
score index values for each model.

ML algorithms, both classical and language model, were used in the first stage
of the architecture. Based on the data set characteristics, the way to measure
those algorithms performance was chosen.

On the Kaggle data set the algorithms were examined according to the accuracy
index. Among all the models tested, the language model (BERT) has the highest
index value as shown in Figure 3 but not by a large margin from the Logistic
Regression model, which leads all the other classical models. The results are
consistent with the fact that a language model encompasses both words and
context, so it is more developed and may lead to better predictions.

However, on the Snapshot data set the algorithms were examined according to
the F1 score index. It is evident in Figure 4 that there are no huge
differences in the quality of predictions between the three classical
algorithms. Random Forest and Multinominal NB received identical index values
while the index value of the Logistic Regression was slightly better. In this
study as well, BERT has received the highest index value. However, BERT’s
results are not very high. As mentioned earlier, the data source is a platform
for community members to formulate their own proposals, so the data set may
contain low-quality text which can reduce the metric value.

###  7.2 The Sampling Module

Figure 5: Example of the chosen triangle function for bERT with fixed average
attention size of 0.4: the x-axis represents the estimation of the probability
of proposal acceptance and the y-axis is the attention size needed for the
vote. Here is an example of the BERT triangle function for a fixed average
attention size of 0.4 - the blue triangle is for the Kaggle data set and the
red triangle is for the Snapshot data set.

In the Sampling Module stage, the functions selected based on the
probabilities from the previous phase (ML module). Using five fixed attention
sizes (0.1, 0.2, 0.3, 0.4, 0.5), we selected the functions that were most
suitable under the conditions we assumed. Since the functions are selected by
the probabilities from the ML module stage, there are such functions for each
classifier. Figure 5 illustrates an example of these functions. It seems from
Figure 6, which represents the (c,d) values of the best functions, that for
both data sets, in most cases, the higher the average sample size, the higher
the d value. C value, on the other hand, does not show any trend.

Figure 6: The functions selected for each criterion value and for each
algorithm: the left column in the table represents the criterion value -
average sample size. For each algorithm the values (c, d) of the selected
function are represented. The blue values refer to the Kaggle data set and the
red to the Snapshot data set. Figure 7: Quality VS Fixed Average Attention
Size: the x-axis represents fixed average attention size that was determined.
The y-axis is the highest quality (how accurate was the model) that the model
received.

###  7.3 Quality Of The Overall Architecture

For each classifier from the ML module phase and for each fixed average
attention size, the appropriate function for the sampling module phase was
found. This function was chosen because it gave the highest quality among the
functions tested. Based on Figure 7, we can see that the quality ratio, that
is, the proportion between the amount of decisions that were consistent with
the original decisions and the total decisions, is relatively high. Moreover,
when examining quality against the fixed average attention size, there is an
upward trend.

##  8 Conclusions, Limitations, and Outlook

Attention-aware social choice refers to tackling the conflict rooted at the
desire of a community to include its participants in the decision making
processes despite their limited time and attention. Our work examines a
possible solution that eventually can be used by digital sovereign communities
that face this problem; it proceeds by a combination of two techniques:
Natural Language Processing (NLP) and sampling. The system initially include a
trained NLP model, to which the governance proposal to change the status quo
are sent, that predict the probability that the proposal would pass if a
regular vote were held. Based on these probabilities, the sampling module
determines the number of participants that each proposal will receive, whose
votes will be tallied and a majority decision will be made based on these
votes. We have shown the feasibility of our architecture by implementing
several realization of the proposed architecture and evaluating them using
computer-based simulations on real-world data.

We do have some limitations, though. Next, we discuss some conclusions from
our work (Section 8.1); some of its limitations (Section 8.2); and, finally,
some avenues for future research (Section 8.3).

###  8.1 Conclusions

We discuss some conclusions stemming from our work:

  * •

Given the relatively similar quality of using the simple NLP model and the
more sophisticated one, it seems that, in some sense, the more sophisticated
one is an unnecessary complication for some settings.

###  8.2 Limitations

We discuss some of the limitations of our work:

  * •

Our method may suffer from bias. In particular, as we use learning algorithms
that learn from past decisions towards the prediction of future decisions, we
may use a learning algorithm that is skewed and biased towards the errors of
the past.

One possible remedy to this problem may be societal: Whenever there is a very
controversial decision, the community may inform the algorithm to down-bias
its learning. A complete solution along these lines may naturally be more
involved, though.

  * •

Our method may open new attack vectors on the governance system. In
particular, a malicious agent may submit their proposal over and over, hoping
for the randomness in the prediction algorithm to come in her favor once in a
while.

While a careful analysis is in place, one immediate – albeit only partial –
solution may be to use the prediction algorithm only as a suggesting tool and
let a different institution (e.g., a pre-selected agent committee) make the
sampling decisions.

###  8.3 Outlook

Below we discuss some avenues for future research.

  * •

Further comparisons We have concentrated on one family of (triangle)
functions. Naturally, more families may prove to be more suitable for
different settings; e.g., more continuously-looking Gaussian-shaped functions.

  * •

Further realizations of the architecture: Here we proposed several
realizations for the different modules that comprise the overall architecture.
Developing further realizations for the estimation module, the sampling
module, and the decision module is a natural future direction of research.
This include, in particular, more conservative solutions: Note that, here, we
defined the quality of a solution as simply the fraction of proposals on which
the solution is correct (with respect to the correct decisions). In certain
cases it may be more desirable to optimize a more “conservative” solution, as
a proposal that is accepted even though it should not have been may be more
damaging than vice versa. One possible solution for this would be to evaluate
a different realization of the decision module, e.g., one that requires a
supermajority for a proposal to be accepted.

  * •

A dynamic architecture: We envision a system based on the architecture
presented here to be used by communities in a dynamic way. I.e., while the
treatment here considers a static set of proposals – and thus the required
average attention can be a-priori known – in the real world a dynamic
algorithm to set the specific triangle functions of the sampling module to be
used for each incoming proposal shall be dynamically be computed. Related, a
dynamic generalization of the architecture may use a self-improving
realization, e.g., by using reinforcement learning to tune the realizations of
the different modules as more proposals arrive. (A more advanced possibility
is to pick, once in a while, artificially large sample vote sizes to re-train
the NLP models.)

  * •

Incentivizing participation: If and when a tool based on the architecture
outlined here is to be employed in the real world, certain issues can arise.
First, here we implicitly assumed that the community is very “disciplined”, in
the sense that when the sampling module declares a required sample size indeed
there are sample-size-many voters who show up to vote. Practically, some
economic incentivization may be needed.

## References

  * [1] Emily Alsentzer, John R Murphy, Willie Boag, Wei-Hung Weng, Di Jin, Tristan Naumann, and Matthew McDermott.  Publicly available clinical bert embeddings.  arXiv preprint arXiv:1904.03323, 0(0):0–0, 2019.
  * [2] Avrim Blum and Tom Mitchell.  Combining labeled and unlabeled data with co-training.  In Proceedings of the eleventh annual conference on Computational learning theory, pages 92–100, Germany, 1998. Springer.
  * [3] Christian Blum and Christina Isabel Zuber.  Liquid democracy: Potentials, problems, and perspectives.  Journal of Political Philosophy, 24(2):162–182, 2016.
  * [4] Casper Solheim Bojer and Jens Peder Meldgaard.  Kaggle forecasting competitions: An overlooked learning opportunity.  International Journal of Forecasting, 37(2):587–603, 2021.
  * [5] Adrian MP Braşoveanu and Răzvan Andonie.  Visualizing transformers for nlp: a brief survey.  In 2020 24th International Conference Information Visualisation (IV), pages 270–279, Germany, 2020. IEEE, Springer.
  * [6] Davide Chicco and Giuseppe Jurman.  The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation.  BMC genomics, 21(1):1–13, 2020.
  * [7] Daniel Dahlmeier.  On the challenges of translating nlp research into commercial products.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 92–96, Germany, 2017\. Springer.
  * [8] Palash Dey and Arnab Bhattacharyya.  Sample complexity for winner prediction in elections.  In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pages 1421–1430, UK, 2015. JAAMAS.
  * [9] Palash Dey, Nimrod Talmon, and Otniel Van Handel.  Proportional representation in vote streams.  arXiv preprint arXiv:1702.08862, 0(0):0–0, 2017.
  * [10] Youssef Faqir-Rhazoui, Javier Arroyo, and Samer Hassan.  A scalable voting system: Validation of holographic consensus in daostack.  HICSS, 0(0):0–0, 2021.
  * [11] Matan Field.  Decentralized governance matters, 2018.
  * [12] G Chowdhury Gobinda.  Natural language processing.  Annual Review of Information Science and Technology, 37:51–89, 2003\.
  * [13] Mohamed Hammad, Monagi H Alkinani, BB Gupta, Abd El-Latif, and A Ahmed.  Myocardial infarction detection based on deep neural network on imbalanced data.  Multimedia Systems, 0(0):1–13, 2021.
  * [14] MV Koroteev.  Bert: A review of applications in natural language processing and understanding, 2021.
  * [15] Rémi Philippe Lebret.  Word embeddings for natural language processing.  Technical report, EPFL, 2016.
  * [16] Tianyang Lin, Yuxin Wang, Xiangyang Liu, and Xipeng Qiu.  A survey of transformers, 2021.
  * [17] Mahesh Pal.  Random forest classifier for remote sensing classification.  International journal of remote sensing, 26(1):217–222, 2005.
  * [18] Juan Ramos et al.  Using tf-idf to determine word relevance in document queries.  In Proceedings of the first instructional conference on machine learning, volume 242, pages 29–48, US, 2003. New Jersey, USA, AI.
  * [19] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks, 2019.
  * [20] Gerard Salton and Christopher Buckley.  Term-weighting approaches in automatic text retrieval.  Information processing & management, 24(5):513–523, 1988.
  * [21] Muhammad Yusril Helmi Setyawan, Rolly Maulana Awangga, and Safif Rafi Efendi.  Comparison of multinomial naive bayes algorithm and logistic regression for intent classification in chatbot.  In 2018 International Conference on Applied Engineering (ICAE), pages 1–5, US, 2018. IEEE.
  * [22] Shuo Xu, Yan Li, and Zheng Wang.  Bayesian multinomial naïve bayes classifier to text classification.  In Advanced multimedia and ubiquitous engineering, pages 347–352. Springer, Germany, 2017.


# SensoDat: Simulation-based Sensor Dataset of Self-driving Cars

Christian Birchler  christian.birchler@zhaw,unibe.ch Zurich University of
Applied Sciences
University of BernSwitzerland ,  Cyrill Rohrbach
<EMAIL_ADDRESS>University of BernSwitzerland ,  Timo Kehrer
<EMAIL_ADDRESS>University of BernSwitzerland and  Sebastiano Panichella
<EMAIL_ADDRESS>Zurich University of Applied SciencesSwitzerland

###### Abstract.

Developing tools in the context of autonomous systems (Khatiri et al., 2024,
2023a) such as self-driving cars (SDCs) is time-consuming and costly since
researchers and practitioners rely on expensive computing hardware and
simulation software. We propose SensoDat, a dataset of 32,580 executed
simulation-based SDC test cases generated with state-of-the-art test
generators for SDCs. The dataset consist of trajectory logs and a variety of
sensor data from the SDCs (e.g., rpm, wheel speed, brake thermals,
transmission, etc.) represented as a time series. In total, SensoDat provides
data from 81 different simulated sensors. Future research in the domain of
SDCs does not necessarily depend on executing expensive test cases when using
SensoDat. Furthermore, with the high amount and variety of sensor data, we
think SensoDat can contribute to research, particularly for AI development,
regression testing techniques for simulation-based SDC testing, flakiness in
simulation, etc.
Link to the dataset: https://doi.org/10.5281/zenodo.10307479

††conference: 21st International Conference on Mining Software Repositories;
April 2024; Lisbon, Portugal

##  1\. Introduction

Testing self-driving cars (SDCs) is crucial for maintaining high security
levels and minimizing potential threats to humans. While infield SDC testing
is costly, simulation technologies offer a safer alternative. However,
conducting simulation-based tests demands increased computational resources,
particularly GPUs for accelerated computation of physical dynamics (Mancini et
al., 2023; Arrieta et al., 2018).

To address the challenge of minimizing the costs of simulation-based testing,
recent research focused on various regression testing techniques for
simulation-based tests. Those techniques aim to test SDCs cost-effectively
while maintaining the system’s safety. For instance, by test prioritization
(Arrieta et al., 2019; Birchler et al., 2023b), in which the tests of the test
suite are prioritized, i.e., sorted in a way so that the testing phase reveals
faults of the system earlier.

However, to conduct research on optimizing simulation-based testing,
researchers rely on expensive test executions in simulation environments
(Mancini et al., 2023; Arrieta et al., 2018). These additional computational
costs are mainly due to the expensive computation simulating the physics of
the environment, which is not the case when testing traditional software
systems. Thus, those tests are expensive and often not affordable when a large
amount of data is required, e.g., for ML and DNNs.

To overcome the issue of running simulations to obtain the execution data of
simulations, researchers can use existing datasets of executed simulation-
based tests. There are only a few datasets available consisting of simulation
data for SDCs (Lu et al., 2023; Christian et al., 2021). Despite the existence
of those datasets, in most cases, the simulators used are not maintained
anymore. Popular simulators like Udacity (Inc., 2017), Apollo (Auto, 2023),
SVL (LGSVL, 2023), and DeepDrive (Deepdrive, 2023), were used in the past for
research purposes but unfortunately, the active development of these
simulators has been stopped by the maintainers or have long release cycles.

We propose a dataset consisting of simulation data of executed SDC test cases
in the BeamNG.tech simulation environment. The BeamNG.tech simulator is known
in academia (Birchler et al., 2023d, a; Panichella et al., 2021; Gambi et al.,
2022; Biagiola et al., 2023; Zohdinasab et al., 2023) and is based on the
popular BeamNG.drive game, which is actively developed and maintained by
BeamNG GmbH. The availability of simulation data of SDCs enhances the research
on SDC testing in simulation. Researchers and practitioners do not rely on
executing expensive test cases to develop and evaluate regression testing
techniques. Having the dataset publicly available eases the research for the
domain of SDC software, especially for researchers and practitioners who can
not afford expensive computing hardware. Furthermore, the availability of an
open dataset improves the reproducibility and comparability of research
results in various areas.

Figure 1. High-level dataset generation process

##  2\. Methodology

As illustrated in Figure 1, to start a data collection process of sensor data
of simulation-based SDC test cases, we have to generate test cases with
different test generators (Section 2.1). After the generation of test cases,
we execute those test cases in a simulation environment (Section 2.2). During
the test execution, we collect the data from various simulated sensors of the
SDC.

###  2.1. Test generation

We generate test cases that are based on three different test generators for
the BeamNG.tech simulator. All of these test generators were developed in the
context of tool competitions at the SBST (Panichella et al., 2021; Gambi et
al., 2022) and SBFT (Biagiola et al., 2023) workshop.

⬇

1{

2 "_id": {...},

3 "OpenDRIVE": {

4 "header": {

5 "@name": "mt_set5/frenetic/test-440",

6 "sdc_test_info": {

7 "@test_id": "mt_set5/frenetic/test-440",

8 "@test_outcome": "PASS",

9 "@predicted_test_outcome": "null",

10 "@test_duration": "11.067427158355713",

11 "@is_valid": "True"

12 },

13 "run_config": {

14 "@rf": "1.5",

15 "@oob": "0.5",

16 "@max_speed": "120",

17 "@obstacles": "False",

18 "@bump_dist": "20",

19 "@delineator_dist": "None",

20 "@tree_dist": "None",

21 "@field_of_view": "120"

22 }

23 },

24 "road": {...}

25 },

26 "execution_data": {...}

27}

Listing 1: Metadata of a test execution consisting of simulation configuration
and test outcome

####  2.1.1. Frenetic

The Frenetic tool (Castellano et al., 2021) uses a genetic algorithm to
minimize the distance between the SDC and the edge of the road. For its
computation, it leverages the concepts of frenet frames and curvature-based
road representations. Thus, it converts the solutions back to the Cartesian
space, which is required for the simulator.

####  2.1.2. FreneticV

The FreneticV tool (Castellano et al., 2022) is an extension of Frenetic,
which is able to reduce the amount of invalid roads. An invalid road is for
example a road which has overly sharp turns, or if it intersects with itself.
This definition of road validity is given by the SBST tool competition (Gambi
et al., 2022), for which the tool was developed for.

####  2.1.3. AmbieGen

The AmbieGen (Humeniuk et al., 2022) tool is a test generator that uses a
multi-objective approach using NSGA-II (Deb et al., 2000). Using the diversity
preservation and the fault revealing power as the objectives, AmbieGen
generates test cases that are more likely to reveal faults based on the OOB
metric.

Table 1. Database storage size Collection | # Documents | Storage size
---|---|---
campaign_2_ambiegen | 973 | 108.36 MB
campaign_2_frenetic | 928 | 109.46 MB
campaign_2_frenetic_v | 944 | 41.93 MB
campaign_3_ambiegen | 964 | 109.85 MB
campaign_3_frenetic | 954 | 112.73 MB
campaign_4_ambiegen | 965 | 111.98 MB
campaign_4_frenetic | 964 | 113.87 MB
campaign_4_frenetic_v | 525 | 63.43 MB
campaign_5_ambiegen | 958 | 109.59 MB
campaign_5_frenetic | 945 | 112.29 MB
campaign_5_frenetic_v | 940 | 112.81 MB
campaign_6_ambiegen | 959 | 111.76 MB
campaign_6_frenetic | 944 | 111.11 MB
campaign_6_frenetic_v | 764 | 91.14 MB
campaign_7_ambiegen | 963 | 110.00 MB
campaign_7_frenetic | 967 | 114.09 MB
campaign_7_frenetic_v | 47 | 5.67 MB
campaign_8_ambiegen | 952 | 110.76 MB
campaign_8_frenetic | 952 | 112.25 MB
campaign_9_ambiegen | 953 | 109.20 MB
campaign_9_frenetic | 964 | 113.57 MB
campaign_10_ambiegen | 971 | 63.95 MB
campaign_11_ambiegen | 973 | 72.79 MB
campaign_11_frenetic | 866 | 66.44 MB
campaign_11_frenetic_v | 953 | 73.52 MB
campaign_12_frenetic | 956 | 110.11 MB
campaign_12_freneticV | 942 | 114.00 MB
campaign_13_ambiegen | 954 | 68.83 MB
campaign_13_frenetic | 959 | 72.52 MB
campaign_13_frenetic_v | 951 | 71.48 MB
campaign_14_ambiegen | 959 | 70.14 MB
campaign_14_frenetic | 866 | 64.05 MB
campaign_14_frenetic_v | 934 | 70.16 MB
campaign_15_ambiegen | 952 | 110.50 MB
campaign_15_frenetic | 870 | 102.61 MB
campaign_15_freneticV | 949 | 114.67 MB
Total | 32,580 | 3.34 GB
Table 2. Overview of available types of sensor data

fuel | steering_input | oil | exhaust_flow | 4x brakeCoreTemperature | left_signal | airspeed | brake_input | signal_l
---|---|---|---|---|---|---|---|---
low fuel | rpm spin | lowhighbeam | fog_lights | 4x brakeThermalEfficiency | signal_r | abs | tcs | parking
gear | airflow speed | lowbeam | fuel_volume | 4x brakeSurfaceTemperature | right_signal | steering | ignition | hazard
odometer | lights | high beam | fuel_capacity | engineRunning | tcs_active | isYCBrakeActive | gearboxMode | clutch_input
brake | horn | brakelight_signal_R | gear_a | running | water_temperature | isTCBrakeActive | lightbar | abs_active
throttle | hasABS | brakelight_signal_L | gear_index | low-pressure | brake_lights | driveshaft | headlights | engine_throttle
parking brake | altitude | lowhighbeam_signal_R | gear_m | rpm | check_engine | wheelspeed | oil_temperature | esc_active
throttle_input | dseColor | lowhighbeam_signal_L | hazard_signal | clutch | clutch_ratio | esc | radiator_fan_spin | avg_wheel_av
reverse | virtualAirspeed | turn signal | is_shifting | parkingbrake_input | engine_load | smoothShiftLogicAV | rpm_tacho | freezeState

###  2.2. Simulation platform

We used the BeamNG.tech (BeamNG GmbH, [n. d.]) simulator with SDC-Scissor
(Birchler et al., 2022) to execute the generated test cases. BeamNG.tech is a
widely used simulator for researching testing of SDCs in academia (Panichella
et al., 2021; Gambi et al., 2022; Biagiola et al., 2023). It is a high-
fidelity soft-body physics simulator that accurately simulates the car’s
dynamics, such as the pressure and deformation on the chassis and engine,
clutch, and its related components. We conjecture obtaining as realistic data
as possible using the BeamNG.tech simulator.

The vendor of BeamNG.tech provides researchers with a free academic license,
which makes it highly accessible for researchers. Furthermore, the simulator
provides a Python API to control the simulation process, including defining
the environment with the road shape and gathering sensor data.

###  2.3. Data generation

####  2.3.1. Process

We generated data by conducting 14 simulation campaigns. Every campaign
follows the process as illustrated in Figure 1. We generate test cases with
test generators (Section 2.1). After the generation, we execute the test cases
in the simulation environment and collect the sensor data at runtime.
Furthermore, we collect the information if the SDC violated the OOB safety
metric and, therefore, passes or fails the test case.

####  2.3.2. Infrastructure

The simulations were executed on a machine with the following hardware
specifications: AMD Ryzen 7 3800X 8 core 16 threads, 64 GB DDR4 RAM, NVIDIA
GeForce GTX 1080 8GB, Windows 11.

campaign_2_freneticcampaign_3_freneticcampaign_4_freneticcampaign_5_freneticcampaign_6_freneticcampaign_7_freneticcampaign_8_freneticcampaign_9_freneticcampaign_11_freneticcampaign_12_freneticcampaign_13_freneticcampaign_14_freneticcampaign_15_freneticcampaign_2_frenetic_vcampaign_4_frenetic_vcampaign_5_frenetic_vcampaign_6_frenetic_vcampaign_7_frenetic_vcampaign_11_frenetic_vcampaign_12_freneticVcampaign_13_frenetic_vcampaign_14_frenetic_vcampaign_15_freneticVcampaign_2_ambiegencampaign_3_ambiegencampaign_4_ambiegencampaign_5_ambiegencampaign_6_ambiegencampaign_7_ambiegencampaign_8_ambiegencampaign_9_ambiegencampaign_10_ambiegencampaign_11_ambiegencampaign_13_ambiegencampaign_14_ambiegencampaign_15_ambiegen$0$$200$$400$$600$$800$$562$$604$$627$$630$$613$$622$$608$$619$$550$$603$$643$$594$$538$$638$$344$$634$$508$$26$$622$$624$$675$$681$$609$$566$$514$$495$$543$$519$$518$$517$$484$$523$$492$$519$$546$$516$$366$$350$$337$$315$$331$$345$$344$$345$$316$$353$$316$$272$$332$$306$$181$$306$$256$$21$$331$$318$$276$$253$$340$$407$$450$$470$$415$$440$$445$$435$$469$$448$$481$$435$$413$$436$#
TestsPASSFAIL

Figure 2. Test outcome distribution among MongoDB collections

##  3\. Data storage

We use MongoDB as database technology to efficiently query and perform
analyses on the data. Since the raw data obtained from the simulations is
mainly stored as JSON files, the mapping of JSON files to documents in MongoDB
is straightforward For each simulation campaign, a dedicated collection is
created in the database.

###  3.1. Database setup

First, to set up the database, download the raw data and code from the public
repository as described in Section 7. Furthermore, it is required to have
Docker (Inc., 2023a) installed on the machine.

⬇

1docker-compose -f ./environment/docker-compose.yml up -d \--build

2docker ps # Verify if container is up and running

3docker cp ./data uploader:/app/data

4docker exec uploader unzip ./data/data.zip -d ./data

5docker cp ./code uploader:/app/code

6docker exec -it uploader python ./code/fill_mongodb.py

Listing 2: Database setup with Docker

A locally deployed MongoDB instance will provide all simulation data by
following the aforementioned instructions. The data is accessible by using a
compatible MongoDB client (e.g., MongoDB Compass (Inc., 2023b)) and the
default credentials (username: msr, password: fooBar).

###  3.2. Database query

Given the database is up and running, querying the data is simple. The queries
must conform to the MongoDB language specification as illustrated in a sample
query in Listing 3. In this example, we use pymongo (Inc., 2023c) as a client
library to connect to MongoDB for the Python programming language. The query
operates on the campaign_2_frenetic database collection and counts the number
of passing test executions. For this specific query, we make use of the
document structure as illustrated in Listing 1.

⬇

1from pymongo.mongo_client import MongoClient

2from pymongo.server_api import ServerApi

3

4uri = "mongodb://msr:fooBar@localhost:27017/?authMechanism=DEFAULT"

5client = MongoClient(uri, server_api=ServerApi(’1’))

6db = client.get_database(’sdc_sim_data’)

7

8query_passing = \

9 {"OpenDRIVE.header.sdc_test_info.@test_outcome": "PASS"}

10nr_passing_tests = db[’campaign_2_frenetic’] \

11 .count_documents(filter=query_passing)

12print(nr_passing_tests)

Listing 3: Sample query with pymongo

Adjusting the query string, a user can also query for specific sensor values.
For example, the user can query data for a time range before a fault has
occurred in the simulation. For this, the user needs to identify the location
of the desired field in the database document and write the appropriate query
string.

##  4\. Data charactaristics

Table 1 overviews the total amount of collections and their corresponding
required storage sizes. From the simulator in which we conducted 14 test
campaigns, we obtained total data from 32,580 test executions. Figure 2
depicts the distribution of failing and passing test executions among 14 test
campaigns. In total, we have 19,926 passing and 12,654 failing test
executions. The distribution depends on the out-of-bound (OOB) metric, which
acts as the oracle. For this dataset, we set $OOB=0.5$, in order to have a
fairly balanced dataset of passing and failing tests.

Next to the metadata (see Listing 1), we obtain data from the simulation as
time series. For each data field, we obtain its state annotated with the
timestamp. The timestamp reflects the delta between the start of the
simulation and the time when the data was retrieved.

We capture two types of data as time series:  (i) sensor-based data, and  (ii)
trajectory data.  For the first type, we collect each sensor in Table 2 its
state at every timestamp. Various sensors provide different data types, such
as binary and continuous. For instance, on one hand, hasABS, parking, or
signal_* are binary and show if certain actions or features were active at the
specific timestamp. On the other hand, rpm, wheelspeed, and steering are
examples of data fields on the continuous range. The second type of data
refers to the monitored trajectory of the SDC. We capture the current position
of the SDC at each timestamp. The trajectories in the dataset are a time
series of logged x,y, and z coordinates on the Cartesian coordinate system of
the simulator. In conjunction with the first type of data, the whole state of
the SDC is preserved for analysis purposes. Such analysis can be made for
various areas to enhance SDC software research as discussed in Section 5.

##  5\. Usage & Implications

The availability of sensor and trajectory data enables the evaluation of
testing methodologies without running expensive simulations. Furthermore, it
lowers the barrier for researchers to conduct research since no complex
infrastructure setup is required for simulating SDCs. In the following, we
present various research directions for which our dataset can contribute.

###  5.1. Driver AI development

The field of SDCs is rapidly advancing, but the availability of AI models
specifically designed for SDCs remains limited. Our goal is to contribute to
the progress of open-source driving AI development. Regrettably, numerous
researchers and practitioners face challenges due to the scarcity of computing
resources required for generating training data for these models. By providing
access to our comprehensive dataset, we aim to empower the research community
to overcome these barriers and facilitate the creation of prototypes for
driving AIs. We believe that this collaborative effort will accelerate
advancements in self-driving technology and foster innovation within the
field.

###  5.2. Regression testing in simulation

Various state-of-the-art regression testing techniques for simulation-based
tests use different features, such as road features (Birchler et al., 2023a,
b). Using these features enables a cost-effective testing of SDCs. However,
using our dataset, we can identify and/or develop new features for new
regression testing approaches.

Additionally, empirical evaluation of regression testing techniques for
simulation-based tests is costly since tests must be executed to obtain the
test outcome. The dataset mitigates the issue of running simulations for this
kind of evaluation purposes of regression testing techniques.

###  5.3. CAN bus protocol support

Most modern vehicles have different components that communicate with each
other over a common shared bus system called CAN bus. Using the dataset
consisting of sensor data, testing relevant CAN devices for the lane-keeping
system is feasible with realistic CAN messages as test inputs. Recent studies
(Birchler et al., 2023d, a) already confirmed that the same type of data to
develop testing approaches for CAN devices based on sensor data retrieved from
simulation environments.

###  5.4. Flakiness in simulation

We encourage researchers to investigate with our dataset the flakiness of
simulation-based tests. In other domains, such as for unmanned aerial vehicles
(UAV), we observe different behaviors between the real world and the simulated
test cases and within simulations (Khatiri et al., 2023b).

These non-deterministic, i.e., flaky, behavior of simulators brings up new
challenges, especially in the context of DevOps pipelines. Having flaky tests
in a continuous integration and delivery (CI/CD) pipeline reduces the time to
deployment since these tests lead to build failures. The presence of flaky
tests leads to failing tests that eventually fail the build process.
Furthermore, flaky tests do not necessarily reveal bugs in the system since
only the test code might cause the flaky behavior. In summary, flaky tests are
a challenging problem for simulation-based tests as they are for traditional
software systems. With the dataset, we aim to provide initial execution data
for comparison with the replicated dataset of other researchers.

##  6\. Remarks

The aim of the dataset is to provide a variety of different sensor and
trajectory data from 32,580 SDC simulations for research purposes. Researchers
can profit from the dataset because they do not necessarily need to execute
expensive simulation-based tests to obtain the required data.

We used the BeamNG.tech simulator since several well-known SDC simulators are
outdated and not maintained anymore, which is crucial for developing new SDC
technologies. BeamNG.tech is currently actively maintained by BeamNG GmbH and
is also widely used in academia.

For future datasets, we suggest increasing the amount of data since training
data for several AI techniques is crucial. Furthermore, we encourage
researchers to replicate the dataset for analyses on research topics mentioned
in Section 5.

##  7\. Data availability

All the data and code to set up the MongoDB and compute the statistics is made
available on Zenodo (Birchler et al., 2023c).

## References

  * (1)
  * Arrieta et al. (2018) Aitor Arrieta, Shuai Wang, Ainhoa Arruabarrena, Urtzi Markiegi, Goiuria Sagardui, and Leire Etxeberria. 2018.  Multi-objective black-box test case selection for cost-effectively testing simulation models. In _Proceedings of the Genetic and Evolutionary Computation Conference, GECCO 2018, Kyoto, Japan, July 15-19, 2018_ , Hernán E. Aguirre and Keiki Takadama (Eds.). ACM, 1411–1418.  https://doi.org/10.1145/3205455.3205490
  * Arrieta et al. (2019) Aitor Arrieta, Shuai Wang, Goiuria Sagardui, and Leire Etxeberria. 2019.  Search-Based test case prioritization for simulation-Based testing of cyber-Physical system product lines.  _J. Syst. Softw._ 149 (2019), 1–34.  https://doi.org/10.1016/J.JSS.2018.09.055
  * Auto (2023) Apollo Auto. 2023\.  _apollo_.  Retrieved December 8, 2023 from https://github.com/ApolloAuto/apollo
  * BeamNG GmbH ([n. d.]) BeamNG GmbH. [n. d.].  _BeamNG.tech_.  https://www.beamng.tech/
  * Biagiola et al. (2023) Matteo Biagiola, Stefan Klikovits, Jarkko Peltomäki, and Vincenzo Riccio. 2023.  SBFT Tool Competition 2023 - Cyber-Physical Systems Track. In _IEEE/ACM International Workshop on Search-Based and Fuzz Testing, SBFT@ICSE 2023, Melbourne, Australia, May 14, 2023_. IEEE, 45–48.  https://doi.org/10.1109/SBFT59156.2023.00010
  * Birchler et al. (2022) Christian Birchler, Nicolas Ganz, Sajad Khatiri, Alessio Gambi, and Sebastiano Panichella. 2022.  Cost-effective Simulation-based Test Selection in Self-driving Cars Software with SDC-Scissor. In _IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2022, Honolulu, HI, USA, March 15-18, 2022_. IEEE, 164–168.  https://doi.org/10.1109/SANER53432.2022.00030
  * Birchler et al. (2023a) Christian Birchler, Sajad Khatiri, Bill Bosshard, Alessio Gambi, and Sebastiano Panichella. 2023a.  Machine learning-based test selection for simulation-based testing of self-driving cars software.  _Empir. Softw. Eng._ 28, 3 (2023), 71.  https://doi.org/10.1007/S10664-023-10286-Y
  * Birchler et al. (2023b) Christian Birchler, Sajad Khatiri, Pouria Derakhshanfar, Sebastiano Panichella, and Annibale Panichella. 2023b.  Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual Environments.  _ACM Trans. Softw. Eng. Methodol._ 32, 2 (2023), 28:1–28:30.  https://doi.org/10.1145/3533818
  * Birchler et al. (2023c) Christian Birchler, Cyrill Rohrbach, Timo Kehrer, and Sebastiano Panichella. 2023c.  _SensoDat: Simulation-based Sensor Dataset of Self- driving Cars_.  https://doi.org/10.5281/zenodo.10307479
  * Birchler et al. (2023d) Christian Birchler, Cyrill Rohrbach, Hyeongkyun Kim, Alessio Gambi, Tianhai Liu, Jens Horneber, Timo Kehrer, and Sebastiano Panichella. 2023d.  TEASER: Simulation-Based CAN Bus Regression Testing for Self-Driving Cars Software. In _38th IEEE/ACM International Conference on Automated Software Engineering, ASE 2023, Luxembourg, September 11-15, 2023_. IEEE, 2058–2061.  https://doi.org/10.1109/ASE56229.2023.00154
  * Castellano et al. (2021) Ezequiel Castellano, Ahmet Cetinkaya, Cédric Ho Thanh, Stefan Klikovits, Xiaoyi Zhang, and Paolo Arcaini. 2021.  Frenetic at the SBST 2021 Tool Competition. In _14th IEEE/ACM International Workshop on Search-Based Software Testing, SBST 2021, Madrid, Spain, May 31, 2021_. IEEE, 36–37.  https://doi.org/10.1109/SBST52555.2021.00016
  * Castellano et al. (2022) Ezequiel Castellano, Stefan Klikovits, Ahmet Cetinkaya, and Paolo Arcaini. 2022.  FreneticV at the SBST 2022 Tool Competition. In _15th IEEE/ACM International Workshop on Search-Based Software Testing, SBST@ICSE 2022, Pittsburgh, PA, USA, May 9, 2022_. IEEE, 47–48.  https://doi.org/10.1145/3526072.3527532
  * Christian et al. (2021) Birchler Christian, Khatiri Sajad, Derakhshanfar Pouria, Panichella Sebastiano, and Panichella Annibale. 2021.  _Dataset Package for the paper ”Single and Multi- objective Test Cases Prioritization for Self- driving Cars in Virtual Environments”_.  https://doi.org/10.5281/zenodo.5771017
  * Deb et al. (2000) Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T. Meyarivan. 2000.  A Fast Elitist Non-dominated Sorting Genetic Algorithm for Multi-objective Optimisation: NSGA-II. In _Parallel Problem Solving from Nature - PPSN VI, 6th International Conference, Paris, France, September 18-20, 2000, Proceedings_ _(Lecture Notes in Computer Science, Vol. 1917)_ , Marc Schoenauer, Kalyanmoy Deb, Günter Rudolph, Xin Yao, Evelyne Lutton, Juan Julián Merelo Guervós, and Hans-Paul Schwefel (Eds.). Springer, 849–858.  https://doi.org/10.1007/3-540-45356-3_83
  * Deepdrive (2023) Deepdrive. 2023\.  _Deepdrive_.  Retrieved December 8, 2023 from https://github.com/deepdrive/deepdrive
  * Gambi et al. (2022) Alessio Gambi, Gunel Jahangirova, Vincenzo Riccio, and Fiorella Zampetti. 2022.  SBST Tool Competition 2022. In _15th IEEE/ACM International Workshop on Search-Based Software Testing, SBST@ICSE 2022, Pittsburgh, PA, USA, May 9, 2022_. IEEE, 25–32.  https://doi.org/10.1145/3526072.3527538
  * Humeniuk et al. (2022) Dmytro Humeniuk, Giuliano Antoniol, and Foutse Khomh. 2022\.  AmbieGen tool at the SBST 2022 Tool Competition. In _15th IEEE/ACM International Workshop on Search-Based Software Testing, SBST@ICSE 2022, Pittsburgh, PA, USA, May 9, 2022_. IEEE, 43–46.  https://doi.org/10.1145/3526072.3527531
  * Inc. (2023a) Docker Inc. 2023a.  _Get Started with Docker_.  Retrieved December 4, 2023 from https://www.docker.com/get-started/
  * Inc. (2023b) MongoDB Inc. 2023b.  _Compass. The GUI for MongoDB._ Retrieved December 4, 2023 from https://www.mongodb.com/products/tools/compass
  * Inc. (2023c) MongoDB Inc. 2023c.  _PyMongo 4.6.1 Documentation_.  Retrieved December 4, 2023 from https://pymongo.readthedocs.io/en/4.6.1/
  * Inc. (2017) Udacity Inc. 2017\.  _A self-driving car simulator built with Unity_.  Retrieved December 10 2023 from https://github.com/udacity/self-driving-car-sim
  * Khatiri et al. (2023a) Sajad Khatiri, Sebastiano Panichella, and Paolo Tonella. 2023a.  Simulation-based test case generation for unmanned aerial vehicles in the neighborhood of real flights. In _2023 16th IEEE International Conference on Software Testing, Verification and Validation (ICST)_.
  * Khatiri et al. (2023b) Sajad Khatiri, Sebastiano Panichella, and Paolo Tonella. 2023b.  Simulation-based Test Case Generation for Unmanned Aerial Vehicles in the Neighborhood of Real Flights. In _IEEE Conference on Software Testing, Verification and Validation, ICST 2023, Dublin, Ireland, April 16-20, 2023_. IEEE, 281–292.  https://doi.org/10.1109/ICST57152.2023.00034
  * Khatiri et al. (2024) Sajad Khatiri, Prasun Saurabh, Timothy Zimmermann, Charith Munasinghe, Christian Birchler, and Sebastiano Panichella. 2024.  SBFT Tool Competition 2024 - CPS-UAV Test Case Generation Track. In _IEEE/ACM International Workshop on Search-Based and Fuzz Testing, SBFT@ICSE 2024_.
  * LGSVL (2023) LGSVL. 2023.  _SVL Simulator: An Autonomous Vehicle Simulator_.  Retrieved December 8, 2023 from https://github.com/lgsvl/simulator
  * Lu et al. (2023) Chengjie Lu, Tao Yue, and Shaukat Ali. 2023.  DeepScenario: An Open Driving Scenario Dataset for Autonomous Driving System Testing. In _20th IEEE/ACM International Conference on Mining Software Repositories, MSR 2023, Melbourne, Australia, May 15-16, 2023_. IEEE, 52–56.  https://doi.org/10.1109/MSR59073.2023.00020
  * Mancini et al. (2023) Toni Mancini, Igor Melatti, and Enrico Tronci. 2023\.  Optimizing Highly-Parallel Simulation-Based Verification of Cyber-Physical Systems.  _IEEE Trans. Software Eng._ 49, 9 (2023), 4443–4455.  https://doi.org/10.1109/TSE.2023.3298432
  * Panichella et al. (2021) Sebastiano Panichella, Alessio Gambi, Fiorella Zampetti, and Vincenzo Riccio. 2021.  SBST Tool Competition 2021. In _14th IEEE/ACM International Workshop on Search-Based Software Testing, SBST 2021, Madrid, Spain, May 31, 2021_. IEEE, 20–27.  https://doi.org/10.1109/SBST52555.2021.00011
  * Zohdinasab et al. (2023) Tahereh Zohdinasab, Vincenzo Riccio, Alessio Gambi, and Paolo Tonella. 2023.  Efficient and Effective Feature Space Exploration for Testing Deep Learning Systems.  _ACM Trans. Softw. Eng. Methodol._ 32, 2, Article 49 (mar 2023), 38 pages.  https://doi.org/10.1145/3544792


then we have the conclusion as desired.

Case {$\textsf{label}(n) = f$}:
By inversion, then $L(n) = q_{v'}$, and $\mathsf{children}(n) = n'$, where $L(n') = q_{v}$ and $\mathsf{SAT}(\varphi \wedge f(v) = v')$.
By IH, $n'[v/x] \Downarrow^\varphi v$.
So we have the following deduction:
\[
\inferrule*
n'[v/x] \Downarrow^\varphi v\\
\mathsf{SAT}(\varphi \wedge f(v) = v')
f\ n'[v/x] \Downarrow^\varphi v'
\]
As $f\ n'[v/x] = (f\ n')[v/x]$, we have $(f\ n')[v/x] \Downarrow^\varphi v'$, as desired.

Case {$\textsf{Label}(n) = (\ast,\ast)$}:
By inversion, then $L(n) = q_{(v_1,v_2)}$, and $\mathsf{children}(n) = [n_1,n_2]$, where $L(n_1) = q_{v_1}$ and $L(n_2) = q_{v_2}$.
By IH, $n_1[v/x] \Downarrow^\varphi v_1$ and $n_2[v/x] \Downarrow^\varphi v_2$.
So we have the following deduction:
\[
\inferrule*
n_1[v/x] \Downarrow^\varphi v_1\\
n_2[v/x] \Downarrow^\varphi v_2
(n_1[v/x],n_2[v/x]) \Downarrow^\varphi (v_1,v_2)
\]

$(n_1[v/x],n_2[v/x]) = (n_1,n_2)[v/x]$, so $(n_1,n_2)[v/x] \Downarrow^\varphi (v_1,v_2)$, as desired.

All remaining cases can proceed similarly to $(\ast,\ast)$.
\end{proof}

\begin{theorem}
If {\sc BuildAngelicFTA}$(v_{in}, \varphi)$ returns $\fta$, then $P \angelicmodels_{v_{in}} \varphi$ for every $P \in \mathcal{L}(\fta)$.
\end{theorem}
\begin{proof}[Proof of \autoref{thm:65}]
Assuming that $P$ terminates on every input, this follows from the previous lemma.

\autoref{fig:angelic-semantics}, \autoref{fig:ftacreationrules}
Assume that {\sc BuildAngelicFTA}$(v_{in}, \varphi)$ returns $\fta$ and let $P \in \mathcal{L}(\fta)$.
\[
P \angelicmodels_{v_{in}} \Spec \ \Longleftrightarrow \  \exists v. \ v \in \SemanticsOf{P}^\Spec(v_{in}) \land \mathsf{SAT}(f(v_{in}) = v \wedge \varphi)
\]

Let $P \in \LanguageOf{A}$, then there is a mapping $L$ such that $(P,L)$ is a run and $L(\mathsf{root}(P)) \in Q_f$.
Then by inversion, it follows that $L(\mathsf{root}(P)) = q_v$ for some $v$ where $\mathsf{SAT}(\varphi \wedge f(v_{in}) = v)$.
\end{proof}

\begin{theorem}
The following theorem generalizes this from individual inputs to ground specifications:
Let $\varphi$ be a ground formula such that:
\[
V = \set{ v_i \ | \ f(v_i) \in \mathsf{Terms}(\varphi)}
\]
Then, if {\sc BuildAngelicFTA}$(v_{i}, \varphi)$ returns $\fta_i$ for inputs $V = \{ v_1, \ldots, v_n\}$, then, for every $P \in \mathcal{L}(\fta_1) \cap \ldots \cap \mathcal{L}(\fta_n) $, we have $P \angelicmodels \varphi$.
\end{theorem}
\begin{proof}[Proof of \autoref{thm:66}]
Follows directly from \autoref{thm:65}.
\end{proof}

\fi

\end{document}