# Dynamic Review-based Recommenders

Kostadin Cvejoski13, Ramsés J. Sánchez12, Christian Bauckhage3 and César
Ojeda4  1Competence Center Machine Learning Rhine-Ruhr  2B-IT, University of
Bonn, Bonn, Germany  3Fraunhofer Center for Machine Learning and Fraunhofer
IAIS, 53757 Sankt Augustin, Germany  4Berlin Center for Machine Learning and
TU Berlin, 10587 Berlin, Germany
{kostadin.cvejoski<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Just as user preferences change with time, item reviews also reflect those
same preference changes. In a nutshell, if one is to sequentially incorporate
review content knowledge into recommender systems, one is naturally led to
dynamical models of text. In the present work we leverage the known power of
reviews to enhance rating predictions in a way that (i) respects the causality
of review generation and (ii) includes, in a bidirectional fashion, the
ability of ratings to inform language review models and vice-versa, language
representations that help predict ratings end-to-end. Moreover, our
representations are time-interval aware and thus yield a continuous-time
representation of the dynamics. We provide experiments on real-world datasets
and show that our methodology is able to outperform several state-of-the-art
models. Source code for all models can be found at [1].

###### Index Terms:

recurrent recommender networks, dynamic language model, attention for
recommendation

##  I Introduction

Following the deep learning agenda, the success of modern recommender systems
heavily relies on their ability to leverage meaningful representations that
allow for the accurate prediction of a purchase or a rating. Fundamentally one
must dwell into interest modeling, as an effective recommendation is such that
it uncovers, for a given user, a hidden interest in an unknown item. It is
natural to study the change of user interest with time and, in the present
work, we seek to incorporate these dynamical notions with those of text
reviews.

Reviews are effectively a form of recommendation, and one that is directly
provided by the user. The challenge however, stems from the unstructured and
ambiguous nature of reviews (and natural language itself). A user might, for
example, simultaneously highlight positive and negative aspects of the
different items she reviews. Following current trends in natural language
processing, we leverage review content through neural models of text and
attention mechanisms, and guarantee the information content of those
representations via reproduction quality. We encourage the dynamical aspect of
text recommendations by learning representations which help predict both when
is the next review arriving and what does it say. One is then led naturally to
dynamical language models, since enforcing good text predictions ensures its
dynamical representation quality.

##  II Related Work

There is a large body of research invested in recommender systems (RS), a big
part of which has lately been devoted to capture the temporal dynamics of both
users and items. One of the first temporal models for recommendation is the
TimeSVD++ [2], which extends the SVD++ matrix factorization algorithm by
introducing time-dependent latent factors. From the neural network
perspective, many models for RS have been developed [3, 4, 5], and Recurrent
Neural Networks (RNNs) have been used to capture time-ordered user activity.
For example, session-based item recommendation use RNNs to infers user
preferences from sessions of user behaviour [6, 7, 8, 9]. Another example,
closer to our work, is the Recurrent Recommender Networks (RRN) which uses two
independent RNNs to model user and item dynamics separately [10].

Just as with user (and item) temporal representations, including review
content representations has also been shown to significantly improve rating
prediction and item recommendation [11, 12, 13, 14]. However, some of these
models break causality, in the sense that they either use the review of the
item whose rate one is predicting, or use item reviews that have not been
received by the time the item of interested was rated.

Finally, a model that combines RRN (a dynamical RS) with character-based
autoregressive language models for reviews has recently been develop [15].
This work however, does not leverage the review content for rating prediction.

In contrast to all these works, we combine dynamical recommender systems with
a dynamical language model that captures review content evolution, and use the
review representations together with the user-item temporal representations in
a causal fashion, to predict the rating of the next review.

##  III Dynamic Review-based Recommenders (DRR)

Figure 1: Dynamic Review-based Recommender. The model consists of three
interacting components: (i) a temporal model composed of two RNNs, one for
users and the other for items, which we called Dynamic Model of Review
Sequences; (ii) a neural language model which leverages the temporal
representations of both user and items, and which we called Dynamic Model of
Review Content; and (iii) a Rating Model which combines the user and item
temporal representations with the review content representations to predict
ratings. Note that when $q=t$ in the language model component, the Dynamic
Review-based Recommender is causal. The model is non-causal when $q=t+1$.

The interests and preferences of users vary as they age, or change their
social status or lifestyle. Exogenous factors like trends or seasons also
affect user preferences. For example, users tend to look for different clothe
types in winter than those they look for in summer. Users also tend to change
their music tastes as they age. Such preference changes are naturally encoded
in the collections of reviews and ratings given by these users over time. Our
goal is to learn representations capturing them. We therefore develop a model
that explicitly uses the text content and ratings of past reviews together
with the history of when those reviews were written to better predict user
interest in unknown items.

Consider a dataset $\mathcal{D}$ with a number of $V$ items (as e.g.
businesses or services, movies, products, etc.) and a number of $U$ users. An
element $e\in\mathcal{D}$ consists of a sequence of $N_{e}$ reviews
$\mathbf{r}_{e}=\\{(\mathbf{x}^{e}_{t},\tau^{e}_{t},\delta^{e}_{t},\mathbf{y}^{e}_{t})\\}_{t=1}^{N_{e}}$,
where the $t$-th review is composed of its text $\mathbf{x}^{e}_{t}$, creation
time $\tau^{e}_{t}$, inter-review time
$\delta^{e}_{t}\equiv\tau^{e}_{t}-\tau^{e}_{t-1}$ and rating vector
$\mathbf{y}^{e}_{t}$.

Such review sequences $\mathbf{r}_{e}$ effectively define time series, and
each of these can either be associate with a user $u$ (in which case we set
$e=u$), or an item $v$ (in which case $e=v$).

Thus the rating vector for user $u$ is such that
$\mathbf{y}^{u}_{t}\in\mathbb{R}^{V}$, with $\mathbf{y}^{u}_{t,v}=p$ if user
$u$ rated item $v$ with rating $p$. Conversely, the rating vector for item $v$
is such that $\mathbf{y}^{v}_{t}\in\mathbb{R}^{U}$. Note that both of these
vectors are large and sparse. To process them efficiently we perform
dimensionality reduction via hashing, following [16].

Our main idea is to model the user and item review sequences separately, via
two independent RNNs which output temporal representations encoding the
nonlinear relations between timing and rating of past reviews. We then feed
these temporal representations to neural models of text, thereby yielding
instantaneous review content models, while simultaneously use them to predict
when are new reviews going to arrive and what are their ratings. The model
thus consists of tree interacting components: a temporal model composed of two
RNNs, one for users and the other for items, which we called Dynamic Model of
Review Sequences, a neural language model which leverages the temporal
representations of both user and items, and which we called Dynamic Model of
Review Content, and a Rating Model which combines the user and item temporal
representations with the review content representations to predict ratings. In
what follows we dwell into the details of these building blocks. Figure 1
summarizes the Dynamic Review-based Recommender (DRR) model.

###  III-A Dynamic Model of Review Sequences

Given a sequence of reviews $\mathbf{r}_{e}$, we process each of its elements
recursively via a RNN with hidden state $\mathbf{h}_{t}^{e}\in\mathbb{R}^{H}$.
At each timestep $t$, we first compute the hidden representation

$\mathbf{z}^{e}_{t}=\mathbf{W}^{e}_{\tau}\tau^{e}_{t}+\mathbf{W}^{e}_{\delta}\delta^{e}_{t}+\mathbf{W}^{e}_{y}\mathbf{y}_{t}^{e}+\mathbf{b}^{e},$
(1)

where $\mathbf{W}^{e}_{\tau},\mathbf{W}^{e}_{\delta},\mathbf{W}^{e}_{y}$ and
$\mathbf{b}^{e}$ are learnable parameters and
$\mathbf{z}^{e}_{t}\in\mathbb{R}^{E}$. We then update the RNN’s hidden state
thus

$\mathbf{h}^{e}_{t}=f_{\theta}^{(e)}(\mathbf{z}^{e}_{t},\mathbf{h}^{e}_{t-1}),$
(2)

where $f_{\theta}^{(e)}$ is implemented by a LSTM network [17].

Note that the superindex $e$ is used here to emphasize that we have two sets
of functions namely, one for the user ($e=u$) and one for the item ($e=v$)
reviews. The temporal representation $\mathbf{h}_{t}^{e}$ thus defined not
only encodes the history of ratings, but also the time lag between past
reviews, thereby yielding a continuous-time representation of the dynamics.

To enforce encoding quality, we first use $\mathbf{h}_{t}^{e}$ to predict the
arrival time of new reviews via a simple Review Creation Model, which we shall
now introduce. Later we will explicitly use $\mathbf{h}_{t}^{u}$ and
$\mathbf{h}_{t}^{v}$ to predict ratings through a Rating Model.

####  III-A1 Review Creation Model

The inter-review times $\delta^{e}_{t}$ can be modeled as following an
exponential distribution whose rate parameter
$\lambda^{(e)}_{\theta}(\mathbf{h}^{e}_{t})$ is a function of the temporal
representation $\mathbf{h}_{t}^{e}$ [18, 19]. In practice we approximate the
function $\lambda_{\theta}^{(e)}:\mathbb{R}^{H}\rightarrow\mathbb{R}_{>0}$
with a multi-layer perceptron. The log-likelihood of the Review Creation Model
is then

$\begin{split}\log{p(\mathbf{\delta}^{e})}=&\sum^{N_{e}}_{t=1}\log{p_{\theta}(\delta_{t+1}^{e}|\mathbf{h}_{t}^{e})}\\\
=&\sum^{N_{e}}_{t=1}\left(\log\lambda_{\theta}^{(e)}\left(\mathbf{h}_{t}^{e}\right)-\lambda_{\theta}^{(e)}\left(\mathbf{h}_{t}^{e}\right)\delta_{t+1}^{e}\right).\end{split}$
(3)

Note that predicting the arrival times of new reviews can be done by either
sampling the exponential distribution, or using the mean of the distribution
directly. In our experiments we use the mean of the distribution.

###  III-B Dynamic Model of Review Content

Consider the $t$-th review in the sequence $\mathbf{r}_{e}$, whose text
content is given by
$\mathbf{x}^{e}_{t}=(\mathbf{w}^{e,t}_{0},\mathbf{w}^{e,t}_{1},\dots,\mathbf{w}^{e,t}_{L^{e}_{t}})$,
where $\mathbf{w}^{e,t}_{j}$ and $L^{e}_{t}$ label the $j$-th word and the
number of words in that review, respectively. To capture how the review
content changes within $\mathbf{r}_{e}$, we define the probability of
observing the word sequence $\mathbf{x}^{e}_{t}$ at the $t$-th review as the
conditional probability $p(\mathbf{x}^{e}_{t}|\mathbf{h}_{t-1})$. Here we
define the global temporal representation $\mathbf{h}_{t}$ encoding the
nonlinear relations between timing and ratings of past reviews as
$\mathbf{h}_{t}\equiv\mbox{concat}([\mathbf{h}_{t}^{u},\mathbf{h}_{t}^{v}])$,
with $\mathbf{h}_{t}^{u,v}$ defined in Eq. 2.

Note that when processing the dataset $\mathcal{D}$, the modeling of review
content does not need to differentiate between user and item. We therefore
drop the superindex $e$ in what follows.

Below we present two models for $p(\mathbf{x}_{t}|\mathbf{h}_{t-1})$, one
based on a Bag-of-Words (BoW) representation, and another on an autoregressive
language model. Both models will be trained by maximising $\log
p(\mathbf{x}_{t}|\mathbf{h}_{t-1})$. These language models will ultimately
allow us to define a vector representation $\bar{\mathbf{s}}_{t}$, summarizing
the content of the $t$-th review, which we will later use as input to our
Rating Model.

####  III-B1 Bag-of-Words Neural Review Model

We assume the words in $\mathbf{x}_{t}$ are generated independently,
conditioned on $\mathbf{h}_{t-1}$, that is
$p(\mathbf{x}_{t}|\mathbf{h}_{t-1})=\prod_{j}^{L_{t}}p_{\theta}(\mathbf{w}^{t}_{j}|\mathbf{h}_{t-1})$,
where we follow [20] and write the probability over words as

$\begin{split}p_{\theta}(\mathbf{w}^{t}_{j}|\mathbf{h}_{t-1})&=\frac{\exp{\\{-a(\mathbf{w}^{t}_{j},\mathbf{h}_{t-1})\\}}}{\sum_{k=1}^{V}\exp{\\{-a(\mathbf{w}^{t}_{k},\mathbf{h}_{t-1})\\}}},\\\
a(\mathbf{w}^{t}_{j},\mathbf{h}_{t-1})&=-\mathbf{h}^{\top}_{t-1}\mathbf{R}\,\mathbf{w}^{t}_{j}-\mathbf{b}\,\mathbf{w}^{t}_{j},\end{split}$
(4)

with $\mathbf{R}\in\mathbb{R}^{2H\times V}$ and $\mathbf{b}\in\mathbb{R}^{V}$
trainable parameters,
$\mathbf{h}_{t}=\mbox{concat}([\mathbf{h}_{t}^{u},\mathbf{h}_{t}^{v}])$ and
$\mathbf{w}^{t}_{j}$ the one-hot representation of the $j$-th word in
$\mathbf{x}_{t}$.

We define the summary representation for $\mathbf{x}_{t}$ as the Bag-of-Words
(BoW) representation $\bar{\mathbf{s}}_{t}\in\mathbb{R}^{V}$, where $V$ is the
vocabulary size [21].

####  III-B2 Autoregressive Review Model

In contrast to the BoW model above, autoregressive language models approximate
the probability over the word sequence $\mathbf{x}_{t}$ as [22]

$p(\mathbf{x}_{t}|\mathbf{h}_{t-1})=\prod^{L_{t}}_{j=1}\,p_{\theta}(\mathbf{w}^{t}_{j}|\mathbf{w}^{t}_{<j},\mathbf{h}_{t-1}),$
(5)

where $\mathbf{w}^{t}_{<j}$ labels all words previous to $\mathbf{w}^{t}_{j}$.

The conditional probability above depends on both $\mathbf{w}^{t}_{<j}$ and
the global temporal representation $\mathbf{h}_{t}$. To model it we take an
approach akin to that of the variational autoencoders of text [23]. That is,
we first concatenate $\mathbf{h}_{t}$ with all word embeddings in
$\mathbf{x}_{t}$, i.e. we define
$\tilde{\mathbf{w}_{j}}=\mbox{concat}[\mathbf{w}_{j},\mathbf{h}_{t-1}]$, and
then process the new vector sequence with a RNN with hidden state
$\mathbf{s}_{k}^{t}\in\mathbb{R}^{S}$, whose update equation reads
$\mathbf{s}^{t}_{j}=g_{\theta}(\tilde{\mathbf{w}_{j}},\mathbf{s}^{t}_{j-1})$.
Here $g_{\theta}$ is implemented by a LSTM network, with equations similar to
those below Eq. (2).

The distribution $p_{\theta}$ is then defined as a categorical distribution
over a vocabulary of size $V$, whose class probabilities are given by
$\boldsymbol{\pi}^{t}_{j}=\mbox{softmax}(\mathbf{W}\,\mathbf{s}^{t}_{j})$,
where $\mathbf{W}\in\mathbb{R}^{V\times S}$ is a learnable matrix.

We now define the summary representation for $\mathbf{x}_{t}$ as a weighted
sum over word representations
$\bar{\mathbf{s}}_{t}=\sum_{j}^{L_{t}}\alpha_{j}^{t}\mathbf{s}^{t}_{j}$ where
the $j$-th weight $\alpha_{j}^{t}$ is calculated with the gated attention
mechanism proposed in [24]

$\begin{split}\alpha_{j}^{t}=&\mbox{softmax}(\mathbf{k}_{j}^{\top}\mathbf{q}),\\\
\mathbf{k}_{j}=&\tanh(\mathbf{M}_{1}\,\mathbf{s}^{t}_{j}+\mathbf{b}_{1})\odot\sigma(\mathbf{M}_{2}\,\mathbf{s}^{t}_{j}+\mathbf{b}_{2}),\end{split}$
(6)

where $\mathbf{M}_{1},\mathbf{M}_{2}\in\mathbb{R}^{A\times S}$ and
$\mathbf{b}_{1},\mathbf{b}_{2}\in\mathbb{R}^{A}$ are learnable parameters,
$\mathbf{q}\in\mathbb{R}^{A}$ can be interpreted as a learnable global query,
$\odot$ denotes element-wise multiplication and $\sigma(\cdot)$ denotes the
sigmoid function. This type of attention is introduced to solve the problem of
the limited expressiveness of the $\tanh(\cdot)$ to capture complex relations,
due to the fact of approximate linearity in the region $[-1,1]$.

As we shall see below, this attentive summary representation allows us to
track the most relevant words affecting the rating of a given item as time
evolves.

###  III-C Combining temporal and summary representations

Given the temporal representations for user and item reviews (i.e.
$\mathbf{h}_{t}^{u}$, $\mathbf{h}_{t}^{v}$), and the summary representation
for review content $\bar{\mathbf{s}}_{t}$, we want to predict the rating
$\hat{y}^{uv}_{t}\in\mathbb{R}$ that user $u$ gives to item $v$. There is,
however, still the question of how to combine $\mathbf{h}_{t}^{u}$ and
$\mathbf{h}_{t}^{v}$ with $\bar{\mathbf{s}}_{t}$. After exploring different
possibilities we found two optimal solutions namely,

####  III-C1 DRR-BoW

For the Bow Neural Review Model we augment Eq. (1) and define
$\tilde{\mathbf{z}}_{t}^{e}=\mathbf{z}_{t}^{e}+\mathbf{W}^{e}_{s}\,\bar{\mathbf{s}}_{t}$,
where $\mathbf{W}^{e}_{s}\in\mathbb{R}^{H\times S}$ is an additional learnable
weight, to get
$\tilde{\mathbf{h}}_{t}^{e}=f_{\theta}^{(e)}(\tilde{\mathbf{z}}^{e}_{t},\tilde{\mathbf{h}}^{e}_{t-1})$,
where $f_{\theta}^{(e)}$ remains the same as in Eq. (2). The new
representation $\tilde{\mathbf{h}}_{t}^{e}$ now encodes the nonlinear
interaction between timing, rating and text of past reviews.

####  III-C2 DRR-LM

For the Autoregressive Review Model we instead define

$\tilde{\mathbf{h}}_{t}^{e}=\mathbf{W}^{(e)}\mbox{concat}([\mathbf{h}_{t}^{e},\bar{\mathbf{s}}_{t}])+\mathbf{b}^{(e)},$
(7)

with
$\mathbf{W}^{(e)}\in\mathbb{R}^{H\times(H+S)},\mathbf{b}^{(e)}\in\mathbb{R}^{H}$
learnable. The resulting representation $\tilde{\mathbf{h}}_{t}^{e}$ also
encodes the interaction between timing, rating and text, albeit through a
different route.

###  III-D Rating Model

We have now all ingredient to predict the rating
$\hat{y}^{uv}_{t}\in\mathbb{R}$ that user $u$ gives to item $v$. We compute
$\hat{y}^{uv}_{t}$ with a factorization machine (FM) [25], here defined as

$\begin{split}\hat{y}^{uv}_{t+1}(\mathbf{h})&=w_{0}+\sum_{i=1}^{2H}w_{i}h_{i}+\sum_{i=1}^{2H}\sum_{j=i+1}^{2H}\langle\mathbf{v}_{i},\mathbf{v}_{j}\rangle
h_{i}h_{j},\\\ \end{split}$ (8)

where
$\mathbf{h}\in\mathbb{R}^{2H}\equiv\mbox{concat}([\tilde{\mathbf{h}}_{t}^{u},\tilde{\mathbf{h}}_{t}^{v}])$,
$w_{0}\in\mathbb{R}$, $\mathbf{w}\in\mathbb{R}^{2H}$ and
$\mathbf{V}\in\mathbb{R}^{2H\times K}$ are learnable parameters, $K$ is set to
10 and $\langle\cdot,\cdot\rangle$ denotes dot product.

We choose the loss function of the Rating Model to be the mean square error
function between $\hat{y}^{uv}_{t}$ and our prediction
$\hat{y}^{uv}_{t}(\mathbf{h})$.

###  III-E DRR Loss function

The complete loss function of the DRR model has therefore three components:
the loss of the Rating Model, the loss of the Dynamic Model of Review
Sequences, which is the negative log-likelihood of an exponential, and the
loss of the Dynamical Model of Review Content, which is the negative log-
likelihood of our word sequence model. Explicitly we write

$\begin{split}\mathcal{L}=&\frac{1}{|\mathcal{D}|}\sum_{u,v\in\mathcal{D}}\sum_{t}(y^{uv}_{t}-\hat{y}^{uv}_{t=1})^{2}\\\
-&\lambda_{1}\sum_{e\in\mathcal{D}}\sum^{N_{e}}_{t=1}\log{p_{\theta}(\delta_{t+1}^{e}|\mathbf{h}_{t}^{e})}-\lambda_{2}\sum_{t}\log
p_{\theta}(\mathbf{x}_{t}|\mathbf{h}_{t-1}),\end{split}$ (9)

where $\lambda_{1},\lambda_{2}\in\mathbb{R^{+}}$ are hyperparameters.

TABLE I: Datasets statistics. The mean and the standard deviation of the number of reviews, sentences and words per review with respect to the user and item. | Automotive | Digital Music | Tools and Home | Pet Supplies
---|---|---|---|---
| user/item | user/item | user/item | user/item
| mean | std | mean | std | mean | std | mean | std
reviews | 6.1/9.3 | 1.7/5.5 | 7.8/9.0 | 6.2/6.5 | 7.8/10.6 | 5.3/9.0 | 7.44/13.7 | 4.4/14.2
sentences | 8.7/9.7 | 6.3/7.5 | 6.3/4.9 | 11.9/10.2 | 8.4/7.5 | 8.1/7.8 | 7.8/6.5 | 7.7/6.9
words | 89.8/101.6 | 68.3/82.5 | 52.5/38.2 | 106.4/94.6 | 80.3/70.5 | 85.6/83.3 | 68.9/55.9 | 71.1/66.2

##  IV Causality

By construction, both DRR-BoW and DRR-LM models above preserve causality — the
models do not use any information from the future to predict ratings. As
mentioned in the introduction, however, most recommender system models that
leverage review content use the review $\mathbf{x}^{v}_{t+1}$, written by user
$u$, to predict the rating $y^{uv}_{t+1}$ given by this same user to the item
$v$. In order to fairly compare our methodology with such models, we use the
degrees of freedom available within the definition of the DRR-LM model and
redefine

$\tilde{\mathbf{h}}_{t}^{e}=\mathbf{W}^{(e)}\mbox{concat}([\mathbf{h}_{t}^{e},\bar{\mathbf{s}}_{t+1}])+\mathbf{b}^{(e)}.$
(10)

This new representations encodes $\bar{\mathbf{s}}_{t+1}$, the summary
representation of the review whose rating it predicts, and breaks causality.
Below we refer to the model using the causal representation Eq. (7) as DRR-
LM-C, whereas we denote the model using the non-causal expression Eq. (10) as
DRR-LM-NC.

Naturally, the causal model is to be preferred as we normally do not have
review content about the item whose rating we want to predict. Nevertheless,
we shall see that the non-causal model lends itself when one is interested in
tracking the words which most affect the rating of a given item as time
evolves.

##  V Experiments and Results

TABLE II: Mean-square error on the rating prediction (* results taken from [12]). | static | non-causal models | causal-models
---|---|---|---
Datasets | PMF* | DeepCoNN* | D-ATT* | AHN* | DRR-LM-NC | RRN | DRR-BoW | DRR-LM-C
A | 0.9187 | 0.7809 | 0.7654 | 0.7314 | 0.7791 | 1.0927 | 0.7838 | 0.8171
DM | 0.8788 | 0.8754 | 0.8506 | 0.8172 | 0.7250 | 0.7961 | 0.7723 | 0.7801
TH | 1.1182 | 0.9856 | 0.9850 | 0.9671 | 0.9264 | 1.0896 | 1.0406 | 1.0656
PS | 1.4340 | 1.2598 | 1.2730 | 1.2515 | 1.0500 | 1.1970 | 1.1734 | 1.1918

Data set To test our model we choose the Amazon dataset [26]. We pick four
5-core subcategory datasets namely, Automotive (A), Digital Music (DM), Tools
and Home (TH) and Pet Supplies (PS). The review creation time is defined as
the difference in days between the original timestamp and the timestamp of the
first review in the dataset. Next we group reviews by day, since the
granularity of the timestamps is day based. All users or items with less than
5 days (i.e. time series with less than 5 points) are removed from the
dataset. The autoregressive language models use the review raw text, changed
into lower case. Preprocessing scripts can be found at [1]. Statistics of the
preprocessed data is summarized in Table I.

Training Our model predicts ratings through the user and item dynamic
representations, which come from two independent RNNs. Simply applying
backpropagation through both sequences is computationally forbidden. In order
to overcome this problem, we train the user and item RNNs alternately. We
first freeze the parameters of e.g. the items’ RNN, and only update those of
the users’ RNN, while back-propagating the gradients of all ratings for a user
batch. The items’ dynamic representations are taken to be fixed. We then
repeat these operations but now with the user parameters and user
representations frozen.

Model Configuration We split each dataset along the time dimension into three
parts: training set (80%), validation set (10%) and test set (10%). We use
grid search on the validation set for hyperparameter tuning. We set the hidden
dimension $H$ of the temporal representation $\mathbf{h}_{t}^{e}$ to 32, and
the embedding dimension $E$ of $\mathbf{z}^{e}_{t}$ to 100. Regarding the
review content models, we set the vocabulary size $V$ to 2000 for DRR-BoW and
to 5000 for DRR-LM. In the latter case we also use GloVe word embeddings [27]
(these corresponds to the $\mathbf{w}_{j}^{t}$ in Eq. (5)) with dimension
$300$. For DRR-LM we also set the attention dimension $A$ to 64 and the
embedding dimension $H^{\prime}$ of the (concatenation of the) temporal and
summary representations to $64$. We use Adam [28] with learning rate 0.0002
and $\beta_{1}=0.9$ and limit the review length to 150 tokens. All methods are
implemented using PyTorch v1.3111https://pytorch.org/. Source code for all
models can be found at [1].

Figure 2: Upper Left: Dynamic attention on the words ’COUNTRY_A’ and ’smell’
for an item in ’Pet Supplies’ dataset. Upper Middle: Review sample from the
beginning of the time series. Upper Right: Review sample from the end of the
time series. **The real names of the countries are replaced with masks
’COUNTRY_A’ and ’COUNTRY_B’ for fairness. Lower Left: Dynamic attention on the
words ’comfortable’ and ’ear’ for an item in the ’Tools and Home’ dataset.
Lower Middle: Review sample from the beginning of the time series. Lower
Right: Review sample from the end of the time series. The darker the highlight
color for a word, the higher its attention value.

Results Given an user and item of interest, the DRR model predicts the arrival
time, rating and the probability over the word sequence of the next review,
and we optimize the model to give the best performance on the rating
prediction task.

Our methodology incorporates modeling the dynamic aspects of user-item
interaction with neural models of review content. To test the importance of
each of these components for the problem of rating prediction, we test our
models against (i) the Probabilistic Matrix Factorization (PMF) [29], which is
a static recommender system which does not model review content; (ii) the RRN
[10], a causal model which learns dynamic user/item representations (albeit
non-continous), but does not model review content; and (iii) three static
models which do leverage review content, namely DeepCoNN [14], D-ATT [13] and
AHN [12]. These last three models are non-casual since they either use the
review of the item whose rate they predict, or use item reviews that have not
been received by the time the item of interest was rated. Table II shows
results for all models on the chosen datasets. We use boldface to highlight
best results in both causal and non-causal cases.

Let us start by focusing on the causal models. First we note that both DRR-BoW
and DRR-LM-C outperform the RRN model, which confirms the known fact that
review content helps in rating prediction tasks. We remark however that in
this case the models in questions are dynamic, and it is the content of past
reviews what is successfully being used. Interestingly, DRR-BoW beats DRR-LM-C
which may hint at the fact that it is enough to know that certain key words
are present in the review, as opposite to e.g. word order, to better predict
the rating. Regarding the non-causal models, DRR-LM-NC outperforms all other
models in almost all datasets, which shows that one indeed needs to not only
account for review content, but also for its dynamic character. Remarkably,
both causal models DRR-BoW and DRR-LM-C perform better than all their non-
causal competitors in two of the datasets (see the Digital Music and Pet
Suplies rows in the table), and comparable to them in the others.

We can conclude that our models successfully learn both temporal user/item
representations and review content representation which together are useful
for rating prediction.

Let us now consider the dynamic attention mechanism of the DRR-LM-NC, which
allows us to e.g. follow in time the weights $\alpha_{j}^{t}$ (defined in Eq.
6) of the words in the reviews for the item whose rate we aim at predicting.
The higher the weight of a word, the stronger its relevance to the rating
prediction. Figure 2 Upper Left shows the attention weights on the words
‘COUNTRY_A’ and ‘smell’ as time evolves for a given product in the ‘Pet
Supplies’ dataset. One can see that although at the start of the time series
the word ‘smell’ was important for determining the rating, its relevance
decreases as the weight on the word ‘COUNTRY_A’ increases. After a closer look
at the reviews we learn that at the start of the time series most reviews were
related to the smell of the product (e.g. whether the dogs were liking the
product’s smell). Later on, however, the manufacturing company moved the
product production to COUNTRY_A, and this event was successfully captured by
our attention model. Figure 2 Upper Middle shows an example review for the
item in question, from the start of the time series. Words with darker
highlights mean here words with higher attention weight. One can see that the
word ‘smell’ is highlighted as important. In contrast, Figure 2 Upper Right
displays a review sampled from the end of the time series, in which one sees
the word ‘COUNTRY_A’ has more relevance than the word ‘smell’. Similarly, the
Lower row of Figure 2 shows the attention weights on the words ‘comfortable’
and ‘ear’ as time evolves for a given product in the ‘Tools and Home’ dataset.

##  VI Conclusion and Feature Work

In this work we proposed a recommender system model which accounts for the
dynamic aspects of user preferences, as reflected in their history of reviews
and ratings. We explicitly learn continuous-time representations for both
users and items, and use these to define dynamic language models for review
content. The latter provided us with review content representations which,
when combined with the temporal user/item representations, proved to be useful
in predicting the hidden interest of users in unknown items. Indeed, our
results outperformed several state-of-the-art recommender system models in
rating prediction tasks, in different datasets.

We also introduced a new dynamic attention mechanism which allowed us to track
the most relevant words for a given rating of an item of interest at a given
instant of time.

Future directions of work include developing attention mechanisms between
reviews with different timestamps, and learning more generic dynamic
representations able to characterize hidden dynamics global to all users.

## Acknowledgment

The authors of this work were supported by the Competence Center for Machine
Learning Rhine Ruhr (ML2R) which is funded by the Federal Ministry of
Education and Research of Germany (grant no. 01—S18038A). Part of the work was
also funded by the BIFOLD-Berlin Institute for the Foundations of Learning and
Data (ref. 01IS18025A and ref. 01IS18037A). We gratefully acknowledge this
support.

## References

  * [1] “Source code,” https://figshare.com/s/2e19d38501d275944487.
  * [2] Y. Koren, “Collaborative filtering with temporal dynamics,” _Commun. ACM_ , vol. 53, no. 4, p. 89–97, Apr. 2010. [Online]. Available: https://doi.org/10.1145/1721654.1721677
  * [3] X. He, L. Liao, H. Zhang, L. Nie, X. Hu, and T.-S. Chua, “Neural collaborative filtering,” in _Proceedings of the 26th international conference on world wide web_ , 2017, pp. 173–182.
  * [4] R. Salakhutdinov, A. Mnih, and G. Hinton, “Restricted boltzmann machines for collaborative filtering,” in _Proceedings of the 24th international conference on Machine learning_ , 2007, pp. 791–798.
  * [5] S. Sedhain, A. K. Menon, S. Sanner, and L. Xie, “Autorec: Autoencoders meet collaborative filtering,” in _Proceedings of the 24th international conference on World Wide Web_ , 2015, pp. 111–112.
  * [6] B. Hidasi, A. Karatzoglou, L. Baltrunas, and D. Tikk, “Session-based recommendations with recurrent neural networks,” _arXiv preprint arXiv:1511.06939_ , 2015.
  * [7] M. Quadrana, A. Karatzoglou, B. Hidasi, and P. Cremonesi, “Personalizing session-based recommendations with hierarchical recurrent neural networks,” in _Proceedings of the Eleventh ACM Conference on Recommender Systems_ , 2017, pp. 130–137.
  * [8] Y. K. Tan, X. Xu, and Y. Liu, “Improved recurrent neural networks for session-based recommendations,” in _Proceedings of the 1st Workshop on Deep Learning for Recommender Systems_ , 2016, pp. 17–22.
  * [9] B. Twardowski, “Modelling contextual information in session-aware recommender systems with neural networks,” in _Proceedings of the 10th ACM Conference on Recommender Systems_ , 2016, pp. 273–276.
  * [10] C.-Y. Wu, A. Ahmed, A. Beutel, A. J. Smola, and H. Jing, “Recurrent recommender networks,” in _Proceedings of the Tenth ACM International Conference on Web Search and Data Mining_. ACM, 2017, pp. 495–503.
  * [11] R. Catherine and W. Cohen, “Transnets: Learning to transform for recommendation,” in _Proceedings of the Eleventh ACM Conference on Recommender Systems_. ACM, 2017, pp. 288–296.
  * [12] X. Dong, J. Ni, W. Cheng, Z. Chen, B. Zong, D. Song, Y. Liu, H. Chen, and G. de Melo, “Asymmetrical hierarchical networks with attentive interactions for interpretable review-based recommendation,” _ArXiv_ , vol. abs/2001.04346, 2019.
  * [13] S. Seo, J. Huang, H. Yang, and Y. Liu, “Interpretable convolutional neural networks with dual local and global attention for review rating prediction,” in _Proceedings of the eleventh ACM conference on recommender systems_ , 2017, pp. 297–305.
  * [14] L. Zheng, V. Noroozi, and P. S. Yu, “Joint deep modeling of users and items using reviews for recommendation,” in _Proceedings of the Tenth ACM International Conference on Web Search and Data Mining_. ACM, 2017, pp. 425–434.
  * [15] C.-Y. Wu, A. Ahmed, A. Beutel, and A. J. Smola, “Joint training of ratings and reviews with recurrent recommender networks,” 2016.
  * [16] K. Weinberger, A. Dasgupta, J. Langford, A. Smola, and J. Attenberg, “Feature hashing for large scale multitask learning,” in _Proceedings of the 26th annual international conference on machine learning_ , 2009, pp. 1113–1120.
  * [17] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, pp. 1735–80, 12 1997.
  * [18] K. Cvejoski, R. J. Sanchez, B. Georgiev, J. Schuecker, C. Bauckhage, and C. Ojeda, “Recurrent point processes for dynamic review models,” in _Workshop on Interactive and Conversational Recommendation Systems at AAAI_ , 2020.
  * [19] K. Cvejoski, R. J. Sánchez, B. Georgiev, C. Bauckhage, and C. Ojeda, “Recurrent point review models,” in _2020 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2020, pp. 1–8.
  * [20] Y. Miao, L. Yu, and P. Blunsom, “Neural variational inference for text processing,” in _International conference on machine learning_ , 2016, pp. 1727–1736.
  * [21] G. E. Hinton and R. R. Salakhutdinov, “Replicated softmax: an undirected topic model,” in _Advances in Neural Information Processing Systems 22_ , Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta, Eds. Curran Associates, Inc., 2009, pp. 1607–1614. [Online]. Available: http://papers.nips.cc/paper/3856-replicated-softmax-an-undirected-topic-model.pdf
  * [22] T. Mikolov, M. Karafiát, L. Burget, J. Cernocký, and S. Khudanpur, “Recurrent neural network based language model.” in _INTERSPEECH_ , T. Kobayashi, K. Hirose, and S. Nakamura, Eds. ISCA, 2010, pp. 1045–1048.
  * [23] S. Bowman, L. Vilnis, O. Vinyals, A. Dai, R. Jozefowicz, and S. Bengio, “Generating sentences from a continuous space,” in _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_ , 2016, pp. 10–21.
  * [24] M. Ilse, J. M. Tomczak, and M. Welling, “Attention-based deep multiple instance learning,” _arXiv preprint arXiv:1802.04712_ , 2018.
  * [25] S. Rendle, “Factorization machines,” in _2010 IEEE International Conference on Data Mining_. IEEE, 2010, pp. 995–1000.
  * [26] R. He and J. McAuley, “Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering,” in _proceedings of the 25th international conference on world wide web_ , 2016, pp. 507–517.
  * [27] J. Pennington, R. Socher, and C. D. Manning, “Glove: Global vectors for word representation,” in _Empirical Methods in Natural Language Processing (EMNLP)_ , 2014, pp. 1532–1543. [Online]. Available: http://www.aclweb.org/anthology/D14-1162
  * [28] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [29] A. Mnih and R. R. Salakhutdinov, “Probabilistic matrix factorization,” in _Advances in neural information processing systems_ , 2008, pp. 1257–1264.


theoremTheorem[section] propositioncnttheorem
proposition[propositioncnt]Proposition lemmacnttheorem lemma[lemmacnt]Lemma
corollarycnttheorem corollary[corollarycnt]Corollary factcnttheorem
assumptioncnttheorem remarkcnttheorem notationcnttheorem definitioncnttheorem
requirementcnttheorem

# Responsibility-Sensitive Safety
an Introduction with an Eye to Logical Foundations and Formalization

Ichiro Hasuo1,2 1National Institute of Informatics, Hitotsubashi 2-1-2, Tokyo
101-8430, Japan<EMAIL_ADDRESS>(The Graduate University for
Advanced Studies), Hayama, Japan.

###### Abstract

_Responsibility-sensitive safety_ (RSS) is an approach to the safety of
automated driving systems (ADS). It aims to introduce mathematically
formulated _safety rules_ , compliance with which guarantees collision
avoidance as a mathematical theorem. However, despite the emphasis on
mathematical and logical guarantees, the logical foundations and formalization
of RSS are largely an unexplored topic of study. In this paper, we present an
introduction to RSS, one that we expect will bridge between different research
communities and pave the way to a logical theory of RSS, its mathematical
formalization, and software tools of practical use.

##  I What is RSS?

_Responsibility-sensitive safety_ (RSS) is an approach to the safety of
automated driving systems (ADS) that has been attracting growing attention.
Safety is a central theme in every aspect of automated driving system (ADS)
research. RSS is unique among these approaches in that it provides _safety
rules_ that are rigorously formulated in mathematical terms. Unlike most
algorithms and techniques studied for ADS, RSS is not so much about _how to
drive safely_ as about _breaking down the ultimate goal_ (namely safety) _into
concrete and checkable conditions_. The goal of RSS is that those safety rules
guarantee ADS safety in the rigorous form of _mathematical proofs_. How far
RSS has gone towards this ultimate goal of ADS safety proofs, and what is
still needed for this ultimate goal, are questions that we will discuss later
in Section VI.

The methodology and basic techniques of RSS were first introduced by
researchers at Mobileye/Intel [ShalevShwartzSS17RSS]. The safety rules derived
in RSS have a variety of possible usages (as we will discuss in Section V),
giving RSS a multifaceted character. Moreover, its presentation to non-expert
audience such as Mobileye/Intel’s
webpage333https://www.mobileye.com/responsibility-sensitive-safety tends to
emphasize RSS’s ideological aspects. All these have somewhat blurred the
theoretical substance of RSS.

In this introductory paper, in contrast, we aim at a theoretical account of
RSS mainly from the viewpoints of logic and software science. We expect that
such an introductory account will bridge between different research
communities and pave the way to a logical theory, formalization, and software
tools—aspects of the RSS study that are largely unexplored at this moment.
Specifically, in the current paper, we 1) formulate two requirements of safety
rules (Sections II-A and II-B), and 2) formulate their assume-guarantee
relationship (see Fig. 2).

######  Remark  (RSS terminologies).

Although there are already a number of research works on RSS as well as
practical works (including the efforts to use it in international standards),
comprehensive technical accounts on RSS are scarce. In particular,
terminologies for the core concepts in RSS do not seem to be fixed in the
literature.

In this paper, we will use the terminologies listed in Table I. Note that they
may differ from the terminologies used in the literature (which vary from a
paper to another). The notions in the table will be introduced in due course.
Their correspondence to those used in the original paper
[ShalevShwartzSS17RSS] is also shown in the table.

TABLE I: RSS terminologies in this paper |  in [ShalevShwartzSS17RSS] |
---|---|---
RSS safety rule |  (no explicit name) |  A pair of an RSS safety condition and a proper response
RSS safety
condition |  safe distance |  An “instantly-checkable” condition that guarantees safety henceforth. Often stated in terms of a safety distance
proper response |  proper response |  A control strategy that realizes safety
RSS responsibility principle |  safety rule,
common sense rule |  A high-level and informal principle used as assumptions on the behaviors of vehicles

##  II Elements of RSS

###  II-A RSS Safety Rules

The central constructs in RSS are _RSS safety rules_. Here is their semi-
formal definition—a concrete example will appear in Section IV.

######  Definition  (RSS safety rule, RSS safety condition, proper response).

An _RSS safety rule_ is a pair $R=(C,P)$ of an RSS safety condition $C$ and a
proper response $P$, designed specifically for each driving scenario (straight
road or crossing, other cars driving in the same direction or another, etc.).
Here,

  * •

an _RSS safety condition_ $C$ is a rigorous condition formulated in
mathematical terms, which must be _instantly checkable_ (in the sense that it
mentions the current values of physical quantities, not the future ones); and

  * •

a _proper response_ $P$ is a control strategy that achieves safe driving,
typically until the vehicle comes to a halt.

Here is the first fundamental requirement of an RSS safety rule.

######  Requirement  (safety assurance in RSS).

An RSS safety rule $R=(C,P)$ is required to satisfy the following:

> _Whenever the RSS condition $C$ is satisfied, executing the proper response
> $P$ from that moment leads to safe driving (i.e. driving with no
> collision)._

By considering the beginning (at time $0$) of the execution of $P$, the above
requirement implies, in particular, that

> _Whenever the RSS condition $C$ is satisfied, there is no collision at that
> moment._

Ideally, this safety assurance should be mathematically proved, building on
the rigorous formulation of the RSS safety condition $C$ as well as that of
the proper response $P$. Such a mathematical proof makes the requirement a
_theorem_.

A crucial feature of an RSS safety rule $R=(C,P)$ is that it reduces the
problem of safety _in the future_ to a condition that can be verified _at
present_ , namely the RSS safety condition $C$. The RSS safety condition $C$
ensures the safety of executions of the proper response $P$, much like a
_precondition_ in a program logic ensures the safety of executions of a
program (see e.g. [Winskel93]).

Such reduction of the future to the present is not easy. In fact, ensuring
future safety is impossible without posing suitable assumptions about the
behaviors of other vehicles. Figure 1 is a situation that is often used as an
example in the literature (including [ShalevShwartzSS17RSS]): a collision
happens if the other vehicles come close to the subject vehicle (SV); and the
SV can do nothing to avoid it.

Figure 1: the subject vehicle (SV)is surrounded by other vehicles. When they
come close, there is nothing that the SV can do to avoid a collision.

###  II-B RSS Responsibility Principles

It is therefore needed to formulate certain behavioral constraints that all
traffic participants are expected to respect. Such constraints can be thought
of as _contracts_ in driving situations, when the latter are identified with
multi-agent systems. Each traffic participant must respect these constraints;
moreover, each agent can act assuming that all the other agents obey these
constraints. In RSS, these behavioral constraints are derived from a high-
level idea of _responsibility_.

Specifically, RSS expresses its idea of responsibility in the following
informal form of _RSS responsibility principles_. (Note that the name “RSS
responsibility principle” differs from those which appear in the
literature—see Remark I.)

######  Definition  (RSS responsibility principles [ShalevShwartzSS17RSS]).

  1. 1.

Don’t hit the car in front of you

  2. 2.

Don’t cut in recklessly

  3. 3.

Right of way is given, not taken

  4. 4.

Be cautious in areas with limited visibility

  5. 5.

If you can avoid a crash without causing another one, you must

Note that these principles indeed embody a natural idea of responsibility, or
“duties of care,” in driving. They are close to those rules commonly expected
of human drivers.

The uses of the five RSS responsibility principles are twofold. Firstly, they
can be used to establish Requirement II-A—more specifically, in a mathematical
proof of the satisfaction of Requirement II-A, one can derive some assumptions
from the informal principles in Definition II-B, and require them of the other
vehicles’ behaviors.

The second use of the RSS responsibility principles is that, as a participant
in a traffic scenario (that is thought of as a multi-agent system), the
subject vehicle (SV)must itself respect those principles:

######  Requirement  (responsibility in RSS).

An RSS safety rule $R=(C,P)$ is required to satisfy the following:

> Let $E$ be an arbitrary execution of the proper response $P$; assume that
> the execution $E$ starts at a state in which the RSS safety condition $C$ is
> true. Then the execution $E$ respects the RSS responsibility principles
> (Definition II-B).

Requirement II-B is expected to be mathematically proven, too. In doing so,
one has to turn the RSS responsibility principles (that are only informally
stated, Definition II-B) into certain rigorous mathematical conditions, in a
way specific to the driving scenario in question. See Section IV for an
example.

##  III The RSS Framework

The elements of RSS that we have described constitute what we call the _RSS
framework_ , depicted in Figure 2. Each RSS safety rule $R=(C,P)$ is
formulated for an individual driving scenario (“single-lane same-direction,”
“cut in,” “crossing,” etc.)—the design of the rule, as well as the proofs that
it satisfies Requirements II-A–II-B, is heavily dependent on the choice of a
driving scenario. An example will be presented in Section IV.

Figure 2: the RSS framework. An RSS safety rule is formulated for an
individual driving scenario. Each rule is expected to satisfy safety
(Requirement II-A) and responsibility (Requirement II-B). The latter
_guarantees_ that the subject vehicle acts responsibly; conversely, the safety
proof assumes that the other vehicles act responsibly.

In the proof of safety, it is assumed that other vehicles respect the RSS
responsibility principles (Definition II-B)—otherwise safety is often
unachievable, see Figure 1. Conversely, it is needed to show that the SV
respects the RSS responsibility principles, so that other vehicles can rely on
it when they plan their behaviors. This is Requirement II-B (responsibility).

While concrete usages of the RSS framework are discussed later in Section V,
we can already see that conceptual benefits of RSS are significant.

For one, the RSS safety rules give a precise recipe for safe driving, in which
1) triggering conditions and control strategies are specified in mathematical
terms, and 2) safety is established through a mathematical proof. RSS can
therefore be seen as a promising approach to the goal of _formal verification
of ADS safety_.

It is, however, unrealistic to expect that RSS realizes a world with zero
traffic accidents. Even if the SV acts according to the RSS safety rules,
other vehicles may not, especially those driven by human drivers. RSS is still
useful in such situations with traffic accidents. The RSS responsibility
principles explicate natural “contracts” that each traffic participant is
expected to follow. These principles can therefore be used to identify who is
liable for an accident—namely the one who did not respect them.

The last argument can be turned upside down and yield the following:

> _a vehicle is not responsible for an accident as long as it respects the RSS
> responsibility principles_.

This is an answer to a major challenge that is currently hindering large-scale
deployment of automated driving, namely

> _the difficulty of determining the boundary of the responsibilities of
> automated driving systems and their manufacturers._

The above boundary of responsibilities is currently vague for ADS, which
exposes the ADS manufacturers to the risk of unexpected and exceeding
liabilities. This risk is a big burden when a company wants to run an ADS
business, potentially blocking the development and deployment of the ADS
technology. Clarification of the boundary of responsibilities is therefore
pursued by many parties, including standardization bodies such as ISO, UL, and
SAE. Indeed, the use of RSS is often advocated in these standardization
efforts, among which the efforts towards the IEEE 2846 standard are
particularly notable.444https://sagroups.ieee.org/2846/

##  IV Example: an RSS Safety Rule for the Single-Lane Same-Direction Scenario

We exhibit an example of an RSS safety rule and its safety and responsibility
proofs. The example is one of the first examples in the literature and is
taken from [ShalevShwartzSS17RSS].

###  IV-A The Scenario $S_{\mathrm{slsd}}$

The driving scenario in question, denoted by $S_{\mathrm{slsd}}$, has a single
lane and two vehicles driving in the same direction. See Figure 3: the vehicle
behind is the one under our control (the subject vehicle (SV)); the SV drives
behind another vehicle that is called the _principal other vehicle (POV)_.

Figure 3: the single-lane same-direction scenario. The subject vehicle (SV)is
following the other vehicle. The latter is called the _principal other vehicle
(POV)_.

The goal is to avoid collision regardless of the behavior of the principal
other vehicle (POV). Here, however, we rule out some unrealistic but
physically possible behaviors of the POV from our consideration—such as the
POV being hit by a comet and suddenly coming to a halt.555The comet here is an
example of the legal notion of _act of God_.
https://en.wikipedia.org/wiki/Act_of_God The worst case within our
consideration is the POV engaging emergency braking and coming to a halt in a
short moment. We want the SV to stop without colliding with the POV. We want
the SV to do so by comfortable braking and not by emergency braking. Moreover,
we have to take into account the _response time_ , the time between the moment
the POV starts braking and the moment the SV starts its response. All these
require a suitable distance between the POV and the SV; the question is how
large exactly the distance should be.

Note that the above restriction of the scenario’s scope (“no comet”) is
similar to the use of the RSS responsibility principles, although there is no
explicit RSS principle that corresponds to the “no comet” assumption. The
argument here is “if a collision happens because of a comet, the SV is not
held responsible,” which is logically parallel to one that uses RSS
responsibility principles such as “if a collision happens because of another
vehicle’s reckless cut in, the SV is not held responsible.”

###  IV-B The RSS Safety Condition and the Proper Response

In [ShalevShwartzSS17RSS], an RSS safety condition $C_{\mathrm{slsd}}$ and a
proper response $P_{\mathrm{slsd}}$ are given as follows. They together
constitute an RSS safety rule
$R_{\mathrm{slsd}}=(C_{\mathrm{slsd}},P_{\mathrm{slsd}})$.

####  IV-B1 The RSS Safety Condition $C_{\mathrm{slsd}}$

The condition $C_{\mathrm{slsd}}$ is given by

$\displaystyle
x_{f}-x_{r}\;>\;\max\left(\,0,\,v_{r}\rho+\frac{1}{2}a_{\mathrm{max}}\rho^{2}+\frac{(v_{r}+a_{\mathrm{max}}\rho)^{2}}{2a_{\mathrm{brake,min}}}-\frac{v_{f}^{2}}{2a_{\mathrm{brake,max}}}\,\right).$
(1)

Here, the following are dynamic parameters that describe the current state of
the driving situation:

  * •

$x_{f},x_{r}$ are the positions of the front vehicle (POV) and the rear
vehicle (SV), respectively666We ignore the lengths of the cars for simplicity.
; and

  * •

$v_{f},v_{r}$ are their velocities, respectively, modeled in the 1-dimensional
lane coordinate.

Besides, the following are static parameters for the driving scenario—they do
not change from one state to another. Their values are decided according to
traffic laws, regional customs, vehicle specs, etc.:

  * •

$\rho$ is the maximum _response time_ that the rear vehicle might take to
initiate the required braking;

  * •

$a_{\max}$ is the maximum (forward) acceleration rate of the rear vehicle;

  * •

$a_{\mathrm{brake,min}}$ is the maximum comfortable braking rate for the rear
vehicle; and

  * •

$a_{\mathrm{brake,max}}$ is the maximum emergency braking rate for the front
vehicle. We assume $0<a_{\mathrm{brake,min}}<a_{\mathrm{brake,max}}$.

An example of these parameter values is found in [XuWW20TRB]: $\rho=0.3$ s,
$a_{\max}=2$ $\mathrm{m}\mathrm{/}\mathrm{s}^{2}$, $a_{\mathrm{brake,min}}=4$
$\mathrm{m}\mathrm{/}\mathrm{s}^{2}$, and $a_{\mathrm{brake,max}}=8$
$\mathrm{m}\mathrm{/}\mathrm{s}^{2}$.

####  IV-B2 The Proper Response $P_{\mathrm{slsd}}$

A proper response is a control strategy that is expected to _avoid any
collision in the future_ , _no matter how other vehicles would behave_ (within
a prescribed range of possible behaviors—recall the “no comet” argument in the
above). The proper response $P_{\mathrm{slsd}}$ in [ShalevShwartzSS17RSS] is

> to engage the maximum comfortable braking, after a response time that is no
> bigger than $\rho$. The behavior during the response time is arbitrary.

The response time is included since otherwise the control strategy would be
unrealizable. Note that there are in fact many constraints on the behavior
during the response time: some are physical (an instant acceleration to the
speed of light is ruled out, for example); others come from traffic laws, the
design of a car, etc. Among all the possible behaviors during the response
time, the worst case behavior is accelerating at the maximum rate $a_{\max}$.

We note that proper responses in RSS—such as $P_{\mathrm{slsd}}$ in the
above—may be quite harsh, in the sense that they are undesirable in view of
other quality metrics than safety (such as comfort and fuel efficiency). We
will discuss, later in Section V-D, how frequent deployment of proper
responses can be avoided.

###  IV-C The Safety Proof

[safety of $R_{\mathrm{slsd}}$] Let $s$ be a state of the driving scenario
$S_{\mathrm{slsd}}$, and let the positions and the velocities of the vehicles
in $s$ be denoted by $x_{f},x_{r},v_{f},v_{r}$ as described in Section IV-B.

Assume that the state $s$ satisfies the RSS safety condition
$C_{\mathrm{slsd}}$. Consider an execution $e$ of the proper response
$P_{\mathrm{slsd}}$ that starts at $s$ and ends when the SV comes to a halt or
collides into the POV. Then, no collision occurs in this execution $e$ of
$P_{\mathrm{slsd}}$. The proof here follows the outline of the one presented
in [ShalevShwartzSS17RSS].

###### Proof.

In the execution $e$ of $P_{\mathrm{slsd}}$ in question, there are two
arbitrary components: the behavior of the SV during the response time, and the
behavior of the POV. It is however obvious that the worst choices in terms of
safety are 1) the SV accelerates at the maximum rate $a_{\max}$, at the
beginning of $e$, for the longest possible response time $\rho$, and 2) the
POV engages the maximum braking, namely at the rate $a_{\mathrm{brake,max}}$,
for the whole period of $e$. In the rest of the proof, we assume these
behaviors of the SV and the POV without loss of generality.

We study the sign of the relative velocity of the SV with respect to the POV.
To do so, we list up the possible patterns of the time-varying relationship
between the velocities of the SV and the POV. See Figure 4.

  * •

In Case 1, the relative velocity of SV is always positive, therefore the
_inter-vehicle distance_ between the SV and the POV becomes minimum when the
SV comes to a halt.

  * •

In Cases 2–3, the relative velocity of SV turns from negative to positive.
Therefore the inter-vehicle distance becomes minimum either at the start (at
time $0$) or when the SV comes to a halt.

  * •

In Case 4, the relative velocity of SV is always negative, therefore the
inter-vehicle distance becomes minimum at the start (at time $0$).

It is easy to see that the four cases are exhaustive (the assumption
$0<a_{\mathrm{brake,min}}<a_{\mathrm{brake,max}}$ is crucial here). It
therefore suffices to ensure $x_{f}>x_{r}$ both at time $0$ and at the time
when the SV comes to a halt.

A necessary and sufficient condition for avoiding collision at time $0$ is,
obviously,

$x_{f}-x_{r}>0.$ (2)

A condition for avoiding collision at the time when the SV comes to a halt is
derived as follows.

  * •

The SV travels the distance $v_{r}\rho+\frac{1}{2}a_{\mathrm{max}}\rho^{2}$
during the response time, at the end of which its velocity reaches
$v_{r}+a_{\mathrm{max}}\rho$. The subsequent braking phase takes the time
$\frac{v_{r}+a_{\mathrm{max}}\rho}{a_{\mathrm{brake,min}}}$, during which the
SV travels the distance
$\frac{1}{2}(v_{r}+a_{\mathrm{max}}\rho)\frac{v_{r}+a_{\mathrm{max}}\rho}{a_{\mathrm{brake,min}}}$.

  * •

To compute the distance that the POV travels until the SV comes to a halt,
note first that we can restrict to Cases 1–3 in Figure 4—this is because the
minimum inter-vehicle distance is at time $0$ (not when the SV comes to a
halt) in Case 4. We can easily see that, in Cases 1–3, the POV comes to a halt
earlier than the SV does. Therefore the traveled distance for POV (until the
SV comes to a halt) is $\frac{1}{2}v_{f}\frac{v_{f}}{a_{\mathrm{brake,max}}}$.

  * •

We conclude that, in Cases 1–3 (Case 4 can be ignored as argued in the above),
the positions of the SV and the POV when the SV comes to a halt are

$x_{r}+v_{r}\rho+\frac{1}{2}a_{\mathrm{max}}\rho^{2}+\frac{(v_{r}+a_{\mathrm{max}}\rho)^{2}}{2a_{\mathrm{brake,min}}}\quad\text{and}\quad
x_{f}+\frac{v_{f}^{2}}{2a_{\mathrm{brake,max}}},$

respectively. There is no collision at that moment if and only if

$x_{f}-x_{r}>v_{r}\rho+\frac{1}{2}a_{\mathrm{max}}\rho^{2}+\frac{(v_{r}+a_{\mathrm{max}}\rho)^{2}}{2a_{\mathrm{brake,min}}}-\frac{v_{f}^{2}}{2a_{\mathrm{brake,max}}}.$
(3)

The RSS safety condition (1) implies both (2–3), and thus ensures that there
is no collision at the beginning or at the end of an execution of
$P_{\mathrm{slsd}}$, which in turn ensures that there is no collision at _any_
moment during the execution. This concludes the proof. ∎

0$\rho$velocitytimeSVPOV

(a) Case 1

0$\rho$velocitytimeSVPOV

(b) Case 2

0$\rho$velocitytimeSVPOV

(c) Case 3

0$\rho$velocitytimeSVPOV

(d) Case 4

Figure 4: the velocities of the SV and the POV in Theorem IV-C. Note that the
assumption $0<a_{\mathrm{brake,min}}<a_{\mathrm{brake,max}}$ is crucial here.

###  IV-D The Responsibility Proof

The following is not a “theorem” in a rigorous sense—this is because the RSS
responsibility principles (Definition II-B) are not formally defined
conditions. We nevertheless provide an argument for it; such arguments are of
great practical values, when it comes to such matters as explainability to the
public and attribution of liability.

[responsibility of $R_{\mathrm{slsd}}$] Assume the same setting as in Theorem
IV-C. Then the execution of the proper response $P_{\mathrm{slsd}}$ satisfies
the RSS responsibility principles.

###### Proof.

(an informal argument) The principles 1 and 5 are ensured by safety (Theorem
IV-C). The other principles (2–4) do not apply to the current driving scenario
$S_{\mathrm{slsd}}$. ∎

##  V Usages of RSS

Some usages of the RSS framework (Figure 2) have been already hinted in the
above. Here we go into their details, providing some pointers to related
scientific studies and ongoing practical efforts at the same time.

###  V-A Attribution of Liability

Attribution of liability is one of the first applications that were pursued
using RSS. We have already discussed it briefly in Section III: when a
collision happened, the traffic participant that did not comply with the RSS
responsibility principles is held liable. The validity of this reasoning is
supported by the mathematical elements of RSS: the RSS safety rules,
especially their safety and responsibility requirements, ensure that there is
no collision as long as all the traffic participants adhere to the RSS
responsibility principles.

Here, the roles of RSS safety rules are twofold. For one, an RSS safety rule
can be seen as a mathematical incarnation of the RSS responsibility
principles, one that enables a rigorous safety proof (the safety theorems in
Figure 2). Another important role is as an _evidence of acting responsibly_.
The RSS responsibility principles are informal conditions; in contrast, RSS
safety rules are rigorous and mathematically checkable. Demonstration of the
compliance with RSS safety rules is therefore a strong evidence of acting
responsibly, via the responsibility requirements for those rules (the
responsibility theorems in Figure 2).

A comprehensive case study of such use of RSS is found in [ShashuaSS18NHTSA],
where the RSS framework is applied to NHTSA pre-crash scenarios.

###  V-B As a Safety Metric

Safety metrics for ADS are an active topic of research, see e.g.
[SilberlingWAKL20, WangLS20RealisticSingleShotLongTerm, AltekarEWCWCRJ20,
ZhaoZMSLLZ_IV20, WengRDSB20IV]. The question here is how to evaluate the
safety of a trajectory of a vehicle. A naive metric can be given by the
minimum distance—the minimum distance to obstacles and other traffic
participants exhibited during the trajectory—but its limitations are obvious,
too, such as the insensitivity to the vehicle speed. A safety metric that has
been commonly used is the _time to collision (TTC)_.

It is straightforward to derive another safety metric from the last usage of
RSS (Section V-A). This _RSS safety metric_ measures the degree with which a
trajectory satisfies RSS safety rules: the margin with which the inequalities
in the RSS safety conditions are satisfied is its safety score. Violation of
the RSS safety conditions results in a negative safety score.

Multiple variations and extensions are possible for the above (informal and
rudimentary) notion of RSS safety metric. Several of such are presented in
[AltekarEWCWCRJ20]; among them are a metric that takes into account whether a
proper response was engaged when needed.

###  V-C Formal Verification of ADS Safety

Another obvious usage of RSS is for _formal verification_ of ADS safety. By
proving that a vehicle’s control complies with RSS safety rules, one can
conclude the safety of the vehicle via the safety theorems of the RSS safety
rules. Here, the “compliance with RSS safety rules” means the following.

######  Definition  (compliance with RSS safety rules).

Let $\mathcal{R}=(R_{1},R_{2},\dotsc,R_{n})$ be a list of RSS safety rules,
where $R_{i}=(C_{i},P_{i})$ for each $i\in\\{1,2,\dotsc,n\\}$. We say that a
vehicle’s trajectory _complies with $\mathcal{R}$_ if, at each moment in the
trajectory, either

  * •

the RSS safety condition $C_{i}$ is true for some $i\in\\{1,2,\dotsc,n\\}$, or

  * •

the proper response $P_{i}$ is being executed for some
$i\in\\{1,2,\dotsc,n\\}$. Additionally, we require that this execution of
$P_{i}$ started in a state in which the corresponding RSS safety condition
$C_{i}$ was true.

[safety of an RSS-compliant trajectory] Let
$\mathcal{R}=(R_{1},R_{2},\dotsc,R_{n})$ be a list of RSS safety rules, where
$R_{i}=(C_{i},P_{i})$ for each $i\in\\{1,2,\dotsc,n\\}$. Assume further that
each rule $R_{i}$ satisfies the safety requirement (Requirement II-A). If a
vehicle’s trajectory $T$ complies with $\mathcal{R}$ (Definition V-C), then
$T$ exhibits no collision.

###### Proof.

Consider an arbitrary moment of the trajectory $T$. If some RSS safety
condition $C_{i}$ is true at that moment, then there is no collision at that
moment—this follows from the remark immediately after Requirement II-A. If
some proper response $P_{i}$ is being executed at that moment, the safety
guaranteed in Requirement II-A implies that no collision occurs then. ∎

Towards the goal of mathematically proving safety of an ADS, it suffices to
ensure that every trajectory of the ADS complies with some list of RSS safety
rules in the sense of Definition V-C. This is the consequence of Theorem V-C.
However, there are two major challenges in ensuring the above.

  * •

An ADS controller is a complex system, involving a number of numeric
optimization algorithms and statistical machine learning. Therefore, proving
its compliance with RSS safety rules is hard.

  * •

Compliance in the sense of Definition V-C requires a comprehensive list
$\mathcal{R}$ of RSS safety rules—otherwise there will be moments that are not
covered by any $C_{i}$ or $P_{i}$. Obtaining such a comprehensive list will
take an enormous effort: a safety proof for a single safety rule is nontrivial
already for a simple scenario (Section IV); and the number of driving
scenarios to be covered is huge.

###  V-D Safety Architecture

A promising “workaround” to the last two challenges in formal verification
(Section V-C) is the use of RSS in a _safety architecture_.

Figure 5: the simplex architecture

A prototypical safety architecture is the _simplex architecture_ shown in Fig.
5 [CrenshowGRSK07, SetoKSC98]. Here, the _advanced controller_ (AC) is a
complex controller that pursues not only safety but also performance (such as
comfort, progress, and fuel efficiency); the _baseline controller_ (BC) is a
simpler controller with a strong emphasis on safety; and the _decision module_
(DM) switches between the two controllers. The DM tries to use the AC as often
as possible for its superior performance. However, when the DM finds that the
current situation is safety critical, it switches to the BC whose behaviors
are more predictable and easier to analyze.

A notable feature of safety architectures such as the simplex architecture is
that they enable formal verification of a system that contains a black-box
component (such as the AC in the simplex architecture). It does so by
“wrapping” the black-box component with safety-centric components that closely
monitor the black-box component and overtake control when it is needed.

The elements of RSS map naturally to the simplex architecture:

  * •

The DM makes use of RSS safety conditions. When they are about to be violated,
the DM switches the control from the AC to the BC.

  * •

The BC implements proper responses. It thus executes a control strategy whose
safety is guaranteed.

  * •

If it so happens that the BC restores the RSS safety condition, then the DM
switches the control back to the AC.

The way the resulting _RSS-supervised controller_ operates reflects the
argument in Section V-C: it lets the AC control as long as some RSS safety
condition $C_{i}$ is true; once it comes close to violating all the RSS safety
conditions, it executes a proper response $P_{i}$ to maintain safety. The
possibility of switching back to the AC is advantageous for performance.

Such use of RSS in a safety architecture successfully addresses the two
challenges in formal verification (Section V-C).

  * •

The complexity of the ADS controller becomes no problem since it is confined
to the AC—the safety of the whole architecture is ensured exclusively by the
properties of the DM and the BC.

  * •

On the difficulty of obtaining a comprehensive list of RSS safety rules, a
safety architecture offers a _proactive_ and _best-effort_ alternative to
formal verification. Even if a rule list
$\mathcal{R}=(R_{1},R_{2},\dotsc,R_{n})$ does not cover all possible driving
scenarios, $\mathcal{R}$ can still be implemented in a safety architecture and
ensure the safety of those scenarios which are covered by $\mathcal{R}$. (For
those driving scenarios which are not covered, we just let the AC do its best
to maintain safety). After all, formal verification is about analyzing a given
controller (Section V-C), while safety architectures are about modifying the
controller (identified with the AC) and making it safe.

Such use of RSS in a safety architecture is advocated increasingly often. In
fact, some recent papers such as [OborilS20IV, Fig. 1] present RSS in the
format of safety architectures, although they may not explicitly refer to the
term “safety architecture.”

##  VI Current State and Future Directions

###  VI-A Making a Rule Set Comprehensive

Towards the ultimate goal of ADS safety proofs, one important area of RSS that
requires further work is the formulation of RSS rules. As we already
discussed, RSS rules are formulated and proved correct in a scenario-specific
manner, and there are a huge number of driving scenarios.

While a comprehensive RSS rule set is obviously important and desired, the
path to it might seem endless. We nevertheless believe that efforts in this
direction are worthwhile.

  * •

One reason is the use of the rules in a safety architecture (Section V-D).
Even if the rule list $\mathcal{R}$ at hand is not comprehensive (and hence
not enough to prove the overall safety), the “best-effort” usage in a safety
architecture ensures safety at least for the scenarios that are already
covered by $\mathcal{R}$.

  * •

Another reason is that RSS rules are irrevocable, with their correctness being
mathematically established. Once derived, they can be used for the coming
dozens of years or even longer—they can be seen as common assets of humankind.
With more efforts thrown in for the formulation of RSS rules, the rule set
grows monotonically.

###  VI-B Tool Support for Using, Deriving, and Verifying RSS Rules

Tool support for RSS has been pursued actively in recent years. One example is
an implementation of some RSS rules: it is offered as a library
[GassmannOBLYEAA19IV] that can be used in combination with simulation
environments such as Baidu Apollo. This implementation is for _using_ RSS
rules that have been already derived and verified.

Another area that calls for tool support is the derivation of RSS rules. As we
discussed in Section V-D, RSS rule derivation requires systematic and
organized efforts, much like formal verification by theorem proving. Therefore
the task needs tool support—much like tool support for theorem proving is
given by _proof assistants_ such as Coq [Coq] and Isabelle/HOL
[nipkow2002isabelle]—otherwise the efforts will be marred with human errors.

Yet another area that calls for tool support is the (formal) verification of
RSS rules. Existing correctness proofs for RSS rules (such as the ones in
Section IV) are mathematical yet not formalized or mechanized; it is not hard
to imagine human errors in such proofs, especially when working with complex
driving scenarios. The problem of giving formalized and mechanized proofs to
RSS rules is investigated in [RoohiKWSL18arxiv]. Their trial is based on a
rigorous notion of signal; they argue that none of the existing _automated_
verification tools is suited for the verification problem. We believe that the
use of theorem provers that allow human interaction should be pursued. An
example of such provers is KeYmaera X [FultonMQVP15].

Note that the last two directions (derivation and verification) are the
motivation of the current introduction (namely building a logical theory and
formalization). We believe that our identification of

  * •

two requirements (Sections II-A and II-B) and

  * •

their assume-guarantee relationship (Fig. 2)

in this paper is a necessary first step in the two directions.

###  VI-C Permissive, and Thus Practical, RSS Rules

The scope of RSS rules is principally safety. They can sacrifice other
practical performance metrics such as comfort, progress, and fuel efficiency.
For greater practical utility, it is desired that RSS rules are less
conservative and more permissive.

One work in this direction is [RSSplusIV2021accepted]; it proposes an
extension of RSS in which the balance between safety and other performance
metrics is proactively adjusted. Other promising directions seem to include
the following.

  * •

_Fine-grained formalization_ of the RSS responsibility principles, so that the
formalization respects different traffic circumstances (urban, rural, or
highway), different regions and countries, and different driving cultures.

  * •

_RSS that is aware of intentions and knowledge_. Responsibilities of other
vehicles will be totally different depending on whether the SV’s turn signal
is blinking or not. Moreover, when SV makes a maneuver with its turn signal
blinking, it is also desired that the SV makes sure that other vehicles are
aware of the turn signal.

  * •

_Enlarging the variety of proper responses_. For example, [deIacoSC20IV]
proposes an extension of the RSS rule in Section IV in which swerves (in
addition to braking) is allowed as evasive maneuvers [deIacoSC20IV]. This
makes the corresponding safety condition much weaker, and the RSS rule more
permissive.

###  VI-D Perceptual Uncertainties

In the presence of perceptual uncertainties (such as errors in position
measurement and object recognition), the perceived values that RSS safety
conditions depend on can be erroneous. Making RSS rules tolerant of such
perceptual uncertainties is pursued in [SalayCEASW20PURSS]; the methodology
used there is more generally formulated in [DBLP:conf/nfm/KobayashiSHCIK21]
and used in combination with the modeling and verification framework Event-B
[AbrialBHHMV10].

A big source of perceptual uncertainties in modern ADS is its _statistical
machine learning_ components (neural networks to be specific). One way to
lessen such ML-related uncertainties is to enrich the output of an ML
component. For example, the use of DNNs’ confidence scores is proposed in
[AngusCS19arxiv]. In [ChowRWGJALKC20], it is proposed to look at
inconsistencies between perceptual data of different modes.

## Acknowledgment

Thanks are due to Clovis Eberhart, James Haydon, Jérémy Dubut, Rose Borher,
Tsutomu Kobayashi, Sasinee Pruekprasert, Ahmet Cetinkaya, Xiaoyi Zhang, and
Akihisa Yamada for discussions. The author is supported by ERATO HASUO
Metamathematics for Systems Design Project (No. JPMJER1603), JST.

## References

  * [1] Jean-Raymond Abrial, Michael Butler, Stefan Hallerstede, Thai Son Hoang, Farhad Mehta, and Laurent Voisin.  Rodin: an open toolset for modelling and reasoning in Event-B.  STTT, 12(6):447–466, 2010.
  * [2] Niraj Altekar, Maria Elli, Jack Weast, Yan Chen, Jeffrey Wishart, Steven Como, Brendan Russo, and Emmanuel James.  Driving safety performance assessment metrics for ADS-equipped vehicles.  SAE International Journal of Advances and Current Practices in Mobility, 2(5):2881–2899, apr 2020.
  * [3] Matt Angus, Krzysztof Czarnecki, and Rick Salay.  Efficacy of pixel-level OOD detection for semantic segmentation.  CoRR, abs/1911.02897, 2019.
  * [4] Jeff Chow, Valerie Richmond, Mike Wang, Uriel Guajardo, Daniel Jackson, Nikos Arechiga, Geoffrey Litt, Soonho Kong, and Sergio Campos.  Certified Control: A New Safety Architecture for Autonomous Vehicles, 2020.  preprint.
  * [5] T. L. Crenshaw, E. Gunter, C. L. Robinson, L. Sha, and P. R. Kumar.  The simplex reference model: Limiting fault-propagation due to unreliable components in cyber-physical system architectures.  In 28th IEEE International Real-Time Systems Symposium (RTSS 2007), pages 400–412, 2007.
  * [6] R. de Iaco, S. L. Smith, and K. Czarnecki.  Safe swerve maneuvers for autonomous driving.  In 2020 IEEE Intelligent Vehicles Symposium (IV), pages 1941–1948, 2020.
  * [7] The Coq development team.  The Coq proof assistant reference manual.  LogiCal Project, 2004.  Version 8.0.
  * [8] Nathan Fulton, Stefan Mitsch, Jan-David Quesel, Marcus Völp, and André Platzer.  KeYmaera X: An axiomatic tactical theorem prover for hybrid systems.  In Amy P. Felty and Aart Middeldorp, editors, CADE, volume 9195 of LNCS, pages 527–538. Springer, 2015.
  * [9] Bernd Gaßmann, Fabian Oboril, Cornelius Bürkle, Shuang Liu, Shoumeng Yan, Maria Soledad Elli, Ignacio J. Alvarez, Naveen Aerrabotu, Suhel Jaber, Peter van Beek, Darshan Iyer, and Jack Weast.  Towards standardization of AV safety: C++ library for responsibility sensitive safety.  In 2019 IEEE Intelligent Vehicles Symposium, IV 2019, Paris, France, June 9-12, 2019, pages 2265–2271. IEEE, 2019.
  * [10] Tsutomu Kobayashi, Rick Salay, Ichiro Hasuo, Krzysztof Czarnecki, Fuyuki Ishikawa, and Shin-ya Katsumata.  Robustifying controller specifications of cyber-physical systems against perceptual uncertainty.  In Aaron Dutle, Mariano M. Moscato, Laura Titolo, César A. Muñoz, and Ivan Perez, editors, NASA Formal Methods - 13th International Symposium, NFM 2021, Virtual Event, May 24-28, 2021, Proceedings, volume 12673 of Lecture Notes in Computer Science, pages 198–213. Springer, 2021.
  * [11] Tobias Nipkow, Lawrence C Paulson, and Markus Wenzel.  Isabelle/HOL: a proof assistant for higher-order logic, volume 2283\.  Springer Science & Business Media, 2002.
  * [12] Fabian Oboril and Kay-Ulrich Scholl.  Risk-Aware Safety Layer for AV Behavior Planning.  In 2020 IEEE Intelligent Vehicles Symposium, IV 2020, Las Vegas, United States, October 19-22, 2020, page 7. IEEE, 2020.
  * [13] Fabian Oboril and Kay-Ulrich Scholl.  RSS+: Pro-active risk mitigation for av safety layers based on rss.  In 2021 IEEE Intelligent Vehicles Symposium, IV 2021. IEEE, 2021.
  * [14] Nima Roohi, Ramneet Kaur, James Weimer, Oleg Sokolsky, and Insup Lee.  Self-driving vehicle verification towards a benchmark.  CoRR, abs/1806.08810, 2018.
  * [15] Rick Salay, Krzysztof Czarnecki, Maria Soledad Elli, Ignacio J. Alvarez, Sean Sedwards, and Jack Weast.  PURSS: towards perceptual uncertainty aware responsibility sensitive safety with ML.  In Huáscar Espinoza, José Hernández-Orallo, Xin Cynthia Chen, Seán S. ÓhÉigeartaigh, Xiaowei Huang, Mauricio Castillo-Effen, Richard Mallah, and John McDermid, editors, Proceedings of the Workshop on Artificial Intelligence Safety, co-located with 34th AAAI Conference on Artificial Intelligence, SafeAI@AAAI 2020, New York City, NY, USA, February 7, 2020, volume 2560 of CEUR Workshop Proceedings, pages 91–95. CEUR-WS.org, 2020.
  * [16] D. Seto, B. Krogh, L. Sha, and A. Chutinan.  The simplex architecture for safe online control system upgrades.  In Proceedings of the 1998 American Control Conference. ACC (IEEE Cat. No.98CH36207), volume 6, pages 3504–3508 vol.6, 1998.
  * [17] Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua.  On a formal model of safe and scalable self-driving cars.  CoRR, abs/1708.06374, 2017.
  * [18] A. Shashua, S. Shalev-Shwartz, and S. Shammah.  Implementing the RSS model on NHTSA pre-crash scenarios.  Technical report, Mobileye, 2018.
  * [19] Jordan Silberling, Paul Wells, Atul Acharya, Joseph Kelly, and John Lenkeit.  Development and application of a collision avoidance capability metric.  In WCX SAE World Congress Experience. SAE International, apr 2020\.
  * [20] Lingguang Wang, Carlos Fernandez Lopez, and Christoph Stiller.  Realistic Single-Shot and Long-Term Collision Risk for a Human-Style Safer Driving.  In 2020 IEEE Intelligent Vehicles Symposium, IV 2020, Las Vegas, United States, October 19-22, 2020, page 8. IEEE, 2020.
  * [21] Bowen Weng, Sughosh Jagannatha Rao, Eeshan Deosthale, Scott Schnelle, and Frank Barickman.  Model Predictive Instantaneous Safety Metric for Evaluation of Automated Driving Systems.  In 2020 IEEE Intelligent Vehicles Symposium, IV 2020, Las Vegas, United States, October 19-22, 2020, page 8. IEEE, 2020.
  * [22] Glynn Winskel.  The Formal Semantics of Programming Languages.  The MIT Press, 1993.
  * [23] X. Xu, X. Wang, and X. Wu.  Calibration and evaluation of responsibility-sensitive safety model on autonomous car-following maneuvers using naturalistic driving study data.  Transportation Research Board (TRB) Annual Meeting, 2020.
  * [24] Hengyu Zhao, Yubo Zhang, Pingfan Meng, Hui Shi, Erran Li, Tiancheng Lou, and Jishen Zhao.  Safety score: A quantitative approach to guiding safety-aware autonomous vehicle computing system design.  In 2020 IEEE Intelligent Vehicles Symposium, IV 2020, Las Vegas, United States, October 19-22, 2020. IEEE, 2020.