# When to Retrieve: Teaching LLMs to Utilize Information Retrieval Effectively

Tiziano Labruna0000-0001-7713-7679 Corresponding Author. Email:
<EMAIL_ADDRESS>Jon Ander Campos0000-0002-1447-5870  Gorka
Azkune0000-0002-2506-7426  University of Bozen-Bolzano  Fondazione Bruno
Kessler  HiTZ Center - Ixa, University of the Basque Country UPV/EHU

###### Abstract

In this paper, we demonstrate how Large Language Models (LLMs) can effectively
learn to use an off-the-shelf information retrieval (IR) system specifically
when additional context is required to answer a given question. Given the
performance of IR systems, the optimal strategy for question answering does
not always entail external information retrieval; rather, it often involves
leveraging the parametric memory of the LLM itself. Prior research has
identified this phenomenon in the PopQA dataset, wherein the most popular
questions are effectively addressed using the LLM’s parametric memory, while
less popular ones require IR system usage. Following this, we propose a
tailored training approach for LLMs, leveraging existing open-domain question
answering datasets. Here, LLMs are trained to generate a special token,
$\langle$RET$\rangle$, when they do not know the answer to a question. Our
evaluation of the Adaptive Retrieval LLM (Adapt-LLM) on the PopQA dataset
showcases improvements over the same LLM under three configurations: (i)
retrieving information for all the questions, (ii) using always the parametric
memory of the LLM, and (iii) using a popularity threshold to decide when to
use a retriever. Through our analysis, we demonstrate that Adapt-LLM is able
to generate the $\langle$RET$\rangle$ token when it determines that it does
not know how to answer a question, indicating the need for IR, while it
achieves notably high accuracy levels when it chooses to rely only on its
parametric memory.

123

##  1 Introduction

The task of question answering (QA) remains a focal point in Natural Language
Understanding research. There are many different datasets serving as
benchmarks for evaluating QA models, such as Natural Questions (NQ) [17],
SQuAD [24] or QuAC [7], just to mention a few. Nowadays, Large Language Models
(LLMs) consistently outperform traditional methods on these benchmarks,
showcasing remarkable performance.

Typically, there are two primary approaches to utilize LLMs for question
answering:

(i) Closed Book Question Answering: This approach involves strategies like
instruction tuning [31] or few-shot prompting [6] to enhance performance.
Here, the LLM relies solely on its parametric memory to answer questions.
However, these parametric memories have inherent limitations as they are based
entirely on the training corpus, meaning for example that they could be
outdated regarding events occurring after the training process.

(ii) Open Book Question Answering: In this approach, the LLM is coupled with
an Information Retriever (IR) system [12, 35]. By leveraging the IR system,
the LLM can retrieve relevant context to supplement its understanding and
provide more accurate answers.

However, the research conducted by Mallen et al. [21] sheds light on the
complexity of question-answering strategies, challenging the notion that the
optimal approach always involves the utilization of an IR system. Through the
introduction of the PopQA dataset, comprising 14 thousand questions annotated
with popularity scores, they demonstrated that while LLMs relying solely on
their parametric memories excel in addressing high-popularity questions, the
efficacy diminishes for low-popularity questions, where using IR becomes
curcial.

Figure 1: The inference process of Adapt-LLM step-by-step: given a question
(step 1), an LLM decides (step 2) whether to answer the question directly
(step 3) or to ask for additional contextual information, generating the
special $\langle$RET$\rangle$ token; for the later, an off-the-shelf IR system
is used to retrieve relevant context (step 4), which is used alongside the
question to prompt again the LLM for the final answer (step 5).

Their findings underscore the importance of a hybrid approach, where LLMs
utilize parametric memory for high-popularity questions, but use an off-the-
shelf IR system to retrieve relevant context to answer low-popularity
questions. Central to their methodology is the establishment of a fixed
popularity score threshold, which they use to decide whether an IR system has
to be employed.

In many cases, however, question answering datasets do not include popularity
scores, so relying on such scores is not a generalizable approach. Motivated
by this limitation, our study aims to address whether LLMs can autonomously
determine when to employ an IR system for improved question answering. To
investigate this, we conduct an evaluation of an LLM using an open-domain
question answering dataset to identify the questions for which the LLM
provides accurate responses and those where its answers are incorrect.

Specifically, for questions where the LLM’s response is incorrect, we annotate
them with a special token, $\langle$RET$\rangle$, indicating the need for
additional context. Subsequently, we utilize these annotations to construct a
new dataset tailored for training purposes, where we teach an LLM to answer
directly if it is confident about the answer or to require context it believes
is useful for answering the question (see Figure 1). Our hypothesis is that
through this training process, the LLM learns to use an IR system when it
needs extra context to answer a question, thus we name it Adapt-LLM.

To validate our hypothesis, we conducted several experiments on the PopQA
dataset [21], as it provides a suitable platform for benchmarking hybrid
retrieval strategies. As a result of these experiments we find that:

  * •

Adapt-LLM consistently outperforms typical fixed strategies for question
answering, such as (i) using the IR system for all questions and (ii) relying
solely on the parametric memory of the LLM.

  * •

Adapt-LLM demonstrates performance comparable to strategies that rely on
popularity scores to determine when to use an IR system, even without
utilizing any popularity score or similar metric. It’s worth noting that
popularity scores are a unique feature of the PopQA dataset, rendering them
inapplicable to other open-domain question answering datasets.

  * •

When Adapt-LLM decides to retrieve additional information, the results
obtained with the context are significantly better than those without it.
Similarly, when Adapt-LLM directly answers questions relying on its parametric
memory, it achieves high accuracies. These observations indicate that the
model effectively discerns when to retrieve information and when it can answer
a question without further context.

  * •

The primary bottleneck for the performance of Adapt-LLM lies in the IR system.
Adapt-LLM achieves much higher performance with gold passages compared to
passages retrieved by the IR system.

Our findings underscore the significance of adaptive retrieval strategies in
enhancing the performance of LLMs for question answering tasks. By training
Adapt-LLM to dynamically determine when to retrieve additional context, we
demonstrate the feasibility of teaching an LLM how to effectively leverage
external information sources only when necessary.

##  2 Related Work

Retrieval-Augmented Generation (RAG) [18] has shown improvements on a wide
variety of NLP areas, such as question answering [16, 12, 30, 22],
truthfulness [13, 20] and language modelling [11, 5, 25] among others. The
ability to ground model generations on retrieved text chunks has also enabled
smaller models to match the performance of larger ones [2]. Moreover, due to
the extremely high cost of training LLMs, RAG has become the standard way to
maintain them updated with new information, not having to re-train the models
periodically to incorporate new facts [9].

Even if augmenting LLMs with retrieval is an essential step for the current
generation of LLMs [14, 26] it also comes with a cost. Traditional retrieval
methods as TF-IDF or BM-25 [28] are only able to retrieve documents with
keyword overlap and suffer from lexical gap [4]. In order to try to solve this
issue, many pre-trained Transformer encoder based dense models have been
proposed [8, 27, 16, 10]. Trained neural models have shown good performance
over a variety of retrieval benchmarks but they still struggle in the zero-
shot setup for new domains [32]. The quality of the retrieval engine is
essential for retrieval-augmented models as this will set the upper bound of
the model performance. Moreover, the usage of a retrieval engine, especially
when the target document index is huge, can significantly increase the latency
of the model and hurt real time applications user experience [3].

On the other hand, as models keep scaling, the world knowledge encoded in
their parameters does too [15]. Many previous efforts have shown that language
models are able to memorize a significant amount of world knowledge and
achieve competitive performance on tasks such as open-domain question
answering when they just use their parametric knowledge for solving the task
[19, 1, 33, 34].

Motivated by all this, the adaptive approach has been proposed as a new
solution [29, 21]. In this approach, if the solution to the task is encoded in
the parameters of the model, the model will be directly used for generating a
solution. Conversely, if the answer is not encoded in the knowledge of the
model, the answer generation will be augmented with external knowledge.

Recently, Schick et al. [29] proposed the Toolformer, a model that can self
teach how and when to use external tools via simple API calls including a
calculator, search engines, a calendar and so on. The self learning process is
based on a synthetic text only corpus that is enriched by prompting an LLM.
The LLM first adds inline API calls on top of the unsupervised corpus. These
API calls are then validated by evaluating whether the execution of the API
calls is helpful for predicting the future tokens. This unsupervised method
significantly boosts model performance in a variety of tasks when compared
against non augmented LLMs, but it also makes the model over use tools. As an
example, for the QA task the model uses the search engine 99.3% of the cases.
On our work, we try to take advantage of the parametric knowledge of LLMs and
just perform retrieval when needed. Adapt-LLM decreases the usage of IR down
to 83.99% while improving performance over vanilla retrieval.

More similar to our work, Mallen et al. [21] propose a dataset and method for
measuring when non-parametric information needs to be retrieved. They present
the PopQA dataset that contains 14K questions about a set of entities with
varying popularity. The popularity of an entity is measured by the page views
of its Wikipedia page. In order to solve this QA task, they use a popularity
score threshold calculated on the PopQA dataset. If the popularity score of an
individual entity is below the threshold they perform a retrieval step. On the
contrary, if the score is greater than the threshold they directly answer the
question. This method yields better results than vanilla retrieval but it
requires the calculation of a popularity score that is not available in
realistic QA scenarios. Our Adapt-LLM method extends the idea of adaptive
retrieval by generating a synthetic dataset that teaches the LLM whether to
call the IR engine.

##  3 Adaptive Retrieval LLM (Adapt-LLM)

Adaptive retrieval refers to the model’s capability to dynamically determine
whether to retrieve additional context information for generating answers in
question answering tasks. Unlike traditional models that either always
incorporate context or never consider it, adaptive retrieval allows the model
to selectively retrieve context based on the specific requirements of each
question. This adaptive approach aims to optimize performance by leveraging
context only when necessary, thereby enhancing the model’s ability to generate
accurate answers.

As depicted in Figure 1, the process of the Adapt-LLM unfolds in the following
sequence:

  1. 1.

The first prompt containing the question is sent to the model (step 1 of
Figure 1).

  2. 2.

The Adapt-LLM evaluates the prompt to determine whether additional context is
necessary to answer the question effectively (step 2).

  3. 3.

If the model determines that context is not required, it directly produces a
response to the question by leveraging its parametric memory (step 3).

  4. 4.

If context is deemed necessary, the Adapt-LLM model returns a special token,
represented as $\langle$RET$\rangle$, and an off-the-shelf IR system is used
to retrieve pertinent context based on the question (step 4); the context is
then combined with the original question prompt to form a comprehensive
representation for answer generation (step 5).

The decision-making process of Adapt-LLM enables the model to determine the
necessity of context for answering questions through dynamic assessment of
each prompt. This flexible behavior allows the model to strike a balance
between utilizing context for enhanced understanding and delivering direct
answers when sufficient.

###  3.1 Training Adapt-LLM

Here, we delineate the methodology employed to train our Adapt-LLM model. The
process of crafting the training data, denoted as $DS_{Adapt}$, is presented
in Algorithm 1.

We begin by selecting an open-domain question answering dataset containing
questions $Q$, associated context passages $P$, and corresponding answers $A$.
We initialize $DS_{Adapt}$ to an empty set (line 1 of the algorithm). For each
question in $Q$, we leverage the base LLM without any retrieval mechanism to
perform a zero-shot inference (line 3). This step allows us to differentiate
questions for which the model generates correct answers from those where its
responses are inaccurate. This process can be understood as a way to discover
what the base LLM knows due to its parametric memory. For questions where the
model’s response is accurate (line 4), we build a training set instance
incorporating the following prompt, which we call parametric_prompt:

Prompt: Answer the question Q. If you need help answer <RET> to get the
context. Q: …

Alongside this prompt, we include the corresponding question from $Q$ and the
golden answer from $A$, collectively forming the instance (line 5), which is
subsequently appended to the $DS_{Adapt}$ dataset (line 6).

In contrast, if the LLM fails to produce a correct response to the question
(line 8), we build two different instances. The first employs the same
parametric_prompt as previously described, with $\langle$RET$\rangle$
designated as the answer (line 9), indicating the necessity for additional
context. The second prompt, termed context_prompt, encompasses contextual
information alongside the question:

Input: Q: questions, A: answers, P: passages, LLM

Output: $DS_{Adapt}$: A training dataset for Adaptive Retrieval

1 $DS_{Adapt}$ = init_empty()

2 for  _q, gold_ans, pass in (Q, A, P)_ do

3 ans = LLM(q)

4 if  _ans = gold_ans_ then

5 inst = build_instance(’parametric_prompt’, q, gold_ans)

6 $DS_{Adapt}$.add(inst)

7 end if

8

9 else

10 inst1 = build_instance(’parametric_prompt’, q, "<RET>")

11 $DS_{Adapt}$.add(inst1)

12 inst2 = build_instance(’context_prompt’, q, gold_ans, pass)

13 $DS_{Adapt}$.add(inst2)

14 end if

15

16 end for

return $DS_{Adapt}$

Algorithm 1 Training data creation

Prompt: Answer the question Q given the context C. Q: …, C: …

For this instance, we include the prompt, the question from $Q$, the golden
answer from $A$, and the corresponding context passage from $P$ (line 11).

After populating the dataset with both types of prompts for questions where
the LLM could not respond accurately and only the parametric_prompt with
golden answers for all other questions, our training set $D_{Adapt}$ is
prepared for the subsequent fine-tuning phase. The fine-tuning process entails
training the base LLM on our dataset, resulting in the Adapt-LLM model.

This approach ensures that the model effectively learns to discern when
context is necessary for answering questions, or to provide a direct response
when it suffices, as well as answer directly when provided with context.

###  3.2 Inference

In the inference phase, we utilize the fine-tuned model to generate responses
to unseen questions. We employ the same prompts used during the training
phase, as outlined in Section 3.1.

Initially, the model is prompted to either provide a direct response or return
$\langle$RET$\rangle$ if it is unsure of the answer. If the model returns
$\langle$RET$\rangle$, we proceed with information retrieval to acquire
relevant context by means of an off-the-shelf IR system. Subsequently, we
augment the question with the retrieved context and prompt the model again
using the second type of prompt introduced during the training phase.

##  4 Experiments and Results

In this section, we outline the experimental framework aimed at assessing the
performance of the proposed adaptive retrieval approach, Adapt-LLM. We begin
by describing the datasets utilized (Section 4.1), followed by an overview of
our base model (Section 4.2), the different configurations of the base model
(Section 4.3), and the training details (Section 4.4). Subsequently, we
introduce the three primary experiments:

  1. 1.

Evaluation of Adapt-LLM performance compared to the following baseline models:
(i) an LLM that retrieves contextual information for all questions, and (ii)
an LLM that exclusively relies on its parametric memory without using an IR
system for any question (Section 4.5).

  2. 2.

Analysis of Adapt-LLM’s ability to determine when extra context is necessary
to answer a question (Section 4.6).

  3. 3.

Comparison with the state-of-the-art approach for PopQA (Section 4.7).

Training Set | Model configuration | Accuracy
---|---|---
NQ | Never Retrieve | 21.43%
Always Retrieve | 35.86%
Adapt-LLM (ours) | 36.77%
SQuAD | Never Retrieve | 21.22%
Always Retrieve | 36.59%
Adapt-LLM (ours) | 38.15%
Table 1: Performance comparison of Llama-2 models trained on the NQ and SQuAD
datasets using different retrieval configurations (NR-LLM, AR-LLM, and Adapt-
LLM), evaluated on the PopQA test set. Exact match accuracy is reported for
all models.

###  4.1 Datasets

To ensure comprehensive training and evaluation of our models, we specifically
selected three diverse question answering datasets. For training, we chose NQ
[17] and SQuAD [24], as they are widely recognized datasets that assess
factual knowledge and are based on Wikipedia. For evaluation, we opted for
PopQA [21]. Below are brief descriptions of each dataset:

#### NQ

The Natural Questions dataset [17] is a collection of real-world questions
derived from Google search queries, accompanied by long-form text passages
obtained from Wikipedia articles and providing a diverse range of topics and
natural language variations. We utilize this dataset for training our models
in the experiments.

#### SQuAD

The Stanford Question Answering Dataset SQuAD [24] is a widely utilized
dataset in the field of natural language processing and comprises questions
posed by crowdworkers on a diverse range of Wikipedia articles, along with
relevant paragraph passages serving as context. We utilize this dataset for
training our models in the experiments.

#### PopQA

The Popular Questions and Answers dataset [21] consists of curated questions
sourced from various online platforms, encompassing a wide range of domains
and styles. Given the variability in the effectiveness of context retrieval
strategies observed in this dataset, we select PopQA as our test set to
evaluate the language models’ performance in determining when context is
necessary for accurate answer provision.

###  4.2 Base Model

In our experiments, we employ Llama-2 [33] as our base LLM. Llama-2 is an
open-source instruction-based LLM, which comes in versions of 7B, 13B, and 70B
parameters. The model is pretrained on an expanded corpus sourced from
publicly available online data sources. This corpus offers a 40% increase in
size compared to its predecessor, contributing to the model’s enhanced
performance and capabilities.

Additionally, Llama-2 features an extended context length, effectively
doubling its capacity to process and comprehend longer sequences of text.
These enhancements significantly improve the model’s effectiveness across
various natural language understanding tasks. Specifically, for our
experiments, we utilize the Llama-2 model with 7B parameters, leveraging its
robust capabilities for our specific research objectives.

| NQ | SQuAD | PopQA
---|---|---|---
Questions | 58,880 | 87,599 | 14,282
Words/question | 9.20 | 10.06 | 6.62
Words/answer | 2.26 | 3.16 | 2.04
Table 2: Comparison of the three datasets we use for our experiments, i.e. SQuAD, NQ and PopQA. For each of them we provide the number of questions, and the average number of words per question and answer. Training |  $\langle$RET$\rangle$ Usage |  $\langle$RET$\rangle$ | No $\langle$RET$\rangle$
---|---|---|---
|  | Acc. w/ context | Acc. w/o context | Acc. w/ context | Acc. w/o context
NQ | 82.26% | 33.04% | 14.65% | 55.72% | 62.36%
SQuAD | 83.93% | 33.40% | 9.94% | 57.73% | 62.92%
Table 3: Results of the usage of the $\langle$RET$\rangle$ token in the Adapt-
LLM model. The first column shows the percentage of PopQA questions for which
the model requests additional context. The second column focuses on the
questions for which Adapt-LLM asks for context ($\langle$RET$\rangle$),
comparing the performance between answering those questions with and without
context. The last column (No $\langle$RET$\rangle$) is for questions which
Adapt-LLM decides to answer directly. We also compare the performance with and
without the context retrieved by the IR system.

###  4.3 Model Configurations

We conduct the experiments using three different model configurations,
corresponding to the three different ways in which an LLM and an IR system can
be combined:

  * •

Adaptive Retrieval (Adapt-LLM). The Adapt-LLM model dynamically decides
whether to retrieve context based on the question and its perceived need for
contextual information, as explained in Section 3.1. As the IR system, we use
Contriever [10], which is an unsupervised model pretrained on a large corpus,
followed by fine-tuning on MS MARCO [23]. We only retrieve the most relevant
passage according to the IR system to prompt the base LLM for the final
answer.

  * •

Never-Retrieve (NR-LLM). This model configuration is trained to answer
questions solely based on the question text without considering any contextual
information. It serves as the baseline for evaluating the performance of
question answering models in the absence of context.

  * •

Always-Retrieve (AR-LLM). In contrast to the NR-LLM model, this configuration
always retrieves context passages to assist in answering questions. It is
trained to utilize context consistently for generating answers. To ensure a
fair comparison with Adapt-LLM, we also use Contriever [10] as the IR system
and only retrieve the most relevant passage as context.

###  4.4 Training Details

For all three model configurations (Adapt-LLM, AR-LLM and NR-LLM) and both
training sets (SQuAD and NQ), we adhere to the parameter configuration
established in Alpaca-Lora [31] which includes a batch size of 128, three
epochs, and a fixed learning rate of 3e-4. We incorporated LoRA (Low-Rank
Adaptation) regularization, with parameters configured for r=8, alpha=16, and
a dropout rate of 0.05. Training was performed on an NVIDIA A40 GPU, for an
average training time of approximately 8 hours. We do not perform any model
selection and we use the last checkpoint after 3 epochs of training.

###  4.5 Validating the Adaptive Retrieval Approach

In order to assess the effectiveness of our adaptive approach (Adapt-LLM) in
comparison to the NR-LLM and AR-LLM configurations, we conducted fine-tuning
of the Llama-2 model on both the NQ and SQuAD datasets across all three
configurations. For the NR-LLM and AR-LLM configurations, we constructed
training samples by extracting question-answer pairs from the datasets and
incorporating corresponding instruction prompts.

Specifically, prompts for the NR-LLM configuration instructed the model to
answer questions without additional context, whereas prompts for the AR-LLM
configuration included both the question and contextual information. In
contrast, the Adapt-LLM training set was constructed following the approach
outlined in Section 3.1, employing a two-step process. As a result of this
process, the 74.72% of the questions in NQ are marked with the
$\langle$RET$\rangle$ token, whereas the 87.49% questions are marked for
SQuAD.

The trained models were then tested on the PopQA dataset to evaluate their
performance in a real-world question answering scenario. During inference, the
NR-LLM and AR-LLM models were utilized as is, with corresponding instruction
prompts provided, and outputs expected to be answers to the questions.
Conversely, for the Adapt-LLM model, we followed the same prompt procedure as
explained in Section 3.2.

The generated answers are then compared to the set of possible answers for
each question, which are already annotated in the PopQA test set. The
evaluation metric used is Exact Match Accuracy, which measures the percentage
of generated outputs that exactly match one of the possible answers for the
corresponding question.

Table 1 presents the results of this experiment, illustrating the performance
of the Llama-2 model across the different configurations and datasets. Across
both the NQ and SQuAD training datasets, the Adapt-LLM configuration
consistently outperforms the Never Retrieve (NR-LLM) and Always Retrieve (AR-
LLM) configurations on the PopQA test set. As can be observed, NR-LLM exhibits
the lowest performance among the models, with an accuracy difference of
approximately 14 absolute points compared to the other configurations. This
disparity suggests that the parametric memory of Llama-2 alone is not
sufficient for effectively answering PopQA questions.

The differences between AR-LLM and Adapt-LLM are narrower. Specifically, the
Adapt-LLM configuration achieves an accuracy of 36.77% and 38.15% on the PopQA
test set when trained on the NQ and SQuAD datasets, respectively, compared to
35.86% and 36.59% for the AR-LLM configuration. Across both training datasets,
Adapt-LLM outperforms AR-LLM, with the largest difference observed when
trained on SQuAD.

All in all, these results underscore the efficacy of the adaptive retrieval
approach in dynamically determining the necessity of context for accurate
question answering, resulting in improved performance compared to fixed
strategies of always or never retrieving context.

Although the disparity between training Adapt-LLM on NQ or SQuAD is relatively
minor, we try to determine the suitability of a training set for a given
evaluation set. While both training sets (NQ and SQuAD) and the evaluation set
(PopQA) are based on Wikipedia, subtle differences may exist.

Table 2 provides insights into the characteristics of the three datasets
involved in our experimental procedure, including the total number of
questions and the average number of words per question and answer. While NQ
appears to be closer to PopQA in terms of question and answer lengths, the key
factor influencing the better results of training Adapt-LLM on SQuAD may be
the number of questions in the training dataset ($\sim$87K in SQuAD and
$\sim$58K in NQ). Further analyses are required to elucidate the factors that
render a training dataset more suitable for a given target dataset (which is
beyond the scope of our study), but these results suggest that scale may play
once again a crucial role.

Figure 2: Histograms depicting the proportion of questions where Adapt-LLM
trained on NQ (left) and Adapt-LLM trained on SQuAD (right) ask for extra
context for different popularity score intervals.

###  4.6 Contextual Retrieval Decision Analysis

In this experiment, our objective is to once again evaluate the effectiveness
of the Adapt-LLM model, this time focusing on its ability to accurately
determine when additional context is needed. For this purpose, we adhere to
the following steps:

  1. 1.

We conduct inference on the Adapt-LLM model using the PopQA test set,
prompting it to either return an answer directly or indicate the need for
additional context by returning $\langle$RET$\rangle$.

  2. 2.

In the case of receiving a $\langle$RET$\rangle$ response from the Adapt-LLM
model, we proceed with the following steps:

    1. 2.1.

We conduct inference on the Adapt-LLM model, prompting it to return an answer
given the context obtained from the IR system.

    2. 2.2.

We also conduct inference on the NR-LLM model with the instruction to provide
an answer directly without additional context.

  3. 3.

If the Adapt-LLM model decides to answer the question directly relying only on
its parametric memory:

    1. 3.1.

We conduct inference on the Adapt-LLM model, prompting it to return the answer
without providing context.

    2. 3.2.

We conduct inference on the AR-LLM model with the instruction to provide an
answer using the context retrieved by the IR system.

Table 3 presents the results of this experiment. The first thing to note is
that the Adapt-LLM model generates the $\langle$RET$\rangle$ token for
approximately 82-83% of the questions in the PopQA dataset, with similar
ratios observed across both training datasets. This observation aligns with
the low performance of the NR-LLM configuration demonstrated in Table 1.

However, Adapt-LLM consistently determines when additional context is required
to answer a question accurately. Across both the NQ and SQuAD training
datasets, Adapt-LLM exhibits significantly higher accuracy when retrieving
context compared to the NR-LLM model’s accuracy without context (as indicated
in the $\langle$RET$\rangle$ column of Table 3). Specifically, for the NQ
dataset, the accuracy of the Adapt-LLM model when requesting context is
33.04%, whereas the accuracy of the NR-LLM model without context retrieval is
notably lower at 14.65%. Similarly, for the SQuAD dataset, Adapt-LLM achieves
an accuracy of 33.40% with context retrieval, whereas the NR-LLM model’s
accuracy without context is substantially lower at 9.94%.

Passages | SQuAD Dev | NQ Dev
---|---|---
Acc. | Acc.
Gold | 89.42% | 69.76%
Contriever | 22.49 | 27.04%
Table 4: Performance comparison of Adapt-LLM for the SQuAD and NQ dev sets,
when using the gold passages provided by the datasets and when using the best
passage retrieved by Contriever.

Finally, the last column of Table 3 (No $\langle$RET$\rangle$) shows the
performance of Adapt-LLM when answering questions based solely on its
parametric memory. As can be seen, accuracies above 62% are obtained when no
context is utilized, providing further evidence that Adapt-LLM effectively
discerns between retrieving context and providing direct answers to questions.
Additionally, we evaluate the performance of these questions when context is
added to the input, revealing significant decreases in accuracy of up to 7
absolute points.

These findings provide insights into the effectiveness of the decision-making
process employed by the Adapt-LLM model in determining the necessity of
additional context for accurate response generation and present empirical
evidence of the necessity of performing dynamic context retrieval in improving
the accuracy of question answering models.

However, it is notable that the overall performance of the model when
answering questions with retrieved context, as observed in Table 3
(approximately 33%), is relatively low. To further explore this observation,
we conduct an additional experiment: evaluating Adapt-LLM (both versions
trained on NQ and SQuAD) on the NQ and SQuAD development splits, comparing
performance when using the gold passages of the dataset and the context
retrieved by our IR system, Contriever [10]. Unfortunately, PopQA does not
provide the gold passages, so direct evaluation there was not possible.

Table 4 presents the results of this experiment. A significant performance
difference is observed between using the gold passage and the top passage
retrieved by Contriever for both datasets (approximately 67 absolute points
for SQuAD and 42 for NQ). This indicates that Contriever, and current IR
systems in general, do not consistently retrieve the most relevant passage to
answer a given question. This observation underscores the importance of
retrieving multiple documents as context, as seen in the most successful open-
domain QA systems [12], and highlights its impact on the overall performance
of Adapt-LLM in PopQA.

To further validate the behavior of Adapt-LLM when requesting additional
context, Figure 2 illustrates the proportion of questions for which our model
generates the $\langle$RET$\rangle$ token, aggregated by popularity score
intervals (left image for Adapt-LLM trained on NQ and right image for SQuAD).
Mallen et al. [21] suggest that high-popularity questions can be adequately
answered using the parametric memory of the LLM, while lower popularity scores
necessitate extra context. In Figure 2, we observe this pattern for both
versions of Adapt-LLM, indicating that our model, despite lacking access to
popularity scores during training or inference, has learned effective criteria
for requesting additional context.

###  4.7 Comparison with state-of-the-art methods

We conducted a comparative analysis between our Adapt-LLM model and the
current state-of-the-art approach for PopQA proposed by Mallen et al. [21].
Their methodology relies on the popularity score annotated in the PopQA
dataset to determine whether a question requires additional context. To
establish the optimal threshold for determining question popularity, Mallen et
al. [21] split the PopQA dataset into 75% as a development set for threshold
determination and 25% as a test set. In the original paper, they apply this
methodology to various LLMs available at that moment (Llama-2 was not released
yet).

To ensure a fair comparison between Adapt-LLM and the popularity-based method,
we replicated their approach using the Llama-2 7B model to determine the best
popularity score threshold (found to be 707,000) using the same PopQA
development set. This allowed us to obtain results consistent with their
methodology while utilizing our base LLM. Similar to the original results in
Mallen et al. [21] when using smaller models, the popularity score threshold
is almost equivalent to always retrieving contextual information for Llama-2
7B. The IR usage is of 99.86% as presented in Table 5. This clearly shows how
the popularity score method struggles with smaller size models, being GPT-3
davinci-003 the only model to get a IR usage below 80% in the original paper
when using adaptive retrieval with the Contriever. Subsequently, we evaluated
our Adapt-LLM configuration on the same 25% test set split and compared the
outcomes with those obtained using the method described by Mallen et al. [21].
This systematic comparison enabled us to assess the efficacy of our Adapt-LLM
model in relation to the current state of the art.

The results of this experiment are presented in Table 5. We observe comparable
performance between the replicated approach of Mallen et al. [21] and Adapt-
LLM when trained on NQ and SQuAD datasets and tested on the 25% subset of
PopQA. It’s worth mentioning that Adapt-LLM does not utilize any information
from PopQA, unlike Mallen et al. [21], who directly use the popularity score
and a 75% portion of PopQA dataset to find an optimal value for that
popularity score. This methodology is not generalizable to other open-domain
question answering tasks since the popularity score is a unique feature of
PopQA. However, Adapt-LLM can be applied to any similar dataset. Given these
characteristics, we believe that the results obtained by Adapt-LLM are even
more significant, offering comparable performance to an approach that utilizes
dataset-specific information. These findings substantiate the validity of our
approach, demonstrating its effectiveness even when trained on datasets
different from the one used for testing.

Model Configuration | IR usage | Accuracy
---|---|---
Popularity Score | 99.86% | 36.81%
Adapt-LLM (NQ) | 87.22% | 35.30%
Adapt-LLM (SQuAD) | 83.99% | 37.29%
Table 5: Performance comparison of Llama-2 base models trained on the SQuAD
and NQ datasets for the Adapt-LLM and Popularity Score configurations. The
later mimics the methodology proposed by Mallen et al. [21] with the Llama-2
LLM as the base model.

##  5 Conclusions

In this paper, we introduce Adapt-LLM, a LLM which learns to discern when
additional context is necessary for answering a question, rather than relying
solely on its parametric memory. Adapt-LLM is the result of fine-tuning a base
LLM on an open-domain question answering dataset that has been modified to
differentiate between questions answerable with the LLM’s parametric memory
alone and those requiring supplementary context. To construct these training
datasets, we initially subject the base LLM to zero-shot evaluation to
determine its accuracy in answering questions. For questions where the model’s
response is incorrect, we train the LLM to generate a special token,
$\langle$RET$\rangle$, indicating the need for additional context.

Through extensive experiments conducted on the PopQA dataset, we show that
Adapt-LLM performs better than its two fixed alternatives: never retrieving
and always retrieving relevant context information. Furthermore, our findings
highlight Adapt-LLM’s capability to effectively discern the necessity of
additional context, which is the primary objective of this work.

For future investigations, we propose exploring methods to enhance performance
when utilizing an IR system, such as incorporating learnable sequential
retrieval techniques. Furthermore, we believe it would be valuable to conduct
a more in-depth analysis of the interaction between training and testing
datasets in the development of Adapt-LLM systems.

## References

  * Achiam et al. [2023] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Amnon Catav and Roy Miara and Ilai Giloh and Nathan Cordeiro and Amir Ingber [2024] Amnon Catav and Roy Miara and Ilai Giloh and Nathan Cordeiro and Amir Ingber.  RAG makes LLMs better and equal.  2024\.  URL https://www.pinecone.io/blog/rag-study/.
  * Barnett et al. [2024] S. Barnett, S. Kurniawan, S. Thudumu, Z. Brannelly, and M. Abdelrazek.  Seven failure points when engineering a retrieval augmented generation system.  _arXiv preprint arXiv:2401.05856_ , 2024.
  * Berger et al. [2000] A. Berger, R. Caruana, D. Cohn, D. Freitag, and V. Mittal.  Bridging the lexical chasm: statistical approaches to answer-finding.  In _Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 192–199, 2000.
  * Borgeaud, Sebastian and Mensch, Arthur and Hoffmann, Jordan and Cai, Trevor and Rutherford, Eliza and Millican, Katie and Van Den Driessche, George Bm and Lespiau, Jean-Baptiste and Damoc, Bogdan and Clark, Aidan and others [2022] Borgeaud, Sebastian and Mensch, Arthur and Hoffmann, Jordan and Cai, Trevor and Rutherford, Eliza and Millican, Katie and Van Den Driessche, George Bm and Lespiau, Jean-Baptiste and Damoc, Bogdan and Clark, Aidan and others.  Improving language models by retrieving from trillions of tokens.  In _International conference on machine learning_ , pages 2206–2240. PMLR, 2022.
  * Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and others [2020] Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and others.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Choi, Eunsol and He, He and Iyyer, Mohit and Yatskar, Mark and Yih, Wen-tau and Choi, Yejin and Liang, Percy and Zettlemoyer, Luke [2018] Choi, Eunsol and He, He and Iyyer, Mohit and Yatskar, Mark and Yih, Wen-tau and Choi, Yejin and Liang, Percy and Zettlemoyer, Luke.  QuAC: Question Answering in Context.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2174–2184, 2018.
  * Gao et al. [2021] T. Gao, X. Yao, and D. Chen.  Simcse: Simple contrastive learning of sentence embeddings.  In _2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021_ , pages 6894–6910. Association for Computational Linguistics (ACL), 2021.
  * Gao, Yunfan and Xiong, Yun and Gao, Xinyu and Jia, Kangxiang and Pan, Jinliu and Bi, Yuxi and Dai, Yi and Sun, Jiawei and Wang, Haofen [2023] Gao, Yunfan and Xiong, Yun and Gao, Xinyu and Jia, Kangxiang and Pan, Jinliu and Bi, Yuxi and Dai, Yi and Sun, Jiawei and Wang, Haofen.  Retrieval-augmented generation for large language models: A survey.  _arXiv preprint arXiv:2312.10997_ , 2023.
  * Gautier, Izacard and Mathilde, Caron and Lucas, Hosseini and Sebastian, Riedel and Piotr, Bojanowski and Armand, Joulin and Edouard, Grave [2022] Gautier, Izacard and Mathilde, Caron and Lucas, Hosseini and Sebastian, Riedel and Piotr, Bojanowski and Armand, Joulin and Edouard, Grave.  Unsupervised dense information retrieval with contrastive learning.  _Transactions on Machine Learning Research_ , 2022.
  * Guu, Kelvin and Lee, Kenton and Tung, Zora and Pasupat, Panupong and Chang, Ming-Wei [2020] Guu, Kelvin and Lee, Kenton and Tung, Zora and Pasupat, Panupong and Chang, Ming-Wei.  REALM: retrieval-augmented language model pre-training.  In _Proceedings of the 37th International Conference on Machine Learning_. JMLR.org, 2020.
  * Izacard, Gautier and Grave, Edouard [2021] Izacard, Gautier and Grave, Edouard.  Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering.  In _EACL 2021-16th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 874–880. Association for Computational Linguistics, 2021.
  * Ji, Ziwei and Lee, Nayeon and Frieske, Rita and Yu, Tiezheng and Su, Dan and Xu, Yan and Ishii, Etsuko and Bang, Ye Jin and Madotto, Andrea and Fung, Pascale [2023] Ji, Ziwei and Lee, Nayeon and Frieske, Rita and Yu, Tiezheng and Su, Dan and Xu, Yan and Ishii, Etsuko and Bang, Ye Jin and Madotto, Andrea and Fung, Pascale.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38, 2023.
  * Jiang et al. [2024] A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al.  Mixtral of experts.  _arXiv preprint arXiv:2401.04088_ , 2024.
  * Kaplan et al. [2020] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ , 2020.
  * Karpukhin et al. [2020] V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, 2020.
  * Kwiatkowski, Tom and Palomaki, Jennimaria and Redfield, Olivia and Collins, Michael and Parikh, Ankur and Alberti, Chris and Epstein, Danielle and Polosukhin, Illia and Devlin, Jacob and Lee, Kenton and others [2019] Kwiatkowski, Tom and Palomaki, Jennimaria and Redfield, Olivia and Collins, Michael and Parikh, Ankur and Alberti, Chris and Epstein, Danielle and Polosukhin, Illia and Devlin, Jacob and Lee, Kenton and others.  Natural questions: a benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:453–466, 2019.
  * Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and Küttler, Heinrich and Lewis, Mike and Yih, Wen-tau and Rocktäschel, Tim and others [2020] Lewis, Patrick and Perez, Ethan and Piktus, Aleksandra and Petroni, Fabio and Karpukhin, Vladimir and Goyal, Naman and Küttler, Heinrich and Lewis, Mike and Yih, Wen-tau and Rocktäschel, Tim and others.  Retrieval-augmented generation for knowledge-intensive NLP tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474, 2020.
  * Liang et al. [2023] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, et al.  Holistic evaluation of language models.  _Transactions on Machine Learning Research_ , 2023.
  * Lin, Stephanie and Hilton, Jacob and Evans, Owain [2022] Lin, Stephanie and Hilton, Jacob and Evans, Owain.  TruthfulQA: Measuring How Models Mimic Human Falsehoods.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3214–3252, 2022.
  * Mallen et al. [2023] A. T. Mallen, A. Asai, V. Zhong, R. Das, D. Khashabi, and H. Hajishirzi.  When not to trust language models: Investigating effectiveness of parametric and non-parametric memories.  In _The 61st Annual Meeting Of The Association For Computational Linguistics_ , 2023.
  * Nakano, Reiichiro and Hilton, Jacob and Balaji, Suchir and Wu, Jeff and Ouyang, Long and Kim, Christina and Hesse, Christopher and Jain, Shantanu and Kosaraju, Vineet and Saunders, William and others [2021] Nakano, Reiichiro and Hilton, Jacob and Balaji, Suchir and Wu, Jeff and Ouyang, Long and Kim, Christina and Hesse, Christopher and Jain, Shantanu and Kosaraju, Vineet and Saunders, William and others.  Webgpt: Browser-assisted question-answering with human feedback.  _arXiv preprint arXiv:2112.09332_ , 2021.
  * Nguyen, Tri and Rosenberg, Mir and Song, Xia and Gao, Jianfeng and Tiwary, Saurabh and Majumder, Rangan and Deng, Li [2016] Nguyen, Tri and Rosenberg, Mir and Song, Xia and Gao, Jianfeng and Tiwary, Saurabh and Majumder, Rangan and Deng, Li.  Ms marco: A human-generated machine reading comprehension dataset.  2016\.
  * Rajpurkar, Pranav and Zhang, Jian and Lopyrev, Konstantin and Liang, Percy [2016] Rajpurkar, Pranav and Zhang, Jian and Lopyrev, Konstantin and Liang, Percy.  SQuAD: 100,000+ Questions for Machine Comprehension of Text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, 2016.
  * Ram, Ori and Levine, Yoav and Dalmedigos, Itay and Muhlgay, Dor and Shashua, Amnon and Leyton-Brown, Kevin and Shoham, Yoav [2023] Ram, Ori and Levine, Yoav and Dalmedigos, Itay and Muhlgay, Dor and Shashua, Amnon and Leyton-Brown, Kevin and Shoham, Yoav.  In-context retrieval-augmented language models.  _Transactions of the Association for Computational Linguistics_ , 11:1316–1331, 2023.
  * Reid et al. [2024] M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. Lillicrap, J.-b. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, et al.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  _arXiv preprint arXiv:2403.05530_ , 2024.
  * Reimers and Gurevych [2019] N. Reimers and I. Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, 2019.
  * Robertson et al. [2009] S. Robertson, H. Zaragoza, et al.  The probabilistic relevance framework: Bm25 and beyond.  _Foundations and Trends® in Information Retrieval_ , 3(4):333–389, 2009.
  * Schick et al. [2024] T. Schick, J. Dwivedi-Yu, R. Dessì, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom.  Toolformer: Language models can teach themselves to use tools.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Seonwoo, Yeon and Son, Juhee and Jin, Jiho and Lee, Sang-Woo and Kim, Ji-Hoon and Ha, Jung-Woo and Oh, Alice Haeyun [2022] Seonwoo, Yeon and Son, Juhee and Jin, Jiho and Lee, Sang-Woo and Kim, Ji-Hoon and Ha, Jung-Woo and Oh, Alice Haeyun.  Two-Step Question Retrieval for Open-Domain QA.  In _60th Annual Meeting of the Association for Computational Linguistics, ACL 2022_ , pages 1487–1492. Association for Computational Linguistics, 2022.
  * Taori, Rohan and Gulrajani, Ishaan and Zhang, Tianyi and Dubois, Yann and Li, Xuechen and Guestrin, Carlos and Liang, Percy and Hashimoto, Tatsunori B [2023] Taori, Rohan and Gulrajani, Ishaan and Zhang, Tianyi and Dubois, Yann and Li, Xuechen and Guestrin, Carlos and Liang, Percy and Hashimoto, Tatsunori B.  Stanford alpaca: an instruction-following llama model (2023).  _URL https://github. com/tatsu-lab/stanford_alpaca_ , 2023.
  * Thakur et al. [2021] N. Thakur, N. Reimers, A. Rücklé, A. Srivastava, and I. Gurevych.  Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.
  * Touvron et al. [2023a] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023a.
  * Touvron et al. [2023b] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_ , 2023b.
  * Zhu et al. [2021] F. Zhu, W. Lei, C. Wang, J. Zheng, S. Poria, and T.-S. Chua.  Retrieving and reading: A comprehensive survey on open-domain question answering.  _arXiv preprint arXiv:2101.00774_ , 2021.


# Towards a New Configurable and Practical
Remote Automotive Security Testing Platform

Sekar Kulandaivel  Robert Bosch Research & Technology Center
<EMAIL_ADDRESS>Wenjuan Lu  Block Harbor Cybersecurity
<EMAIL_ADDRESS>Brandon Barry  Block Harbor Cybersecurity
<EMAIL_ADDRESS>Jorge Guajardo  Robert Bosch Research & Technology
Center
<EMAIL_ADDRESS>Sekar Kulandaivel,1 Wenjuan Lu,2 Brandon
Barry,2 and Jorge Guajardo1  1Robert Bosch LLC — Research and Technology
Center, 2Block Harbor Cybersecurity
{sekar.kulandaivel<EMAIL_ADDRESS>{wen,
<EMAIL_ADDRESS>
###### Abstract

In the automotive security sector, the absence of a testing platform that is
configurable, practical, and user-friendly presents considerable challenges.
These difficulties are compounded by the intricate design of vehicle systems,
the rapid evolution of attack vectors, and the absence of standardized testing
methodologies. We propose a next-generation testing platform that addresses
several challenges in vehicle cybersecurity testing and research domains. In
this paper, we detail how the Vehicle Security Engineering Cloud (VSEC) Test
platform enables easier access to test beds for efficient vehicle
cybersecurity testing and advanced (e.g., penetration, fuzz) testing and how
we extend such test beds to benefit automotive security research. We highlight
methodology on how to use this platform for a variety of users and use cases
with real implemented examples.

##  1 Introduction

In light of recent security standards [3] and regulations [16], the automotive
cybersecurity industry demands more efficient and widespread cybersecurity
testing of automotive products. For example, ISO/SAE 21434 requires
verification and validation (V&V) testing among the vehicle, its subsystems,
and its components. Types of testing include cybersecurity functional testing,
vulnerability scanning, penetration testing, and fuzz testing. This
cybersecurity testing is further complicated by the current push towards
software-defined vehicles, which weaves safety with security and demands
faster cycles of development (and thus equally faster cycles of testing).

Unfortunately, executing security tests poses a substantial challenge as a
result of limited access to hardware, the lack of configurable networks, the
difficulty of connecting and configuring automotive electronic control units
(ECUs) for non-specialists, and the lack of qualified personnel. Fully-enabled
cybersecurity features often limit the capability of functional test engineers
to be able to measure and calibrate features on the spot, which results in
these critical features being disabled for the majority of the test cycle.
Cybersecurity testing is usually performed after functional and environmental
testing, which can make it challenging for security teams to push for patching
identified vulnerabilities if significant changes are required [19].

In addition to the lack of thorough security feature testing, penetration
testing on complete systems is also an area that needs much improvement. It is
already difficult to find security experts who are well-versed in vehicle
design and infrastructure, and the physical nature of the vehicle means often
these experts must be hands-on with the entire vehicle to find the path of
least resistance for a threat. The size and price tag of the targets makes
penetration testing of vehicles a costly process. Likewise, the ECUs within a
vehicle are complicated to set up outside of a running vehicle, especially
without access to proprietary knowledge (e.g., software, wiring diagrams).

There are existing efforts to combat these challenges and make automotive
security testing easier. Some solutions focus on cloud-based tests with
virtual ECUs [1]; however, these are typically isolated and may not represent
a whole system. Another solution is hardware-in-the-loop (HIL) testing [18,
8]. This type of testing approach is often tailored for testing individual
ECUs or (at most) a couple of ECUs, and it would be difficult and costly to
replicate a whole vehicle network of physical ECUs as it would require several
HIL systems. In addition, custom test beds are often manually built to target
a specific function, but these test beds are not configurable enough to be
used efficiently and cost-effectively for all necessary security testing. Due
to this effort, instrumenting these test beds and connecting them to a larger
testing infrastructure has not traditionally been a common approach.

There has been some significant progress in automotive security testing. The
Eclipse openDuT framework [4] aims to automate testing and validation for
automotive software and applications by offering a description language for
test cases, management of test cases, and orchestration of distributed ECUs.
VitroBench [20] connects ECUs on a vehicle network bus to an FPGA that can
launch a variety of network attacks while simulating signals from vehicle
sensors. CANlay [7] offers a network abstraction layer to facilitate
transmission of controller area network (CAN) data and sensor signals between
(remote) ECUs. Additionally, CRATE [13] offers an all-in-one-box hardware
solution that simplifies the connection to multiple ECUs and vehicle networks
and offers other convenient capabilities, such as dedicated power switching.
While each of these works targets some aspect of automotive security testing,
a platform that combines many of these features into a single configurable,
user-friendly, and practical remote security testing platform still eludes the
industry.

In this paper, we build such a testing platform and explore several testing
methodologies that this platform enables for penetration testers, ECU
engineers, and security researchers. We share our design and vision for this
testing platform, and we posit that such a platform should offer the following
key characteristics: (1) managed multiple user access control in a familiar
environment, (2) access to real hardware with a realistic, configurable
network, (3) hardware support at centralized locations with remote user
access, and (4) ability to add custom applications for security testing.

##  2 Current Security Testing Platforms

Currently, we see two main categories of verification activities for vehicle
cybersecurity that testing platforms have aimed to support. As a part of
automotive engineering development activities, cybersecurity verification is
done as a part of a standard V&V process. The usual procedure for aligning
this with ISO/SAE 21434 Clause 10.4.2 is as follows:

  1. 1.

Verification methods: This type of testing usually follow the ASPICE
engineering process [17] of Unit Testing, Integration Testing, and
Qualification Testing. Cybersecurity requirements are combined with functional
requirements and verified as per engineering process standards. This type of
testing includes requirement-based testing, interface testing, resource
evaluation, control and data flow verification, and static and dynamic
analysis. The techniques used in this type of testing follow general software
testing methodology and can be easily performed by a qualified test engineer.

  2. 2.

Testing for unidentified weakness or vulnerabilities: This type of testing is
unique to the field of cybersecurity and requires the test engineer to have
specialized knowledge of or experience with cyber attacks (e.g., the Sam Curry
attacks [2]) in order to be effective. These activities include targeted
functional testing, vulnerability assessment, fuzz testing, and penetration
testing.

Out of the numerous cybersecurity testing activities discussed above, the
verification testing methods are mostly addressed with a comprehensive
engineering process. These methods stand to benefit from broader acceptance of
other automotive quality-oriented disciplines, such as ASPICE and functional
safety, which should lead to improvements. As the industry adopts better and
more effective software quality engineering, in part, thanks to regulatory
pressure, we are starting to see many tools being developed to address the
activities of vulnerability assessment and fuzz testing, specifically
targeting automotive use cases.

#### Challenges

Even with the entrance of more service providers into the field, automotive
cybersecurity functional testing and penetration testing, particularly for
complex components or systems, still suffer from shortcomings in technical
proficiency and cost effectiveness. Below, we list some of the main challenges
we see in these types of testing:

  1. 1.

Technically qualified automotive cybersecurity testing specialists are hard to
find,

  2. 2.

Testing engagements are difficult to manage due to the (implicit) need to
access hardware remotely, and

  3. 3.

Fully remote penetration testing is still not realistic for most automotive
use cases and difficult to repeat over time.

As evident in other cybersecurity disciplines, such as web security or
Internet-of-Things (IoT) security, a test engineer should readily have access
to targets. However, getting access to a test target is currently much more
difficult for automotive pentesters due to the cost, availability, complexity
of modern vehicle architectures, and equipment needed for car hacking. Due to
the unique skill set of pentesters and the small talent pool, most automotive
OEMs and suppliers do not have the internal capability to perform fuzzing or
penetration testing on every product. Penetration testing requires a
completely different skill set and mindset from the traditional test engineer
or development engineer. This activity is increasingly outsourced to third-
party service providers. Compared to internal V&V, this outsourcing is much
more difficult to manage in the event of project delays, dependency on third-
party expertise and resources, concerns about data security and
confidentiality, and limited hardware availability, which can result in
timeline penalties and potentially cause mistakes in test target
configurations.

There has been some effort in addressing these issues with running software
simulations of ECUs and networks on the cloud. However, these are often
extremely limited and do not represent even a fraction of all attack surfaces
of a modern vehicle [12]. More importantly, any hardware-based weaknesses and
vulnerabilities are often completely absent in software-only simulations.
Traditionally, test beds for security projects are custom built for
penetration testing with a highly customized interface. These test beds have a
short lifetime (i.e., the length of a penetration testing contract), are very
time consuming to build, and require specialized knowledge to interface with
them. Due to the custom nature of these test beds and their fast turnarounds,
it has not been worthwhile to build a generalized testing platform. As
vehicles evolve to become more software-defined, a high demand is placed on
testing far beyond product development and into the production and operation
stages of the vehicle life cycle. Without a test bed to rerun penetration
tests on, identifying new vulnerabilities during the life of a product can be
difficult.

###  2.1 Recent progress

| TESTING PLATFORMS
---|---
KEY ATTRIBUTES | [4] | [13] | [20] | [1] | [7] | This platform
Remote access | ✓ | ✓ |  | ✓ | ✓ | ✓
Rich web interface |  |  |  | ✓ |  | ✓
Physically distributed ECUs | ✓ |  |  |  | ✓ | ✓
Hands-free configurable ECU network |  | ✓ | ✓ |  | ✓ | ✓
Handles physical COTS ECUs | ✓ | ✓ | ✓ |  | ✓ | ✓
TABLE I: A comparison of key attributes in our testing platform versus other
platforms and frameworks

Despite the challenges faced by the industry, we have seen considerable effort
being made recently to address the shortcomings of cybersecurity testing.
Table I compares our platform against several other modern testing platforms
and highlights the key attributes that we offer and also explore in this
paper.

#### Testing hardware

Electronics test and measurement equipment and software manufacturer, Keysight
Technologies, launched the SA8710A Automotive Cybersecurity Test Platform [8]
a few years back with hardware that connects to all relevant interfaces on a
modern vehicle and built-in security scanners and fuzzing tools. Built with
its product line of connectivity and measurement instruments, the Keysight
platform offers hardware that is capable of capturing small variations and
disturbances in communication signals. Last year, we saw the launch of the BG
Networks CRATE [13], an all-in-one hardware box capable of connecting to most
of the vehicle interfaces, with options to expand using USB. The CRATE™ also
offers some software tools to facilitate remote access and fuzz testing. Other
traditional hardware test interface providers do not offer specific hardware
for security testing, but they offer software plug-ins to support limited
security testing [18].

#### Testing frameworks

The Eclipse openDuT [4] framework focuses on automated and repeatable test and
validation executions in the automotive space. This work is still in an
incubating state and eventually aims to support a number of use cases, such as
fully automated gray-box tests, tests across distributed test benches, and
other functional tests. This framework provides end-to-end encryption of a
private network between edge devices (ECUs, Restbus simulations, etc.) with an
underlying control and registration logic. Ideally, network traffic from
different devices under test can be linked together to create a more unified
approach for testing. Additionally, CANlay offers virtual configurable
networks where isolated ECUs can be bridged with sensor simulators for J1939
testing. A subset of ECUs here called overlays can be tested by carrying
network traffic between these isolated ECUs.

#### The ideal solution

In contrast to prior work, we aim to build a platform that is easy to use by
many users without deep hardware knowledge and preferably provide remote
access to enable global teams to work on the same project at the same time.
The platform should be capable of offering multiple test beds and, if
required, the test bed should be configurable so that only a subset of ECUs
can be operated on-demand. This hardware should be located in a central
location with staff on site that specialize in hardware to support remote
users that access the system. It should also be easy for a remote test
engineer to launch their own custom security tests using remotely-programmable
hardware and measurement tools.

##  3 A New Testing Platform

We present a flexible and configurable testing platform for automotive
security by extending a cloud-based framework with additional components to
support a number of testing methodologies. We focus on building a solution
that is useful for many types of users, ranging from penetration testers to
security researchers, and we aim to create a platform that bridges features in
both software and hardware testing. We do not aim to cover all aspects of
functional and safety testing, and we do not attempt to recreate a HIL system.
We implement some of these testing methodologies and discuss practical
benefits of our tests.

###  3.1 Testing platform roles

We design this testing platform with a variety of users and roles in mind:
Penetration testers: Users who need to test quickly and easily launch
automated pentesting scripts, perform wireless tests while remote, access
platforms without needing to ship hardware, connect a local device to a remote
device (e.g., OEM dealership tool), and easily switch between different test
beds from different contracts.
Researchers: Users who need to easily deploy and demonstrate novel security
research on ECUs from multiple OEMs, copies of the same ECU for repeatability,
and multiple model years while performing analysis with measurement tools
(e.g., oscilloscopes, logic analyzers) and launching attacks from a custom ECU
on the network.
Engineers: Users who needs managed access to the platform via scheduling,
global remote access to test beds, quick reconfiguration of a prior project to
handle new vulnerabilities, and a familiar user interface to interact with.
Lab technicians: A role that is responsible for setting up hardware in the
lab, activating hardware-only features in-person, and moving probes for
measurement tools if need.

###  3.2 Web-based remote access

The first step to developing a user-friendly testing platform is providing an
easy-to-use and familiar interface. Vehicle Security Engineering Cloud [6]
(VSEC) is a cloud-based platform that aims to connect and automate various
steps of the vehicle cybersecurity engineering process. A major problem we see
in the V&V processes is the lack of repeatability and frequency of tests. To
address this problem, while maintaining confidentiality, VSEC Test centralizes
the testing process and execution while distributing the efforts across
locations and resources.

#### Managed remote access

Via a web interface, the VSEC Test platform allows users to register and
connect hardware test benches or vehicles to the cloud, which are then shared
and managed centrally under an enterprise account. Authorized users are able
to schedule benches, access all available interfaces of the bench via a Linux
terminal, and control multiple power sources to individual configured targets.
User access is configured and controlled by the lab orchestration software to
prevent unauthorized access. Features such as file explorer and workspace
sharing can be enabled to allow quick ways to reproduce test environments;
otherwise, each user has a sandbox environment in a Docker container to keep
their workspace separated. In addition, VSEC Test also hosts a test management
system that allows for continuous and repeated test setup with connected
benches.

To evaluate the effectiveness and robustness of the VSEC Test platform, we set
up a few bench and vehicle targets in the lab and connected various vehicle
interfaces to a local machine that connects to VSEC Test on the cloud. We used
built-in tests to remotely scan for basic information from a complete vehicle
network, and compared the results with similar open-source tools installed
directly to the local machine. Then we set up a custom test case on VSEC Test
to be ran periodically on the remote target while purposely triggering some
network downtime during test execution to simulate unplanned network
disruptions, and compared the results to a local setup. We found that the
added latency due to the cloud connection was not sufficient to cause a
noticeable negative effect on the effectiveness of the test team. Test results
obtained from the cloud platform are functionally equivalent to those achieved
through a local setup. Moreover, the cloud platform provides the added benefit
of facilitating instant sharing of results among team members and automating
the report generation process, thereby eliminating the need for manual
compilation of results from open-source tools.

###  3.3 Testbed setup

#### Sourcing ECUs

In the context of this work, we define ECUs as production ECUs from the field
(i.e., real vehicles), engineering samples (e.g., A-samples, B-samples), or
even off-the-shelf automotive-grade microcontrollers (MCUs). Depending on the
user, obtaining ECUs for the testing platform will require different
approaches. Where an engineer and pentester would get hardware from their own
supply chain or from the contracted customer, respectively, a researcher would
likely obtain ECUs from a real vehicle from either a salvage yard or buying
ECUs directly from OEM parts shops. Testing a single ECU can be sufficient if
the tests are focused on purely CAN physical layer specifics or other single-
ECU session-based protocols, such as Unified Diagnostic Services (UDS)
protocol and Universal Measurement and Calibration Protocol (XCP). In some
cases, multiple ECUs are needed to wake up certain ECUs (e.g., a powertrain
ECU may require a body ECU and vice versa), although we have reverse-
engineered some of these wake-up messages. For our platform, we source ECUs
across three different OEMs from a local salvage yard and from a real vehicle:
five powertrain ECUs (two pairs of duplicates from different model years), a
gateway ECU, a power steering ECU, and two instrument panel clusters (IPCs).
We also acquire a C-sample gateway ECU from one of the aforementioned OEMs,
which matches the OEM of one of the IPCs. Additionally, we also obtain a CAN-
enabled automotive-grade MCU with full programming capabilities.

#### Configurable network

Most users will often work on projects that require ECUs with different bus
speeds, different OEMs, and different attached hardware (e.g., measurement
tools, actuators, etc.) As such, it is necessary for a testing platform to
provide the ability to configure the platform remotely. The alternative here
is having access to multiple real vehicles or (re)building separate test beds,
which is expensive and harder to manage. We build a prototype of a
configurable CAN bus network system that supports multiple bus speeds,
multiple ECUs from multiple OEMs, and the ability to switch on/off the power
of each ECU on-demand via VSEC Test.

Figure 1:  The entire physical network layout (left) can contain ECUs from a
number of different original equipment manufacturers (OEMs). These ECUs may
connect to CAN buses with different speeds, which ultimately connect to a PC
via a CAN-USB interface. By powering off all other ECUs except those from OEM
#1, we can construct a network consisting of ECUs from just OEM #1 (middle).
Likewise, we can construct a network of ECUs from just OEM #2 (right) by
powering off all other ECUs.

In Figure 1, we depict an example of of a single platform that: (1) hosts
multiple buses with different bus speeds, (2) multiple OEM platforms on one
bench, and (3) the ability to control the network topology. The figure
demonstrates what the entire bench looks like with ECUs from three different
OEMs as well as how our testing platform would bring up a network consisting
of just ECUs from OEM #1 and OEM #2. Using a relay board that powers on/off
each ECU on command, a tester could also isolate a single ECU or subset of
ECUs from the entire set. The ability to slowly ramp up the number of ECUs
under test is useful when developing a technique (e.g., a new attack), and the
ability to test a technique against multiple OEM implementations can determine
the applicability of the technique. For our platform, we construct a
configurable network by attaching the power supply of each ECU to a 16-port
relay board,111The ignition pin can be powered by a separate relay if needed,
thus having up to 16 ports can help control both main power and ignition while
maximizing the number of ECUs under test. which is controlled by an Arduino
Due MCU. A script on the tester PC communicates with this MCU over Serial, and
the code on the Arduino toggles the user-specified ECU to activate/deactivate.
In this work, we focus on CAN buses, and adding a new bus protocol such as
Automotive Ethernet or CAN-FD would be similar.

#### Network simulation

Real CAN traffic from road tests is often only available after a vehicle is in
production. This traffic is usually the only information a researcher can
access to simulate the bus network. Reverse-engineering CAN traffic can
successfully uncover which CAN messages control which vehicle features/sensors
[14, 20]. Engineers and some pentesters (depending on the contract) have
access to proprietary CAN database (DBC) files, which identify the
producer/consumer of each signal for all CAN messages. An alternative source
of information for engineers and pentesters are Restbus simulations, which
provides sufficient CAN traffic for the ECU under test to function as
expected. For our platform, we investigated another method of obtaining a
larger network of ECUs by combining the traffic from multiple ECUs that belong
to the same make and model but are physically-distributed. We implemented a
test bed with an IPC located in Pennsylvania and remotely connected to a
powertrain ECU hosted in Michigan. While there is some latency involved with
forwarding CAN traffic between these two locations, the latency is small
enough to enable access to most UDS commands and exhibit control of the IPC
without any obvious delays.

#### Environment simulation

While reverse-engineering and proprietary DBC files can enable users to
interpret CAN signals, it is still challenging to mimic traffic from road
tests without spending hours driving a vehicle. Users can utilize driving
simulators, such as CARLA [15], to provide hours and miles of simulated road
traffic. Combining these simulators with either a reverse-engineered or
original DBC file can generate realistic CAN bus traffic to the ECUs under
test. Engineers often have access to advanced sensor and actuator models that
can significantly enhance the realism of the CAN bus traffic. While these
models are typically deployed in software-only simulations, they could be
connected to our testing platform to capture results only observed on real
hardware. For our testing platform, we utilize CARLA in combination with a DBC
file to provide realistic speed values to an IPC and then deploy an intrusion
detection system (IDS) from prior work [5] as if it were in a real car.

#### Measurement tools

We expand the capabilities of the platform by connecting measurement tools,
such as oscilloscopes and logic analyzers, and offering software control of
these tools. As these tools are often expensive, a platform with centralized
hardware can reduce overall hardware costs by permitting shared access to
these tools. For our platform, we connect a Picoscope oscilloscope to the
power pins of an ECU and a Saleae logic analyzer to all of our CAN buses. Via
command-line scripts, the Picoscope can capture power profiles as input to the
aforementioned IDS. Additionally, the logic analyzer can observe bit-level CAN
bus traffic for the purpose of building advanced CAN bus attacks [10, 11, 9].

##  4 Enabled Testing Methodologies

###  4.1 Secure Development Lifecycle (SDL) testing

VSEC Test aims to integrate all methods of vehicle cybersecurity testing into
a single easy-to-use platform. As discussed above, a portion of cybersecurity
verification, as required by the ISO/SAE standard, is no different than
software and systems verification required by other industry standards. The
remote capabilities and test management features offered by VSEC Test allows
test engineers to design and set up the test procedures one time in order to
run on a schedule or triggered by other tools as a part of CI/CD frameworks.
This centralized platform for test procedures and results can potentially be
used to exchange data throughout the supply chain and ensure consistent
verification test coverage for the development life cycle of a vehicle.

For example, we perform continuous cybersecurity functional testing on
components with over-the-air (OTA) update functionality as a service. We set
up the test rack and develop the test procedures once on VSEC Test, and it is
performed on a periodic schedule and also whenever a new software update is
applied. Continuous functional testing against cybersecurity requirements
effectively identified vulnerabilities that were overlooked during the
development cycle. In addition, VSEC Test provides a set of built-in tests as
a preliminary scan for known weaknesses and vulnerabilities, which can be used
directly against a target vehicle or ECU. This scan can allow users to easily
and quickly determine if there are large gaps in measures protecting the
interface or if there are known exploits that will work on the target vehicle
or ECU.

###  4.2 Penetration testing

#### Pentesting approaches

Most pentest engagements start with an exploratory phase, where the tester
aims to discover as much information about the target as possible. The
duration and effort in this phase is largely determined by the amount of
information available to the tester. VSEC Test allows users to build a pool of
their own test scripts or use built-in discovery scans specifically targeting
automotive networks, including protocols such as UDS and XCP. These tests can
be queued up and ran in the background or over the weekend, saving time and
allowing a team to focus on more technically challenging tasks or other
projects. The web interface allows testers to access and check on test status
at any time from anywhere to determine the next step of action.

#### Partner Pentesting

A major difficulty in accessing the talent pool of world-class vehicle
penetration testers is physical location. The VSEC Test platform enables any
remote engineer with credentials to connect to the bench and handle any
physical interactions with the component and remotely script and execute tests
from VSEC Test. We call this Partner Pentesting.

Figure 2: The remote test engineer can access either a fully remote testing
platform with access to multiple test beds or work collaboratively with an in-
lab (i.e., local to the lab) test engineer for tests that require hands-on
support. This Partner Pentesting method enables the technical specialist to
focus on their testing strategies while leaving the hardware setup and
physical controls to the lab technician/local test engineer.

To evaluate the Partner Pentesting concept, we set up a scenario as depicted
in Figure 2, where a remote wireless penetration testing specialist is working
with a local test engineer to control a software-defined radio connected to
the cloud and capture communication between the key fob and the car in
addition to any vehicle network messages. The specialist then proceeds to
study and decode the traffic, create scripts to attack the communication, and
work together with the local test engineer to validate findings. In this case,
without physical access to the vehicle and a local partner to perform tasks,
it would have been very difficult to perform a meaningful penetration test
against the vehicle’s wireless entry system.

#### Side-channel analysis

On the other hand, we have other benches in the lab can be tested with little
to no user interactions. For these targets, fully remote network-level
penetration test with a test bench setup can achieve results on-par with on-
site engagements. However, we found that more hands-on penetration testing
procedures such as hardware analysis and side channel attacks are still very
difficult to perform without having an onsite specialist. As a result, while
acceptable for many scenarios, fully remote penetration testing remains very
limited for certain configurations due to manual activation required for
physical interfaces and required hardware and tooling interaction for certain
procedures. However, we are able to make progress with a Partner Pentesting
setup that provides assistance from onsite personnel and achieves comparable
outcomes as a fully onsite engagement.

###  4.3 Research testing

We now discuss how our testing platform can enable a researcher to rapidly
establish and experiment with numerous ECU networks to support their security
research. A testing platform where the ECUs present on the bus can be easily
configured to support the development of a research project is ideal for fine-
tuning an attack or defense.

#### Enhancing test bed functionality

For research use cases, it is critical to have a programmable ECU on the
network to launch attacks or implement defenses. Typically, simulation-based
environments remove real-world characteristics, such as bus voltage, bit-level
CAN bus arbitration, bus errors, etc. Likewise, advanced attacks beyond simple
CAN bus injection often require microcontroller-level timing precision that
cannot be achieved by the PC-USB interface. Thus, our platform includes an
automotive-grade MCU to mimic having programming access to an ECU. For other
testing purposes, a bench with MCUs from several vendors could enable a user
to test with different vendor libraries and MCU features very quickly.

#### Exploring real research use cases

We identify a set of research projects that use a similar test setup and
demonstrate how our testing platform could have made that research easier and
more efficient. To demonstrate the usefulness of this new remote and
configurable platform, we implement and test three different open-source
research implementations using a single bench of ECUs. We remotely run the
CANvas network mapper [10] to correctly identify unique ECUs on two CAN buses
with different speeds and across several configurations network topologies. We
also remotely demonstrate the CANnon bus disruption attack [11] using the
automotive-grade microcontroller on our bench and observe numerous faults on
the CAN bus, which were captured by the attached Saleae logic analyzer.
Finally, we remotely deploy techniques from the CANdid authentication bypass
attack [9] against three configurations of a single bench, where we isolate
three powertrain ECUs from three different vehicle models using our relay-
controlled power inputs and observe the same ability to control the randomness
of the challenge. The CANnon and CANdid attack require launching attacks from
an automotive-grade microcontroller and require an attached oscilloscope or
logic analyzer to fine-tune certain parameters, which are all features offered
by our testing platform.

#### Offering new services

A unique challenge with automotive security research is the lack of hands-on
demonstrators for the community to experiment and test with. We envision the
offering of new services that permit access to research examples of attacks,
defenses, measurement techniques, etc. Instead of limiting the availability of
research to published code and papers, the ability to remotely access a
running sample in a real in-vehicle network environment would strengthen the
research community. In cases where code or hardware descriptions do not want
to be publicly disclosed, our testing platform could enable such access.

##  5 Conclusion & Outlook

With vehicles becoming more software-defined, the need for higher quality and
more automated security testing is evident. A flexible, easy-to-use, and
remote-capable test platform not only improves the current industry testing
capabilities, but it can also potentially provide easier access to
logistically challenging and costly hardware. This allows the industry access
a larger pool of talent for testing and allows more people to learn about
automotive components. Furthermore, we see potential in leveraging this type
of cloud-based platform to reduce costs and increase participation for global
hacking competitions and Bug Bounty programs.

That being said, the ideal testing platform is one that the community will
actually use and implement in their research, development, and production. In
this work, we propose several well-defined methodologies to build a useful and
configurable remote testing platform. However, as the needs of the automotive
security industry changes, this testing platform should be able to adapt and
be as flexible as possible. Once other testing platforms mature, future work
should investigate the different underlying technologies and features used to
build these platforms and explore relevant trade-offs and limitations.

## References

  * [1] CanBusHack.  Cloudcar.  Available at https://cloudcar.canbushack.com/. Accessed on 2024-03-15.
  * [2] Sam Curry.  Web hackers vs. the auto industry: Critical vulnerabilities in ferrari, bmw, rolls royce, porsche, and more.  Available at https://samcurry.net/web-hackers-vs-the-auto-industry/. Accessed on 2024-03-15.
  * [3] International Organization for Standardization.  ISO/SAE 21434: 2021: Road Vehicles: Cybersecurity Engineering.  ISO, 2021.
  * [4] Eclipse Foundation.  Eclipse opendut.  Available at https://projects.eclipse.org/projects/automotive.opendut. Accessed on 2024-03-15.
  * [5] Stefan Gehrer, Jorge Guajardo Merchan, and Shalabh Jain.  System and method for intrusion detection on a physical level using an internal analog to digital converter, March 31 2022.  US Patent App. 17/032,624.
  * [6] Block Harbor.  Block harbor vsec.  Available at https://blockharbor.io/vsec-platform/. Accessed on 2024-03-15.
  * [7] Jake Jepson, Subhojeet Mukherjee, Martin Span, and Jeremy Daily.  Canlay: A network virtualized testbed for vehicle systems–improving system integration and verification efforts.  In INCOSE International Symposium, volume 33, pages 1–16. Wiley Online Library, 2023.
  * [8] Keysight.  Sa8710a automotive cybersecurity test platform.  Available at https://www.keysight.com/us/en/product/SA8710A/automotive-cybersecurity-penetration-test-platform.html. Accessed on 2024-03-15.
  * [9] Sekar Kulandaivel.  Revisiting remote attack kill-chains on modern in-vehicle networks.  PhD thesis, Carnegie Mellon University, 2021.
  * [10] Sekar Kulandaivel, Tushar Goyal, Arnav Kumar Agrawal, and Vyas Sekar.  $\\{$CANvas$\\}$: Fast and inexpensive automotive network mapping.  In 28th USENIX Security Symposium (USENIX Security 19), pages 389–405, 2019.
  * [11] Sekar Kulandaivel, Shalabh Jain, Jorge Guajardo, and Vyas Sekar.  Cannon: Reliable and stealthy remote shutdown attacks via unaltered automotive microcontrollers.  In 2021 IEEE Symposium on Security and Privacy (SP), pages 195–210. IEEE, 2021.
  * [12] Shahid Mahmood, Hoang Nga Nguyen, and Siraj A Shaikh.  Automotive cybersecurity testing: Survey of testbeds and methods.  Digital Transformation, Cyber Security and Resilience of Modern Societies, pages 219–243, 2021.
  * [13] BG Networks.  Bg networks crate.  Available at https://bgnetworks.com/crate/. Accessed on 2024-03-15.
  * [14] Mert D Pesé, Troy Stacer, C Andrés Campos, Eric Newberry, Dongyao Chen, and Kang G Shin.  Librecan: Automated can message translator.  In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 2283–2300, 2019.
  * [15] CARLA Team.  Carla open-source simulator for autonomous driving research.  Available at https://carla.org/. Accessed on 2024-03-15.
  * [16] GRVA WP29 UNECE.  Un regulation no. 155—cyber security and cyber security management system.  Technical report, Technical Report. United Nations, 2021.
  * [17] VDA.  Automotive spice.  Available at https://vda-qmc.de/en/automotive-spice/. Accessed on 2024-03-15.
  * [18] Vector.  Vector vt system.  Available at https://www.vector.com/int/en/products/products-a-z/hardware/vt-system/#c174985. Accessed on 2024-03-15.
  * [19] VicOne.  Vicone automotive cyberthreat landscape report 2023.  Available at https://documents.vicone.com/reports/automotive-cyberthreat-landscape-report-2023.pdf. Accessed on 2024-03-15.
  * [20] Anthony Kee Teck Yeo, Matheus E. Garbelini, Sudipta Chattopadhyay, and Jianying Zhou.  Vitrobench: Manipulating in-vehicle networks and cots ecus on your bench: A comprehensive test platform for automotive cybersecurity research.  Vehicular Communications, 43, 2023.