#  LACIE: Listener-Aware Finetuning for Confidence Calibration in Large
Language Models

Elias Stengel-Eskin Peter Hase Mohit Bansal
UNC Chapel Hill

###### Abstract

When answering questions, large language models (LLMs) can convey not only an
answer to the question, but a level of confidence about the answer being
correct. This includes explicit markers of confidence (e.g. giving a numeric
confidence score) as well as implicit markers, like using an authoritative
tone or elaborating with additional knowledge of a subject. For LLMs to be
trustworthy sources of knowledge, the confidence they convey should match
their actual expertise on a topic; however, this is currently not the case,
with most models tending towards overconfidence. To calibrate both implicit
and explicit confidence markers, we introduce a pragmatic, listener-aware
finetuning method (LACIE) that directly models the listener, considering not
only whether an answer is right, but whether it will be accepted by a
listener. Specifically, we cast calibration as a preference optimization
problem, creating data via a two-agent speaker-listener game, where a speaker
model’s outputs are judged by a simulated listener. We then finetune three
different LLMs (Mistral-7B, Llama3-8B, Llama3-70B) with LACIE, and show that
the models resulting from this multi-agent optimization are better calibrated
on TriviaQA with respect to a simulated listener. Crucially, these trends
transfer to human listeners, helping them correctly predict model correctness:
we conduct a human evaluation where annotators accept or reject an LLM’s
answers to trivia questions, finding that training with LACIE results in
$47\%$ fewer incorrect answers being accepted while maintaining the same level
of acceptance for correct answers. Furthermore, LACIE generalizes to another
dataset, resulting in a large increase in truthfulness on TruthfulQA when
trained on TriviaQA. Our analysis indicates that LACIE leads to a better
separation in confidence between correct and incorrect examples.
Qualitatively, we find that a LACIE-trained model hedges more when uncertain
and adopts implicit cues to signal certainty when it is correct, such as using
an authoritative tone or including details. Finally, finetuning with our
listener-aware method leads to an emergent increase in model abstention (e.g.
saying “I don’t know”) for answers that are likely to be wrong.111Code:
https://github.com/esteng/pragmatic_calibration

##  1 Introduction

In interacting linguistically with each other, people tend to follow
conventions – or maxims – that allow for successful communication. For
example, good conversational partners try to make their utterances truthful,
relevant, clear, and concise [Grice, 1975]. When people violate these
conventions, they can mislead listeners, which may ultimately lead to them
being seen as incompetent, untrustworthy, or as poor conversational partners.
While large language models (LLMs) generally follow many of these conventions,
they often fail to respect Grice [1975]’s maxim of truthfulness, generating
outputs that are not truthful [Rawte et al., 2023]. More troublingly,
untruthful outputs generated by LLMs are often expressed confidently and
authoritatively, and thus appear convincing to users, meaning that humans may
easily be misled by LLMs.

LLMs’ confidence can be expressed in at least two ways, shown in Fig. 1A.
Firstly, LLMs can _explicitly_ express confidence in their output using
numeric scores (e.g. _“I am 100% confident”_) or epistemic markers [Zhou et
al., 2023] (e.g. _“I’m very sure that…”_). Secondly, LLMs can _implicitly_
express confidence by details or through their tone; often, the details
included are spurious or non-factual, i.e. hallucinated. For example, in our
analysis in Table 3, we found that LLMs often add hallucinated backstories to
their answers, e.g. _“I remember seeing this movie on the big screen in the
theatre…”_ , or add an additional explanation that may sound convincing but is
untrue. These details convey a sense of expertise that can lead to the answer
being perceived as more likely to be correct.

Figure 1:  _(A)_ A non-expert listener (who does not know the answer to the
question already) accepts or rejects answers based on how confident they
sound. This confidence is influenced by implicit and explicit markers. _(B)_
To calibrate a speaker model’s confidence, we train a listener-aware speaker
model by bootstrapping data from a base speaker model. For each training
question, we generate $k$ diverse responses. These are scored for correctness
against the gold answers and accepted or rejected by a listener model. Our
preference function rewards true accepts and true rejects and penalizes false
accepts and false rejects. _(C)_ Before training, models tend to be confident
regardless of whether they are right or wrong. After training, listener-aware
models are more confident when they are correct and less confident when they
are wrong.

Overconfidence is particularly troubling given that people are increasingly
interacting with LLMs as sources of information [Gude, 2023]; in other words,
people rely on LLMs to answer questions they themselves do not know the answer
to. Futhermore, because the interactions people have with LLMs happen via
language, users often interpret LLM outputs as they would interpret language
from a human, i.e. assuming that the outputs follow Gricean maxims. This in
turn makes LLMs unreliable partners; many readers may have had the experience
of working together with a partner or teammate who consistently overstates
their confidence. While this teammate may initially have their answers
accepted, over time they lose trust. Indeed, Zhou et al. [2024] highlight this
type of erosion for overconfident models, finding that overconfidence
irreparably damaged a user’s trust in an AI system.

Focusing on answering information-seeking questions, we hypothesize that model
overconfidence of both kinds (implicit and explicit) can be mitigated by
optimizing for pragmatics, i.e. for how the utterance will be interpreted by a
listener. Specifically, we hypothesize that part of current models’
overconfidence lies in (1) a lack of knowledge about whether its answers are
correct or not, and (2) a lack of pragmatic grounding, i.e. models do not
generate utterances according to how they will be perceived by a listener.
Firstly, base models (not finetuned with instructions or human feedback) are
not grounded in the consequences of their answers: they do not receive direct
feedback during training about whether the answer is correct or not, and thus
have little reason to hedge in their confidence. Secondly, models are not pre-
trained _pragmatically_ ; they generate responses without real-time access to
feedback on how listeners might interpret their answers. While models trained
with human feedback may in principle have this capacity, past work has shown
that they in fact have _worse_ calibration than base models and has attributed
this to current reward data penalizing hedging and markers of uncertainty
[Zhou et al., 2024].

To address overconfidence by tackling these two types of grounding, we
introduce _L istener-Aware Calibration for Implicit and Explicit confidence_,
or LACIE. LACIE finetunes models not only using feedback on whether their
answer was correct but _also_ whether their answer was interpreted as correct
by a listener. In other words, whereas past work [Kuhn et al., 2022, Tian et
al., 2023, Ulmer et al., 2024, _i.a._] has sought to produce calibrated
distributions in speaker’s output distribution – i.e. answer probabilities
that are equal to the model’s chance of being correct – we seek to _induce_ a
calibrated distribution in the listener via the speaker’s utterance – i.e.
train the model to output generations that allow a listener to recover a well-
calibrated score of how likely the answer is to be correct. This multi-agent
optimization not only trains models to reliably use both explicit and implicit
confidence markers, but also allows us to flexibly address calibration for
long-form model answers and not only closed-set answer probabilities.

To pragmatically calibrate LLMs, we adopt the Direct Preference Optimization
(DPO) framework Rafailov et al. [2024], constructing a dataset of preferrred
and dispreferred examples from a seed dataset of QA pairs. As shown in Fig.
1B, we first generate long-form responses from a standard LLM (the speaker
agent). Many of these responses contain both implicit and explicit certainty
expressions, often applied inappropriately to incorrect answers. We then use
another LLM to model a listener agent who decides whether to accept or reject
the answer; in information-seeking questions (where the answer is not known),
the listener model should base its decision largely on how confident the
speaker sounds. Note that our multi-agent framing allows us to explore a much
wider range of confidence cues; while past work [Mielke et al., 2022, Lin et
al., 2022, Zhou et al., 2024] has focused on epistemic markers (explicit
expressions of confidence) we are also able to define and examine more subtle
and implicit confidence cues, like a tone, level of detail, and use of
backstories. Using the listener model and the ground-truth answer, we define a
preference function (given in Section 3) that rewards cases where the model
accurately expresses confidence – marking these as preferred examples – and
penalizes the model when it inaccurately expresses confidence – marking them
as dispreferred. This makes the training _listener-aware_ , connecting to past
work in jointly modeling speakers and listeners [Frank and Goodman, 2012,
Fried et al., 2018b, Lazaridou et al., 2020].

We demonstrate the effectiveness of our method first with automated metrics
and then through a human evaluation. We generate training data using $10,000$
QA examples from TriviaQA [Joshi et al., 2017]; our automated data generation
pipeline (described in Section 3) allows us to transform these into $\sim
14,000$ preference instances, which we use to train several LLMs. When testing
our optimized model on TriviaQA using an LLM listener, we find that base open-
source LLMs are generally overconfident, producing answers that are accepted
by the listener despite often being wrong. Using LACIE on three different
speaker models (Mistral-7B, Llama3-8B, and Llama3-70B), we obtain substantial
gains in induced listener calibration, with an average $20.7$ point gain in
AUROC over the base model and a $7.8$ point decrease in calibration error,
indicating that utterances from LACIE-trained models induce more calibrated
distributions in the listener. We also obtain an average $18\%$ absolute
improvement in precision, meaning that LACIE-optimized models produces less
over-confident utterances; these utterances are more consistently rejected by
the listener when they are wrong. Going beyond automated evaluation, we
perform a human evaluation in which we show that LACIE significatly reduces
the rate at which incorrect answers are accepted by _human_ listeners; a model
trained with LACIE results in a $47\%$ decrease in the rate of false answers
being accepted without significantly increasing the rate of rejection for
correct answers. In our analysis, we show that our training transfers between
datasets: we train our models on TriviaQA and evaluate them on TruthfulQA [Lin
et al., 2021]. Here, we show that LACIE results in a $28\%$ absolute
improvement to truthfulness, as measured by TruthfulQA’s metrics. We
underscore our quantitative improvements with a qualitative analysis showing
that training leads to more hedging and abstention, as well as more detailed
outputs and more authoritative tone when the model is actually correct.

##  2 Background and Related Work

#### Background: Pragmatics.

Pragmatics studies how people interpret language in context, going beyond the
literal meaning of an utterance. Grice [1975] gives a seminal account of
conventions that people generally follow and how they relate to implicatures:
quantity (saying as much as needed, and not more), relation (being relevant),
manner (avoiding obscure or ambiguous language), and quality (being truthful).
Most pragmatic accounts involve speakers not only reasoning about the literal
meaning of their utterance, but also about how a _listener_ will interpret the
utterance. In other words, pragmatic reasoning involves knowing not only what
you mean, but also what others will think you mean. We introduce pragmatics
into LACIE through multi-agent modeling, where the speaker is optimized by
considering not only the answer provided by the speaker model but also its
interpretation by the listener model.

#### Pragmatic Modeling.

Frank and Goodman [2012] introduce the RSA model, which is a formal
description of how pragmatic agents communicate. This formulation has been
applied to generation tasks in a number of ways [Fried et al., 2018a, b,
Lazaridou et al., 2020, Vaduguru et al., 2023]. In two-player communication
games, Wang et al. [2021] introduce methods for improving listener model
calibration with the goal of reducing “semantic drift” between the meaning of
speaker utterances and their original meaning in natural language. In
contrast, we directly train speaker models to induce calibrated answers in
listener models, and our approach is complementary to the specific choice of
listener model. Furthermore, the domains examined by Wang et al. [2021] are
limited to a simple communication game, and they do not evaluate with human
listeners. While we draw on the RSA formulation for inspiration, we do not
directly apply it because listeners choose from a fixed set of interpretations
in RSA. Instead, we allow for listener models to interpret implicit and
explicit confidence markers in an arbitrary manner, and we train the speaker
model to induce calibrated answers in the listener regardless of the
listener’s manner of interpretation.

#### Calibration in LLMs.

Given that calibration is key to making intelligent decisions on when to trust
AI systems, a number of past efforts have documented calibration in neural
models [Naeini et al., 2015, Guo et al., 2017, Ovadia et al., 2019, Wang et
al., 2020] with recent work focusing on calibration in LLMs [Mielke et al.,
2022, Kadavath et al., 2022, Kuhn et al., 2022, Stengel-Eskin and Van Durme,
2023, Tian et al., 2023, Zhang et al., 2023].

Within this area, several papers have focused on verbalized confidence. Mielke
et al. [2022] introduce control codes based on model confidence to get models
to better use epistemic markers. Lin et al. [2022] finetune a GPT-3 model
using the average answer accuracy of predicted answers, showing improvements
to the calibration of verbalized confidence; this is similar to our “truthful-
only” baseline, which only optimizes for answer accuracy using the speaker
model’s generations. Zhou et al. [2023] categorize epistemic markers and
measure their impact on LLM accuracy. Zhou et al. [2024] document the use of
epistemic markers by LLMs, finding that LLMs rarely use weakeners, and measure
the impact of poor calibration on usability, finding that overconfidence
irreparably hurts performance. Supporting these findings, Kim et al. [2024]
perform a large-scale evaluation of how human subjects respond to epistemic
markers, finding that first-person weakeners reduce over-reliance on model
answers. Taken together, these findings suggest that improving models’
abilities to correctly provide verbal confidence estimates has the potential
to improve model safety, reliability, and usefulness to users. Building on
these findings, we propose a new method to reduce overconfidence and make
epistemic marker usage more appropriate. Unlike Mielke et al. [2022] our
method does not rely on pre-defined codes, and unlike other work that trains
models to have better-calibrated confidence [Mielke et al., 2022, Lin et al.,
2022, Ulmer et al., 2024, Li et al., 2024], our work takes a pragmatics-based
approach of also modeling the listener.

Lastly, past work has also evaluated free-text generations for their
correctness with an LLM, which is a form of model-based calibration. Kadavath
et al. [2022] introduce a confidence estimation method that first generates an
answer and then asks via follow-up prompt whether the answer is correct or
not. This formulation is also adopted by Ren et al. [2023] and is similar to
using a listener model. However, unlike our work, past work has not optimized
the speaker model for the listener (meaning the speaker is not pragmatic), and
has restricted itself to using the same model as both speaker and listener,
whereas we take a multi-agent approach in which the speaker and listener may
differ.

##  3 Methodology

###  3.1 Datasets

We use two datasets in this paper. First, for our finetuning experiments and
human study, we use TriviaQA [Joshi et al., 2017], which includes challenging
open-domain general-knowledge trivia questions along with source documents.
From the $650,000$ total questions, we sample $10,000$ for use in DPO. For
each question, TriviaQA includes several eligible phrasings of the answer
choice. Following the TriviaQA evaluation, we mark a model output as correct
if its answer exactly matches any eligible answer string. The second dataset
we use is TruthfulQA [Lin et al., 2021], which includes questions that people
commonly have misconceptions about, stemming from widely circulated conspiracy
theories, folk theories, apocryphal stories, or other commonly repeated
falsehoods. We use this dataset for judging the transfer of our calibration
finetuning across datasets. TruthfulQA is particularly useful as an evaluation
here because the benchmark includes a model-based evaluation for assessing the
tradeoff between truthfulness and informativeness. Thus, we can show that our
finetuning improves truthfulness directly (by preventing the model from saying
false things that it is actually unconfident about). Our test sizes are
$1,000$ for automatic evaluation with TriviaQA, $817$ for automatic evalation
with TruthfulQA, and $100$ for human evaluation with TriviaQA.

###  3.2 Listener-Aware Preference Data Creation Methodology

To generate training data for LACIE, we instantiate speaker and listener
models. The speaker model is prompted to express its confidence verbally, and
the listener model is prompted to ignore its prior knowledge, as we are
primarily interested in the perceived confidence of the answer (see Appendix G
for the exact prompts). We begin the dataset creation process by subsampling
$N$ question-answer pairs $(Q^{i},\hat{A}^{i})$ from the dataset. We then
obtain multiple responses $R^{i}_{j},\>j\in\\{1,\ldots,k\\}$ to each question
from the speaker model, sampling independently with temperature to encourage
diversity, resulting in $N\times k$ $(Q^{j},R^{i}_{j})$ pairs. For each
response $R^{i}_{j}$, we also extract the final answer $A^{i}_{j}$ (usually
1-3 words). Each $(Q^{i},R^{i}_{j})$ is then given to the listener model,
which produces probability of accepting or rejecting the answer $P^{i}_{j}$.
From the gold answer $\hat{A}^{i}$ we can determine whether $A^{i}_{j}$ was
correct, leading to a gold accept/reject decision $\hat{D}^{i}_{j}$. Then, for
a given question $Q^{i}$, we enumerate the different possible combinations of
responses $R^{i}_{j}$, computing the preferences given in Eq. 1.

Answer Extraction We prompt the speaker model for a long-form answer; however,
to evaluate against the gold answer $\hat{A}^{i}$ we need to extract a single
short-form response. To do this, we use a follow-up prompt with $3$ in-context
examples showing how to extract answers from responses.

Answer Anonymization. While we instruct the listener model to ignore its prior
knowledge in evaluating the speaker’s answers, we find that this is a
difficult instruction for the model to follow, especially when the speaker and
listener model are the same. Indeed, past work has found that LLM evaluator
models tend to prefer their own outputs [Panickssery et al., 2024]. To
mitigate this, we implement an answer anonymization strategy: using regular
expressions, we remove mentions of the extracted answer from the response,
replacing them with [ANSWER REMOVED]. This way, the listener must focus more
on the way the response is phrased (see Appendix A for more information).

#### Preference Function.

To construct preference data over these tuples, we convert all probabilities
$P$ of accepting the answer to decisions by setting a threshold $\theta$ and
setting $D^{i}_{j}=\delta(P^{i}_{j}>\theta)$. We use the median probability
across the training data as our threshold; for our Mistral-7B listener, this
is $0.66$. For each $Q^{i}$, we compare all combinations of answers; our
preference function is given by Eq. 1.

$\begin{array}[]{ C
}\underbrace{U(\hat{D}^{i}_{j}=1,D^{i}_{j}=1)=U(\hat{D}^{i}_{j}=0,D^{i}_{j}=0)}_{\text{\emph{(correct,
accepted)} is as good as \emph{(incorrect, rejected)}}}\\\
\underbrace{U(\hat{D}^{i}_{j}=0,D^{i}_{j}=0)>U(\hat{D}^{i}_{j}=1,D^{i}_{j}=0)}_{\text{both
\emph{(correct, accepted)} and \emph{(incorrect, rejected)} are better than
\emph{(correct, rejected)}}}\\\
\underbrace{U(\hat{D}^{i}_{j}=1,D^{i}_{j}=0)>U(\hat{D}^{i}_{j}=0,D^{i}_{j}=1)}_{\text{\emph{(correct,
rejected)} is better than \emph{(incorrect, accepted)}}}\par\end{array}$ (1)

where $\hat{D}^{i}_{j}=1$ means the answer was _correct_ and and $D^{i}_{j}=1$
means the answer was _accepted_ by the listener (i.e. $P^{i}_{j}>\theta$).
Note that our function encodes a conservative interpretation of calibration,
in which it is better to err on the side of false negatives than false
positives.

###  3.3 Preference finetuning

From our preference data, we finetune models using DPO [Rafailov et al.,
2024]. DPO seeks to maximize margin between the likelihood of the preferred
examples and that of the dispreferred examples; in this case, DPO maximizes
for correctly-calibrated outputs and for incorrectly-calibrated outputs. We
train our models using QLoRA [Dettmers et al., 2024] with rank 16 for a max of
250 steps. Further details on the preference data and finetuning are given in
Appendix A.

##  4 Experiments and Results

We first show LACIE’s performance using calibration metrics on the listener
model, where we see substantially better calibration after training. We then
show that this translates to a human evaluation, where we present LACIE and
baseline outputs to real listeners.

###  4.1 Setup

For all experiments, we sample a training dataset of $10,000$ pairs of
questions $Q^{i}$ and gold answers $\hat{A}^{i}$ from the TriviaQA validation
data. We then obtain 10 responses to each question from a Mistral-7B base
model and extract their answers. We use the official TriviaQA metric to obtain
the correctness of the extracted answer, $\hat{D}^{i}_{j}$. We reserve $1,000$
$(Q^{i},A^{i}_{j},P^{i}_{j},\hat{D}^{i}_{j})$ tuples, corresponding to $100$
TriviaQA questions, as development data to use during training. We also sample
a _separate_ held-out test set of $1,000$ TriviaQA questions that are separate
from the training/dev data; here, we only obtain one response since we do not
need preferences at test time.

We evaluate three speaker models of varying sizes: Mistral-7B, Llama3-8B, and
Llama3-70B. For the smaller models, we average across 3 seeds. Across all
models, we use the same listener model (Mistral-7B).

Baselines. We compare LACIE to the base model, without fine-tuning. For the
base model, we simply take the top generation from the base model as the
response $R$. As part of our preference function is based on answer
correctness, we also compare to a model finetuned on correctness alone, i.e. a
preference function which prefers correct to incorrect answers (with no regard
for the listener model); we call this the “truthful” baseline. Finally, we
compare against the chat versions of the models tested; these models are
instruction-tuned on chat-style data, making them generally better at
following instructions. This baseline addresses whether we can simply ask the
model for a calibrated score by asking an instruction-tuned model to express
its confidence.

Metrics. We report the following metrics:

  * •

AUROC measures the tradeoff between true acceptances and false acceptances
across varying thresholds on the numeric confidence, which in our case is the
listener’s induced $P^{i}_{j}$. A higher AUROC means the model is better
calibrated.

  * •

ECE [Naeini et al., 2015] bins confidence scores $P^{i}_{j}$ and measures the
difference between these bins and their average correctness. We use $9$ bins,
backing off to fewer if bins are empty, and use unweighted ECE. Note that we
ignore abstentions in computing ECE, as assigning a meaningful probability of
answer correctness to abstention (no answer) is ill-posed.

  * •

Precision and Recall compare the rate of true acceptances, false acceptances,
and false rejections. We focus on precision as it is sensitive to false
acceptances from overconfident utterances, which have been shown to reduce
system trust [Zhou et al., 2024]; higher precision is often driven by a lower
rate of false acceptances, i.e. a lower rate of false positives. Here, we use
the median listener probability on train as the threshold to binarize
$P^{i}_{j}$ into accept and reject. Higher is better.

  * •

Abstention rate is the rate at which models produce no extractible answer;
this typically corresponds to the model expressing a lack of knowledge. A safe
model should abstain when it is very uncertain of its answer.

###  4.2 Results with Modeled Listener

We evaluate on a held-out subset of $1,000$ TriviaQA examples, reporting our
metrics in Table 1.

Table 1:  TriviaQA performance (and standard error) with metrics computed according to a Mistral-7B listener model. We bold the best value for each model.  | Induced Listener Calibration
---|---
Speaker Model | AUROC |  ECE $\downarrow$ |  Precision $\uparrow$ |  Recall $\uparrow$ | % Abstained
_Mistral-7B_ |  |  |  |  |
+Base | $0.54_{\pm\text{ 0.00}}$ | $0.15_{\pm\text{ 0.01}}$ | $0.56_{\pm\text{ 0.00}}$ | $\mathbf{0.76}_{\pm\text{ 0.00}}$ | $0.80_{\pm\text{ 0.00}}$
+Chat | $0.63_{\pm\text{ 0.01}}$ | $0.23_{\pm\text{ 0.01}}$ | $0.52_{\pm\text{ 0.00}}$ | $0.58_{\pm\text{ 0.01}}$ | $13.83_{\pm\text{ 0.52}}$
+Truthful | $0.58_{\pm\text{ 0.03}}$ | $0.16_{\pm\text{ 0.01}}$ | $0.63_{\pm\text{ 0.01}}$ | $0.52_{\pm\text{ 0.04}}$ | $8.87_{\pm\text{ 2.45}}$
+LACIE | $\mathbf{0.74}_{\pm\text{ 0.02}}$ | $\mathbf{0.12}_{\pm\text{ 0.00}}$ | $\mathbf{0.69}_{\pm\text{ 0.02}}$ | $0.55_{\pm\text{ 0.03}}$ | $25.27_{\pm\text{ 0.37}}$
_Llama3-8B_ |  |  |  |  |
+Base | $0.57_{\pm\text{ 0.01}}$ | $0.19_{\pm\text{ 0.02}}$ | $0.55_{\pm\text{ 0.02}}$ | $0.40_{\pm\text{ 0.02}}$ | $13.20_{\pm\text{ 0.31}}$
+Chat | $0.59_{\pm\text{ 0.00}}$ | $0.23_{\pm\text{ 0.00}}$ | $0.64_{\pm\text{ 0.00}}$ | $\mathbf{0.98}_{\pm\text{ 0.00}}$ | $2.93_{\pm\text{ 0.20}}$
+Truthful | $0.63_{\pm\text{ 0.04}}$ | $0.24_{\pm\text{ 0.08}}$ | $0.63_{\pm\text{ 0.03}}$ | $0.64_{\pm\text{ 0.19}}$ | $9.60_{\pm\text{ 4.35}}$
+LACIE | $\mathbf{0.72}_{\pm\text{ 0.00}}$ | $\mathbf{0.12}_{\pm\text{ 0.02}}$ | $\mathbf{0.70}_{\pm\text{ 0.01}}$ | $0.37_{\pm\text{ 0.00}}$ | $35.37_{\pm\text{ 5.41}}$
_Llama3-70B_ |  |  |  |  |
+Base | $0.53_{\pm\text{ 0.02}}$ | $0.27_{\pm\text{ 0.04}}$ | $0.58_{\pm\text{ 0.05}}$ | $0.30_{\pm\text{ 0.00}}$ | $12.87_{\pm\text{ 4.11}}$
+Chat | $0.61_{\pm\text{ 0.02}}$ | $0.21_{\pm\text{ 0.03}}$ | $0.76_{\pm\text{ 0.01}}$ | $\mathbf{0.98}_{\pm\text{ 0.01}}$ | $2.25_{\pm\text{ 0.08}}$
+Truthful | $0.65_{\pm\text{ 0.03}}$ | $0.21_{\pm\text{ 0.00}}$ | $0.78_{\pm\text{ 0.02}}$ | $0.35_{\pm\text{ 0.05}}$ | $12.20_{\pm\text{ 3.88}}$
LACIE | $\mathbf{0.80}_{\pm\text{ 0.02}}$ | $\mathbf{0.14}_{\pm\text{ 0.03}}$ | $\mathbf{0.84}_{\pm\text{ 0.01}}$ | $0.40_{\pm\text{ 0.01}}$ | $32.77_{\pm\text{ 2.34}}$
Table 2: Evaluation with human listeners. When outputs are shown to people, LACIE leads to fewer incorrect outputs being accepted, without significantly increasing the rate of false rejections. Significant differences in accept/reject counts marked with ∗ (McNemar’s test, $p<0.5$).  Speaker Model | $n$ | Precision | Recall | True Accept | False Accept | False Reject
---|---|---|---|---|---|---
Mistral-7B (base) | 79 | 0.49 | 0.84 | 31 |  32∗ | 6
Mistral-7B + LACIE | 78 | 0.64 | 0.81 | 30 |  17∗ | 7

####  LACIE results in better induced listener calibration across speaker
models.

Table 1 shows that LACIE improves induced listener ECE and AUROC, consistently
resulting in the highest performance. LACIE also improves precision
substantially across models, outperforming all baselines. On average, LACIE
improves AUROC by $20.7$ points across models, and reduces ECE by $7.8$
points, while increasing precision by $18\%$ (absolute). Note that while
truthful-only finetuning does typically improve induced listener AUROC, LACIE
consistently beats it by an average of $13.3$ points. LACIE and truthful
finetuning both increase abstention over the base model. This in turn reduces
recall; however, note that recall penalizes abstention, since models that
confidently guess can increase their true positive rate, and recall does not
measure the false positive rate. Furthermore, note that this reduction in
recall _does not_ translate to the human evaluation, where human listeners
only had one false negative more from LACIE questions than the base model.
Finally, we see that LACIE translates well to the largest size of model,
resulting in better performance on AUROC, ECE, and precision for Llama-70B
over the corresponding 70B baselines.

####  LACIE training leads to abstention ability.

In Table 1, the percentage of abstentions increases with LACIE compared to any
baselines; qualitatively, the model often expresses a lack of knowledge (cf.
Table 4). In Appendix C we find that the base model’s accuracy on examples
where LACIE models abstained is substantially lower, i.e. abstention is
correctly correlated with cases where the model does not know the correct
answer. We also see that when considering only non-abstained examples, model
accuracy is generally comparable after LACIE training. Interestingly, the
abstention behavior seen here is emergent, in that it is not seen during
training. All training samples have valid full answer outputs, with no
examples of abstention. Nevertheless, LACIE training leads to a more
conservative model that produces abstaining outputs far more often than the
base model.

###  4.3 Human Evaluation

Table 1 show that finetuning using LACIE leads to models that are better-
calibrated w.r.t. a modeled listener. However, the ultimate test of this
tuning is whether these benefits transfer to real human listeners, rather than
simulated models of human listeners. To test this, we ask human annotators to
accept or reject answers to a subset of TriviaQA questions. To foreshadow the
results, we find that LACIE training reduces false accepts by $47\%$,
improving human precision by $15$ points.

#### Setup.

We use the outputs from the Mistral-7B model, and sample 100 test questions,
pairing each with the LACIE-trained model’s answer as well as the base model’s
answer, resulting in 200 total items. Additional data and annotation details
can be found in Section D.1. For each question-answer item, we ask annotators
to accept or reject the answer. We also ask annotators to what degree they
know the answer to the question. We exclude answers where annotators know the
answer, as their decision to accept or reject here will be based on their
knowledge and not on the confidence expressed by the model. The task is framed
as a trivia game, and answers are presented as coming from a teammate. The
full annotation interface is shown in Section D.3. We recruited annotators on
Amazon’s Mechanical Turk with stringent qualification requirements; we then
filtered annotators based on a qualification task, described in Section D.2.
We also include attention checks in each annotation task and excluded
annotators who failed any attention checks; each task had $\sim 10\%$ of
examples as attention checks. In total, 5 annotators participated in our task.
Annotators knew the answers to 21 of the reference questions and 22 of the
LACIE questions.222These numbers differ because different annotators were
shown the LACIE and reference questions. These were excluded from the
analysis, leaving 79 and 78 total.

####  LACIE improves induced listener calibration for human listeners.

Table 2 shows that finetuning using LACIE transfers well to people, with
precision increasing by $15$ points over the base model. The increase in
precision is driven by a $47\%$ reduction in false positives (statistically
significant at $p<0.05$ by McNemar’s test), i.e. the LACIE model was better
able to express low confidence when its answer was wrong. At the same time,
recall remained roughly the same, indicating that human listeners do not
perceive the LACIE model as underconfident on all samples. Indeed, the LACIE
model only had one additional false negative, a difference that is not
statistically significant ($p=1.0$). This is particularly promising, as it
indicates that the decreases in recall see in Table 1 do not translate to
humans, while the increases in precision do.

##  5 Discussion and Analysis

#### Effect of training on listener probabilities.

Fig. 2 shows the average induced listener probability assigned to correct and
incorrect samples. For baseline speaker models, this probability is
approximately equal whether an example is correct or not, indicating that the
baselines use roughly the same high level of confidence to express both
correct and incorrect answers. For LACIE, the probability for incorrect
answers is substantially lower than for correct ones, indicating the speaker
is modulating confidence correctly to express uncertainty (as captured by the
listener model).

Figure 2: Induced listener probabilities for LACIE-trained and baseline models
(Mistral-7B). Baselines have similar scores for correct and incorrect
examples; LACIE results in significantly lower scores for incorrect answers.

#### Out-of-distribution generalization.

Table 3 shows LACIE’s performance on TruthfulQA’s evaluation split of 817
questions, which is out-of-distribution (since LACIE model was only trained
with preferences sourced only from TriviaQA). TruthfulQA’s evaluation is
model-based, using fine-tuned judge models. As a sanity-check, the truthful-
only baseline improves truthfulness from 0.27 to 0.39, indicating Mistral-7B
trained to be truthful on TriviaQA generally is rated as more truthful by the
judge model. However, LACIE training _further_ improves truthfulness to 0.55,
reflecting the additional benefit of our pragmatic training. This improvement
comes at a $9$-point cost to informativeness, reflecting a natural tradeoff
between being maximally informative and maximally truthful; note that similar
to recall, abstention impacts informativeness, since informativeness is a
recall-oriented metric.333Note that answers like _“I don’t know”_ generally
receive zero score from the informativeness model. These results demonstrate
that LACIE training imbues LLMs with an ability to produce better-calibrated
outputs not only for the data distribution they were trained on, but also on
out-of-domain questions where the evaluation directly addresses truthfulness.

Table 3: TruthfulQA performance using Mistral-7B.  Setting | Truth. | Info.
---|---|---
base | 0.27 | 0.99
truthful | 0.39 | 0.97
LACIE | 0.55 | 0.90

Qualitative Analysis. In Fig. 3, we manually annotate a subsample of 100
examples, coding each for several kinds of implicit and explicit confidence
markers. These include adding details about the topic, other implicit markers
(e.g. a personal backstory), explicit confidence markers (including epistemic
markers), concise answers (no details or markers), hedging, irrelevant
answers, and abstention. We find that after training, the LACIE model adds
more details than the baseline model but only on examples that are correct.
This reflects that confident answers are often supported by additional
details, and corresponds to a lower rate of concise answers. The LACIE model
also makes greater use of hedging, and abstains on several incorrect examples,
whereas the reference model never abstains. Additionally, LACIE leads to an
increased use of explicit markers; note that this does not mean the trained
model is overconfident, as we treat both confident and unconfident markers as
explicit, i.e. _“I’m 20% confident”_ counts as an explicit marker. In Table 4,
we show three qualitative examples from the reference and tuned model.
Qualitatively, we find that the LACIE model shows more examples of hedging and
saying “I don’t know”, as seen in the first example. However, we also note
that, as shown in the second example, LACIE training sometimes results in a
more authoritative tone when the model is correct; this tone might be
perceived as rude. Overall, these qualitative results highlight the benefit of
a method that can incorporate both implicit and explicit cues: LACIE results
in more hedging and abstention, and in a variety of confidence cues that
communicate both certainty _and_ uncertainty.

Figure 3: Frequency of qualitative categories in trained and reference models. LACIE training results in more hedging and abstaining for incorrect examples and more detailed answers for correct ones. Table 4: Qualitative examples of the reference and trained models taken from Mistral-7B. Examples shown demonstrate a range of correctness and confidence, including abstention, overconfidence, and calibrated confidence. Question |  Reference |  LACIE |  Answer
---|---|---|---
_What Michelle Pfeiffer movie got a boost from the Coolio song Gangsta’s Paradise?_ |  Michelle Pfeiffer’s appearance in “I’m Your Woman” was boosted by the Coolio song “Gangsta’s Paradise.” [Incorrect & Confident] |  I think it was a Batman movie. But I’m not too sure.\n and I don’t even know what the movie was about so I don’t have a lot of confidence in my response. [Incorrect & Unconfident] |  _Dangerous Minds_
_In what month was the attack on Pearl Harbor?_ |  I believe the attack on Pearl Harbor occurred in December. I am confident in my answer because I remember reading about the event in school. [Correct & Confident] |  December\n Confidence: 100%\n Rationale: Pearl Harbor. Duh. placeholderplaceholder [Correct & Confident] |  _December_
_Which US No 1 single came from Diana Ross’s platinum album Diana?_ |  Diana’s 1980 album Diana was a huge hit. It spawned three US No 1 singles, including “Upside Down.” The other two singles were “I’m Coming Out” and “My Old Piano.” [Correct & Confident] |  I’m not sure, I’m not familiar with her work. Can you give me more information about the album? placeholderplaceholder [Abstention & Unconfident] |  _Upside Down_

##  6 Conclusion

We introduced a new method, LACIE, for training models to accurately express
confidence, both explicitly and implicitly. Our method makes use of pragmatic,
listener-aware, mulit-agent training to inform a speaker model both about
whether the answer it generated was right _and_ whether the answer was phrased
in a way that a listener would accept it. We demonstrated the effectiveness of
LACIE using automated evaluation as well as a human evaluation, where LACIE
led to a large reduction in the number of incorrect answers accepted by human
annotators. We then qualitatively showed that LACIE leads to increased hedging
on incorrect examples, while increasing the use of confidence markers on
correct examples.

## Limitations and Broader Impacts

#### Limitations.

While we are able to show improvements in model calibration as measured via a
user study, our utility function may not be ideal for different downstream
human-AI team settings. Specifically, in our DPO setup, we define our utility
function to encourage both correctness and calibration in the listener model,
while favoring calibration. Further, for a fair comparison, we do not perform
any supervised finetuning with ground-truth labels, but only construct
preference pairs out of model-generated data. This means that (1) our utility
function may not exactly represent user utility in downstream human-AI team
settings, as our utility ordering of answers may not exactly reflect a true
user rank ordering, and (2) models may be finetuned to achieve higher
accuracies on tasks like TriviaQA and TruthfulQA by leveraging ground-truth
supervision, which we do not do in order to ensure a fair comparison with DPO-
based calibration methods. Our setup shows how to improve implicit and
explicit answer calibration, but a method that is specialized for maximal
performance in a particular setting would need to fully exploit available
label supervision and be tailored to listener utility in that setting.
Similarly, we note that the conservative reward function we develop
(preferring false rejections to false acceptances) may not be optimal for all
applications, such as those where usability is preferred over over safety.
Furthermore, we note that, while our training results in better calibration,
it may not always result in a teammate people would want to work with.
Qualitatively, the confident outputs from the LACIE-trained model often
emphasize how easy or obvious questions are; while this may lead to short-term
benefits in getting correct answers accepted by the listener, users may find
this behaviour annoying or offputting. We leave simultaneously optimizing for
calibration and politeness to future work.

#### Broader Impacts.

Improving model calibration has important implications for model safety and
user satisfaction when LLMs are deployed in settings involving humans
[Hendrycks et al., 2021]. Human overreliance on AI systems is a widely
documented phenomenon [Zhou et al., 2024, Kim et al., 2024]. The most
immediate risk of overreliance is that humans might automatically accept model
outputs even when they are wrong [Cummings, 2004], but more pernicious risks
include humans shirking responsibility for high-stakes decisions as well as
humans gradually losing comptence in key decision-making processes as
seemingly capable AI systems take over the processes [Borg et al., 2024]. To
combat all of these risks, it is crucial that we shape AI systems to more
appropriately interact with humans by training them to calibrate their answers
based on who is on the other end of the conversation. This work aims to
improve calibration of LLM systems by more fully accounting for pragmatic
aspects of text-based human-AI interaction.

## Acknowledgements

This work was supported by DARPA ECOLE Program No. HR00112390060, NSF-AI
Engage Institute DRL-2112635, DARPA Machine Commonsense (MCS) Grant
N66001-19-2-4031, and a Google PhD Fellowship. The views contained in this
article are those of the authors and not of the funding agency.

## References

  * Borg et al. [2024] Jana Schaich Borg, Walter Sinnott-Armstrong, and Vincent Conitzer.  _Moral AI: And How We Get There_.  Random House, 2024.
  * Chow [1957] Chi-Keung Chow.  An optimum character recognition system using decision functions.  In _IRE Transactions on Electronic Computers_ , pages 247–254. IEEE, 1957.
  * Cummings [2004] Mary Cummings.  Automation bias in intelligent time critical decision support systems.  _AIAA 1st Intelligent Systems Technical Conference_ , 2004.  doi: 10.2514/6.2004-6313.  URL https://arc.aiaa.org/doi/abs/10.2514/6.2004-6313.
  * Dettmers et al. [2024] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  QLoRA: Efficient finetuning of quantized LLMs.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Frank and Goodman [2012] Michael C Frank and Noah D Goodman.  Predicting pragmatic reasoning in language games.  _Science_ , 336(6084):998–998, 2012.
  * Fried et al. [2018a] Daniel Fried, Jacob Andreas, and Dan Klein.  Unified pragmatic models for generating and following instructions.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1951–1963, 2018a.
  * Fried et al. [2018b] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell.  Speaker-follower models for vision-and-language navigation.  _Advances in neural information processing systems_ , 31, 2018b.
  * Geifman and El-Yaniv [2017] Yonatan Geifman and Ran El-Yaniv.  Selective classification for deep neural networks.  _Advances in neural information processing systems_ , 30, 2017.
  * Grice [1975] Herbert P Grice.  Logic and Conversation.  In _Speech acts_ , pages 41–58. Brill, 1975.
  * Gude [2023] Vinayaka Gude.  Factors influencing ChatGPT adoption for product research and information retrieval.  _Journal of Computer Information Systems_ , pages 1–10, 2023.  URL https://www.tandfonline.com/doi/abs/10.1080/08874417.2023.2280918.
  * Guo et al. [2017] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  In _International conference on machine learning_ , pages 1321–1330. PMLR, 2017.
  * Hendrycks et al. [2021] Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt.  Unsolved problems in ML safety.  _arXiv preprint arXiv:2109.13916_ , 2021.
  * Joshi et al. [2017] Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer.  Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_ , Vancouver, Canada, July 2017. Association for Computational Linguistics.
  * Kadavath et al. [2022] Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield Dodds, Nova DasSarma, Eli Tran-Johnson, et al.  Language models (mostly) know what they know.  _arXiv preprint arXiv:2207.05221_ , 2022.
  * Kim et al. [2024] Sunnie SY Kim, Q Vera Liao, Mihaela Vorvoreanu, Stephanie Ballard, and Jennifer Wortman Vaughan.  " I’m not sure, but…": Examining the impact of large language models’ uncertainty expression on user reliance and trust.  _arXiv preprint arXiv:2405.00623_ , 2024.
  * Kuhn et al. [2022] Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar.  Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Lazaridou et al. [2020] Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman.  Multi-agent communication meets natural language: Synergies between functional and structural language learning.  _arXiv preprint arXiv:2005.07064_ , 2020.  URL https://arxiv.org/pdf/2005.07064.
  * Li et al. [2024] Xiang Lisa Li, Urvashi Khandelwal, and Kelvin Guu.  Few-shot recalibration of language models.  _arXiv preprint arXiv:2403.18286_ , 2024.
  * Lin et al. [2021] Stephanie Lin, Jacob Hilton, and Owain Evans.  TruthfulQA: Measuring how models mimic human falsehoods.  _arXiv preprint arXiv:2109.07958_ , 2021.  URL https://arxiv.org/pdf/2109.07958.
  * Lin et al. [2022] Stephanie Lin, Jacob Hilton, and Owain Evans.  Teaching models to express their uncertainty in words.  _Transactions on Machine Learning Research_ , 2022.
  * Mielke et al. [2022] Sabrina J Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau.  Reducing conversational agents’ overconfidence through linguistic calibration.  _Transactions of the Association for Computational Linguistics_ , 10:857–872, 2022.
  * Naeini et al. [2015] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using Bayesian binning.  In _Twenty-Ninth AAAI Conference on Artificial Intelligence_ , 2015.
  * Ovadia et al. [2019] Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek.  Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift.  _Advances in neural information processing systems_ , 32, 2019.
  * Panickssery et al. [2024] Arjun Panickssery, Samuel R Bowman, and Shi Feng.  LLM evaluators recognize and favor their own generations.  _arXiv preprint arXiv:2404.13076_ , 2024.
  * Rafailov et al. [2024] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Rawte et al. [2023] Vipula Rawte, Amit Sheth, and Amitava Das.  A survey of hallucination in large foundation models.  _arXiv preprint arXiv:2309.05922_ , 2023.  URL https://arxiv.org/pdf/2309.05922.
  * Ren et al. [2023] Allen Z Ren, Anushri Dixit, Alexandra Bodrova, Sumeet Singh, Stephen Tu, Noah Brown, Peng Xu, Leila Takayama, Fei Xia, Jake Varley, et al.  Robots that ask for help: Uncertainty alignment for large language model planners.  In _Conference on Robot Learning_ , pages 661–682. PMLR, 2023.
  * Stengel-Eskin and Van Durme [2023] Elias Stengel-Eskin and Benjamin Van Durme.  Calibrated interpretation: Confidence estimation in semantic parsing.  _Transactions of the Association for Computational Linguistics_ , 11:1213–1231, 2023.
  * Tian et al. [2023] Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D Manning.  Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 5433–5442, 2023.
  * Ulmer et al. [2024] Dennis Ulmer, Martin Gubri, Hwaran Lee, Sangdoo Yun, and Seong Joon Oh.  Calibrating large language models using their generations only.  _arXiv preprint arXiv:2403.05973_ , 2024.
  * Vaduguru et al. [2023] Saujas Vaduguru, Daniel Fried, and Yewen Pu.  Generating pragmatic examples to train neural program synthesizers.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Wang et al. [2021] Rose Wang, Julia White, Jesse Mu, and Noah Goodman.  Calibrate your listeners! Robust communication-based training for pragmatic speakers.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 977–984, 2021.
  * Wang et al. [2020] Shuo Wang, Zhaopeng Tu, Shuming Shi, and Yang Liu.  On the inference calibration of neural machine translation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3070–3079, 2020.
  * Wolf et al. [2020] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_ , pages 38–45, 2020.
  * Zhang et al. [2023] Hanning Zhang, Shizhe Diao, Yong Lin, Yi R Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang.  R-tuning: Teaching large language models to refuse unknown questions.  _arXiv preprint arXiv:2311.09677_ , 2023.
  * Zhou et al. [2023] Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto.  Navigating the grey area: How expressions of uncertainty and overconfidence affect language models.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 5506–5524, Singapore, December 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.335.  URL https://aclanthology.org/2023.emnlp-main.335.
  * Zhou et al. [2024] Kaitlyn Zhou, Jena D Hwang, Xiang Ren, and Maarten Sap.  Relying on the unreliable: The impact of language models’ reluctance to express uncertainty.  _arXiv preprint arXiv:2401.06730_ , 2024.

##  Appendix A Methodological Details

#### Dataset Details.

The preference function in Eq. 1 encodes multiple preferences falling into
five categories: $U(\lnot C,\lnot A)>U(\lnot C,A)$, $U(C,A)>U(\lnot C,A)$,
$U(C,A)>U(C,\lnot A)$, $U(C,\lnot A)>U(\lnot C,A)$, $U(\lnot C,\lnot
A)>U(C,\lnot A)$, where $C$ means $\hat{D}^{i}_{j}=1$ and $A$ means
$D^{i}_{j}=1$. We balance our data across these categories by limiting all
categories to the size of the least-represented one ($2,757$). This results in
$13,785$ total preference pairs.

#### Training Details.

We train our models using QLoRA [Dettmers et al., 2024] with rank 16 for 250
steps. We use a batch size of 12 and gradient accumulation, with updates every
10 batches. Every 10 updates, we validate; we subsample 5 validation batches
to decode and measure listener precision and recall, which we use for
checkpoint selection.

#### Answer Anonymization.

To keep the listener from relying too much on prior knowledge, we remove the
extracted answer from the generated output using exact match. Qualitatively,
we found that without this removal, the listener generally assigned a high
probability of acceptance to answers that it itself generated, i.e. when the
speaker and listener model were the same, the listener model accepted most
outputs. By removing the exact answer string, we can mitigate this problem.

#### TruthfulQA Evaluation

TruthfulQA’s original evaluation used OpenAI’s Curie model, which is no longer
accessible. For maximum reproducibility, we use open-source variants of these
judge models from https://huggingface.co/allenai/truthfulqa-info-judge-
llama2-7B. These models are comparable to the original TruthfulQA judge
models.

##  Appendix B Scaling

A key question is how LACIE performance changes with increasing data. In Fig.
4, we plot listener precision (Mistral-7B) as we increase the number of
training questions (from which we construct our outputs and preference pairs)
from $2,000$ to $10,000$. Generally, performance increases with more data,
with the best precision coming at $10,000$ examples. Fig. 4 shows tapering
towards the upper range of data, indicating that $10,000$ examples is likely
sufficient.

Figure 4: Precision and AUROC as the size of the training data increases.
LACIE generally continues improving with more data.

##  Appendix C Accuracy results

#### Accuracy and Abstention.

Table 6 shows that the overall accuracy of the model decreases with training.
However, further inspection reveals that this drop in accuracy is largely
driven by the model producing a larger rate of responses with no extractible
answer after training, e.g. responses like _“I don’t know”_. We refer to these
responses as “abstentions”, borrowing the term from selective prediction
[Chow, 1957, Geifman and El-Yaniv, 2017]. Rather than make a prediction on
every test instance, selective prediction aims to develop models which make
predictions only in cases where they are likely to be correct. This relates
closely to our setting, in which we seek to align the model’s verbal
confidence with its likelihood of being correct. One way this can be
manifested is via selectively producing no output, i.e. if the output is very
uncertain, the model should not produce an answer at all (rather than produce
one that risks misleading the listener). The “Predicted Data” accuracy in
Table 6 shows the accuracy of each model only when considering the examples
for which the model made a prediction,

Table 5: Speaker accuracy for all data and for the subset of data for which
the model made a prediction (i.e. did not abstain). LACIE’s predicted accuracy
is generally comparable to the baselines. The drop in accuracy overall can be
attributed to the increase in abstention.

| Speaker Accuracy
---|---
Speaker Model | All Data | Predicted Data
_Mistral-7B_ |  |
+Base | $0.55_{\pm\text{ 0.00}}$ | $0.59_{\pm\text{ 0.01}}$
+Chat | $0.43_{\pm\text{ 0.00}}$ | $0.50_{\pm\text{ 0.01}}$
+Truthful | $0.56_{\pm\text{ 0.03}}$ | $0.61_{\pm\text{ 0.01}}$
+LACIE | $0.45_{\pm\text{ 0.01}}$ | $0.60_{\pm\text{ 0.01}}$
_Llama3-8B_ |  |
+Base | $0.49_{\pm\text{ 0.00}}$ | $0.56_{\pm\text{ 0.01}}$
+Chat | $0.62_{\pm\text{ 0.00}}$ | $0.64_{\pm\text{ 0.00}}$
+Truthful | $0.58_{\pm\text{ 0.03}}$ | $0.64_{\pm\text{ 0.02}}$
+LACIE | $0.39_{\pm\text{ 0.04}}$ | $0.61_{\pm\text{ 0.02}}$
_Llama3-70B_ |  |
+Base | $0.58_{\pm\text{ 0.06}}$ | $0.66_{\pm\text{ 0.04}}$
+Chat | $0.74_{\pm\text{ 0.01}}$ | $0.76_{\pm\text{ 0.01}}$
+Truthful | $0.68_{\pm\text{ 0.03}}$ | $0.78_{\pm\text{ 0.00}}$
+LACIE | $0.46_{\pm\text{ 0.02}}$ | $0.69_{\pm\text{ 0.03}}$

Table 6: Base model accuracy on examples where the trained model abstained vs.
predicted. Across models, astained examples are far more likely to be wrong.

Model | Acc (abstain) | Acc (predict)
---|---|---
Mistral-7B | $33.4$ | $61.6$
Llama3-8B | $35.7$ | $55.7$
Llama3-70B | $46.9$ | $63.6$

while the % Absention column in Table 1 shows the rate at which the model
abstains. In general, the base model rarely abstains; with truthfulness-only
training, this rate increases by an average of $3\%$. Finally, with pragmatic
training, the abstention rate increases to an average of $26.9\%$. Crucially,
we see that, while the overall accuracy decreases dramatically, the accuracy
on the non-abstained examples generally only decreases slightly. In other
words, when the LACIE-trained model does make a prediction, it is roughly as
likely to be correct as the base model; this can be attributed to the fact
that we train on model-generated data, meaning that the trained model has
roughly the same underlying knowledge as the base model. Our training does not
remove that knowledge, but rather primarily changes how it is expressed. Note
that some reduction in overall accuracy is always expected in selective
prediction, as the model cannot possibly increase accuracy by abstaining, and
it is rewarded for guessing on every example when considering only accuracy.

In Table 6 we show the performance of the base model on the subset of examples
for which the LACIE model abstained or made a prediction. Here, we see that
the base model accuracy is far lower on the abstained examples than the
predicted ones; in other words, LACIE training informs the model about which
examples are likely to succeed and which are not, with the unlikely examples
leading to more abstention.

##  Appendix D HIT Details

###  D.1 Data

We sample 100 test questions from TriviaQA, pairing each with two answers (one
from the base model, one from the LACIE-trained model). 50 of the questions
were answered correctly by the base model, and 50 incorrectly. We use
stratified sampling based on the listener model, binning the listener’s
$P^{j}_{i}$ scores into 5 bins and sampling 20 examples from each bin. In the
annotation task, we shuffle items and then group them into batches of 20, so
that each annotator sees at least 20 items; we add 2 attention checks to each
batch. These are examples like those used in the pilot, where the decision to
accept or reject should be fairly obvious. Any batch with failed attention
checks is discarded, and annotators who failed any attention checks were
removed from the annotation pool.

###  D.2 Annotators and Pilot HIT

We recruited annotators on Amazon’s Mechanical Turk, restricting to annotators
with approval rates over $98\%$ and with the Master qualification who are
located in the US. We then conducted a further pilot HIT to qualify
annotators; we created four examples with fairly obvious responses; 2 examples
were questions annotators would know (e.g. _“What is the capital of the
USA?”_) and two were extremely obscure information. Furthermore, two were
confident and two were clearly unconfident. This allowed us to ensure that
annotators understood which samples to accept and reject and how to indicate
their knowledge. Only annotators who scored 100% on the pilot were allowed to
participate in the main HIT. Annotator pay was $\$1.66$ per batch of 20,
estimated to be $\$15$ per hour.

Figure 5: Annotation instructions.

###  D.3 Template and Instructions

Our interface asks four questions. The instructions can be seen in Fig. 5, and
the full interface can be seen in Fig. 6. First, the annotator chooses whether
to accept or reject the answer; they then provide a confidence rating
indicating how confident they are in their decision to accept or reject. Next,
they are asked to indicate to what degree they know the answer; this is
included so that we can exclude examples for which annotators already know the
answer, as their decision to accept or reject here will be based on their
knowledge and not on the confidence expressed by the model. Finally, we ask
the annotators how convincing their interlocutor was. Annotators are
instructed not to look up the answers to any of the questions, and that their
reward is not contingent on their team performance; they are also explicitly
instructed that false rejections are better than false accepts.

##  Appendix E Computational Resource Details

Our training generally requires two processes: one to host the speaker, and
one to host the listener. All 7B models are run on 2 GPUs (NVIDIA A6000s with
48Gb memory and L40s with 40Gb memory). These resources are sufficient for
both inference and training with QLoRA. The 70B model requires 4 GPUs (A6000
or L40) running in parallel, with both the speaker and listener model
parallelized across GPUs.

All models are accessed via Huggingface’s Transformers library [Wolf et al.,
2020]. For LoRA training and quantization, we use bitsandbytes and accelerate.

##  Appendix F License

We make our code and models publicly accessible. We use an Apache license 2.0
license and include the following links to the licenses for datasets, code,
and models used in this paper. For further information, please refer to the
links below.

PyTorch: BSD-style

Huggingface Transformers: Apache 2.0

Huggingface Accelerate: Apache 2.0

bitsandbytes: MIT

TriviaQA: Apache 2.0

TruthfulQA: Apache 2.0

Figure 6: Annotation interface

##  Appendix G Prompts

The prompts are shown in Appendix G, Appendix G, and Appendix G.

Table 7: QA prompt used across models, both to generate preference data and to
prompt models at test-time.

⬇ You will be asked trivia questions. Please respond to the best of your
ability. Your response should reflect how confident you are in your answer,
and why you believe your answer is right. Your response should be more than a
single word, but limited to 1-2 sentences. Question: {question} Response:
---

Table 8: Listener prompt with two manually-constructed examples.

⬇ Pretend you know nothing about the world. Based only on how the answer is
phrased, would you accept this final answer? If the answer sounds confident,
you should accept it. Otherwise, you should reject it. Don’t consider whether
you think the answer is right or not, focus only on how it is phrased. The
answer will be obscured, so that you make your decision only on the tone of
the answer. Answer just "yes" or "no". Examples: Question: Who wrote Paradise
Lost? Response: I’m 100\% sure that [ANSWER REMOVED] wrote Paradise Lost. Do
you accept the answer? Response: yes Question: Who wrote Paradise Lost?
Response: I have no idea but I will randomly guess [ANSWER REMOVED]. Do you
accept the answer? Response: no Question: {question} Response: {response}
Final answer: {final_answer} Do you accept the answer? Response:
---

Table 9: Answer extraction prompt with 3 manually-constructed examples ⬇
Please extract a single answer from the following response to a question. If
no answer is present, please write "NONE". Question: Who wrote Paradise Lost?
Response: The author of Paradise Lost was John Milton, who published the book
in 1667. Final answer: John Milton Question: Which colonial power did Algeria
gain independence from in 1962? Response: Algeria gained independence from
France in 1962 after years of bloody conflict. Final answer: France Question:
How many planets are in our solar system? Response: Please respond to the
survey link below: https://www.surveymonkey.com/r/5VZ7Z6P Final answer: NONE
Question: {question} Response: {response} Final answer:
---


# A Multi-Modal Simulation Framework to Enable
Digital Twin-based V2X Communications
in Dynamic Environments

Lorenzo Cazzella, Francesco Linsalata, Maurizio Magarini,
Matteo Matteucci, and Umberto Spagnolini

###### Abstract

Digital Twins (DTs) for physical wireless environments have been recently
proposed as accurate virtual representations of the propagation environment
that can enable multi-layer decisions at the physical communication equipment.
At high-frequency bands, DTs can help to overcome the challenges emerging in
high mobility conditions featuring vehicular environments. In this paper, we
propose a novel data-driven workflow for the creation of the DT of a Vehicle-
to-Everything (V2X) communication scenario and a multi-modal simulation
framework for the generation of realistic sensor data and accurate mmWave/sub-
THz wireless channels. The proposed method leverages an automotive simulation
and testing framework based on the Unreal Engine game engine and an accurate
ray-tracing channel simulator. Simulations over an urban scenario show the
achievable realistic sensor and channel modelling both at the infrastructure
and at ego-vehicles.

###### Index Terms:

Digital Twin, V2X, data-driven, mmWave/sub-THz

##  I Introduction

The sixth generation (6G) technology targets to revolutionize the mobility
industry enhancing the role of wireless connections in car manufacturing.
Indeed, connected vehicles are based on the vehicle-to-everything (V2X)
technology to communicate with other road users employing future 6G networks.
The communication module of a connected vehicle working at high frequencies—in
the mmWave and sub-THz bands—can offer a true Gb/s experience and deliver
information more efficiently than through sensor detection and processing [1].

Figure 1: Interaction between a real-world V2X network and its high-fidelity
digitized representation. The urban environment and the vehicles’ movements
govern wireless propagation in the real world. The digital replica uses high-
definition reconstructed 3D maps, realistic sensor simulations, and accurate
3D ray tracing to control the communication equipment.

However, the use of higher frequency bands makes the wireless links prone to
line-of-sight (LOS) blockages. The vehicles’ movement and the rising density
of urban buildings, which mostly determine the signal degradation, are the
primary causes of discontinuous connectivity in V2X environments [2]. As a
possible countermeasure, the latest 6G paradigm envisions merging the physical
and digital worlds [3], and it is widely acknowledged that the Digital Twin
(DT) will be a key enabling technology to achieve this goal [4].

The DT aims at providing a high-fidelity digital representation of physical
phenomena. This is obtained not only by integrating simulations and available
data but also by accounting for the entire phenomenon life-cycle, which
provides up-to-date insights about the physical entity. Thus, a DT-enabled V2X
network can get meaningful knowledge of the surrounding environment by
collecting and processing a huge amount of data, and providing real-time
accurate channel estimation [4]. To create such an accurate real-time digital
reproduction of the physical environment, the envisioned DT has to use high-
definition 3D maps and combine multi-modal sensory data from scattered devices
and infrastructure nodes. The DT needs to be continuously improved to enhance
its decision accuracy and to refine its approximation of the physical world,
as depicted in Fig. 1. Indeed, the demands for data reliability are stricter
in dynamic scenarios than in static ones. Particularly, in scenarios with
stringent time requirements and severe tolerance constraints, as for V2X
networks where even a small model inaccuracy can have a remarkable impact on
any decision process.

Real-time DTs have been researched in a variety of fields, including real-time
remote monitoring and control in industry, risk assessment in transportation,
and smart scheduling in smart cities. The use of DT in the network has mostly
concentrated on operation issues for wireless communications, including edge
computing, network optimization, and service management [3].

The availability of multi-modal data for communications-related decisions
(e.g., beam prediction and localization) can greatly reduce the required
decision time where an exhaustive search over all the candidates would be
required by the standard [5]. To foster the application of multi-modal
datasets to decision-making in communications, Gu et al. present in [5] an
experimentally obtained dataset aiding beam selection in V2X mmWave bands. By
contrast, our paper focuses on simulating the real vehicular world using a
real-time DT, with attention to the environment’s physical modelling and
wireless signal propagation. The main contributions of our paper can be
summarized as it follows:

  * wide

a workflow for the creation of the DT of a vehicular urban/suburban
environment from multi-sensor data acquired at the communication
infrastructure or ego-vehicles;

  * wide

a data-driven multi-modal simulation framework for V2X wireless communications
leveraging a realistic automotive 3D simulation and testing software based on
the renowned Unreal Engine game engine for the generation of realistic sensor
data, and on an accurate ray-tracing wireless channel simulator. Finally, we
present simulation results over an urban environment showing the sensing and
channel generation realism achievable through the proposed framework.

##  II Digital Twin-aided Communications

(a)

(b)

Figure 2: Geometric variation of the most powerful ray-tracing channel paths
between an RSU and a VE operating at 28 GHz in an urban environment when
affected (blue) or not affected (orange) by vehicular blockage from a truck in
terms of (a) geometrical configuration of the main paths in the environment,
and (b) directions of arrival and delays of the simulated channel paths.

The V2X umbrella includes the two main subcategories of Vehicle-to-Vehicle
(V2V) and Vehicle-to-Infrastructure (V2I) communications. V2V, or sidelink,
refers to direct communication between vehicles, without any support from the
network infrastructure, while V2I includes communication between vehicles and
road infrastructure, e.g., road side units (RSUs). Narrow-beam and high
frequencies communications are foreseen as candidate solutions to high
spectral and energy efficiency KPIs of future 6G V2X networks [5].

We consider a multi-user V2X communication system in which the communication
infrastructure and the vehicular user equipments (VEs) are capable of multi-
modal sensing, i.e., they are equipped with multiple sensors (e.g., camera,
LiDAR, radar) to perceive their surroundings. VEs can be also equipped with
sensors to estimate their state—e.g., positioning systems and inertial
measurement units (IMUs),—in terms of acting forces, position, and
orientation.

Owing to highly dynamic environments, V2X communications experience rapidly
varying communication link conditions, which can greatly affect communication
performance and call for novel channel estimation techniques. As an example,
Fig. 2 shows the deep change in electromagnetic wireless propagation between a
road-side unit and a VE when affected by the vehicular blockage phenomenon.
The differences in terms of angles and delays of the propagation paths clearly
show the variation in the spatial and temporal propagation channel patterns in
the blocked and non-blocked conditions. As a result, dynamic environment
conditions affect wireless propagation in vehicular settings, severely
changing the propagation structure between the transmitter and the receiver
for both V2I and V2V communications. The latter scenario is the most
challenging, since both the communication nodes are dynamic in this case.

Recent contributions in the literature, e.g., [6, 7], have emphasized the
advantages of integrated communication and multi-modal sensing to provide
effective solutions over a wide variety of tasks, including beam prediction,
channel estimation, blockage prediction/hand-off, VE positioning, and object
detection. We believe that the availability of multiple sensors at the
communication infrastructure and/or at VEs, in challenging channel estimation
conditions, can benefit from the automotive simulation and testing framework.
The latter are commonly used for the validation of Autonomous Vehicles (AV)
and Advanced Driver Assistance Systems (ADAS) and enable the construction of
the DT of vehicular scenarios aimed at controlling physical communication
equipment based on virtual sensors simulation and accurate channel generation.

##  III A multi-modal simulation framework for V2X communications

Figure 3: DT construction workflow. Acquisitions from multiple sensors at the
vehicle or the communication infrastructure are preprocessed by sensor fusion
and semantic instance segmentation to determine the components of the 3D scene
and their materials. Procedural digital reconstruction is performed and
utilized for multi-modal sensor simulation within an Unreal Engine-based
automotive framework and for accurate wireless propagation simulation through
a ray-tracing simulator.

In this section, we propose a workflow for the construction of the DT of a
physical urban/suburban environment and a multi-modal simulation framework to
enable wireless communications. Figure 3 depicts the proposed workflow. The
procedure is intended as an iterative approach, which constantly updates the
DT from the acquired data.

###  III-A Digital Twin construction

In the following, we briefly describe the sensory data acquisition,
preprocessing and digital reconstruction steps, focusing on multi-sensor
simulation and ray-tracing channel generation in the reconstructed 3D
environment, which are the target of this contribution.

Sensory data acquisition (Figs. 3a and 3b): the environment surrounding a
sensing-capable network component (a VE or at the communication
infrastructure) is perceived employing multiple sensors, e.g., camera, LiDAR,
and radar. Positioning and inertial measurement systems are also common on
automated vehicles, providing information on the forces acting on the vehicle,
its position, and its orientation.

Preprocessing of the acquired data (Figs. 3c and 3d): the acquired data is
preprocessed through sensor fusion techniques to leverage the joint
availability of multiple sensor streams. Semantic instance segmentation is
then performed to determine the 3D scene components—e.g., buildings, vehicles,
fences, lamp posts, road marks, and vegetation. An effective method for LiDAR
3D point-cloud semantic segmentation is proposed by Zhuang et al. in [8],
where the authors exploit perceptual information from RGB images and spatial-
depth information from point clouds to devise a collaborative fusion scheme.

Digital 3D reconstruction (Fig. 3e): the identified 3D scene components are
mapped to known 3D object models by means of procedural 3D reconstruction.
Procedural modelling for urban environments leverages the patterns of
architectural designs to synthesize realistic 3D models. Shape grammars [9]
have been proposed to effectively reconstruct real-environment models. In
shape grammars, prior knowledge of the architectural design principles is
described utilizing formal grammars. Shape grammars have been successfully
integrated with data-driven processes to effectively reconstruct city models
from sensor data. Nishida et al. propose in [10] a system to generate a
procedural model of a building from a single camera image, automatically
estimating camera parameters and generating the building windows and doors
geometry. Smelik et al. provide in [11] a more general survey on accurate
procedural modelling of virtual twins, focusing on the degree of interactivity
and control offered by the described methods.

Materials matching (Fig. 3f): a material is associated to each 3D component
class to enable the reconstructed 3D scenario for electromagnetic (EM)
propagation simulations. We decompose a 3D vehicle models into bodies,
windows, bumpers, wheel rims, and tires. Shape components for the 3D models
within the reconstructed scenario can be derived from procedural modelling
grammar. For a thorough description of the EM properties of several materials,
we refer the reader to the ITU-R recommendation P.1238 [12].

Multi-modal simulation (Fig. 3g): the reconstructed 3D environment is
integrated within the Midgard automotive software framework, designed by
AnteMotion for testing, simulating and validating Autonomous Vehicles (AV) and
Advanced Driver Assistance Systems (ADAS). In particular, Midgard provides
state-of-the-art realistic urban/suburban 3D simulation over different weather
conditions relying on the Unreal Engine [13] real-time game engine. Midgard
allows the simulation or integration of a wide variety of sensors—both static
(for simulations at the communications infrastructure) and dynamic (moving
jointly with the simulated vehicles):

  * •

RGB camera, provides standard coloured images and is defined for both a
pinhole camera model and for real-life ultra wide-angle lenses (fisheye
camera);

  * •

identification/segmentation cameras, identify objects perceived in the camera
image, classifying each pixel based on the object type (semantic
segmentation), separating pixels belonging to different observed objects
(instance segmentation), or providing the bounding boxes associated to each
object (multi-object identification);

  * •

optical flow camera, allows representing the object movements associating to
each image pixel its vertical and horizontal velocity and encoding them into
RGB channels;

  * •

LiDAR, provides a 3D point cloud for a given number of channels (stacked
vertical lasers) and a specified horizontal angular resolution;

  * •

inertial measurement unit, provides information on the dynamic state of the
vehicle to which it is attached;

  * •

mmWave radar, simulates radar sensing through specialized ray-tracing
software, e.g., Remcom WaveFarer.

Accurate wireless channel simulation (Fig. 3h): to achieve an accurate
simulation of the wireless propagation channel, we utilized the Remcom
Wireless InSite specialized ray-tracing simulation software. Alternatively,
Sionna RT, a real-time ray-tracing channel simulation software leveraging GPU
hardware acceleration, has also been recently introduced. Ray-tracing
propagation models have been proven to effectively simulate multi-path
propagation and space-time channel dispersion. The considerable progress in
computing architectures and the formulation of improved computational methods
enable real-time performance for multi-sensor simulations and ray-tracing
wireless propagation simulation.

The reconstructed 3D model of the urban/suburban environment is imported
within Wireless InSite, associating the corresponding material to each
environment component. The position and orientation of moving vehicles are
modified frame-by-frame, jointly with the automotive multi-sensor simulation.
Wireless InSite provides GPU acceleration and multi-threading and allows 3D
ray tracing considering different types of interaction with the
environment—i.e., reflections, diffraction, scattering, and transmission. For
each propagation ray between transmitter and receiver, the information
provided by Wireless InSite includes the punctual details on received power,
phase, direction of departure (DoD), direction of arrival (DoA), delay and
Doppler shift (for moving Tx or Rx equipment).

###  III-B Datasets generation

Besides controlling physical equipment, the proposed simulation framework
enables the generation of realistic datasets of synchronized multi-sensor
data. Accurate communication channels at the infrastructure and at the ego-
vehicles are the basis for realistic benchmarks definition and machine/deep
learning algorithms training.

A vehicular traffic simulator—e.g, the Simulation of Urban MObility (SUMO)
open-source software—can be utilized to generate realistic vehicular traffic
data. The use of a vehicular traffic simulator requires the availability of an
accurate map of the road network. To this aim, we propose to extend the
preprocessing phase proposed in Fig. 3 with a further step for road mark
identification and HD lanes reconstruction. A multi-stage pipeline relying on
clothoidal spline models is proposed in [14] for the definition of lane-level
high-definition maps. We endowed the SUMO vehicle motion model with a
kinematic bicycle model for the estimation of the front wheels’ rotation.

We integrated the SUMO simulator with the Midgard automotive simulation
framework and with Remcom Wireless InSite to achieve dynamic multi-sensor and
ray-tracing channel simulations. The generation of vehicular traffic and
wireless channel data can be integrated with network-layer simulations through
event-based network simulators, like OMNeT++ and NS-3. 5G New Radio-enabled
vehicular networks can be simulated, e.g., using the 5G-LENA NS-3 module or
through the Simu5G OMNeT++-based simulator.

We notice that procedural 3D generation allows building a wide variety of
combinations of 3D and environmental features (e.g., building 3D structures,
textures, weather conditions) to generate several diversified datasets.
Machine learning methods for decision-making in vehicular networks can benefit
from the availability of massive diversified datasets that faithfully
represent the environment dynamics to solve common vehicular communications
tasks—e.g., initial access and beam prediction—at different layers in the
communication stack [15]. The multitude of procedural 3D scenarios can be
utilized to validate communication protocols on heterogeneous vehicular
networks and diverse configurations of the wireless propagation environment.

##  IV Digital Twin simulation

Figure 4: Simulation of RGB camera, LiDAR and semantic segmentation camera
sensors and ray-tracing channel at a simulation frame for the selected urban
scenario both at the communication infrastructure and at an ego-vehicle.

In this section, we demonstrate the proposed multi-modal simulation framework
over an urban vehicular scenario. We assume that the sensory data acquisition,
preprocessing and digital reconstruction procedures have been performed, and
the 3D models of vehicles and simulation scenario are available.

The first three columns of Fig. 4 present the simulation results for three
different virtual sensors—i.e., RGB camera (pinhole camera model), LiDAR unit
and semantic segmentation camera—in Midgard from the point of view of an RSU
and of a vehicle moving in the scenario in the neighbourhood of the RSU. The
camera resolution has been set up to $1920\times 1080$ pixels, while the focal
length is 16 mm and the camera sensor size is 36 mm x 20.25 mm. The LiDAR unit
emulates a realistic radar with 64 channels, a rotation frequency of 10 Hz and
a horizontal resolution of 0.6 deg, providing a 3D point cloud of the RSU and
vehicle surroundings. The semantic segmentation camera annotates each RGB
camera pixel with a different color for each object type, distinguishing
vehicles, buildings, curbs, pavements, fences and vegetation. The figure shows
the real-time realism and accuracy reached by the Midgard Unreal Engine-based
automotive framework during simulation.

The generation of the ray-tracing channel through Remcom Wireless InSite is
presented in the last column of Fig. 4 for both a transmitter at the
infrastructure and at an ego-vehicle. The simulated RSU and VEs operate at 28
GHz carrier frequency. The figure shows the most powerful generated rays. Ray-
tracing has been performed considering isotropic Tx and Rx antennas, and a
maximum of 7 reflections, 6 diffractions and 1 scattering interactions between
transmitter and receiver. The simulation of MIMO channels over antenna arrays
can be achieved by applying the array response vectors or precoding/combining
schemes to the simulated propagation paths according to the communication
system architecture.

During the materials matching phase, concrete has been selected for buildings
structures, glass for vehicles and building windows, wood for fences, perfect
electric conductor (PEC) for lamp posts, vehicle bodies, vehicle bumpers and
wheel rims, and rubber for vehicle tires.

##  V Conclusion

In this paper, we proposed a novel workflow and a multi-modal simulation
framework for the construction of a realistic DT of a physical scenario from
multi-sensor acquisitions in dynamic vehicular environments. The developed DT
is aimed at improving existing V2X communications by controlling physical
equipment based on accurate virtual sensors and channel data. Moreover, the
proposed multi-modal simulation framework allows the generation of realistic
multi-sensor and ray-tracing channel datasets, enabling the definition of
realistic benchmarks and the training of machine/deep learning algorithms.
Simulations over an urban vehicular environment show the sensing realism
achievable through the Unreal Engine-based automotive simulation framework and
ray-tracing channel generation results at the communication infrastructure and
at an ego-vehicle.

##  VI Acknowledgment

This work was partially supported by the European Union under the Italian
National Recovery and Resilience Plan (NRRP) of NextGenerationEU, partnership
on “Telecommunications of the Future” (PE00000001 - program “RESTART”,
Structural Project 6GWINET) and by the Joint Lab between Huawei Milan Research
Center and Politecnico di Milano.

## References

  * [1] X. Liao, Z. Wang, X. Zhao, K. Han, P. Tiwari, M. J. Barth, and G. Wu, “Cooperative ramp merging design and field implementation: A digital twin approach based on vehicle-to-cloud communication,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 5, pp. 4490–4500, 2021.
  * [2] F. Linsalata, S. Mura, M. Mizmizi, M. Magarini, P. Wang, M. N. Khormuji, A. Perotti, and U. Spagnolini, “LoS-Map Construction for Proactive Relay of Opportunity Selection in 6G V2X Systems,” _IEEE Transactions on Vehicular Technology_ , pp. 1–15, 2022.
  * [3] L. U. Khan, W. Saad, D. Niyato, Z. Han, and C. S. Hong, “Digital-Twin-Enabled 6G: Vision, Architectural Trends, and Future Directions,” _IEEE Communications Magazine_ , vol. 60, no. 1, pp. 74–80, 2022.
  * [4] C. Ding and I. W.-H. Ho, “Digital-Twin-enabled City-model-aware Deep Learning for Dynamic Channel Estimation in Urban Vehicular Environments,” _IEEE Transactions on Green Communications and Networking_ , vol. 6, no. 3, pp. 1604–1612, 2022.
  * [5] J. Gu, B. Salehi, D. Roy, and K. R. Chowdhury, “Multimodality in mmWave MIMO Beam Selection Using Deep Learning: Datasets and Challenges,” _IEEE Communications Magazine_ , vol. 60, no. 11, pp. 36–41, 2022.
  * [6] S. Jiang, G. Charan, and A. Alkhateeb, “Lidar aided future beam prediction in real-world millimeter wave V2I communications,” _IEEE Wireless Communications Letters_ , 2022.
  * [7] W. Xu, F. Gao, X. Tao, J. Zhang, and A. Alkhateeb, “Computer Vision Aided mmWave Beam Alignment in V2X Communications,” _IEEE Transactions on Wireless Communications_ , 2022.
  * [8] Z. Zhuang, R. Li, K. Jia, Q. Wang, Y. Li, and M. Tan, “Perception-aware multi-sensor fusion for 3d lidar semantic segmentation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 16 280–16 290.
  * [9] G. Stiny and J. Gips, “Shape grammars and the generative specification of painting and sculpture.” in _IFIP congress (2)_ , vol. 2, no. 3. Citeseer, 1971, pp. 125–135.
  * [10] G. Nishida, A. Bousseau, and D. G. Aliaga, “Procedural modeling of a building from a single image,” in _Computer Graphics Forum_ , vol. 37, no. 2. Wiley Online Library, 2018, pp. 415–429.
  * [11] R. M. Smelik, T. Tutenel, R. Bidarra, and B. Benes, “A survey on procedural modelling for virtual worlds,” in _Computer Graphics Forum_ , vol. 33, no. 6. Wiley Online Library, 2014, pp. 31–50.
  * [12] “ITU-R Recommendation P.1238-11,” www.itu.int/rec/R-REC-P.1238-11-202109-I/en, accessed on 17 March 2023.
  * [13] “Unreal Engine,” www.unrealengine.com, accessed on 17 March 2023.
  * [14] P. Cudrano, B. Gallazzi, M. Frosi, S. Mentasti, and M. Matteucci, “Clothoid-based lane-level high-definition maps: Unifying sensing and control models,” _IEEE Vehicular Technology Magazine_ , vol. 17, no. 4, pp. 47–56, 2022.
  * [15] H. Ye, L. Liang, G. Y. Li, J. Kim, L. Lu, and M. Wu, “Machine learning for vehicular networks: Recent advances and application examples,” _IEEE Vehicular Technology Magazine_ , vol. 13, no. 2, pp. 94–101, 2018.