# A General Calibrated Regret Metric for Detecting and Mitigating Human-Robot
Interaction Failures

Kensuke Nakamura1 Ran Tian2 Andrea Bajcsy1
1Carnegie Mellon University 2 UC Berkeley
{kensuken<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Robot decision-making increasingly relies on expressive data-driven human
prediction models when operating around people. While these models are known
to suffer from prediction errors in out-of-distribution interactions, not all
prediction errors equally impact downstream robot performance. We identify
that the mathematical notion of regret precisely characterizes the degree to
which incorrect predictions of future interaction outcomes degraded closed-
loop robot performance. However, canonical regret measures can be poorly
calibrated across diverse deployment interactions. We derive a calibrated
regret metric that evaluates the quality of robot decisions in probability
space rather than reward space. With this transformation, our metric removes
the need for explicit reward functions to calculate the robot’s regret,
enables fairer comparison of interaction anomalies across disparate deployment
contexts, and facilitates targeted dataset construction of “system-level”
prediction failures. We experimentally quantify the value of this high-regret
interaction data for aiding the robot in improving its downstream decision-
making. In a suite of closed-loop autonomous driving simulations, we find that
fine-tuning ego-conditioned behavior predictors exclusively on high-regret
human-robot interaction data can improve the robot’s overall re-deployment
performance with significantly ($77\%$) less data.

##  I Intro

Data-driven human models have transformed how robots interact with people in
settings ranging from autonomous driving [13], to tabletop manipulation [19,
20], and even game-play [1]. Despite their widespread adoption, these models
are still susceptible to mis-predicting human behavior, potentially causing
robots to make unsafe decisions and raising the need for detecting and
mitigating such prediction failures. The predominate approach for quantifying
predictor performance is to analyze the model in isolation (i.e., component-
level analysis). For example, predictor failures can be measured by
sufficiently large deviations between the predicted versus real human behavior
[33]; in turn, this data could be used to develop benchmarks, anomaly
detectors, or re-train the predictor itself. While this approach is automatic
and easy to implement, it omits the fact that prediction errors at the
component-level do not always lead to system-level robot performance
degradation [7], For example, in the left image of Figure LABEL:fig:regret,
the parked white vehicle was erroneously predicted to start slowly moving;
however, from the blue robot’s perspective, its performance was not impacted
by these mis-predictions. Consequently, a predictor does not need to waste its
limited capacity being re-trained on such mis-predictions, and anomaly
detectors should not unnecessarily activate. However, automatically detecting
such system-level prediction failures from human–robot interaction data
requires a system-level metric that rigorously quantifies the impact of
prediction failures on robot decisions.

In this work, our key insight is that system-level human prediction failures
are precisely characterized by the mathematical notion of regret. Canonically,
regret measures the difference between the maximum reward decision in
hindsight given complete information and the realized reward from its actual
decision made under uncertainty [43]. In theory, identifying all deployment
interactions with sufficiently high regret would automatically extract all
system-level prediction failures. Unfortunately, the standard regret
definition can be poorly calibrated across disparate deployment contexts. This
is because the reward gap between the hindsight-optimal decision and executed
decision varies widely between different deployment contexts (e.g., sparse
versus crowded highway driving). Thus, we propose that

> evaluating the quality of a robot decision by its likelihood rather than its
> absolute reward calibrates regret across disparate contexts.

This idea is inspired by mathematical psychology and econometrics, wherein an
agent’s decisions are interpreted as noisily-optimal under some reward,
normalized by the reward of alternative decisions in that context [25, 3]. It
is precisely this normalization that calibrates the perceived quality of a
decision relative to the context-dependent reward scale. Thus, even if two
contexts have wildly different reward scales (e.g., sparse versus crowded
highway), a likelihood model is a principled way to place all decision
comparisons on the same scale: a value between zero and one. With this
probabilistic model, we derive a calibrated regret metric that enables
meaningful comparisons of how prediction failures degrade closed-loop human-
robot interaction performance across diverse deployment contexts. Moreover,
since our regret metric only requires evaluating decision quality via
likelihoods instead of direct rewards, we generalize the use of regret for
reward-free robot interaction planners, such as generative neural planners
[30].

In a series of closed-loop simulation experiments in the autonomous driving
domain, we automatically extract high-regret human-robot interaction data,
revealing flawed aspects of the human prediction model that lead to system-
level robot failures (e.g., a false sense of influence and a misunderstanding
stranded agents, shown on the right of Figure LABEL:fig:regret). We further
investigate the value of this high-regret data for improving a robot’s
downstream decision-making. We find that fine-tuning the robot’s human
behavior predictor on a small subset ($23\%$) of the total deployment data
yields consistent improvements during robot re-deployment when the subset
contains only high-regret data, unlike other possible subsets such as a random
selection or only low-regret interactions. Furthermore, fine-tuning on high-
regret data is competitive with the performance improvements achieved by fine-
tuning on the full dataset and does not exhibit any degradation artifacts in
nominal interaction settings. Ultimately, our analysis reveals that the very
presence of high-regret data during human behavior predictor fine-tuning is
one of the most important indicators of downstream robot decision-making
improvement.

While we take a first step towards continually improving robot interactions
with people, our core contribution is a general regret-based data selection
approach for automatically identifying system-level prediction failures [31].
Beyond our demonstration of behavior predictor fine-tuning, our approach
suggests several interesting directions for future work, including using
historical data of system-level prediction failures to construct runtime
monitors and automatically explaining the semantics of the failures
themselves.

##  II Related Work

Planning-aware Prediction. A standard method for capturing the influence of a
robot’s actions on other agents is to learn a conditional behavior prediction
model [13, 37, 19]. However, training and evaluating conditional prediction
models in the same manner as nonconditional models may lead to overly
optimistic estimations of how an ego agent may influence the behavior of other
agents [34], which can have poor downstream effects on planning. One way to
combat this is to augment the purely data-driven conditional behavior
prediction approach with additional modeling assumptions such as adversarial
predictions [21] or treating a predictions as a reference for multiagent
trajectory optimization [8]. However, quantifying the likelihood of these
model-based modifications is difficult without counterfactual interaction
examples in the predictor’s training data [8]. Most related to our work is the
task-aware metric proposed in [17] for evaluating prediction models, which
learns a high-level cost-function with hand-designed features for evaluating
planner-relevant prediction failures. Similar to this work, our proposed
metric is able to capture asymmetries in downstream tasks, is task-aware, and
is computationally feasible to compute. However, our regret metric adjusts the
severity of prediction failures directly to the robot’s planning ability,
generalizes the notion of regret to no longer require explicit reward (or
cost) functions, and is calibrated to enable fair comparison of failures
between diverse robot deployment contexts.

Detecting Impactful Anomalies for Robot Interaction. A long-standing challenge
in machine learning is detecting when test data is “out-of-distribution”
relative to the model’s training data [23, 14] and has been tackled with a
suite of approaches in a variety of domains such as computer vision [40, 29,
26, 6] and natural language processing [42]. However, assessing the impact of
anomalies in robotics poses unique challenges, specifically because anomalous
inputs in one part of the autonomy stack (e.g., predictor) may or may not
manifest in the overall system performance [2, 31]. As such, these kinds of
anomalies are often called “system-level” anomalies or failures. In the
context of interaction-relevant human prediction failures, previous approaches
sought to detect such “system-level” failures online via p-quantile anomalies
[11], learned Bayes filters [33], or even language models [10]. Once detected,
poor interactions can be avoided by switching predictor models or applying a
safety backup policy [12, 36, 28, 33]. Anomaly detection can also be useful in
the offline setting, wherein datasets are mined using heuristic features that
express safety-critical interactions [32] or counterfactual safety and
responsibility metrics [15]. Our approach is also utilized in an offline
setting, but mines for human-robot interactions from deployment data that were
high-regret from the robot’s perspective via our novel regret metric.
Furthermore, we demonstrate that fine-tuning a predictor using only these
high-regret scenarios can improve robot performance when redeployed while only
using a small subset of the total deployment data.

Iterative Robot Interaction Improvement. Continual learning for human-robot
interaction is, in general, an open research problem. In the context of
behavior prediction, one dominant paradigm is to improve or adapt a pre-
trained predictor with new human behavior data [35, 24, 18]. In the context of
collaborative robotics, Abeyruwan et al. demonstrated the efficacy of learning
by repeatedly deploying the robot to interact with people in an embodied ping-
pong game [1]. Here, the human and robot ping-pong play policies are
iteratively learned by alternating between training in simulation and real-
world deployments. In a similar vein, our method leverages both pre-trained
behavior prediction models and real (but targeted) human interaction data from
robot deployment to fine-tune the robot’s prediction model and improve re-
deployment performance.

##  III Problem Formulation

Our problem formulation is grounded in trajectory forecasting and prediction-
aware robot planning.

Agent States, Actions, and Context. Let the robot state be denoted by
$s^{\mathrm{R}}$, and the state of each human agent in the scene be denoted by
$s^{\mathrm{H}_{i}}$, $i\in\\{1,\ldots,M\\}$. For example, for an autonomous
driving application domain, each agent’s state can be represented as position,
heading, and velocity. All agents evolve their states according to their
respective control actions: $a^{\mathrm{R}}$ for the robot (e.g., acceleration
and turning rate) and $a^{\mathrm{H}_{i}}$ for the $M$ humans. For shorthand,
we denote the joint human-robot state at any time $t$ as
$s_{t}=[s^{\mathrm{R}}_{t},s^{\mathrm{H}_{1}}_{t},\ldots,s^{\mathrm{H}_{M}}_{t}]$
and a $T$-length future sequence of actions starting at time $t$ for any human
or robot agent as $\bm{a}_{t}=[a_{t},\ldots,a_{t+T}]$. Finally, let $C$ denote
the scene or environment context (e.g., semantic map of the environment) used
for both human prediction and robot decision-making.

Human Prediction & Robot Planning. We assume access to a pre-trained human
behavior prediction model $P_{\theta}$ with parameters $\theta$ capable of
conditional generations [37, 30]. Mathematically, this model is represented as

$P_{\theta}(\bm{a}^{\mathrm{H}_{1}:\mathrm{H}_{M}}_{t}\mid\bm{a}^{\mathrm{R}}_{t},s_{t},C).$
(1)

The model can generate $T$-horizon predictions of the $M$ humans in the scene,
$\bm{a}^{\mathrm{H}_{1}:\mathrm{H}_{M}}_{t}:=[\bm{a}_{t}^{H_{1}},...,\bm{a}_{t}^{H_{M}}]$,
conditioned on the scene context $C$, current111The predictor $P_{\theta}$ can
also take in a history of past states, $s_{t-h:t}$ where $h$ is the number of
previous timesteps that the predictor considers. human-robot state $s_{t}$,
and a candidate robot action trajectory $\bm{a}^{\mathrm{R}}_{t}$. We are also
given access to the robot’s task-driven planner

$\bm{a}^{\mathrm{R}}_{t}=\pi_{\phi}(s_{t},P_{\theta},C),$ (2)

with parameters $\phi$. It takes as input the human prediction model
$P_{\theta}$, the current joint state $s_{t}$, and scene context $C$ to
generate a best-effort plan $\bm{a}^{\mathrm{R}}_{t}$ with respect to the
current predictions. Within our framework, the robot’s planner could be a
reward-based planner (e.g., $\phi$ are the weights of a pre-specified reward
function’s features) or a neural planner (e.g., $\phi$ are the neural network
weights). At deployment time, the robot’s planning model can be re-queried
given new observations and predictions. We assume that the planner is optimal
under the given reward function or the available data for planner design.

Problem Statement. We deploy the robot for $\hat{T}$ time steps in $N$
scenarios (e.g., different initial conditions, environments, etc.) and collect
an aggregate deployment dataset

$\mathcal{D}=\\{(\hat{\bm{s}},\hat{\bm{a}}^{\mathrm{R}},\hat{\bm{a}}^{\mathrm{H}_{1}:\mathrm{H}_{M}})_{n}\\}_{n=1}^{N},$
(3)

consisting of observed joint state, robot action, and $M$ human action
trajectories (denoted by hat notation). Importantly, this observed data
exhibits how real interactions between the humans and robot occur, and are a
function of the robot’s predictive human model used during planning. In
general, as the robot continues to be deployed, $\mathcal{D}$ grows larger and
larger. We seek to automatically identify a subset of deployment data,
$\mathcal{D}_{\mathcal{I}}\subset\mathcal{D}$, that captures system-level
prediction failures: deployment interactions wherein human prediction failures
significantly impacted closed-loop robot performance. In addition to studying
the types of system-level failures induced by erroneous human predictors (end
of Section IV), we further investigate the use of this informative dataset as
a data-efficient way to refine the robot’s predictive model $P_{\theta}$ such
that, in subsequent deployments, the robot’s decision-making performance is
improved (Section VII).

##  IV A Calibrated Regret Metric for Detecting System-Level Prediction
Failures

Our core idea to detect system-level prediction failures is grounded in the
notion of regret: high-regret interactions are those in which incorrect human
predictions negatively impacted robot decision-making. Intuitively, by ranking
all the human-robot trajectory data from $\mathcal{D}$ by the robot’s
deployment regret, we can quantify the most anomalous interactions from the
perspective of the robot.

Regret Calibration Challenges in Reward-based Planning. The canonical measure
of regret is defined in reward space and considers the difference between the
maximum possible reward that the robot could incur in hindsight and the
realized reward from its actual decision [43]. However, even with a reward-
based planner in which the rewards are readily queryable, directly using the
canonical measure of regret may not always rank interactions in a desired way.
This is because an underlying assumption in this definition is that the reward
is perfectly calibrated between deployment scenarios or contexts, making two
high-regret interactions from different deployment contexts directly
comparable. Unfortunately, in practice, when robots are deployed in a wide
variety of scenarios and contexts $C$, it is extremely hard to have such a
perfectly calibrated reward, complicating the straightforward application of
absolute reward-based regret for failure detection. We illustrate this
technical point with a representative example drawn from our simulation
experiments (details in Section VI).

Illustrative Example. Consider the first column of Figure 1, where the same
robot is deployed in two different contexts: one where the robot drives behind
a stopped truck ($C_{1}$) and the other where the robot approaches an
intersection ($C_{2}$). The robot’s reward function is a linear combination of
control effort, collision penalty, road progress, and lane-keeping. In
scenario $C_{1}$, the robot mispredicts the stopped truck begins moving and
initiates a lane change, and in scenario $C_{2}$, it takes an aggressive
maneuver to avoid nearby traffic. The reward distribution of the executed and
hindsight-optimal action in each scenario is shown on the right of Figure 1. A
traditional regret metric assigns scenario $C_{1}$ a regret of $11.4$ and
$C_{2}$ a regret of $11.7$. However, it is only in scenario $C_{1}$ where the
suboptimal behavior led to the robot taking an extremely poor action. In
reality, despite $C_{2}$ taking an unnecessarily aggressive action, the robot
was able to safely proceed through the intersection. While the robot could
have taken a more optimal action in both scenarios, canonical measures of
regret are unable to identify scenario $C_{1}$ as being a worse decision in
that context.

Figure 1: Illustrative Example. Left column shows the robot’s predictions and
deployment-time decision. Right column shows the counterfactual analysis given
observed human behavior. Top Row ($C_{1}$): The robot mispredicts a stopped
vehicle to begin moving and chooses to change lanes. In reality, the truck
remained stopped. Bottom Row ($C_{2}$): The robot mispredicts a stopped car to
stay stopped and swerves offroad to avoid a collision. The human’s executed
actions instead drove forward. Canonical measures of regret rate scenario
$C_{1}$ as equivalent to $C_{2}$, even though the robot’s behavior $C_{1}$ led
to qualitatively worse behavior. Each hindsight-optimal action and executed
action’s reward is shown on the right scale.

A General Calibrated Regret Metric. To appropriately account for the context-
dependent reward distributions, we propose that during regret computation,
decision quality should be evaluated in probability-space. That is, instead of
looking at the absolute robot reward, we consider the reward of a candidate
decision normalized by the reward of all other possible decisions in that
scenario. Specifically, we utilize a likelihood model that can compute the
counterfactual probability of any candidate robot action sequence,
$\bm{a}^{\mathrm{R}}$, given the observed actions of the other human agents,
$\hat{\bm{a}}^{\mathrm{H}_{1}:\mathrm{H}_{M}}$, and the observed joint state
trajectory $\hat{\bm{s}}$ in that context $C$:

$P_{\phi}(\bm{a}^{\mathrm{R}}\mid\hat{\bm{a}}^{\mathrm{H}_{1}:\mathrm{H}_{M}},\hat{\bm{s}},C).$
(4)

By normalizing over alternative options in the scene, this likelihood model
enables fairer comparison of executed robot behaviors,
$\hat{\bm{a}}^{\mathrm{R}}$, across different scenarios irrespective of the
absolute reward the robot incurred. Note that here the likelihood model shares
the same parameters as the robot’s planner ($\phi$). For example, if the robot
is using a reward-based planner, then this probability distribution is shaped
according to the same reward weights and features that the robot uses for
decision-making.

For any deployment scenario $d\in\mathcal{D}$, we quantify the deployment
regret at timestep $t$ as:

$\displaystyle\mathcal{R}_{t}(d):=\max_{\bm{a}^{\mathrm{R}}_{t}}$
$\displaystyle\big{[}P_{\phi}(\bm{a}^{\mathrm{R}}_{t}\mid\hat{\bm{a}}^{\mathrm{H}_{1}:\mathrm{H}_{M}}_{t},\hat{\bm{s}}_{t},C)\big{]}$
(5) $\displaystyle-
P_{\phi}(\hat{\bm{a}}^{\mathrm{R}}_{t}\mid\hat{\bm{a}}^{\mathrm{H}_{1}:\mathrm{H}_{M}}_{t},\hat{\bm{s}}_{t},C).$

Intuitively, if the other agents behaved exactly as the robot’s model
$P_{\theta}$ expected, then the robot would have executed a trajectory deemed
high-performance that incurs low regret. More subtly, if the predictions
failed to capture human behavior in a way that was irrelevant to the robot’s
performance, then the robot’s executed trajectory will still have low regret.
Ultimately, the robot will only have a high regret if its planner was
blindsided by a mis-prediction that impacted its ability to execute a high-
performance trajectory.

Illustrative Example. Going back to the reward-based planning setting in
Figure 1, let us define the probability of each robot decision using the Luce-
Shepard choice rule [25], which places exponentially more probability on robot
decisions that are high-reward under its reward function. Recall that the
canonical regret metric evaluated the unnecessarily aggressive (but safe)
maneuver in $C_{2}$ as approximately the same regret as the unsafe lane change
into a stopped truck in $C_{1}$. In contrast, our calibrated regret metric,
which evaluates the decision likelihood of each action, results in an
intuitive separation: $C_{1}$ has a higher regret of $0.56$ compared to $0.34$
in $C_{2}$.

Extracting High-Regret Interactions ($\mathcal{D}_{\mathcal{I}}$). Finally,
leveraging our general calibrated regret metric from Equation 5, we score each
scene $d\in\mathcal{D}$ by taking the average of the per-timestep
regrets222Additional variants for computing the total regret of a scene are
explored in the supplementary material.. Akin to Farid et al. [11], we utilize
these rankings to form an empirical regret distribution and pick the top
$p$-quantile of scenarios with highest regret to constitute our system-level
failure dataset, $\mathcal{D}_{\mathcal{I}}$, where $p$ is a design choice. In
Figure LABEL:fig:regret, we visualize a histogram of the deployment regret
distribution extracted from our closed-loop autonomous-driving simulation
experiments, demonstrating the infrequent but high-regret nature of system-
level prediction failures (more details in Section VII). Interesting scenarios
emerge as high-regret, such as the robot believing it can influence stopped
traffic to start moving if it starts driving (Figure LABEL:fig:regret,
middle), or being unable to understand how a stranded truck will not move out
of the way (Figure LABEL:fig:regret, right). For further comparisons on the
differences between the scenarios extracted by our metric and prior approaches
such as L2 prediction error, canonical regret, and the approach from Farid et
al. [11], please refer to the supplementary Section X-C.

##  V Learning to Predict High-Regret Interactions for Deployment Improvement

Thus far we have proposed a calibrated detection method for identifying
system-level prediction failures, but we are also interested in the potential
for robot performance improvement. Here, the fundamental challenge is that the
robot has an incomplete understanding of how its behavior may influence other
agents. One approach could be to retrain (or fine-tune) the prediction model
$P_{\theta}$ on all the recorded deployment data, $\mathcal{D}$. However, our
regret metric just revealed that only a subset of mis-predicted human
behaviors and robot interactions really mattered for closed-loop robot
performance. As a result, this may indicate that the prediction model’s
limited capacity should not be wasted by treating every data point as equal.
Treating all data as equally valuable would only increase training costs as
the size of $\mathcal{D}$ grows and could lead to unexpected model changes in
critical scenarios. In fact, we hypothesize that the scenarios we identified
with our regret metric, $\mathcal{D}_{\mathcal{I}}\subset\mathcal{D}$, are
precisely the interactions that the robot’s predictive model was missing. In
other words, if the robot’s predictor cannot understand what robot actions can
lead to poor interaction outcomes, the planner will be unable to evaluate
which actions to avoid. By focusing the model on learning to predict high-
regret interactions, we hypothesize this will enable the model to expand its
understanding of how the interaction could evolve (even if it leads to
undesirable outcomes) and at re-deployment time, the robot can actively steer
away from these possibilities. In Section VI, we detail a straightforward
approach for incrementally improving robot performance by fine-tuning its
prediction model $P_{\theta}$ on only the system-level failure dataset
$\mathcal{D}_{\mathcal{I}}$.

##  VI Experimental Setup

To evaluate the potential of our approach in data-rich human-robot interaction
deployment contexts, we ground our experiments in driving interactions between
an autonomous vehicle and human-driven vehicles. Here we detail the specifics
of our robot planner, human predictor and associated fine-tuning methodology,
and state-of-the-art simulator which enables us to test closed-loop
“deployment” of the robot under different controlled conditions.

Robot Policy: Tree Policy Planning (TPP). Our planner $\pi_{\phi}$ is a tree-
structured contingency planner developed by Chen et al. [7]. This reward-based
planner is compatible with state-of-the-art conditional behavior prediction
models to account for the influence of the robot’s actions when planning. The
planner first generates a tree of randomly sampled multi-stage dynamically
feasible trajectories, after which each branch is fed into an ego-conditioned
predictor $P_{\theta}$ to generate a scenario tree comprised of multimodal
predictions. The optimal robot trajectory is obtained by maximizing a hand-
designed reward function which is a linear combination of collision costs,
lanekeeping costs, control costs, and distance-traveled reward (full details
can be found in the original paper [7]). This reward function is fixed
throughout the entire robot deployment and human predictor fine-tuning
process.

Base Human Predictor. For our prediction model $P_{\theta}$, we train an ego-
conditioned variant of the state-of-the-art Agentformer model [41] where the
context $C$ is the semantic map. The training loss is a weighted sum of three
terms: $1)$ prediction loss that penalizes incorrect predictions from the
ground truth, $2)$ ego-conditioned collision (EC) loss that penalizes
collisions between the predictions and conditioned ego-action. $3)$ other
regularization losses such as diversity and KL divergence loss. Our base
prediction model is trained on the Nuscenes dataset [5], consisting of 1,000
20-second driving logs collected in Boston and Singapore. During this phase of
training, we use a learning rate of 1e-4.

Fine-tuning the Predictor. For fine-tuning, we unfreeze only the last 1.5M
parameters of the 17.7M parameter Agentformer model. This corresponds to the
last half of the future decoder module [41]. Across all our baselines, we
fine-tune the predictor to see the data for 25 epochs at a reduced learning
rate of 5e-5 held constant throughout the 25 epochs. Furthermore, when fine-
tuning we eliminate the diversity loss and EC collision loss, focusing the
model’s efforts to only accurately predict the ground-truth deployment data.
Due to the large size of the Nuscenes data, we trained on two Nvidia A6000
GPUS with a batchsize of 14 (effective batch size 28) and gradient
accumulation of 5 steps.

| ADE / FDE ($\downarrow$) |
---|---|---
Fine-tuning Data | Nuscenes | High-Regret | Low-Regret |
None (Base model) | 0.371 / 1.919 | 0.639 / 2.999 | 0.610 / 2.375 |
Low-regret-FT | 0.353 / 1.554 | 0.386 / 2.333 | 0.386 / 1.783 |
Random-FT | 0.315 / 1.435 | 0.366 / 2.100 | 0.382 / 1.738 |
High-regret-FT | 0.327 / 1.520 | 0.333 / 2.014 | 0.366/1.713 |
All-FT | 0.297 / 1.381 | 0.326 / 1.973 | 0.340 / 1.665 |
TABLE I: Fine-tuned Open-loop Prediction Performance. Average ADE / FDE of the
base prediction model and the fine-tuned models on log-replay data. Columns of
table indicate different log-replay validation data splits: original Nuscenes
data, closed-loop simulation data that exhibited system-level failures (i.e.
high-regret), and closed-loop simulation data identified as low-regret. Lowest
error is denoted in italics-bold, and second lowest in bold. Each baseline was
trained with three random seeds. The reported numbers come from the model with
the lowest High-regret ADE/FDE.

Closed-loop Simulation. In our setting, it is insufficient to simulate human
agents using log replay from the original Nuscenes data, since we are
specifically looking to identify prediction failures that are coupled with the
robot’s decision-making. As such, human agents must be allowed to react to the
robot’s actions, which will inevitably differ from the original driving logs.
To capture this, we use the state-of-the-art BITS simulation environment [39]
and use 100 held-out scenes from the original Nuscenes dataset (not used
during training) to initialize each simulation. All non-ego agents are
simulated with the off-the-shelf BITS policy, which generates short-horizon
goal locations for the agent and corresponding trajectories via reward-based
action selection. We constructed the closed-loop deployment dataset
$\mathcal{D}$ by running simulations for an interaction horizon of 20 seconds
in each of the 100 held-out Nuscenes initializations, matching the length of
each scene in the Nuscenes dataset.

Calibrated Regret Computation. Our regret metric requires access to a
conditional probability distribution over robot trajectories given observed
human behavior. Since this case study considers a reward-driven planner, our
likelihood model follows from the Luce-Shepard choice rule [25] where the
parameters $\phi$ denote the weights of the reward function:

$\displaystyle
P_{\phi}(\bm{a}^{\mathrm{R}}_{t}\mid\hat{\bm{a}}^{\mathrm{H}_{1}:\mathrm{H}_{M}}_{t},\hat{\bm{s}}_{t},C)=\frac{1}{Z}e^{R_{\phi}(\bm{a}^{\mathrm{R}}_{t},\hat{\bm{a}}^{\mathrm{H}_{1}:\mathrm{H}_{M}}_{t},\hat{\bm{s}}_{t},C)},$
(6)

where
$Z:=\sum_{\tilde{\bm{a}}^{\mathrm{R}}_{t}}e^{R_{\phi}(\tilde{\bm{a}}^{\mathrm{R}}_{t},\hat{\bm{a}}^{\mathrm{H}_{1}:\mathrm{H}_{M}}_{t},\hat{\bm{s}}_{t},C)}$.
At each timestep in scene $d\in\mathcal{D}$ where the robot originally
replanned, we re-compute the reward of each candidate ego action with respect
to the ground truth behavior of the other agents, instead of predictions. This
allows us to compute the conditional probability of each candidate ego action
and obtain the scene’s regret $\mathcal{R}_{t}(d)$.

Dataset Construction. Using the deployment dataset $\mathcal{D}$, we discarded
four due to the absence of non-ego agents, resulting $N=96$ total closed-loop
deployment scenarios each of length $20$ seconds. We ranked the ego agent’s
average regret and chose the top $p=20$-quantile333The percentage of the
deployment data to be deemed as system-level failures is a design decision. We
chose $p=20$ in part due to the small size of the closed-loop deployment
dataset. of scenarios to be our set of scenes that suffered from interaction-
relevant prediction failures, yielding a total of 20 high-regret scenarios in
$\mathcal{D}_{\mathcal{I}}$. The rest of the 76 scenarios were deemed
sufficiently low-regret. We hold out 20% of both high-regret
$\mathcal{D}_{\mathcal{I}}$ and low-regret
$\mathcal{D}\setminus\mathcal{D}_{\mathcal{I}}$ scenes (3 and 17
respectively). The remainder of the deployment data is used for fine-tuning
the model.

##  VII Experimental Results

To investigate how learning to predict high-regret interactions can improve
the robot’s behavior, we perform a series of experiments wherein we fine-tune
the predictive model $P_{\theta}$ with different subsets of the deployment
dataset, $\mathcal{D}$, and then re-deploy the robot in our simulation
environment to measure its changes in closed-loop performance. Specifically,
we hold the robot’s planner constant and we redeploy the robot in the
simulator on initializations of the held-out 20 validation scenarios
(consisting of 3 high-regret scenes and the rest of the 17 as low-regret
scenes). Importantly, since the behavior of the robot changes as a function of
its predictive model in the closed-loop simulation, we once again allow the
non-ego agents to respond to the robot via the BITS simulation model.

We compare our method, called High-Regret-FT which fine-tunes $P_{\theta}$
exclusively on high-regret data $\mathcal{D}_{\mathcal{I}}$ containing the 17
identified high-regret scenes not held out for evaluation, to the same
predictor architecture but fine-tuned on three different datasets: Random-FT
uses the randomly samples a subset of 17 scenes from the deployment data
$d\sim\mathrm{Unif}[\mathcal{D}]$, All-FT uses the entire deployment dataset
$\mathcal{D}$ consisting of the 76 scenarios not held out for validation, and
Low-Regret-FT uses only 17 low-regret scenarios that did not exhibit any
system-level failures, sampled from
$\mathcal{D}\setminus\mathcal{D}_{\mathcal{I}}$. Since the last baseline does
not have any high-regret scenarios, this serves as our control the measure the
impact of high-regret scenes being present in the data at all. Each baseline
took about 1.5 hours to finetune, except All-FT which took about 7 hours due
to the larger dataset size. We fine-tuned each baseline across 3 independent
random seeds.

Takeaway 1: High-Regret-FT improves log-replay prediction accuracy in high-
regret scenarios, without degrading performance on low-regret scenarios and
the original Nuscenes validation set.

We first evaluate the predictors “open-loop” on log-replay data from three
different validation datasets: the original Nuscenes validation data, the 3
high-regret scenes in the deployment validation dataset, and the 17 low-regret
scenes in the deployment validation dataset. We report the Average
Displacement Error (ADE) and Final Displacement Error (FDE) for each
baseline’s best performing seed (standard practice in the trajectory
forecasting literature) in Table I. The base predictor (trained only on
Nuscenes) doubles its ADE and FDE on the simulated deployment data due to a
distribution shift in the driving behavior between Nuscenes and the simulated
BITS policy. Intuitively, High-Regret-FT reduces the ADE and FDE on high-
regret validation data the most. More interestingly, it also improves open-
loop prediction accuracy on low-regret scenes over the base prediction model
(40% ADE reduction, 28% FDE reduction). We hypothesize this is because the
high-regret validation data leaks enough information about the general
distribution shift between the Nuscenes and the BITS simulation to enable a
model that only saw high-regret interactions to improve on nominal low-regret
interaction data as well. While All-FT achieves the best prediction accuracy
across the board, it is important to note that this is an information-
advantaged baseline: it has the ability to learn from both system-level and
non-system-level failures the deployment data. Since the amount of simulated
deployment data in our experiments is quite small relative to the Agentformer
prediction model’s size, the model has the capacity to “memorize” all the
deployment data. However, High-Regret-FT is fine-tuned on $77\%$ less data
than All-FT, making it much more data efficient to train than the All-FT
approach, while still performing the second-best in terms of quantitative
metrics. As the deployment dataset $\mathcal{D}$ grows and training time and
costs increase along with it, this can be a promising avenue to alleviate
these challenges while still achieving high predictive performance. Finally,
the least amount of improvement over the base model is exhibited by Low-
Regret-FT and followed by Random-FT. This is particularly important, since it
demonstrates the quantitative value of attending to the high-regret
interaction data.

Figure 2: Qualitative Re-Deployment Result. Blue robot must navigate around a
new stopped truck while being mindful of other moving cars (white). Small
deviations in the white vehicles are the result of inherent variation in
closed-loop simulation. As the prediction model $P_{\theta}$ is targettedly
re-trained on more high-regret interactions, the closed-loop performance
naturally improves (top to bottom). Even though High-Regret-FT trains on
$77\%$ less data than the All-FT approach, it still achieves qualitatively
comparable behavior in re-deployment.

Takeaway 2: High-Regret-FT improves closed-loop performance metrics such as
collision cost and average regret in all re-deployment scenes, even compared
to All-FT.

| High-Regret Scenarios | Low-Regret Scenarios
---|---|---
Fine-tuning Data | Col Cost ($\downarrow$) | Col Severity ($\downarrow$) | Regret ($\downarrow$) | Col Cost ($\downarrow$) | Col Severity ($\downarrow$) | Regret ($\downarrow$)
Base Model | 10.829 | 0.848 | 0.034 | 0.411 | 0.051 | 0.006
Low-regret-FT | 7.489 ($\pm$ 0.408) | 0.794 ($\pm$ 0.059) | 0.019 ($\pm$ 0.002) | 0.670 ($\pm$ 0.206) | 0.354 ($\pm$ 0.493) | 0.009 ($\pm$ 0.000)
Random-FT | 7.209 ($\pm$ 0.883) | 0.743 ($\pm$ 0.064) | 0.016 ($\pm$ 0.003) | 0.495 ($\pm$ 0.213) | 0.076 ($\pm$ 0.054) | 0.006 ($\pm$ 0.000)
High-regret-FT | 3.763 ($\pm$ 0.965) | 0.588 ($\pm$ 0.101) | 0.016 ($\pm$ 0.003) | 0.397 ($\pm$ 0.814) | 0.062 ($\pm$ 0.031) | 0.006 ($\pm$ 0.001)
All-FT | 1.176 ($\pm$ 0.775) | 0.377 ($\pm$ 0.067) | 0.013 ($\pm$ 0.002 ) |  0.486 ($\pm$ 0.131) | 0.060 ($\pm$ 0.030) |  0.007 ($\pm$ 0.001)
TABLE II: Closed-Loop Re-Deployment Performance. Average robot performance
metrics for each fine-tuned predictor (standard deviation over three seeds of
fine-tuning in parentheses). Two main columns are evaluations on high-regret
and low-regret held-out validation scenes. Italics-bold indicates best, and
bold indicates second best. Across all metrics and settings, High-regret-FT
consistently improves robot performance, and is competitive with the
privileged All-FT approach.

We use three metrics to evaluate the closed-loop re-deployment performance of
our robot using each candidate fine-tuned predictor. First, we measure Total
Collision Cost, which is the total cost the ego incurred during the simulation
from collisions. Second, Collision Severity which is the intensity of a
collision measured by $\frac{\text{Total Collision Cost}}{\\#\text{Frames in
Collision}}$. This metric is used to disentangle high total collision costs
resulting from 1-2 timesteps of extremely poor behavior (e.g., quickly driving
through a stopped vehicle) versus many timesteps of small collision costs
(e.g., stopped bumper-to-bumper at an intersection). Finally, we measure
average Regret as defined in Section IV. We report the average closed-loop
performance for each method across three seeds of fine-tuning in Table II.

We find that High-Regret-FT decreases collision cost, severity, and average
regret compared to the base model across both high- and low-regret scenes. As
hypothesized, most closed-loop performance gains are observed in the high-
regret scenes, decreasing robot regret by $-{53}\%$ and collision cost by
$-{65}\%$ (shown in Table II). By being able to see all the deployment data,
All-FT observes the highest performance gains in high-regret scenes but shows
performance degradation in low-regret scenes (increasing robot regret by about
$+31\%$ and collision cost by $+7\%$). However, we note that the performance
degradation in low-regret scenarios is not statistically significant. For each
baseline, the results from the baseline model are within 1 standard deviation
(except for the regret of Low-Regret-FT). In contrast, the improved
performance of High-Regret-FT is more than one standard deviation better than
the performance of Low-Regret-FT and Random-FT with respect to the collision-
based metrics.

Takeaway 3: Closed-loop and open-loop performance are only correlated on high-
regret scenes.

We find that in the high-regret scenes, the open-loop performance (Table I) is
positively correlated with the closed-loop planner performance (Table II). We
hypothesize that this is because these scenes are much more challenging (e.g.,
highly interactive), encode more nuanced human behavior, and affect the
planner the most; thus, the prediction improvements in such scenarios indicate
that $P_{\theta}$ has learned to capture interaction better which naturally
improves the planning performance.

##  VIII Preliminary Results: Detecting Interaction Failures of Generative
Planners

One consequence of our likelihood-based approach is that it generalizes a
notion of calibrated regret to be readily compatible with generative neural
planners. These neural planners learn a joint distribution of multi-agent
behavior and execute the most likely ego plan under the joint distribution
[27, 30]. However, this means that they do not maintain explicit rewards,
prohibiting the application of prior system-level anomaly detection methods
[11]. With the rise of these types of models in robotics, we see this as an
important step towards uniting core decision-theoretic concepts with
increasingly powerful but opaque planning paradigms.

To substantiate the generality of our metric across both application domains
and type of planner, we further demonstrate how our regret metric can
automatically identify system-level prediction failures when the robot’s
planner is a generative model in a simulated social navigation setting.

Setup. The robot is modeled as a Dubins’ car dynamical system [9] and is
tasked with reaching a goal while navigating around a planar pedestrian that
can freely move in any direction. The robot is given one of two goals
$g_{R}\in\\{\texttt{top},\texttt{right}\\}$ and is allowed to observe an
initial low-dimensional behavior cue of the human (e.g., eye gaze, head tilt)
denoted by $\delta_{\mathrm{H}}$. The robot must reach its intended goal
unless it predicts that doing so will intersect with the human’s predicted
path; in this case, the robot switches to navigate to the other goal (Figure
3). The robot’s generative planner $\pi_{\phi}(\cdot)$ is a Vector-quantized
Variational Autoencoder (VQ-VAE) [38] trained on simulated multi-agent
interaction data described below.

Dataset Generation. We constructed a dataset of $|\mathcal{D}_{train}|=10,000$
human-robot trajectories based on a small set of simple rules in order to
train our generative planner. The positions of the two goal locations and the
robot and human’s start locations were fixed in all training examples (shown
in Figure 3), but the choice of goal $g$ and initial human behavior cue
$\delta_{\mathrm{H}}$ were uniformly distributed. The synthetic human
trajectory generation abides by the following rule:
$\bm{a}^{\mathrm{H}}=PC(\texttt{left})$ if $\delta_{H}+\epsilon<1/3$,
$PC(\texttt{right})$ if $\delta_{\mathrm{H}}+\epsilon>2/3$,
$PC(\texttt{down})$ otherwise. Here, $PC(\cdot)$ denotes a proportional
controller noisily executing a 6-timestep action trajectory that guides the
human to the prespecified goal points to the left, right, or below the human’s
initial position, and $\epsilon$ is zero-mean noise.

The synthetic robot trajectory generation follows a similar process, but
conditions on the desired goal $g_{R}$ and the synthetic human behavior
$\bm{a}^{\mathrm{H}}$. The robot’s action $\bm{a}^{\mathrm{R}}$ is a noisily
executed 6-timestep action trajectory given by a proportional controller that
guides the robot to $g_{R}$, unless this brings the robot in close proximity
to the human trajectory $\bm{a}^{\mathrm{H}}$ in which case the robot’s
proportional controller guides it to the other goal with 80$\%$ probability.
This dataset synthesis leads to six different high-level outcomes of the
interaction depending on which directions the human and robot travel. Figure 3
shows three of these joint human-robot interaction outcomes.

Generative Planner Architecture. The robot’s generative planner $\pi_{\phi}$
predicts the joint distribution of future human-robot behavior conditioned on
the robot’s observation and the robot’s goal. We used an encoder-decoder
architecture based on the VQ-VAE, which is a variant of the traditional VAE
that uses vector quantization to discretize the latent representation into $K$
latent embedding vectors $\\{z_{k}\\}_{k=1}^{K}$. The encoder
$P_{\alpha}(z\mid\delta_{\mathrm{H}},g)$ with learnable parameters $\alpha$
takes $\delta_{\mathrm{H}}$ and $g_{R}$ as inputs to encode the input and
produce a categorical distribution over latent embedding through vector
quantization. We further inject a small amount of zero-mean Gaussian noise
into quantized embedding $z\in\\{z_{k}\\}_{k=1}^{K}$ to encourage diverse
outputs during the joint trajectory decoding phase. This decoder
$P_{\beta}(\bm{a}^{\mathrm{R}},\bm{a}^{\mathrm{H}}\mid z)$ with learnable
parameters $\beta$ takes a sampled latent vector $z$ as input (sampled from
$P_{\alpha}(z\mid\delta_{\mathrm{H}},g)$ with additional noise injection) to
approximate the joint distribution over actions
$\bm{a}^{\mathrm{R}},\bm{a}^{\mathrm{H}}$. In our setting, we set $K=6$ and
initialized the latent embeddings with K-means clustering to encourage each of
the K embedding vectors to correspond to one of the six joint human-robot
action trajectories.

By designing the generative planner in this way, it is straightforward to
generate action trajectories conditioned on not only the robot’s observation
and goal but also sample action trajectories conditioned on one of the six
high-level interaction outcomes. This generative model does planning and
prediction simultaneously conditioned on the state $s_{t}=\delta_{\mathrm{H}}$
and context $C=g_{R}$ and connects to the policy $\pi_{\phi}$ and predictor
$P_{\theta}$ as defined in Section III where the neural network weights of the
policy and predictor are equivalent (i.e., $\theta\equiv\phi$) and
specifically consist of $\alpha$ and $\beta$.

Figure 3: Our Regret Metric with a Generative Planner. Even when using a
generative model for joint planning and prediction, our generalized regret
metric can correctly detect poor behavior. (left) Human behaved as predicted,
and the robot incurred low regret. (middle) The robot mispredicted the human’s
behavior, but the robot’s actions were still likely under the ground-truth
joint behavior and incurred low regret. (right) The robot mispredicted the
human’s behavior in a way that led to a low-probability human-robot joint
trajectory and incurred high regret.

Calibrated Regret Computation. Since the behavior cue $\delta_{\mathrm{H}}$
does not solely determine $\bm{a}^{\mathrm{H}}$, it is possible that the
generative planner mispredicts the human’s executed actions and thus executes
a poor plan. Suppose the robot samples latent embedding $\tilde{z}$ from
$P_{\alpha}$ and conditions on this to generate the joint action trajectory
$\tilde{\bm{a}}_{\mathrm{R}},\tilde{\bm{a}}_{\mathrm{H}}$ from $P_{\beta}$.
The robot’s executed trajectory is precisely this action trajectory,
$\hat{\bm{a}}^{\mathrm{R}}:=\tilde{\bm{a}}_{\mathrm{R}}$; however, the ground-
truth trajectory of the human, $\hat{\bm{a}}^{\mathrm{H}}$, may not be aligned
with the original action trajectory that the robot anticipated
($\tilde{\bm{a}}_{\mathrm{H}}$).

Recall that our regret computation in Equation 5 hinges on evaluating the
likelihood of executed robot decisions given the true human behavior and the
most likely hindsight decision. To perform this computation, we must first
obtain the counterfactual conditional probability density of the robot’s
trajectory:
$P(\hat{\bm{a}}^{\mathrm{R}}\mid\hat{\bm{a}}^{\mathrm{H}},\delta_{\mathrm{H}},g)$.
We leverage our ability to sample from the decoder $P_{\beta}$ to form a
kernel density estimate of
$P_{\beta}(\hat{\bm{a}}^{\mathrm{R}},\hat{\bm{a}}^{\mathrm{H}}\mid\tilde{z})$
by querying the decoder 100 times. We then approximate the conditional
probability of $\hat{\bm{a}}^{\mathrm{R}},\hat{\bm{a}}^{\mathrm{H}}$ by
integrating the KDE around a small neighborhood of the ground-truth joint
action trajectory. Furthermore, the conditional probability of
$\hat{\bm{a}}^{\mathrm{H}}$ can be approximated by marginalizing over all
possible $\bm{a}^{\mathrm{R}}$. Leveraging the approximate probabilities, we
compute the counterfactual probability for any $\hat{\bm{a}}^{\mathrm{R}}$

$\displaystyle
P(\hat{\bm{a}}^{\mathrm{R}}\mid\hat{\bm{a}}^{\mathrm{H}},\delta_{\mathrm{H}},g_{\mathrm{R}})=$
$\displaystyle\quad\quad\quad\frac{\sum_{z}P_{\beta}(\hat{\bm{a}}^{\mathrm{R}},\hat{\bm{a}}^{\mathrm{H}}\mid
z)P_{\alpha}(z\mid\delta_{\mathrm{H}},g_{\mathrm{R}})}{\int_{\tilde{\bm{a}}_{\mathrm{R}}}\sum_{z}P_{\beta}(\hat{\bm{a}}^{\mathrm{H}},\tilde{\bm{a}}_{\mathrm{R}},\mid
z)P_{\alpha}(z\mid\delta_{\mathrm{H}},g_{\mathrm{R}})}.$

Note that this is a particular instantiation of (4) where the planner and
predictor parameters are the same ($\phi\equiv\theta$) and are parameterized
by $\alpha$ and $\beta$; we then compute the desired probability using Bayes’
theorem. We repeat this process for the maximum probability action trajectory
$\bm{a}^{\mathrm{R}}$444In practice, finding this maximizing
$\bm{a}^{\mathrm{R}}$ is approximated by searching over a prespecified set of
action trajectories. to compute our regret metric from Equation 5.

Results. The results of the regret metric in the generative planning domain
are shown in Figure 3. Our regret metric only flags interactions as anomalous
when mispredictions lead to unexpectedly close proximity interactions between
the robot and human (right, Figure 3). In both nominal interactions (left,
Figure 3) and interactions where mispredictions did not lead to close-
proximity interactions (center, Figure 3), the deployment regret is quite low.
Importantly, we highlight that our generalized calibrated regret metric
captures the intuition on what is a system-level prediction failure without
the need for manually specifying an evaluative reward-function for either the
high-level or low-level robot and human behaviors.

##  IX Conclusion & Discussion

Conclusion. We present a generalized calibrated regret metric for detecting
and mitigating human-robot interaction failures. We utilized the proposed
regret metric to construct a targeted dataset of system-level prediction
failures. In a suite of closed-loop autonomous driving simulations with a
cost-based planner, we demonstrated how fine-tuning ego-conditioned behavior
predictors exclusively on this high-regret human-robot interaction data can
both quantitatively and qualitatively improve the robot’s overall re-
deployment performance with significantly ($77\%$) less data. While fine-
tuning and incremental model improvement are open areas of research, our
results indicate that even straightforward predictor fine-tuning approaches
can show improvements in open-loop and closed-loop metrics when high-regret
human-robot interaction data is highly represented in the fine-tuning dataset.
We also showed early results demonstrating the applicability of our
generalized calibrated regret metric for identifying planning failures from
generative models.

Discussion. At its core, our proposed method is an offline approach, since the
counterfactual regret computation requires access to the ground truth data
observed during deployment. In certain domains, like trajectory prediction in
autonomous driving, real human-robot interaction data is already increasingly
available as autonomous vehicles are deployed at scale. However, studying our
metric in other domains, such as state-of-the-art human-robot collaborative
manipulation datasets, is an exciting frontier [19]. Relatedly, applying this
metric beyond human prediction failures, to say, identifying system-level
perception failures is an interesting future direction. Finally, future work
should also investigate the use of offline datasets—like the ones constructed
with our regret-based approach—to inform runtime monitors that can anticipate
system-level prediction failures online.

## References

  * Abeyruwan et al. [2023] Saminda Wishwajith Abeyruwan, Laura Graesser, David B D’Ambrosio, Avi Singh, Anish Shankar, Alex Bewley, Deepali Jain, Krzysztof Marcin Choromanski, and Pannag R Sanketi.  i-sim2real: Reinforcement learning of robotic policies in tight human-robot interaction loops.  In _Conference on Robot Learning_. PMLR, 2023.
  * Antonante et al. [2023] Pasquale Antonante, Sushant Veer, Karen Leung, Xinshuo Weng, Luca Carlone, and Marco Pavone.  Task-aware risk estimation of perception failures for autonomous vehicles.  _arXiv preprint arXiv:2305.01870_ , 2023.
  * Baker et al. [2009] Chris L Baker, Rebecca Saxe, and Joshua B Tenenbaum.  Action understanding as inverse planning.  _Cognition_ , 113(3):329–349, 2009.
  * Bansal et al. [2017] Somil Bansal, Mo Chen, Sylvia Herbert, and Claire J Tomlin.  Hamilton-jacobi reachability: A brief overview and recent advances.  In _2017 IEEE 56th Annual Conference on Decision and Control (CDC)_. IEEE, 2017.
  * Caesar et al. [2020] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020.
  * Chakraborty and Bansal [2023] Kaustav Chakraborty and Somil Bansal.  Discovering closed-loop failures of vision-based controllers via reachability analysis.  _IEEE Robotics and Automation Letters_ , 2023.
  * Chen et al. [2023] Yuxiao Chen, Peter Karkus, Boris Ivanovic, Xinshuo Weng, and Marco Pavone.  Tree-structured policy planning with learned behavior models.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , 2023.
  * Chen et al. [2024] Yuxiao Chen, Sushant Veer, Peter Karkus, and Marco Pavone.  Interactive joint planning for autonomous vehicles.  _IEEE Robotics and Automation Letters_ , 2024.
  * Dubins [1957] L. E. Dubins.  On curves of minimal length with a constraint on average curvature, and with prescribed initial and terminal positions and tangents.  _American Journal of Mathematics_ , 79(3):497–516, 1957.  ISSN 00029327, 10806377.  URL http://www.jstor.org/stable/2372560.
  * Elhafsi et al. [2023] Amine Elhafsi, Rohan Sinha, Christopher Agia, Edward Schmerling, Issa AD Nesnas, and Marco Pavone.  Semantic anomaly detection with large language models.  _Autonomous Robots_ , 47(8), 2023.
  * Farid et al. [2023] Alec Farid, Sushant Veer, Boris Ivanovic, Karen Leung, and Marco Pavone.  Task-relevant failure detection for trajectory predictors in autonomous vehicles.  In _Conference on Robot Learning_. PMLR, 2023.
  * Fridovich-Keil et al. [2020] David Fridovich-Keil, Andrea Bajcsy, Jaime F Fisac, Sylvia L Herbert, Steven Wang, Anca D Dragan, and Claire J Tomlin.  Confidence-aware motion prediction for real-time collision avoidance.  _The International Journal of Robotics Research_ , 2020.
  * Hagedorn et al. [2023] Steffen Hagedorn, Marcel Hallgarten, Martin Stoll, and Alexandru Condurache.  Rethinking integration of prediction and planning in deep learning-based automated driving systems: a review.  _arXiv preprint arXiv:2308.05731_ , 2023.
  * Hendrycks and Gimpel [2016] Dan Hendrycks and Kevin Gimpel.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _arXiv preprint arXiv:1610.02136_ , 2016.
  * Hsu et al. [2023] Kai-Chieh Hsu, Karen Leung, Yuxiao Chen, Jaime Fernàndez Fisac, and Marco Pavone.  Interpretable trajectory prediction for autonomous vehicles viacounterfactual responsibility.  In _IEEE/RSJ Int. Conf. on Intelligent Robots & Systems_, 2023.
  * Ivanovic and Pavone [2019] Boris Ivanovic and Marco Pavone.  The trajectron: Probabilistic multi-agent trajectory modeling with dynamic spatiotemporal graphs.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019.
  * Ivanovic and Pavone [2022] Boris Ivanovic and Marco Pavone.  Injecting planning-awareness into prediction and detection evaluation.  In _2022 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2022.
  * Ivanovic et al. [2023] Boris Ivanovic, James Harrison, and Marco Pavone.  Expanding the deployment envelope of behavior prediction via adaptive meta-learning.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023.
  * Kedia et al. [2023a] Kushal Kedia, Atiksh Bhardwaj, Prithwish Dan, and Sanjiban Choudhury.  Interact: Transformer models for human intent prediction conditioned on robot actions.  _arXiv preprint arXiv:2311.12943_ , 2023a.
  * Kedia et al. [2023b] Kushal Kedia, Prithwish Dan, Atiksh Bhardwaj, and Sanjiban Choudhury.  Manicast: Collaborative manipulation with cost-aware human forecasting.  _arXiv preprint arXiv:2310.13258_ , 2023b.
  * Kedia et al. [2023c] Kushal Kedia, Prithwish Dan, and Sanjiban Choudhury.  A game-theoretic framework for joint forecasting and planning.  _International Conference on Intelligent Robots and Systems_ , 2023c.
  * Kingma and Welling [2014] Diederik P. Kingma and Max Welling.  Auto-Encoding Variational Bayes.  In _2nd International Conference on Learning Representations, ICLR 2014_ , 2014.
  * Lee et al. [2018] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin.  A simple unified framework for detecting out-of-distribution samples and adversarial attacks.  _Advances in neural information processing systems_ , 31, 2018.
  * Li et al. [2023] Yiheng Li, Seth Z Zhao, Chenfeng Xu, Chen Tang, Chenran Li, Mingyu Ding, Masayoshi Tomizuka, and Wei Zhan.  Pre-training on synthetic driving data for trajectory prediction.  _arXiv preprint arXiv:2309.10121_ , 2023.
  * Luce [2005] R Duncan Luce.  _Individual choice behavior: A theoretical analysis_.  Courier Corporation, 2005.
  * Luo et al. [2023] Rachel Luo, Rohan Sinha, Yixiao Sun, Ali Hindy, Shengjia Zhao, Silvio Savarese, Edward Schmerling, and Marco Pavone.  Online distribution shift detection via recency prediction, 2023.
  * Luo et al. [2022] Wenjie Luo, Cheol Park, Andre Cornman, Benjamin Sapp, and Dragomir Anguelov.  JFP: Joint future prediction with interactive multi-agent modeling for autonomous driving.  In _6th Annual Conference on Robot Learning_ , 2022.  URL https://openreview.net/forum?id=Y42uoIekm5b.
  * Nakamura and Bansal [2023] Kensuke Nakamura and Somil Bansal.  Online update of safety assurances using confidence-based predictions.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , 2023.
  * Rahman et al. [2021] Quazi Marufur Rahman, Peter Corke, and Feras Dayoub.  Run-time monitoring of machine learning for robotic perception: A survey of emerging trends.  _IEEE Access_ , 9, 2021.  ISSN 2169-3536.
  * Seff et al. [2023] Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S Refaat, Rami Al-Rfou, and Benjamin Sapp.  Motionlm: Multi-agent motion forecasting as language modeling.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023.
  * Sinha et al. [2022] Rohan Sinha, Apoorva Sharma, Somrita Banerjee, Thomas Lew, Rachel Luo, Spencer M Richards, Yixiao Sun, Edward Schmerling, and Marco Pavone.  A system-level view on out-of-distribution data in robotics.  _arXiv preprint arXiv:2212.14020_ , 2022.
  * Stoler et al. [2023] Benjamin Stoler, Ingrid Navarro, Meghdeep Jana, Soonmin Hwang, Jonathan Francis, and Jean Oh.  Safeshift: Safety-informed distribution shifts for robust trajectory prediction in autonomous driving, 2023.
  * Sun et al. [2021] Liting Sun, Xiaogang Jia, and Anca D Dragan.  On complementing end-to-end human behavior predictors with planning.  _Robotics: Science and Systems_ , 2021.
  * Tang et al. [2022] Chen Tang, Wei Zhan, and Masayoshi Tomizuka.  Interventional behavior prediction: Avoiding overly confident anticipation in interactive prediction.  In _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022.
  * Thakkar et al. [2023] Neerja Thakkar, Karttikeya Mangalam, Andrea Bajcsy, and Jitendra Malik.  Adaptive human trajectory prediction via latent corridors, 2023.
  * Tian et al. [2022] Ran Tian, Liting Sun, Andrea Bajcsy, Masayoshi Tomizuka, and Anca D Dragan.  Safety assurances for human-robot interaction via confidence-aware game-theoretic human models.  In _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022.
  * Tolstaya et al. [2021] Ekaterina Tolstaya, Reza Mahjourian, Carlton Downey, Balakrishnan Vadarajan, Benjamin Sapp, and Dragomir Anguelov.  Identifying driver interactions via conditional behavior prediction.  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021.
  * Van Den Oord et al. [2017] Aaron Van Den Oord, Oriol Vinyals, et al.  Neural discrete representation learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Xu et al. [2023] Danfei Xu, Yuxiao Chen, Boris Ivanovic, and Marco Pavone.  Bits: Bi-level imitation for traffic simulation.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023.
  * Yang et al. [2022] Jie Yang, Ruijie Xu, Zhiquan Qi, and Yong Shi.  Visual anomaly detection for images: a systematic survey.  _Procedia Computer Science_ , 2022.
  * Yuan et al. [2021] Ye Yuan, Xinshuo Weng, Yanglan Ou, and Kris Kitani.  Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2021.
  * Zheng et al. [2020] Yinhe Zheng, Guanyi Chen, and Minlie Huang.  Out-of-domain detection for natural language understanding in dialog systems.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2020.
  * Zinkevich [2003] Martin Zinkevich.  Online convex programming and generalized infinitesimal gradient ascent.  In _Proceedings of the 20th international conference on machine learning (icml-03)_ , 2003.

##  X Supplementary Material

| High-Regret Scenarios | Low-Regret Scenarios
---|---|---
Fine-tuning Data | Col Cost | Col Severity | Regret | Col Cost | Col Severity | Regret
Base Model | 10.829 | 0.848 | 0.034 | 0.411 | 0.051 | 0.006
AvgReg | 4.286 (-60.4%) | 0.690 (-18.7%) | 0.014 (-58.7%) | 0.347 (-15.5%) | 0.039 (-24.9%) | 0.005 (-11.1%)
WstReg | 4.713 (-56.6%) | 0.520 (-38.7%) | 0.020 (-43.3%) | 0.565 (+37.4%) | 0.093 (+80.9%) | 0.006 (+2.02%)
AvgReg (+col, +div) | 14.505 (+34.0 %) | 0.960 (+13.2 %) | 0.048 (+39.4 %) | 0.437 (+6.4 %) | 0.082 (+60.3 %) | 0.006 (+9.1 %)
AvgReg (+div) | 7.477 (-30.9 %) | 0.654 (-23.0 %) | 0.014 (-60.6 %) | 0.366 (-10.83 %) | 0.0578 (+12.4 %) | 0.005 (-6.06 %)
TABLE III: Finetuning when ranking with average regret (2nd row, also reported
in Table II as High-regret-FT) vs worst single-timestep regret (3rd row). We
also ablated the choice of loss function on closed-loop performance by using
all of the original loss functions (4th row) and only removing the ego-
conditioned collision loss (5th row).

###  X-A Perception-enabled Robot Planner

We demonstrate a toy setting where our regret metric is able to identify task-
relevant detection failures when using a Conditional Variational Autoencoder
(CVAE)-based planner for navigating to a goal.

Problem Setup. A robot is deployed to reach a goal point $g_{1}$ (top flag in
Figure 4). It has dynamics
$\dot{\mathbf{x}}=[\dot{x},\dot{y},\dot{\theta}]^{\top}=[\cos\theta,\sin\theta,u]^{\top}$,
where $u$ is the control input. It is also equipped with a sensor
$P_{sense}(C)$ that returns $1$ if it detects an obstacle $\mathcal{O}$, and
$0$ otherwise. In the absence of an obstacle, it should always try to reach
$g_{1}$. However, if an obstacle is blocking the robot’s path to $g_{1}$, it
should instead navigate to an alternate goal location $g_{2}$ (right flag in
Figure 4).

To train our robot planner, we generated a dataset of
$|\mathcal{D}_{train}|=10000$ trajectories. For simplicity, the robot’s
starting state and the locations of $g_{1}$, $g_{2}$, and $\mathcal{O}$ (if
present) were the same across the entire training dataset. In each scene, the
demonstrated trajectories were obtained by applying a small amount of Gaussian
noise to a proportional controller guiding the robot to its goal location.

Figure 4: Generative Planner: Perception Failure. The robot’s training data
demonstrated that the robot should aim for the top flag in the absence of an
obstacle, but move toward the rightmost flag if an obstacle is detected. The
green flag denotes the robot’s perception-conditioned planned goal. Orange
obstacle denotes that there was actually an obstacle present, and grey
obstacle denotes that the robot falsely detected an obstacle. Our regret
metric correctly flags detection failures and measured nominal scenarios as
having low regret.

Inducing Generative Planner Failures. We wanted to identify which robot
behaviors (if any) were assigned high regret by our regret metric. To do this,
we deployed the robot four times. In two of the scenarios, there was the
obstacle $\mathcal{O}$ (top right, and bottom left of Figure 4). However, in
one of those two scenarios, we injected a perception failure (i.e., manually
setting the sensor observation to be incorrect) that prevented the robot from
detecting $\mathcal{O}$. We similarly deployed the robot in a setting with no
obstacle two times but injected a perception failure that falsely detected
$\mathcal{O}$ in one of the deployments (bottom right in Figure 4).

Trajectory Probabilities. To simulate the robot, we sampled 500 trajectories
conditioning on $P_{sense}(C)=1$ and 500 on $P_{sense}(C)=0$. For each
condition, we took the average of the 500 respective trajectories as an
exemplar for deployment. However, despite the ease of sampling from the CVAE,
finding the probability of a particular sample from a generative model is
known to be a difficult problem [22]. To approximate each exemplar
trajectory’s probability under the ground truth presence/absence of the
obstacle, we formed a per-timestep kernel density estimate (KDE) of the
conditional likelihood using the 1000 previously sampled trajectories (For
more details, see Figure 5 of [16]). To compute the trajectory probability, we
integrated the KDE for $\pm\delta$ around the most likely and executed action
for $\delta=0.1$ at each timestep. The trajectory’s regret was computed via
our metric as in Equation 5 by averaging over the per-timestep regret.

Results. As seen in Figure 4, the robot incurs very low regret whenever its
sensor was accurate. However, in both settings where the sensor failed, the
robot incurs a much higher regret. We highlight that nowhere in this
formulation (demonstrations nor generative planner) was there a reward
function that determined the robot’s behavior. Furthermore, this toy example
demonstrates that our general calibrated regret metric may have use beyond the
human-robot interaction setting; for example, for detecting and mitigating
failures in other autonomy modules beyond agent prediction (e.g., perception
module) in a task-aware manner [2].

###  X-B Regret Formulation and Mitigation Methods

In our formalism from Section IV, we defined the regret on a per-timestep
basis. For the experiments, we ranked the scenarios by the average regret
incurred over the 20-second interaction horizon $\hat{T}$. However, inspired
by the safety analysis literature that considers only the worst safety
violation incurred over the time horizon [4], we also experimented with
ranking scenarios by the worst single-timestep regret incurred over $\hat{T}$.
The closed-loop simulation results are shown in the second row of Table III.

Furthermore, during predictor fine-tuning described in Section VI, we chose to
remove the ego-conditioned collision loss and diversity loss during the fine-
tuning. We found that when fine-tuning with the ego-conditioned (EC) collision
loss, the predictor was unable to learn from the high-regret data. This is
because much of the high-regret data was induced by ego collisions with other
agents. Thus, the EC collision loss directly conflicts with learning from this
new data. The results of fine-tuning with EC collision loss and diversity loss
are shown in the third row of Table III. We also observed that training
without diversity loss leads to slightly better closed-loop performance. The
baseline fine-tuning without EC collision loss but with diversity loss is
shown in the fourth row of Table III.

###  X-C Comparison to Other Prediction Anomaly Metrics

In this section, we qualitatively and quantitatively compare the scenarios
deemed anomalous by our metrics against others. Namely, we compare our
calibrated regret metric against selecting for scenarios with the highest ADE
(a component-level metric), canonical regret (an uncalibrated system-level
metric), and the task-relevant failure detection (TRFD) method of Farid et al.
[11] (another uncalibrated metric). The two main differences between our
metric and TRFD are 1) Our approach calculates regrets on a per-timestep
basis, making it amenable to detecting failures of closed-loop motion plans.
For example, one critical assumption of TRFD is that anomaly detection must be
conducted with respect to an open-loop motion plan that the robot may not
deviate significantly from. However, the ability to replan is crucial when
deploying robots in highly interactive scenarios such as driving. 2) Our
metric measures the performance difference in hindsight between the actions
the robot took and other actions the robot could have taken. TRFD only
measures the difference between the predicted cost incurred by a single open-
loop motion plan and the costs it actually incurred. This means that in the
event that the robot executed a risky action that assigned sufficient
probability mass to a poor outcome, it would not be flagged as an interaction-
relevant prediction failure even if there was a poor outcome, so long as it
was not a $p$-quantile anomaly. Our method would still assign high regret to
this outcome, assuming another action that incurred lower costs was possible.

$\%$ Overlap of Scenarios | Ours | TRFD∗ | Canonical Regret | L2
---|---|---|---|---
Ours | 100 | 100 | 95 | 35
TRFD∗ |  | 100 | 100 | 100
Canonical Regret |  |  | 100 | 35
L2 |  |  |  | 100
TABLE IV: Percent overlap of the top 20 anomalous scenarios for all pairs of
metrics. Note that TRFD does provide a quantitative score to each scenario as
the other metrics do. However, in our experiments, TRFD flagged every scenario
as anomalous, so it has 100$\%$ overlap with every other metric.

We apply each metric to the initial closed-loop deployment data as described
in Section VI. For our metric, L2 error, and the canonical regret metrics, we
compare the top 20 scenes as scored by the metric. TRFD differs from the other
methods in that it does not assign scores to scenarios. It simply assigns a
label of whether or not that scenario was anomalous. The pairwise overlap
between each metric is shown in Table IV

####  X-C1 Calibrated Regret vs TRFD

We find that in our simulated closed-loop deployments, TRFD assigns every
scenario as being anomalous. We hypothesize that this is due to the
distribution shift between the Nuscenes data that the baseline Agentformer
model was trained on, and the simulated behavior of the BITS simulated agents.
This indicates that TRFD is sensitive to the quality of the baseline
prediction model when identifying anomalous scenarios.

####  X-C2 Calibrated Regret vs Canonical Regret

Our proposed calibrated regret metric has 95$\%$ overlap with canonical
regret. Each of the overlapping scenarios suffers from at least one timestep
of collision, indicating that both metrics can identify system-level failures.
The scenario with high canonical regret that was not deemed anomalous by our
calibrated metric was an instance of the robot simply driving near unmoving
vehicles and showing nominal, albeit slightly conservative behavior. The
individual scenario captured by our calibrated regret metric and not by the
canonical metric is, in contrast, still clearly a system-level failure where
the robot drives through stopped traffic at an intersection.

####  X-C3 Calibrated Regret vs L2 Error

Our proposed metric has only 35$\%$ overlap with the scenes with the highest
L2 error. Qualitatively, the scenarios with high L2 error tend to have many
agents or a single agent that the baseline Agentformer model predicts
extremely incorrectly. However, these scenarios with high L2 error only
overlap with our calibrated regret metric if the highly erroneous
mispredictions actually impact the ego agent. These scenarios tended to be in
scenarios where the robot was crossing an intersection or attempting to merge
lanes into crowded traffic.