# Wasserstein Distributionally Robust Control Barrier Function using
Conditional Value-at-Risk with Differentiable Convex Programming

Alaa Eddine Chriat111Ph.D student, Aerospace Engineering Department,
Mississippi State University, Starkville, MS 39759, USA. and Chuangchuang
Sun222Assistant professor, Aerospace Engineering Department, Mississippi State
University, Starkville, MS 39759, USA.

###### Abstract

Control Barrier functions (CBFs) have attracted extensive attention for
designing safe controllers for their deployment in real-world safety-critical
systems. However, the perception of the surrounding environment is often
subject to stochasticity and further distributional shift from the nominal
one. In this paper, we present distributional robust CBF (DR-CBF) to achieve
resilience under distributional shift while keeping the advantages of CBF,
such as computational efficacy and forward invariance.

To achieve this goal, we first propose a single-level convex reformulation to
estimate the conditional value at risk (CVaR) of the safety constraints under
distributional shift measured by a Wasserstein metric, which is by nature tri-
level programming. Moreover, to construct a control barrier condition to
enforce the forward invariance of the CVaR, the technique of differentiable
convex programming is applied to enable differentiation through the
optimization layer of CVaR estimation. We also provide an approximate variant
of DR-CBF for higher-order systems. Simulation results are presented to
validate the chance-constrained safety guarantee under the distributional
shift in both first and second-order systems.

##  1 Introduction

Autonomous systems are nowadays ubiquitous in the world, from daily life
assistance for household chores, and industrial productions, to space
explorations, which have significantly changed human society. For example, in
2020, there are 276 million motor vehicles registered in the U.S.[1], and the
global market of industrial robots was estimated at around $\$55$ billion and
is projected to surpass $\$165$ billion in 2028 [2]. However, there can often
be perturbations, noise, or malicious attacks during sensing and perception,
communication, and actuation. This issue is further exacerbated in
unstructured and dynamical environments, such as off-road vehicles, multi-
domain operations, and space explorations.

Consider a challenging rescue mission with a robot system after an earthquake.
Due to the damage to the cyber-physical infrastructure, communication with the
agent can be very limited/ corrupted. Also, the mission can have many
unanticipated scenarios, such as road closure. Moreover, in future
battlefields, military forces will be deployed in complex environments,
including multi-domain operations (MDO) against adversaries by combining the
traditional domains (e.g., air, maritime) and the information and
electromagnetic domains. In MDO, the goal is to achieve superiority by
"connecting distributed sensors, shooters, and data from all domains to joint
forces, enabling the coordinated exercise of authority to integrate planning
and synchronize convergence in time, space, and purpose" ([3], page 6). In
such operations, pervasive uncertainty/perturbation from the environments,
intentional/ stealthy/ deceptive adversarial attacks from the opponents, and
cyber-physical dysfunctionality can all possibly lead to mission failures.

In aerial flights and space explorations, such scenarios are also common. A
quadcopter, deployed in the open world for wild wire monitoring and disaster
relief, can often encounter unexpected gusts. Moreover, the CADRE (Cooperative
Autonomous Distributed Robotic Exploration) program from NASA JPL aims to
achieve collaborative, autonomous exploration, and formation sensing with an
integrative pipeline of sensing, perception, communication, computing, and
decision-making. Small rovers will be deployed, which will explore the lunar
surface, share data, and cooperatively make decisions to eventually accomplish
tasks, such as generating a digital elevation map. Moreover, the first Mars
helicopter Ingenuity was launched with Mars rovers (e.g., Perseverance) for
more effective Mars exploration. Intuitively, the condition is much more
restrictive on the Moon/ Mars, regarding atmospheric data, communication
quality, the toughness of terrain, and eventually the degradation of all
equipment (e.g., perception sensors) on board without maintenance. As a
result, the noise and uncertainty from sensing and perception, inter-vehicle
communication, planning and control, and actuation will bring significant
challenges to collaborative decision-making and coordination. In other words,
while a prior of those quantities of interest is available, such probabilistic
distribution can shift. In summary, decision-making in autonomous systems with
rigorous robustness and resilience is highly desired in many applications,
especially those with humans in the loop. To this end, our overarching goal is
to develop a distributed approach for distributionally robust control and
decision-making for autonomous systems with applications in space exploration.

Due to the importance of decision-making under distributional shift, there
exist many works of distributionally robust control for a single agent in the
framework of model predictive control (MPC), mostly also using chance
constraints such as conditional value-at-risk [4, 5, 6, 7, 8]. There are also
works based on approximate dynamic programming to achieve distributional
robustness [9]. However, those methods often require solving a complex
optimization problem, possibly in the minimax form, making it inapplicable for
online efficient control. Comparatively, control barrier functions(CBF [10])
have attracted much attention with diverse variants in different settings. One
of the reasons is their computational efficacy, which only requires
successively solving a convex quadratic programming regarding one time step
for general control affine systems. Moreover, the forward invariance of safety
satisfaction is also guaranteed via the control barrier condition. However,
exactly combining chanced-constrained distributional optimization and control
barrier function is not straightforward and we aim to bridge the gaps.

Consider a safety constraint $h(x,w)\leq 0$, where $x$ is the state and $w$ is
the noise subject to distributional shift. The chance-constrained safety
specification, $\mathrm{CVaR}\circ h(x,w)$, is often estimated by conditional
value at risk (CVaR) by solving optimization problems. Here “$\circ$" denotes
function composition. It comes to the first issue: estimating the CVaR under
distributional shift is nontrivial; it is by nature a tri-level problem: CVaR
optimization and the primal-dual optimization considering the distributional
constraint. To address this issue, we present an approximate approach to keep
tractability while avoiding over-conservatism, which eventually leads to
single-level convex programming. Subsequently, it comes to the forward
invariance and satisfaction of the $\mathrm{CVaR}\circ h(x,w)$ by control
barrier functions. Naturally, the control barrier condition (CBC) that
enforces the forward invariance of the safety constraints under distributional
shift is in the form of $\mathrm{CBC}\circ\mathrm{CVaR}\circ h(x,w)$. It is
known that CBC needs the differentiation of its argument, which in this case
$\mathrm{CVaR}\circ h(x,w)$. However, with $\mathrm{CVaR}(\bullet)$ as an
optimization layer, it is not immediately clear how to differentiate through
it to construct the CBC. To circumvent this issue, work in [11] estimates the
conditional value at risk of the control barrier condition, instead of the
CVaR estimate of the original constraint $h(x,w)$. That is to say, a relaxed
criterion is imposed by enforcing the chance-constrained control barrier
condition (i.e., $\left.\mathrm{CVaR}_{\alpha}\circ\mathrm{CBC}\circ
h(x,w)\right)$, instead of enforcing the forward variance of the real chance-
constrained safety constraint (i.e.,
$\mathrm{CBC}\circ{\mathrm{CVaR}}_{\alpha}\circ h(x,w)$ ). As a result,
$\mathrm{CVaR}_{\alpha}\circ\mathrm{CBC}\circ$ $h(x,w)$ only needs to
differentiate $h(x,w)$ itself, which is much easier compared to the
differentiation through the optimization layer $\mathrm{CVaR}_{\alpha}\circ
h(x,w)$ over $x$.

###  1.1 Contributions

To enable distributional robustness while keeping the advantages of the
control barrier function, we make the following contributions to bridge the
aforementioned gaps with distributionally robust control barrier functions.

  * •

We present a formulation to simply the tri-level optimization problem for
estimating the $\mathrm{CVaR}$ into a single-level convex program, then use
differentiable convex programming to enable differentiation through the
optimization layer $\mathrm{CVaR}\circ h(x,w)$

  * •

We construct the control barrier function to enforce the forward invariance of
the chance-constrained safety constraint (i.e., $\mathrm{CVaR}\circ h(x,w)$).
As a result, it makes it possible to enforce
$\mathrm{CBC}\circ\mathrm{CVaR}_{\alpha}\circ h(x,w)$ that can capture the
essence of the problem and eventually guarantee the forward invariance and the
satisfaction of the safety specification. We also provide an approximate
method for higher-order systems.

  * •

We present simulation results where the strength of the proposed
distributionally robust control barrier function (i.e., remaining safe under
distributional shift) in stochastic environments is demonstrated compared to
the vanilla CBFs.

###  1.2 Related Works

There have been extensive existing works[12, 13] studying robust learning and
control under perturbations, often formulated as $l_{p}$ balls as
$\left\|x-x_{0}\right\|_{p}\leq\epsilon$, with $x$ and $x_{0}$ as the quantity
of interest and its nominal value respectively and $\epsilon$ as the radius of
the $l_{p}$ ball. More specifically, the perturbations in observations [14],
action [15], model [16], and those in the context of safe reinforcement
learning (RL)[17, 18] have been well studied. In multi-agent RL, model
uncertainty[19], adversary agents [19, 20], and beyond have been considered to
achieve robustness. Correspondingly, the distributional shift defined as
$x\sim p(x),d(p(x),p_{0}(x))\leq\epsilon$, where $p(x)$ and $p_{0}(x)$ are the
real and nominal probability distributions of $x$ with $d(\bullet,\bullet)$ as
a distance measurement between two probability distributions and $\epsilon$
the threshold. Compared to the $l_{p}$ ball perturbation, the distributional
shift admits a much larger space to explore to find the worst adversarial
behaviors, which the agent is expected to mitigate in a principled way. For
distributionally robust single-agent learning and control under distributional
shift, model-based approaches[4] such as approximate dynamic programming [9,
5] and model predictive control [6, 7, 8] have been proposed, with a chance-
constrained criterion under a Wasserstein metric. Specifically, in the
framework of model predictive control, many existing works of distributionally
robust control use conditional value-at-risk [4, 5, 6, 7, 8]. In the model-
free regime, one line of work is to generate environments/ tasks with
distributional shifts, for policy training to achieve robustness[21, 22, 23].
Moreover, to balance the worst-case (robustness) and average performance, [24]
trains policies over task groups by adding regularization to the worst
possible outcomes. Additionally, with the presence of adversaries, the
minimax/ bi-level optimization formulation [25] is often employed for worst-
case robustness, which is challenging to solve.

###  1.3 Organization

To ensure that the research provides a comprehensive understanding of the
topic at hand, this research paper is organized into four main sections. The
first section introduces the background and significance of the research,
highlighting the motivation behind the study and its relevance. The rest of
this paper is organized as follows. In Section II, we revisit some fundamental
concepts and definitions of control barrier functions, differentiable convex
programming, and distributionally robust optimization. In section III, we
start by developing the optimization problem to estimate $\mathrm{CVaR}$ under
distributional shift and calculate the relevant gradients. Then we construct
the control barrier function of the $\mathrm{CVaR}$ for first-order systems
and provide an approximate method for higher-order systems. In section IV, we
present simulation results for first-order and second-order systems
highlighting the advantage of using the Distributionally Robust Control
Barrier Function(DR-CBF). Finally, section V summarizes our research, and
discusses some future ideas that warrant exploration.

##  2 Preliminary

###  2.1 Control Barrier Function

Control Barrier Functions (CBFs) are mathematical tools used to ensure the
safety and stability of dynamic systems. A CBF is a function that maps the
current state of the system to a value that measures how far the system is
from violating the desired safety constraint. The control law is then designed
to enforce the CBF such that it remains within the safe region. The control
law is typically designed using a Lyapunov-based approach, where a Lyapunov
function is chosen to drive the system to the desired behavior and solved in
the form of a Quadratic Program (QP), while The barrier function is
incorporated into the QP to guarantee the desired safety specifications.
Mathematically, consider the nonlinear control-affine system:

$\dot{x}(t)=f(x(t))+g(x(t))u(t)$ (1)

where $f$ and $g$ are globally Lipschitz, $x\in\mathbb{R}^{n}$ and
$u\in\mathbb{R}^{m}$ are the states and control inputs, respectively,
constrained in closed sets, with initial condition $x(t_{0})=x_{0}$.

###### Definition 1

[10]$h:\mathbb{R}^{n}\rightarrow\mathbb{R}$ is a barrier function for the set
$C=\left\\{x\in\mathbb{R}^{n}:h(x)\geqslant 0\right\\}$ if $\exists$
$\mathcal{K}$ function $\alpha(\bullet)$ such that:

$\begin{gathered}\sup_{u\in U}[L_{f}h(x)+L_{g}h(x)u+\alpha(h(x))]\geqslant
0\\\ \inf_{\text{int}(C)}[\alpha(h(x))]\geqslant 0\text{ \quad and
\quad}\lim_{\partial C}\alpha(h(x))=0\end{gathered}$ (2)

Because not all systems are first-order in inputs, we can use higher-order
control barrier functions to constrain higher-order systems.

###### Definition 2

[26]For the non linear system (1) with the $m^{th}$ differentiable function
$h(x)$ as a constraint, we define a sequence of functions $\psi_{i}$ with
$i\in\\{1,2,...,m\\}$, starting from $\psi_{0}=h(x)$:

$\psi_{i}(x,t)=\dot{\psi}_{i-1}(x,t)+\alpha_{i}\left(\psi_{i-1}(x,t)\right)$
(3)

the function $h(x)$ is a high order control barrier function if $\exists$
$\mathcal{K}$ functions $\alpha_{i}(\bullet)$ such that:

$\psi_{m}(x,t)\geqslant 0$ (4)

By solving the following Quadratic Program, leveraging the power of CBF, we
can enforce safety constraints, maintain stability, and prevent undesirable
behavior

$\displaystyle\min_{u\in[\underline{u},\bar{u}]}$ $\displaystyle J(x,u)$ (5)
s.t. $\displaystyle\frac{\partial h(x)}{\partial
x}(f(x)+g(x)u)+\kappa(h(x))\geq 0$

###  2.2 Differentiable Convex Programming

Differentiable convex programming is a powerful technique that enables the
computation of gradients for the objective function of an optimization problem
with respect to its parameters. This is achieved by applying matrix
differentiation to the Karush-Kuhn-Tucker (KKT) conditions. A notable example
of a differentiable optimization method is OPTNET [27], which incorporates
differentiable optimization problems within the architecture of neural
networks. During training, the gradients of the objective function are
computed and back-propagated through the network. In a broader sense, this
methodology can be applied to differentiate through disciplined convex
programs [28] by initially mapping them into cone programs [29], computing the
gradients, and subsequently mapping back to the original problem. A common
application of differentiable programming is learning the constraints of an
optimization problem, such as convex polytopes or ellipsoid projections. The
major advantage of differentiable optimization methods, such as OPTNET, lies
in their ability to optimize a wide range of challenging convex objectives
that are typically difficult to handle using traditional optimization
approaches. It is worth noting that the convex quadratic program (QP) in
equation (5) can be differentiated through the KKT conditions [27], which
serve as equivalent conditions for global optimality. According to the KKT
conditions, at the optimal solution, the gradient of the Lagrangian function
with respect to the program’s input and parameters must be zero. Consequently,
by taking the partial derivative of the Lagrangian function with respect to
the input and extending it through the chain rule to the program’s parameters,
their gradients can be obtained. We have integrated differentiable
optimization using the cvxpylayers package
333https://github.com/cvxgrp/cvxpylayers which is an extension to the cvxpy
package with an affine-solver-affine (ASA) approach. The ASA consists of
taking the optimization problem’s objective and constraints and mapping them
to a cone program. For a generalized QP:

$\displaystyle\min_{x}$ $\displaystyle\ \ \frac{1}{2}x^{T}Qx+q^{T}x$ (6) s.t.
$\displaystyle\ \ Ax=b$ $\displaystyle\ \ Gx\leq h,$

we can write the Lagrangian of the problem as:

$L(z,\nu,\lambda)=\frac{1}{2}z^{T}Qz+q^{T}z+\nu^{T}(Az-b)+\lambda^{T}(Gz-h)$
(7)

where $\nu$ are the dual variables on the equality constraints and
$\lambda\geq 0$ are the dual variables on the inequality constraint. Using the
KKT conditions for stationarity, primal feasibility, and complementary
slackness.

$\displaystyle Qz^{\star}+q+A^{T}\nu^{\star}+G^{T}\lambda^{\star}$
$\displaystyle=0$ (8) $\displaystyle Az^{\star}-b$ $\displaystyle=0$
$\displaystyle D\left(\lambda^{\star}\right)\left(Gz^{\star}-h\right)$
$\displaystyle=0$

By differentiating these conditions, we can shape the Jacobian of the problem
as follows.

$\left[\begin{array}[]{l}d_{z}\\\ d_{\lambda}\\\
d_{\nu}\end{array}\right]=-\left[\begin{array}[]{ccc}Q&G^{T}D\left(\lambda^{\star}\right)&A^{T}\\\
G&D\left(Gz^{\star}-h\right)&0\\\
A&0&0\end{array}\right]^{-1}\left[\begin{array}[]{c}\left(\frac{\partial\ell}{\partial
z^{\star}}\right)^{T}\\\ 0\\\ 0\end{array}\right]$ (9)

Furthermore, via chain rule, we can get the derivatives of any loss function
of interest regarding any of the parameters in the QP.

###  2.3 Distributionally Robust Optimization and Conditional Value at Risk

Distributionally Robust Optimization (DRO) is an approach to optimization
under uncertainty that aims to find solutions that are robust against a wide
range of possible probability distributions. Unlike traditional optimization
methods that assume a known probability distribution for uncertain parameters,
DRO takes a more cautious approach by considering a set of possible
distributions and optimizing for the worst-case scenario within that set. In
DRO, the uncertain parameters are typically modeled as random variables with
unknown distributions. The goal is to find a solution that performs well
across all possible distributions within a certain ambiguity set. The
ambiguity set represents the range of possible distributions and is defined
based on certain metrics, such as statistical moments or the Wasserstein
metric. The key idea in DRO is to find a solution that minimizes the expected
cost or maximizes the expected reward under the worst-case distribution within
the ambiguity set. This approach provides a robust solution that performs well
regardless of the actual distribution of the uncertain parameters. DRO has
applications in various domains, including operations research, finance, and
machine learning. It can be used to optimize decisions in settings with
uncertain data, such as supply chain management, portfolio optimization, or
predictive modeling with uncertain inputs. However, solving DRO problems can
be challenging due to the increased complexity introduced by worst-case
optimization. The optimization problem typically becomes non-convex and
computationally demanding. Various techniques, such as convex relaxations,
scenario approximations, or sample-based methods, are used to handle the
computational challenges associated with DRO. In general, the value at risk of
a random quantity $h(x,w)$ (with $h$ as the shorthand notation) with a
confidence level of $\alpha$ is :

$\operatorname{VaR}_{\alpha}(h):=\min\\{\eta\in\mathbb{R}\mid\mathbb{P}(h\leq\eta)\geq\alpha\\}$
(10)

which can be interpreted as the worst-case scenario risk with probability
$\alpha$. Due to the complexity of solving for the VAR, we define a more
efficient version, the conditional value at risk which can be formulated as
the following convex program:

$\mathrm{CVaR}_{\alpha}(h):=\min_{\eta\in\mathbb{R}}\mathbb{E}\left[\eta+\frac{(h-\eta)_{+}}{1-\alpha}\right]$
(11)

which can be subsequently reformulated into a tractable linear program:

$\displaystyle\mathrm{CVaR}_{\alpha}\left(h\right)\approx$
$\displaystyle\min_{\eta_{i},s_{i}}\quad\eta_{i}+\frac{1}{(1-\alpha)N_{s}}\sum_{m=1}^{N_{s}}s_{i}^{m}$
(12) $\displaystyle\text{ s.t. }\quad h-\eta_{i}\leq s_{i}^{m},\quad\forall
m\in\mathbb{I}_{1:N_{s}}$ $\displaystyle\quad\quad\quad 0\leq
s_{i}^{m},\quad\forall m\in\mathbb{I}_{1:N_{s}}$

where $I\in\mathbb{N}$ is the set of estimated samples. In order to take into
account the unmeasured distributions, we introduce the Wasserstein metric and
build an ambiguity set. This enables solving the problem for the worst-case
scenario. For all distributions
$\mathcal{P}_{1},\mathcal{P}_{2}\in\mathcal{P}(\mathbb{W})$ we can define the
Wasserstein metric as:

$d_{\mathrm{W}}\left(\mathcal{P}_{1},\mathcal{P}_{2}\right):=\min_{\kappa\in\mathcal{P}\left(\mathbb{W}^{2}\right)}\left\\{\int_{\mathbb{W}^{2}}\left\|w_{1}-w_{2}\right\|\mathrm{d}\kappa\left(w_{1},w_{2}\right)\right.\left.\times\mid\Pi^{l}\kappa=\mathcal{P}_{l},l=1,2\right\\}$
(13)

integrating the Wasserstein metric into the $\mathrm{CVaR}$ linear program to
optimize over the whole ambiguity set results in the following optimization
problem [6]:

$\sup_{\mathcal{P}\in\mathbb{D}}\mathrm{CVaR}_{\alpha}^{\mathcal{P}}(h)=\inf_{\lambda\geq
0}\left\\{\lambda\epsilon+\frac{1}{N_{s}}\sum_{m=1}^{N_{s}}\sup_{w\in\mathbb{W}}\left\\{[h-\eta]_{+}-\lambda\left\|w-w^{m}\right\|\right\\}\right\\}$
(14)

Overall, distributionally robust optimization provides a principled approach
to decision-making under uncertainty, offers robustness guarantees, and can
lead to more reliable and resilient solutions in uncertain environments.

##  3 Chance Constrained Distributionally Robust Control Barrier Functions
with a Wasserstein Metric

###  3.1 The Estimation of Conditional Value-at-Risk under Distributional
Shifts: A Simplified Formulation

Consider a general dynamical system in the following form

$\dot{x}=f\left(x,u\right)$ (15)

where
$x\in\mathbb{R}^{n},u\in\mathbb{R}^{m},f:\mathbb{R}^{n+m}\rightarrow\mathbb{R}^{n}$
are the state, control input, and the dynamical transition function,
respectively. Naturally, in many scenarios, there arise safety constraints,
such as obstacle avoidance formulated as $h(x)\leq 0$. Without loss of
generality, only one constraint is considered here and our approach can be
easily extended to multiple-constraint cases.

As the dynamics in (15) are deterministic, we consider stochastic constraints
with additive noise $h(x,w)\leq 0$. As a result, it is desirable to satisfy
the constraint with as high a probability as possible, with a prerequisite
first to estimate the worst case $h(x,w)$ under stochasticity. Then value-at-
risk and its more tractable approximation conditional value-at-risk (CVaR)
[30] are often used to measure the risks. Mathematically,
$\lim_{\alpha\rightarrow 1}\mathrm{CVaR}_{\alpha}(h(x,w))\leq$ 0 means that
the constraint is satisfied with a probability of at least $\alpha$ (i.e.,
$\mathbb{P}(h(x,w)\leq 0)\geq\alpha$ ). With $N_{s}$ independent and
identically distributed (i.i.d.) samples of the disturbance
$\left\\{w^{m}\right\\}_{m=1}^{N_{s}}$, we can get the corresponding samples
of $x$ at the current time step based on the dynamics (15) and the control
input $u$ of last time-step in an online setting. Time dependency is omitted,
as it is applicable for all time instances. Then the $\mathrm{CVaR}_{\alpha}$
can be estimated by solving the following linear programming [30] with
auxiliary variables $s^{m}$

$\min_{\eta,s}\left\\{\eta+\frac{1}{(1-\alpha)N_{s}}\sum_{m\in\left[N_{s}\right]}s^{m}\mid\text{
s.t. }h\left(x^{m},w^{m}\right)-\eta\leq s^{m},s^{m}\geq 0,\forall
m\in\left[N_{s}\right]\right\\},$ (16)

where $\left[N_{s}\right]=1,\ldots,N_{s}$ is the index set for the samples.

To estimate the risk constraint $\mathrm{CVaR}_{\alpha}(h(x,w))$ via (16),
there often requires many samples (i.e., large $N_{s}$ ) of the random
variables, such as sample average approximation (SAA [31]) in data-driven
stochastic optimization. However, collecting samples of the disturbance for
the physical robotic systems is too restrictive and unsafe, especially with
humans in the loop. As such, limited samples will usually not be able to
capture the proper distribution of the stochastic variable, admitting an
ambiguity set. In other words, the actual distribution might shift from the
estimated one. To effectively ensure robustness under such a distributional
shift, Distributionally Robust Optimization (DRO [32, 33, 34] will be employed
to solve the stochastic optimization problem by considering the worst case
within the ambiguity set. Intuitively, the metrics to measure the distance
between two probabilistic distributions are used to parameterize the ambiguity
set, including the Kullback-Leibler divergence [35] and the Wasserstein metric
[36]. Here the latter is adopted, as distributional robust optimization
provides a probabilistic guarantee of out-of-sample performance under a
Wasserstein metric [37]. In the following, we will show how
$\mathrm{CVaR}_{\alpha}$ can be estimated with the Wasserstein ambiguity set.

Denote $p_{0}(w)$ as the empirical distribution of the random variables $w$
estimated from samples $\left\\{w^{m}\right\\}_{m=1}^{N_{s}}$. Then the
ambiguity set of the perturbed distribution $p(w)$ from the nominal
distribution $p_{0}(w)$ under a Wasserstein metric [36] is expressed as
$\mathcal{P}=\left\\{p(w)\mid
W_{d}\left(p(w),p_{0}(w)\right)\leq\rho\right\\}$, with
$W_{d}(\bullet,\bullet)$ as the Wasserstein distance and $\rho$ as the
threshold of such a shift. To satisfy the constraint, we have the following
worst-case scenario in the ambiguity set as
$H(x):=\sup_{p\in\mathcal{P}}\operatorname{CVaR}_{\alpha}^{p}(h(x,w))\leq 0$.
Then based on the definition of $\mathrm{CVaR}_{\alpha}$, it can be further
reformulated as

$\displaystyle\sup_{p\in\mathcal{P}}\operatorname{CVaR}_{\alpha}^{p}(h(x,w))\leq$
$\displaystyle\min_{\eta}\left\\{\eta+\frac{1}{1-\alpha}\sup_{p}\left\\{[h(x,w)-\eta]_{+}-\lambda
W_{d}\left(p(w),p_{0}(w)\right)\right\\}\right\\}$ (17) $\displaystyle=$
$\displaystyle\min_{\eta}\left\\{\eta+\frac{1}{1-\alpha}\frac{1}{N_{s}}\sum_{m\in\left[N_{s}\right]}\sup_{w}\left\\{[h(x,w)-\eta]_{+}-\lambda\left\|w-w^{m}\right\|\right\\}\right\\}$

where $[\bullet]_{+}=\max(0,\bullet)$. In the first inequality, we take the
ambiguity set constraint as a penalty with $\lambda>0$ to reduce one layer of
the optimization problem (i.e., eliminating the minimization over $\lambda$).
However, the inner maximization is on the infinite-dimensional probability
measure of $w$ and is hence intractable. With Kantorovich duality [37], it is
further equivalently reformulated as an optimization problem on the finite
space of $w$ in the equality. Note that $[\bullet]_{+}$can be transformed as
linear constraints with slack variables as (16), resulting in an equivalent
convex quadratic programming (QP) problem from (17). Note that the problem in
(17) will lead to a bi-level optimization problem if combined with optimal
control design. Due to the difficulty of solving bi-level optimization, most
existing works plainly list the constraints in (17) by removing the
minimization over $\eta$ in the overall optimal control problem. This lead to
conservatism (e.g., replacing $min_{x}f(x)\leq 0$ by $f(x)\leq 0$). Here we
will address this issue principally to keep tractability while mitigating
over-conservatism, by efficiently solving the supremum problem in (17) to
further reduce it from a bi-level problem to a single-level convex
programming. To achieve this, it is further assumed that the noise is additive
as $h(x,w)=h(x)+w$, where $w\in\mathbb{R}^{1}$ is in a closed convex set
subject to a Gaussian distribution. This can lead to the analytical solution
for the supremum problem. with two cases considering the $[\bullet]_{+}$
operator.

Case 1: $h(x)+w-\eta\leq 0$.
In this case, the optimal solution for the
$\sup_{w}\left\\{-\lambda\left\|w-w^{m}\right\|\right\\}$ regarding $w$ is
achieved with $w^{*}=w^{m}$. Hence, it results in the following linear
program:

$\displaystyle\sup_{p\in\mathcal{P}}\operatorname{CVaR}_{\alpha}^{p}(h(x,w))=\min_{\eta}$
$\displaystyle\left\\{\eta+\frac{1}{1-\alpha}\frac{1}{N_{s}}\sum_{m\in\left[N_{s}\right]}s^{m}\right\\}$
(18) s.t. $\displaystyle h(x)+w^{*}-\eta\leq s^{m},$ $\displaystyle
h(x)+w^{*}-\eta\leq 0,$ $\displaystyle 0\leq s^{m}.$

In this case, $s^{m}$ will be zeros (due to the minimization and nature of the
first two constraints). Then the estimation of
$\sup_{p\in\mathcal{P}}\operatorname{CVaR}_{\alpha}^{p}(h(x,w))$ will be the
worst case of $h(x)+w^{m}$, intuitively leading to conservatism under the
distributional shift of $w$.

Case 2: $h(x)+w-\eta\geq 0$.
In this case, the optimal solution for the linear program
$\sup_{w}h(x,w)-\eta-\lambda\left\|w-w^{m}\right\|$ regarding $w$ is achieved
at the vertices of the polytope feasible sets formulated via the linear
constraints, including the bounds. Therefore, the value $w^{m}$, or the bounds
of the set $\underline{w},\bar{w}$ are possible solutions for the supremum
operator. As a result, we can arrive at the following linear program:

$\displaystyle\sup_{p\in\mathcal{P}}\operatorname{CVaR}_{\alpha}^{p}(h(x,w))=\min_{\eta}$
$\displaystyle\left\\{\eta+\frac{1}{1-\alpha}\frac{1}{N_{s}}\sum_{m\in\left[N_{s}\right]}(s^{m}+L^{m})\right\\}$
(19) s.t. $\displaystyle\left\\{\begin{array}[]{l}h(x)+\bar{w}-\eta\leq
s^{m},\\\ h(x)+\bar{w}-\eta-\lambda(\bar{w}-w^{m})\leq L^{m},\\\
h(x)+\bar{w}-\eta\geq 0,\\\ \end{array}\right.$
$\displaystyle\left\\{\begin{array}[]{l}h(x)+\underline{w}-\eta\leq s^{m},\\\
h(x)+\underline{w}-\eta+\lambda(\underline{w}-w^{m})\leq L^{m},\\\
h(x)+\underline{w}-\eta\geq 0,\\\ \end{array}\right.$
$\displaystyle\left\\{\begin{array}[]{l}h(x)+w^{m}-\eta\leq s^{m},\\\
h(x)+w^{m}-\eta\leq L^{m},\\\ h(x)+w^{m}-\eta\geq 0,\end{array}\right.$

For each potential solution, we get a set of constraints to satisfy in (19).
For the term with absolute value operator, i.e., $-\left\|w-w^{m}\right\|$, it
is further reduced as $(\underline{w}-w^{m})$ and $-(\bar{w}-w^{m})$, with
$\underline{w}$ and $\bar{w}$ as the lower and upper bound of $w$,
respectively. Also, since $h(x)+\underline{w}-\eta\geq 0$ and
$h(x)+\underline{w}-\eta\leq s^{m}$, the constraint $0\leq s^{m}$ becomes
trivial and thus can be removed. Note that the problem in (17) is only to
estimate the risk constraint and it will lead to a bi-level optimization
problem if combined with optimal control design. Due to the difficulty of
solving bi-level optimization, most existing works just plainly list the
objective (without the minimization) and constraints in (17) as extra
constraints over $\eta$ in the overall optimal control problem. This lead to
conservatism (e.g., replacing $\min_{\eta}f(\eta)\leq 0$ by $f(\eta)\leq 0$).
Here we will address this issue principally to keep tractability while
mitigating over-conservatism by integrating it with control barrier functions.
As a result, we will present how we can use the optimal value of
$\sup_{p\in\mathcal{P}}\operatorname{CVaR}_{\alpha}^{p}(h(x,w))$ and its
derivative to construct the control barrier conditions in the following
section.

###  3.2 Distributionally Robust Control Barrier Functions via Differentiable
Convex Programming

Consider the following non-linear control-affine system $\dot{x}=f(x)+g(x)u$,
where $f$ and $g$ are locally Lipschitz, $x\in D\subset\mathbb{R}^{n}$ is the
state and $u\in U\subset\mathbb{R}^{m}$ is the set of admissible inputs. The
safety set is defined as $\mathcal{C}=\left\\{x\in D\subset\mathbb{R}^{n}\mid
h(x,w)\leq 0\right\\}$ with $\mathcal{C}\subset D$. Then $h$ is a zeroing
control barrier function (CBF) [10] if there exists an extended class-
$\kappa_{\infty}$ function $\kappa$ such that for the above control system

$\sup_{u\in U}\left(L_{f}h(x,w)+L_{g}h(x,w)u+\kappa(h(x,w))\right)\leq
0,\forall x\in D$ (20)

where $L_{f}h(x)=\left(\frac{\partial h(x,w)}{\partial x}\right)^{T}f(x)$ is
the Lie derivative. Note that $h(x,w)\leq 0$, instead of $h(x,w)\geq 0$ in the
CBF literature, defines the safety set for consistency here. As such, it is "
$\leq$ " rather than " $\geq$ " in (20). The control barrier condition (CBC)
in (20) will ensure the forward invariance of the constraint $h(x,w)$ and has
been extensively studied with many variants. Forward invariance means that the
violation of the safety constraint will only become smaller and smaller if
starting outside the safety set, and will remain inside otherwise.

In terms of distributionally robust CBF, work in [11] estimates the
conditional value at risk of the control barrier condition in (20), instead of
the CVaR estimate of the original constraint $h(x,w)$ as in (17). That is to
say, the former is applying a relaxed criterion by enforcing the chance-
constrained control barrier condition (i.e.,
$\left.\mathrm{CVaR}_{\alpha}\circ\mathrm{CBC}\circ h(x,w)\right)$, instead of
enforcing the forward variance of the real chance-constrained safety
constraint (i.e., $\mathrm{CBC}\circ{\mathrm{CVaR}}_{\alpha}\circ h(x,w)$ ).
Here we use " $\circ$ " to denote function composition to avoid many layers of
parentheses. However, while $\mathrm{CBC}\circ\mathrm{CVaR}_{\alpha}\circ
h(x,w)$ can capture the essence of the problem, it brings new challenges. As
to differentiate through the optimization layer $\mathrm{CVaR}_{\alpha}\circ
h(x,w)$ over $x$ (non-trivial), while
$\mathrm{CVaR}_{\alpha}\circ\mathrm{CBC}\circ$ $h(x,w)$ only needs to
differentiate $h(x,w)$ itself (much easier). we combine distributionally
robust control with control barrier functions to enforce the forward
invariance of the risk estimate $\mathrm{CVaR}_{\alpha}\circ h(x,w)$. As
discussed before, the estimate of $\mathrm{CVaR}_{\alpha}(h(x,w))$ in (17) is
a convex quadratic program, which we need to differentiate through over $x$ to
construct the control barrier condition in (20). Leveraging recent advances in
differentiable convex optimization [38, 39, 40], we can formulate our problem
as a disciplined parameterized program and use the cvxpylayers package to map
our problem into a cone program and differentiate the KKT conditions at the
optimal solution to get the partial derivatives of the solution of the
$\mathrm{CVaR}$ with respect to the problem’s parameters; see details in
Section 2.2. As a result, we are able to calculate
$\frac{\partial\mathrm{CVaR}_{\alpha}(h(x,w))}{\partial x:}$ and construct the
control barrier condition as follows

$\min_{u}\left\\{J\left(x,u\right)\mid\text{ s.t.
}\frac{\partial\mathrm{CVaR}_{\alpha}(h(x,w))}{\partial
x}\left(f\left(x\right)+g\left(x\right)u\right)+\kappa\left(\operatorname{CVaR}_{\alpha}(h(x,w))\right)\leq
0,\forall m\in\left[N_{s}\right]\right\\},$ (21)

where $J\left(x,u\right)$ is the loss function, such as reference trajectory
tracking (e.g., $\left\|x-x_{ref}\right\|_{2}^{2}$ ), a Lyapunov function for
goal-reaching, optimal fuel consumption (e.g., $\left\|u\right\|_{2}^{2}$ ),
etc. Problem (21) is often a convex quadratic programming, with a quadratic
loss function $J\left(x,u\right)$ as exemplified and a linear CBC constraint
inheriting from the general CBF context in (20). Note that the dynamics are
not explicitly included in the optimization, and thus not variables, as CBF
only considers a single time-step forward. The single-step system propagation
will follow the dynamics in (15) to get the next state with the control input
$u$ from (21). In this way, we implicitly integrate the optimization problem
in (17) for risk estimate as part of the optimal control problem (21), rather
than just listing the constraints therein. Algorithm1 summarizes the steps for
the distributionally robust control barrier function.

Algorithm 1 Distributionally Robust Control Barrier Function

1:Require: $\alpha,\lambda,$ samples

2:while Not converged do

3: Initialize $x$, $w^{N_{s}}$, $h(x,w)$

4: Calculate $\mathrm{CVaR}$ using (18) or (19)

5: Get $\frac{\partial\mathrm{CVaR}_{\alpha}(h(x,w))}{\partial x}$ by
backpropagating through the QP using differentiable convex programming
(Section 2.2)

6: Solve for optimal input $u^{*}$ using the QP in (21) with $\mathrm{CVaR}$
and its derivative $\frac{\partial\mathrm{CVaR}_{\alpha}(h(x,w))}{\partial x}$

7: Update $x\leftarrow x+f(x,u)\Delta t$

8: if $x==x_{\text{final}}$ then

9: Break

10: end if

11:end while

###  3.3 High-Order System: an Approximate Method

Higher-order control barrier functions are an extension of traditional control
barrier functions used for higher-order systems. HOCBF incorporates higher-
order derivatives of the system’s states which allows the consideration of
more complex dynamics. Hence, handling more intricate safety requirements and
enabling systems to avoid undesirable behaviors. Using the same approach for a
higher-order system proves challenging due to the need for two successive
differentiation through the Linear programs (18)and (19). However, we can get
a good approximation by differentiating $h(x,w)$ analytically first to get the
first layer of the HOCBF 2, then we can calculate the
$\mathrm{CVaR}(\dot{h}(x,w)$ and backpropagate to get the gradients. We can
construct the first Barrier analytically:

$\psi(x,w)=\dot{h}(x,w)+\mathrm{\kappa}(\dot{h}(x,w))$ (22)

then we can calculate the $\mathrm{CVaR}$ using:

$\sup_{p\in\mathcal{P}}\operatorname{CVaR}_{\alpha}^{p}(\psi(x,w))=\min_{\eta}\left\\{\eta+\frac{1}{1-\alpha}\frac{1}{N_{s}}\sum_{m\in\left[N_{s}\right]}\sup_{w}\left\\{[\psi(x,w)-\eta]_{+}-\lambda\left\|w-w^{m}\right\|\right\\}\right\\}$
(23)

It can be further integrated t in the optimal control problem as in (21) in
the following way

$\min_{u}\left\\{J\left(x,u\right)\mid\text{ s.t.
}\frac{\partial\mathrm{CVaR}_{\alpha}(\psi(x,w))}{\partial
x}\left(f\left(x\right)+g\left(x\right)u\right)+\kappa\left(\operatorname{CVaR}_{\alpha}(\psi(x,w))\right)\leq
0,\forall m\in\left[N_{s}\right]\right\\}$ (24)

Although the resulting CBF is an approximation as
$\mathrm{CBC}\circ\mathrm{CVaR}_{\alpha}\circ\mathrm{CBC}\circ h(x,w)$ instead
of $\mathrm{CBC}\circ\mathrm{CBC}\circ\mathrm{CVaR}_{\alpha}\circ h(x,w)$, its
performance was comparable to the first order DR-CBF. We defer the exact
method for high-order systems, which requires higher-order differentiable
convex programming techniques, to future works.

##  4 Simulations and Results

In this section, we assess the performance of DR-CBF in several scenarios
involving first-order systems and second-order systems. Our approach is then
compared to a conventional CBF approach while keeping all other configurations
identical. The advantages of the proposed DR-CBF on maintaining safety under
distributional shift are presented. Note that in both these simulations, we
use the first approximation of $\mathrm{CVaR}$(18), for its computational ease
and compromise between optimality and robustness.

###  4.1 Dubins Car: A First-Order System Case Study

To evaluate the DR-CBF, we used the first-order Dubins car environment with
the following kinematics:

$\left(\begin{array}[]{c}\dot{x}\\\ \dot{y}\\\
\dot{\theta}\end{array}\right)=\left[\begin{array}[]{ccc}\cos\theta&-\sin\theta&0\\\
\sin\theta&\cos\theta&0\\\
0&0&1\end{array}\right]\left(\begin{array}[]{c}v_{x}\\\ v_{y}\\\
\omega\end{array}\right),$ (25)

where $v_{x}$ and $v_{y}$ are the velocities along the $x$ and $y$ axes of the
car’s frames, $\theta$ is the heading angle, and $\omega$ is the angular
velocity.

In order to go from an initial state $r_{0}=[x_{0},y_{0},\theta_{0}]^{T}$ to a
final state $r_{f}=[x_{f},y_{f},\theta_{f}]^{T}$, we use a Lyapunov function
$\frac{1}{2}(r-r_{f})^{2}$, resulting in
$J(r,u)=(r-r_{f})u+\frac{1}{2}(r-r_{f})^{2}$. We describe the safe region by
the area outside a circular obstacle in the middle of the car’s trajectory and
an additive noise $h(r,w)=\rho^{2}-\left\|r-r_{obs}\right\|_{2}^{2}+w\leq 0$
To keep the QP in (21) in a standard Control Lyapunov Function(CLF) form, we
rewrite it as an explicit Quadratic Program and integrate the DR-CBF:

$\displaystyle\min_{u\in[\underline{u},\bar{u}],\delta}$ $\displaystyle\ \ \
u^{T}Qu+q^{T}\delta^{2}$ (26) s.t. $\displaystyle\ \ \
(r-r_{f})u+\frac{1}{2}(r-r_{f})^{2}\leq\delta$ $\displaystyle\ \ \
\frac{\partial\mathrm{CVaR}_{\alpha}(h(x,w))}{\partial
x}\left(f\left(x\right)+g\left(x\right)u\right)+\kappa\left(\operatorname{CVaR}_{\alpha}(h(x,w))\right)\leq
0$

where $\delta$ is a relaxation factor for the CLF, to allow some divergence
from reaching the final point when the safety of the car is compromised and
the CBF needs to take over.

Figure 1: The normal CBF trajectory crosses the noisy region and fluctuates
while the DR-CBF avoids the noisy region by a safe margin.

Figure 1 presents trajectories for the standard CBF and our distributionally
robust CBF, while the shaded area shows the contours of the circumference
noise. The standard CBF successfully avoids the obstacle but stays amid the
noisy region, resulting in a fluctuated trajectory affected by the noise. On
the other hand, the DR-CBF takes a more conservative trajectory avoiding the
noisy region as well.

###  4.2 Quadcopter: A Second-Order System Case Study

The approximate method for high-order systems was demonstrated on a 2D
Quadcopter environment, with the following kinematics:

$\displaystyle\ddot{x}$ $\displaystyle=\frac{T_{r}+T_{l}}{m}\sin{\theta}$ (27)
$\displaystyle\ddot{y}$ $\displaystyle=\frac{T_{r}+T_{l}}{m}\cos{\theta}-g$
$\displaystyle\ddot{\theta}$ $\displaystyle=(T_{r}-T_{l})\frac{L}{J}$

where $x$ is the horizontal distance of the quadcopter’s frame, $y$ is the
vertical distance, $\theta$ is the orientation of the quadcopter,
$T_{r},T_{l}$ are the right and left rotors thrust/control inputs, $L$ is the
arm length of the quadcopter, and $J$ is its moment of inertia. We simulate a
circular trajectory reference problem, where the quadcopter would come across
four obstacles along its circular path, with added noise to their
circumference $h(r,w)=\rho^{2}-\left\|r-r_{obs}\right\|_{2}^{2}+w\leq 0$, with
$r=[x,y]$. This will require twice differentiations to get the control input
in the CBF as this is a second-order system. We evaluate the performance of
the CBF and DR-CBF in tracking the trajectory and avoiding the obstacles by a
safe margin.

Figure 2: The DR-CBF follows a more conservative trajectory to avoid the
potentially unsafe region compared to normal CBF.

In Figure 2, it is demonstrated that the CBF stays close to the obstacle
crossing the noisy region (magnified picture) while the DR-CBF starts steering
clear of the obstacle earlier to avoid the noise, and keeps a safer distance
from it. Due to the agility of the Quadcopter, we see a throwing motion after
avoiding the obstacle for both algorithms, which reflects a slight delay in
returning back to the reference trajectory.Table 1 summarizes the values for
the coefficients used in each problem.

##  5 Conclusion

In this paper, we devise a distributionally robust control barrier function
for stochastic constraints, using the conditional value at risk, and convex
differentiable programming. The proposed framework results in a safer and more
robust variant of CBF. In further work, we want to explore methods for the
following settings. (1) More complex, non-additive, and multidimensional
noise, which leads to a harder problem for solving the supremum problem. (2)
Solving the optimization under distributional constraints exactly with the
primal-dual method without taking the dual variable as a constant penalty
coefficient. This requires further work to re-cast the optimization over the
dual variable in a tractable way. (3) Exact methods for higher-order systems,
which require higher-order differentiable convex programming techniques.

## Appendix

Table 1: Parameters Parameters | Description | Value
---|---|---
$\alpha$ | Confidence level | $0.95$
$\lambda$ | Penalty coefficient for Wasserstein metric | $1$
$\mu$ | Mean of samples | $0.0$
$\sigma$ | Variance of samples | $0.1$
$\mathrm{\kappa_{1}}$ | Dubins linear kappa coefficient | $1$
$\mathrm{\kappa_{2}}$ | Quadcopter linear kappa coefficient | $6$
$\mathrm{\kappa_{3}}$ | Quadcopter linear kappa coefficient | $2$
$\mathrm{\kappa_{4}}$ | Quadcopter linear kappa coefficient | $12$
$\mathrm{\kappa_{5}}$ | Quadcopter linear kappa coefficient | $4$
$\mathrm{\kappa_{6}}$ | Quadcopter linear kappa coefficient | $15$
$\mathrm{\kappa_{7}}$ | Quadcopter linear kappa coefficient | $5$
$\mathrm{\kappa_{8}}$ | Quadcopter linear kappa coefficient | $8$
$\mathrm{\kappa_{9}}$ | Quadcopter linear kappa coefficient | $5$

## References

  * Mot [2023] “Motor vehicle registered,” , 2023.  URL https://www.statista.com/statistics/183505/number-of-vehicles-in-the-united-states-since-1990/, accessed on May 25, 2023.
  * ind [2023] “Industrial-robot-market-size-worldwide,” , 2023.  URL https://www.statista.com/statistics/728530/industrial-robot-market-size-worldwide/#:~:text=In%202020%2C%20the%20size%20of,surpass%20165%20billion%20U.S.%20dollars./, accessed on May 25, 2023.
  * USA [2021] “Department of the Air Force Role in Joint All Domain Operations (JADO). Air Force Doctrine Publication (AFDP) 3-99. Maxwell Air Force Base, AL,” , 2021.  URL https://www.doctrine.af.mil/Doctrine-Publications/AFDP-3-99-DAF-Role-in-Jt-All-Domain-Ops-JADO/, accessed: 2022-10-19.
  * Van Parys et al. [2015] Van Parys, B. P., Kuhn, D., Goulart, P. J., and Morari, M., “Distributionally robust control of constrained stochastic systems,” _IEEE Transactions on Automatic Control_ , Vol. 61, No. 2, 2015, pp. 430–442.
  * Hakobyan and Yang [2021] Hakobyan, A., and Yang, I., “Wasserstein distributionally robust motion control for collision avoidance using conditional value-at-risk,” _IEEE Transactions on Robotics_ , Vol. 38, No. 2, 2021, pp. 939–957.
  * Bahari Kordabad et al. [2022] Bahari Kordabad, A., Wisniewski, R., and Gros, S., “Safe Reinforcement Learning Using Wasserstein Distributionally Robust MPC and Chance Constraint,” 2022.
  * Coulson et al. [2021] Coulson, J., Lygeros, J., and Dörfler, F., “Distributionally robust chance constrained data-enabled predictive control,” _IEEE Transactions on Automatic Control_ , Vol. 67, No. 7, 2021, pp. 3289–3304.
  * Coppens and Patrinos [2021] Coppens, P., and Patrinos, P., “Data-driven distributionally robust MPC for constrained stochastic systems,” _IEEE Control Systems Letters_ , Vol. 6, 2021, pp. 1274–1279.
  * Yang [2020] Yang, I., “Wasserstein distributionally robust stochastic control: A data-driven approach,” _IEEE Transactions on Automatic Control_ , Vol. 66, No. 8, 2020, pp. 3863–3870.
  * Ames et al. [2016] Ames, A. D., Xu, X., Grizzle, J. W., and Tabuada, P., “Control barrier function based quadratic programs for safety critical systems,” _IEEE Transactions on Automatic Control_ , Vol. 62, No. 8, 2016, pp. 3861–3876.
  * Long et al. [2022] Long, K., Yi, Y., Cortes, J., and Atanasov, N., “Safe and stable control synthesis for uncertain system models via distributionally robust optimization,” _arXiv preprint arXiv:2210.01341_ , 2022.
  * Madry et al. [2017] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A., “Towards deep learning models resistant to adversarial attacks,” _arXiv preprint arXiv:1706.06083_ , 2017.
  * Lütjens et al. [2020] Lütjens, B., Everett, M., and How, J. P., “Certified adversarial robustness for deep reinforcement learning,” _Conference on Robot Learning_ , PMLR, 2020, pp. 1328–1337.
  * Liu et al. [2022] Liu, Z., Guo, Z., Cen, Z., Zhang, H., Tan, J., Li, B., and Zhao, D., “On the robustness of safe reinforcement learning under observational perturbations,” _arXiv preprint arXiv:2205.14691_ , 2022.
  * Tessler et al. [2019] Tessler, C., Efroni, Y., and Mannor, S., “Action robust reinforcement learning and applications in continuous control,” _International Conference on Machine Learning_ , PMLR, 2019, pp. 6215–6224.
  * Mankowitz et al. [2019] Mankowitz, D. J., Levine, N., Jeong, R., Shi, Y., Kay, J., Abdolmaleki, A., Springenberg, J. T., Mann, T., Hester, T., and Riedmiller, M., “Robust reinforcement learning for continuous control with model misspecification,” _arXiv preprint arXiv:1906.07516_ , 2019.
  * Brunke et al. [2022] Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S., Panerati, J., and Schoellig, A. P., “Safe learning in robotics: From learning-based control to safe reinforcement learning,” _Annual Review of Control, Robotics, and Autonomous Systems_ , Vol. 5, 2022, pp. 411–444.
  * Achiam et al. [2017] Achiam, J., Held, D., Tamar, A., and Abbeel, P., “Constrained policy optimization,” _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , JMLR. org, 2017, pp. 22–31.
  * Zhang et al. [2020] Zhang, H., Chen, H., Xiao, C., Li, B., Liu, M., Boning, D., and Hsieh, C.-J., “Robust deep reinforcement learning against adversarial perturbations on state observations,” _Advances in Neural Information Processing Systems_ , Vol. 33, 2020, pp. 21024–21037.
  * Sun et al. [2022] Sun, C., Kim, D.-K., and How, J. P., “ROMAX: Certifiably Robust Deep Multiagent Reinforcement Learning via Convex Relaxation,” _2022 International Conference on Robotics and Automation (ICRA)_ , IEEE, 2022, pp. 5503–5510.
  * Ren and Majumdar [2022] Ren, A. Z., and Majumdar, A., “Distributionally robust policy learning via adversarial environment generation,” _IEEE Robotics and Automation Letters_ , Vol. 7, No. 2, 2022, pp. 1379–1386.
  * Morrison et al. [2020] Morrison, D., Corke, P., and Leitner, J., “Egad! an evolved grasping analysis dataset for diversity and reproducibility in robotic manipulation,” _IEEE Robotics and Automation Letters_ , Vol. 5, No. 3, 2020, pp. 4368–4375.
  * Wang et al. [2019] Wang, D., Tseng, D., Li, P., Jiang, Y., Guo, M., Danielczuk, M., Mahler, J., Ichnowski, J., and Goldberg, K., “Adversarial grasp objects,” _2019 IEEE 15th International Conference on Automation Science and Engineering (CASE)_ , IEEE, 2019, pp. 241–248.
  * Xu et al. [2022] Xu, M., Huang, P., Niu, Y., Kumar, V., Qiu, J., Fang, C., Lee, K.-H., Qi, X., Lam, H., Li, B., et al., “Group Distributionally Robust Reinforcement Learning with Hierarchical Latent Variables,” _arXiv preprint arXiv:2210.12262_ , 2022.
  * Yang et al. [2022] Yang, B., Zheng, L., Ratliff, L. J., Boots, B., and Smith, J. R., “Stackelberg MADDPG: Learning Emergent Behaviors via Information Asymmetry in Competitive Games,” 2022.
  * Xiao and Belta [2021] Xiao, W., and Belta, C., “High-order control barrier functions,” _IEEE Transactions on Automatic Control_ , Vol. 67, No. 7, 2021, pp. 3655–3662.
  * Amos and Kolter [2017] Amos, B., and Kolter, J. Z., “Optnet: Differentiable optimization as a layer in neural networks,” _arXiv preprint arXiv:1703.00443_ , 2017.
  * Agrawal et al. [2019a] Agrawal, A., Amos, B., Barratt, S., Boyd, S., Diamond, S., and Kolter, J. Z., “Differentiable convex optimization layers,” _Advances in neural information processing systems_ , Vol. 32, 2019a.
  * Agrawal et al. [2019b] Agrawal, A., Barratt, S., Boyd, S., Busseti, E., and Moursi, W. M., “Differentiating through a cone program,” _arXiv preprint arXiv:1904.09043_ , 2019b.
  * Rockafellar et al. [2000] Rockafellar, R. T., Uryasev, S., et al., “Optimization of conditional value-at-risk,” _Journal of risk_ , Vol. 2, 2000, pp. 21–42.
  * Kleywegt et al. [2002] Kleywegt, A. J., Shapiro, A., and Homem-de Mello, T., “The sample average approximation method for stochastic discrete optimization,” _SIAM Journal on optimization_ , Vol. 12, No. 2, 2002, pp. 479–502.
  * Sinha et al. [2017] Sinha, A., Namkoong, H., and Duchi, J., “Certifiable distributional robustness with principled adversarial training,” _arXiv preprint arXiv:1710.10571_ , Vol. 2, 2017.
  * Blanchet and Murthy [2019] Blanchet, J., and Murthy, K., “Quantifying distributional model risk via optimal transport,” _Mathematics of Operations Research_ , Vol. 44, No. 2, 2019, pp. 565–600.
  * Rahimian and Mehrotra [2019] Rahimian, H., and Mehrotra, S., “Distributionally robust optimization: A review,” _arXiv preprint arXiv:1908.05659_ , 2019.
  * Hu and Hong [2013] Hu, Z., and Hong, L. J., “Kullback-Leibler divergence constrained distributionally robust optimization,” _Available at Optimization Online_ , Vol. 1, No. 2, 2013, p. 9.
  * Givens and Shortt [1984] Givens, C. R., and Shortt, R. M., “A class of Wasserstein metrics for probability distributions.” _Michigan Mathematical Journal_ , Vol. 31, No. 2, 1984, pp. 231–240.
  * Mohajerin Esfahani and Kuhn [2018] Mohajerin Esfahani, P., and Kuhn, D., “Data-driven distributionally robust optimization using the Wasserstein metric: Performance guarantees and tractable reformulations,” _Mathematical Programming_ , Vol. 171, No. 1-2, 2018, pp. 115–166.
  * Amos et al. [2018] Amos, B., Jimenez, I., Sacks, J., Boots, B., and Kolter, J. Z., “Differentiable mpc for end-to-end planning and control,” _Advances in neural information processing systems_ , Vol. 31, 2018.
  * Agrawal et al. [2019c] Agrawal, A., Barratt, S., Boyd, S., Busseti, E., and Moursi, W. M., “Differentiating through a cone program,” _arXiv preprint arXiv:1904.09043_ , 2019c.
  * Agrawal et al. [2020] Agrawal, A., Barratt, S., Boyd, S., and Stellato, B., “Learning convex optimization control policies,” _Learning for Dynamics and Control_ , PMLR, 2020, pp. 361–373.