# Learning Responsibility Allocations for Safe Human-Robot Interaction with
Applications to Autonomous Driving

Ryan K. Cosner, Yuxiao Chen, Karen Leung, and Marco Pavone  Ryan K. Cosner is
with the California Institute of Technology<EMAIL_ADDRESS>Yuxiao Chen
is with NVIDIA<EMAIL_ADDRESS>Karen Leung is with the University of
Washington and NVIDIA<EMAIL_ADDRESS><EMAIL_ADDRESS>Marco Pavone is
with Stanford University and NVIDIA<EMAIL_ADDRESS><EMAIL_ADDRESS>Ryan’s work was performed during an internship with NVIDIA.

###### Abstract

Drivers have a responsibility to exercise reasonable care to avoid collision
with other road users. This assumed responsibility allows interacting agents
to maintain safety without explicit coordination. Thus to enable safe
autonomous vehicle (AV) interactions, AVs must understand what their
responsibilities are to maintain safety and how they affect the safety of
nearby agents. In this work we seek to understand how responsibility is shared
in multi-agent settings where an autonomous agent is interacting with human
counterparts. We introduce Responsibility-Aware Control Barrier Functions (RA-
CBFs) and present a method to learn responsibility allocations from data. By
combining safety-critical control and learning-based techniques, RA-CBFs allow
us to account for scene-dependent responsibility allocations and synthesize
safe and efficient driving behaviors without making worst-case assumptions
that typically result in overly-conservative behaviors. We test our framework
using real-world driving data and demonstrate its efficacy as a tool for both
safe control and forensic analysis of unsafe driving.

##  I Introduction

Drivers have a duty to exercise reasonable care when interacting with other
road users. The assumption that other road users will exercise responsible
behaviors enables everyone to maintain safety without explicit coordination.
Thus, for autonomous vehicles (AVs) to safely and seamlessly interact with
other road users, they must utilize a safety framework that can interpretably
codify basic safe driving behaviors (such as those laid out by the IEEE safe
driving standards [1]), and exercise duty of care, i.e., drive responsibly.
However, the responsibility for ensuring safety is shared asymmetrically and
the allocation is influenced by context-dependent social driving norms. For
example, drivers are generally more responsible for avoiding collisions with
vehicles in front of them than behind them (see Figure 1).

The goal of this work is to develop a rigorous yet flexible safety framework
that is capable of codifying and synthesizing safe and responsible driving
behaviors [1]. Unfortunately, most existing models and techniques for ensuring
AV safety tend to make strong assumptions about how other drivers behave which
often results in defensive or erratic driving. For instance, many approaches
make worst-case assumptions about the other agents’ behaviors [2, 3, 4, 5];
while these assumptions allow for strong theoretical guarantees, they are
often impractical as they result in infeasible planning problems or induce
overly conservative behaviors. Therefore, a key challenge is developing safety
models that are not only robust to varied driving behaviors, but that are also
capable of accounting for context-dependent social norms that effect how
drivers implicitly coordinate.

With that goal in mind, we propose Responsibility-Aware Control Barrier
Functions (RA-CBFs), a responsibility-aware safety paradigm that is conducive
to efficient techniques for safe control synthesis and safety evaluation. A
primary use case of RA-CBFs is their application as a safety filter within an
AV stack, whereby the output of the main AV decision-making and planning
algorithmic pipeline (powered, for example, by high-capacity human behavior
prediction models based on deep learning) is monitored for its compliance with
basic safe and responsible driving behaviors (as encoded by the RA-CBF) and
possibly revised in case of anomalies, in a manner similar to [3] and [5]. As
a proof of concept, we demonstrate this capability in the context of real-time
closed-loop control and post-facto forensic analysis.

Related work. We present relevant work on responsibility-aware safety methods
for multi-agent interactions with an emphasis on AV applications.
Specifically, we consider related work on socially-aware planning, safe multi-
agent control, and safety constraint learning.

Many recent works focus on modeling and estimating drivers’ social preferences
to synthesize socially-aware driving behaviors. For instance, [6] estimates
the Social Value Orientation (SVO) of other drivers and formulates a SVO-based
game-theoretic planner. [7, 8] craft a (potentially learned) planning reward
function that incentivizes an AV to be more cooperative, sympathetic, and/or
courteous. While these approaches demonstrate that accounting for social
preferences can lead to more human-like AV behaviors, they do not provide any
assurances or quantification of AV safety.

Figure 1: In human driving, vehicles can be expected to demonstrate a
reasonable duty of care. For example, a trailing vehicle (green) takes
responsibility for not colliding with the car in front of it (blue) and a
merging vehicle (orange) follows formal and informal rules to avoid a
collision with the vehicles in the lane (pink). How can we ensure that
autonomous vehicles act according to such informal driving etiquette?

In the safety-critical control literature, under decentralized multi-agent
settings (which is more relevant to AV applications), many certifiably safe
multi-agent control methods exist, including Control Barrier Functions (CBFs)
[4, 9, 10, 11], Reciprocal Velocity Obstacles [12], Hamilton-Jacobi
Reachability [13, 14], Responsibility Sensitive Safety [5], and Safety Force
Field [3]. However, these methods rely on worst-case or static assumptions on
how other agents behave, which, unfortunately, do not hold in practice. This
limitation has spurred recent works [15, 16, 17] that investigate collision-
avoidance responsibility and how to decide which agent should take more effort
to avoid collisions. However, these methods consider either centralized
control or centrally defined social preferences, which does not apply to
autonomous driving since the social preferences of other agents are not known
precisely and cannot be assigned.

Learning responsibility allocation influenced by social norms necessitates the
need to combine safety-critical control with data-driven methods. Recent
safety-critical learning methods [18, 19, 20, 21] infer safety constraints
from data. To the best of our knowledge, the consideration of responsibility
has yet to be investigated within a data-driven safe multi-agent control
setting. In this work, we take on a data-driven approach to learn how
responsibility is allocated among multiple agents and synthesize corresponding
safe and responsible controls. We elect to use CBFs as the basis of our method
due to its interpretable and rigorous control-theoretic formulation and
computational tractability.

Contributions: Our contributions are three-fold: (i) We present a novel
concept of Responsibility-Aware Control Barrier Functions (RA-CBFs) which
extends the standard CBF to account for asymmetric sharing of responsibility
between multiple agents. (ii) We propose a data-driven constraint-learning
algorithm to infer the responsibility allocations modeled in the RA-CBF
formulation. (iii) We showcase our method using real-world driving data and
demonstrate the utility of RA-CBFs and the learned responsibility allocations
in safe closed-loop AV control and forensic analysis.

##  II Background

The dynamics of the vehilces are abstracted as a nonlinear control-affine
system of the form:

$\displaystyle\dot{\mathbf{x}}=\mathbf{f}(\mathbf{x})+\mathbf{g}(\mathbf{x})\mathbf{u}$
(1)

where $\mathbf{x}\in\mathbb{R}^{n}$ and
$\mathbf{u}\in\mathcal{U}\subset\mathbb{R}^{m}$ represent the states and
inputs of the system and where $\mathcal{U}$ is assumed to be compact. We
further assume that $\mathbf{f}:\mathbb{R}^{n}\to\mathbb{R}^{n}$ and
$\mathbf{g}:\mathbb{R}^{n}\to\mathbb{R}^{n\times m}$ are locally Lipschitz
continuous. Given a state-feedback controller
$\mathbf{k}:\mathbb{R}^{n}\to\mathbb{R}^{m}$, the closed-loop dynamics are:

$\displaystyle\dot{\mathbf{x}}=\mathbf{f}_{\textrm{cl}}(\mathbf{x})=\mathbf{f}(\mathbf{x})+\mathbf{g}(\mathbf{x})\mathbf{k}(\mathbf{x}).$
(2)

###  II-A Control Barrier Functions

In this section, we define safety as the forward-invariance of some safe set
$\mathcal{C}\subset\mathbb{R}^{n}$ and review Control Barrier Functions (CBFs)
as a tool for synthesizing safe controllers.

######  Definition 1 (Forward Invariance and Safety).

A set $\mathcal{C}\subset\mathbb{R}^{n}$ is forward invariant if for every
$\mathbf{x}(0)\in\mathcal{C}$ the solution to (2) satisfies
$\mathbf{x}(t)\in\mathcal{C}$ for all $t\geq 0$. We call the system (2) safe
with respect to $\mathcal{C}$ if $\mathcal{C}$ is forward invariant.

Let the set $\mathcal{C}$ be the 0-superlevel set of some continuously
differentiable function $h:\mathbb{R}^{n}\to\mathbb{R}$ with $0$ a regular
value111We say that 0 is a regular value of $h$ if
$h(\mathbf{x})=0\implies\frac{\partial h}{\partial\mathbf{x}}\neq 0$.:

$\displaystyle\mathcal{C}$
$\displaystyle\triangleq\\{\mathbf{x}\in\mathbb{R}^{n}~{}|~{}h(\mathbf{x})\geq
0\\}$ (3)

We deem $h$ a CBF if it also satisfies the following condition:

######  Definition 2 (Control Barrier Function [22]).

Let $\mathcal{C}\subset\mathbb{R}^{n}$ be the 0-superlevel set of some
continuously differentiable function $h:\mathbb{R}^{n}\to\mathbb{R}$ with 0 a
regular value. The function $h$ is a Control Barrier Function (CBF) for (1) if
there exists an extended class $\mathcal{K}_{\infty}$ function222A continuous
function $\alpha:\mathbb{R}_{\geq 0}\to\mathbb{R}_{\geq 0}$ is a class
$\mathcal{K}_{\infty}$ function if $\alpha(0)=0$, $\alpha$ is strictly
monotonically increasing, and $\lim_{c\to\infty}\alpha(c)=\infty$. A
continuous function $\alpha:\mathbb{R}\to\mathbb{R}$ is an extended
$\mathcal{K}_{\infty}$ function if $\alpha(0)=0$, $\alpha$ is strictly
monotonically increasing, $\lim_{c\to\infty}\alpha(c)=\infty$, and
$\lim_{c\to-\infty}\alpha(c)=-\infty$.  $\alpha$ such that for all
$\mathbf{x}\in\mathcal{C}$:

$\displaystyle\sup_{\mathbf{u}\in\mathcal{U}}\quad\overbrace{\underbrace{L_{\mathbf{f}}h(\mathbf{x})}_{\frac{\partial
h}{\partial
x}\mathbf{f}(\mathbf{x})}+\underbrace{L_{\mathbf{g}}h(\mathbf{x})}_{\frac{\partial
h}{\partial{x}}\mathbf{g}(\mathbf{x})}\mathbf{u}}^{\frac{dh}{dt}(\mathbf{x},\mathbf{u})}\geq-\alpha(h(\mathbf{x})).$
(4)

Here, Lie derivative notation is used to represent the partial derivatives
$L_{\mathbf{f}}h(\mathbf{x})\triangleq\frac{\partial
h}{\partial\mathbf{x}}\mathbf{f}(\mathbf{x})$ and
$L_{\mathbf{g}}h(\mathbf{x})\triangleq\frac{\partial
h}{\partial\mathbf{x}}\mathbf{g}(\mathbf{x})$. Intuitively, the CBF constraint
(4) limits $\frac{dh}{dt}$, and prevents $h$ from decreasing along the
trajectory when $h(\mathbf{x})=0$, thus rendering $\mathcal{C}$ forward
invariant for (2). This intuition was formalized by [22] in the following
theorem:

######  Theorem 1 (CBF Safety, [22]).

Given a set $\mathcal{C}\subset\mathbb{R}^{n}$ defined as the 0-superlevel set
of a continuously differentiable function $h:\mathbb{R}^{n}\to\mathbb{R}$ with
0 a regular value, if $h$ is a CBF, then any locally Lipschitz controller
$\mathbf{k}:\mathbb{R}^{n}\to\mathbb{R}^{m}$ that satisfies (4) for all
$\mathbf{x}\in\mathcal{C}$, renders the system (2) safe w.r.t. $\mathcal{C}$.

##  III Responsibility-aware Decentralized Multi-agent Safety

In this section, we extend the CBF framework to a decentralized multi-agent
setting and introduce additional terms to account for asymmetrically shared
responsibility.

###  III-A Decentralization Multi-agent CBF

We extend the system dynamics (1) to multiple agents:

$\displaystyle\dot{\mathbf{x}}_{i}=\mathbf{f}_{i}(\mathbf{x}_{i})+\mathbf{g}_{i}(\mathbf{x}_{i})\mathbf{u}_{i}$
(5)

where $\mathbf{x}_{i}\in\mathbb{R}^{n_{i}}$,
$\mathbf{u}_{i}\in\mathcal{U}_{i}\subset\mathbb{R}^{m_{i}}$,
$\mathbf{f}_{i}:\mathbb{R}^{n_{i}}\to\mathbb{R}^{n_{i}}$,
$\mathbf{g}_{i}:\mathbb{R}^{n_{i}}\to\mathbb{R}^{m_{i}}$ represent the state,
input, drift, and actuation matrix of agent $i$. For the entire system of
$N\in\mathbb{N}$ agents, let
$\mathbf{x}=\begin{bmatrix}\mathbf{x}_{1}^{\top}&\cdots&\mathbf{x}_{N}^{\top}\end{bmatrix}^{\top}$
denote the concatenated state and the dynamics for $\mathbf{x}$ be denoted as
in (1):

$\displaystyle\dot{\mathbf{x}}=\underbrace{\begin{bmatrix}\mathbf{f}_{1}(\mathbf{x}_{1})\\\
\vdots\\\
\mathbf{f}_{N}(\mathbf{x}_{N})\end{bmatrix}}_{\mathbf{f}(\mathbf{x})}+\underbrace{\begin{bmatrix}\mathbf{g}_{1}(\mathbf{x}_{1})\\\
\vdots\\\
\mathbf{g}_{N}(\mathbf{x}_{N})\end{bmatrix}}_{\mathbf{g}(\mathbf{x})}\underbrace{\begin{bmatrix}\mathbf{u}_{1}\\\
\vdots\\\ \mathbf{u}_{N}\end{bmatrix}}_{\mathbf{u}}.$ (6)

If the multi-agent system is governed by a centrallized controller, the CBF
inequality can be checked directly and used as a constraint in an
optimization-based controller to obtain safe inputs [23]. However, centralized
control is often unrealizable for AVs due to communication and scalability
issues as well as the presence of human actors. Thus, we focus on a
decentralized variant of the CBF constraint (4) and assume that each agent can
measure the states of the other agents, but independently generates its own
input according to some controller unknown to the other agents.

One common method for retaining safety guarantees in the context of
decentralized control, is to ensure robustness with respect to all possible
controls of the other agents (including the worst-case inputs) as in [4]. In
this case, constraint (4) from the perspective of agent $i$ becomes:

$\displaystyle\sup_{\mathbf{u}_{i}\in\mathcal{U}_{i}}\inf_{\begin{subarray}{c}\mathbf{u}_{j}\in\mathcal{U}_{j},\\\
j\neq
i\end{subarray}}L_{\mathbf{f}}h(\mathbf{x})+L_{\mathbf{g}}h(\mathbf{x})\mathbf{u}\geq-\alpha(h(\mathbf{x})).$
(7)

This is a conservative constraint which ensures the safety of the system even
when other agents act adversarially. For human-interactive systems where
responsibility is shared, this controller is often unnecessarily conservative.

Despite their safety-guarantees, worst-case constraints like (7) are highly
conservative and prevent fluent behaviors [24]. It is therefore desirable to
find a less conservative safety constraint that is more cognizant of the
social interactions between agents even when the controllers of the other
agents are unknown. For this purpose, we consider a novel CBF framework that
models social responsibility.

###  III-B Responsible-Aware Control Barrier Functions

In multi-agent systems of human actors, the responsibility for maintaining
safety is typically shared amongst people. For example, humans exhibit social
behavior in crowd navigation and driving where the burden of maintaining
safety is distributed amongst everyone [6, 25]. Equipped with the notion that
agents share the responsibility of maintaining safety, we move away from
worst-case behavioral assumptions, and instead, learn the responsibility
allocation from data. First, we define responsibility allocation functions:

######  Definition 3 (Responsibility Allocation Function).

A function $\gamma:\mathbb{N}\times\mathbb{R}^{n}\to\mathbb{R}$ is a
responsibility allocation function for $N\in\mathbb{N}$ if for all
$\mathbf{x}\in\mathbb{R}^{n}$:

$\displaystyle\sum_{i\in\\{1,\dots,N\\}}\gamma(i,\mathbf{x})\geq 0,$ (8)

For agent $i$ in a multi-agent system at state $\mathbf{x}$,
$\gamma(i,\mathbf{x})>0$ indicates increased responsibility,
$\gamma(i,\mathbf{x})=0$ indicates evenly shared responsibility, and
$\gamma(i,\mathbf{x})<0$ indicates decreased responsibility. The sum of
$\gamma(i,\mathbf{x})$ is lower bounded by zero to ensure that the total
allocated responsibility must be greater than or equal to that of even
sharing.

Using these responsibility allocation functions we can present our definition
of Responsibility-Aware Control Barrier Functions (RA-CBFs) which consider
responsibility allocation in their decentralized multi-agent safety
constraint:

######  Definition 4 (Responsibility-Aware Control Barrier Function).

Let $\mathcal{C}\subset\mathbb{R}^{n}$ be the 0-superlevel set of some
continuously differentiable function $h:\mathbb{R}^{n}\to\mathbb{R}$ with 0 a
regular value. Additionally, let
$\gamma:\mathbb{N}\times\mathbb{R}^{n}\to\mathbb{R}$ be a responsibility
allocation function for $N\in\mathbb{N}$. The function $h$ is a
Responsibility-Aware CBF for the system (6) and responsibility allocation
function $\gamma$ if there exists an extended class $\mathcal{K}_{\infty}$
function $\alpha$ such that for all $\mathbf{x}\in\mathcal{C}$ and all
$i\in\\{1,\dots,N\\}$ :

$\displaystyle\sup_{\mathbf{u}_{i}\in\mathcal{U}_{i}}\underbrace{\overbrace{L_{\mathbf{g}_{i}}h(\mathbf{x})\mathbf{u}_{i}+\frac{1}{N}\Big{(}\alpha(h(\mathbf{x}))+L_{\mathbf{f}}h(\mathbf{x})\Big{)}}^{\mathbf{c}_{i}(\mathbf{x},\mathbf{u})\triangleq}-\gamma(i,\mathbf{x})}_{\textrm{RA-
CBF Constraint}(i,\mathbf{x},\mathbf{u},\gamma)\triangleq}\geq 0.$ (9)

######  Remark 1.

For generality, RA-CBFs are presented for $N$ agents but in practice it is
common to enforce CBF constraints between each pair of agents where the number
of constraints enforced on agent $i$’s input grows linearly with the number of
agents [26]. In this case there would be several pairwise RA-CBF constraints
with $N=2$. In Section V, we take this approach in our application.

######  Theorem 2 (Responsibility-Aware Safety).

Given a set $\mathcal{C}\subset\mathbb{R}^{n}$ defined as the 0-superlevel set
of a continuously differentiable function $h:\mathbb{R}^{n}\to\mathbb{R}$ with
0 a regular value, if $h$ is an RA-CBF for (6) and the responsibility
allocation function $\gamma:\mathbb{N}\times\mathbb{R}^{n}\to\mathbb{R}$ for
$N\in\mathbb{N}$, then any locally Lipschitz controller
$\mathbf{k}:\mathbb{R}^{n}\to\mathbb{R}^{m}$ that satisfies (9) for all
$\mathbf{x}\in\mathcal{C}$ and $i\in\\{1,\dots,N\\}$, renders system (2) safe
with respect to $\mathcal{C}$.

###### Proof.

First let $\mathbf{c}_{i}(\mathbf{x},\mathbf{k}(\mathbf{x}))\triangleq
L_{\mathbf{g}_{i}}h(\mathbf{x})\mathbf{k}_{i}(\mathbf{x})+\frac{1}{N}\Big{(}\alpha(h(\mathbf{x}))+L_{\mathbf{f}}h(\mathbf{x})\Big{)}$
for all $i\in\\{1,\dots,N\\}$. Since the $\mathbf{k}_{i}$ satisfies (9),

$\displaystyle 0$
$\displaystyle\geq-\mathbf{c}_{i}(\mathbf{x},\mathbf{k}(\mathbf{x}))+\gamma(i,\mathbf{x}),$
(10)
$\displaystyle\geq-\sum_{i\in\\{1,\dots,N\\}}\mathbf{c}_{i}(\mathbf{x},\mathbf{k}(\mathbf{x}))+\sum_{i\in\\{1,\dots,N\\}}\gamma(i,\mathbf{x}),$
(11)
$\displaystyle\geq-\sum_{i\in\\{1,\dots,N\\}}\mathbf{c}_{i}(\mathbf{x},\mathbf{k}(\mathbf{x}))$
(12)

Inequality (11) follows from the decentralized constraint (9) for all $i$ and
(12) holds since $\gamma$ is a responsibility allocation function for $N$.
Since the final inequality (12) is equivalent to the centralized CBF
constraint (4), Theorem 1 implies the safety of system (2) with respect to
$\mathcal{C}$. ∎

In summary, instead of considering the worst-case inputs from other agents,
our RA-CBF approach uses $\gamma(i,\mathbf{x})$ to allow agent $i$’s required
contribution to decentralized multi-agent safety to vary depending on the
state $\mathbf{x}$ of all of the agents in the scene. Also, instead of
explicitly considering the uncertainty in the other agents’ actions, one
perspective of the responsibility allocation function is that it models a
bound on the projection [27] of the other agents’ inputs onto the CBF time
derivative. Thus we can learn the effect of the other agents’ actions as a
scalar adjustment to $\frac{dh}{dt}$ as opposed to predicting their exact
trajectories.

Our responsibility model is similar to that of [15] which instead uses a
multiplicative term and is limited to driftless systems (i.e., systems where
$\mathbf{f}(\mathbf{x})\equiv 0$). By using an additive term, our model is
generally applicable to control affine systems and is capable of accounting
for the effect of responsibility even when the unforced dynamic are unsafe,
i.e. $\alpha(h)+L_{\mathbf{f}}h(\mathbf{x})\leq 0$.

##  IV Learning Responsibility Allocation

In this section we formalize the problem of learning responsibility allocation
functions $\gamma(i,\mathbf{x})$ from data and describe our method for
learning $\gamma$ from expert demonstrations, given a known safe set
$\mathcal{C}$ and associated CBF $h$.

###  IV-A Problem Setting

We assume that agent $i$ strives to minimize some unknown function
$Q_{i}:\mathbb{R}^{n_{i}\times m_{i}}\to\mathbb{R}$ and does so according to a
constrained optimal control policy:

$\displaystyle\mathbf{k}_{i}(\mathbf{x})=\operatorname*{argmin}_{u_{i}\in\mathcal{U}_{i}}$
$\displaystyle\quad Q_{i}(\mathbf{x}_{i},\mathbf{u}_{i})$ (13) s.t.
$\displaystyle\quad\textrm{RA-CBF
Constraint}(i,\mathbf{x},\mathbf{u},\gamma)\geq 0$

where $Q_{i}(\mathbf{x}_{i},\mathbf{u}_{i})$ represents agent $i$’s cost for
input $\mathbf{u}_{i}$ at state $\mathbf{x}_{i}$. Although the cost function
is unknown, we assume that all agents obey the RA-CBF constraint for some
$\gamma$ that we seek to learn, thus framing the problem of learning
responsibility allocations as constraint learning.

###  IV-B Learning Paradigm

Let $\mathcal{D}=\\{\mathbf{u}^{k},\mathbf{x}^{k}\\}_{k=1}^{N_{d}}$ be a
dataset of state-input pairs gathered from expert (human) demonstrations where
$N_{d}$ represents the total number of data points collected. Since the cost
function $Q_{i}$ can vary during data collection, it is possible for a state
to have several associated expert inputs.

Our goal is find some responsibility allocation function $\gamma$ such that
the RA-CBF constraint is satisfied for all state-input pairs in the expert
demonstrations $\mathcal{D}$. This can be written as the constrained
optimization problem:

$\displaystyle\gamma^{*}$
$\displaystyle=\operatorname*{argmin}_{\gamma}\quad\|\gamma\|$ (14)
$\displaystyle\textrm{s.t. }\quad\textrm{RA-CBF
Constraint}(i,\mathbf{x},\mathbf{u},\gamma)\geq 0,\>\forall
i\in\\{1,...,N\\},$
$\displaystyle\quad\sum_{i\in\\{1,\dots,N\\}}\gamma(i,\mathbf{x})\geq
0,\qquad\textrm{for all }(\mathbf{x},\mathbf{u})\in\mathcal{D},$

where the constraints enforce satisfaction of the RA-CBF and ensure that
$\gamma$ is a responsibility allocation function.

To find an approximate solution to this problem we take inspiration from [19,
20] and relax (14) to the following unconstrained loss function:

$\displaystyle\mathcal{L}(\mathcal{D},\gamma)=$
$\displaystyle\|\gamma\|+\lambda_{1}\sum_{(\mathbf{x},\mathbf{u})\in\mathcal{D}}\sum_{i=1}^{N}\bigg{[}-\mathbf{c}_{i}(\mathbf{x},\mathbf{u})+\gamma(i,\mathbf{x})\bigg{]}_{+}$
$\displaystyle+\lambda_{2}\sum_{(\mathbf{x},\mathbf{u})\in\mathcal{D}}\left[\sum_{i=1}^{N}-\gamma_{i}(i,\mathbf{x})\right]_{+}$
(15)

where $\lambda_{1},\lambda_{2},\in\mathbb{R}_{\geq 0}$ are hyperparameters
which adjust the constraint relaxations and
$[\;\cdot\;]_{+}\triangleq\max\\{\cdot,0\\}$. This loss function can then be
used find approximate solutions to (14):

$\displaystyle\gamma^{*}\approx\operatorname*{argmin}$
$\displaystyle\quad\mathcal{L}(\mathcal{D},\gamma)$ (16)

Figure 2: The learned responsibility allocation surface is visualized for a
range of velocities and relative positions. Scene 1:  the ego vehicle (yellow)
is driving on a two lane road. In all cases,
$\gamma(\textrm{ego},\mathbf{x})>0$ indicating a degree of conservative
driving. Generally, $\gamma(\textrm{ego},\mathbf{x})$ is larger when the other
agent (red) is in front of the ego vehivlethan when behind, indicating
increased responsibility when driving behind another vehicle. Scene 2:  The
ego vehicle is stopped at a four-way intersection with the other vehicle
(green) ahead or behind it (and no blue agent). Again the ego vehicle (yellow)
is more responsible when the green vehicle is in front of it than when it is
behind it. Scene 3:  The ego vehicle (yellow) is stopped at a four-way
intersection with the other vehicle (blue) crossing from top to bottom (and no
green agent). $\gamma(\textrm{ego},\mathbf{x})$ is large for all positions and
velocities of the blue vehicle showing that the ego agent takes is more
responsible in this situation.

###  IV-C Responsibility Regularization

However, the loss function used in the unconstrained optimization (16) is
insufficient since, as in Inverse Reinforcement Learning, the problem of
learning the constraint in (13) is poorly defined since the optimal input
generated by (13) is a function of both the unknown cost function $Q_{i}$ and
unknown responsibility allocation function $\gamma$. Intuitively, this is
because we cannot answer the question “did the agent act that way because it
wanted to (cost minimization) or because it had to (safety constraint
satisfaction)?”. To better define the constraint learning problem we take an
approach similar to [28] and regularize $\gamma$ by maximizing the likelihood
that it was used in (13) to generate the expert demonstrations $\mathcal{D}$.

Following the maximum entropy model presented in [29] with the variant for
continuous-time nonlinear systems presented in [30] we wish to solve the
optimization problem:

$\displaystyle\gamma^{*}_{\textrm{reg}}=\operatorname*{argmax}\sum_{(\mathbf{x},\mathbf{u})\in\mathcal{D}}\mathcal{P}(\mathbf{u}~{}|~{}\mathbf{x},\gamma).$
(17)

We approximate the probability of a given $\mathbf{u}$, by choosing
$\textrm{disc}(\mathcal{U})$ to be a finite discretization of $\mathcal{U}$
such as
$\textrm{disc}(\mathcal{U})=\\{\mathbf{u}\in\mathcal{U}\;|\;\delta\lfloor\mathbf{u}/\delta\rceil\\}$
for some $\delta>0$ where $\lfloor\cdot\rceil$ rounds each component to the
nearest integer. Mimicking the forms presented in [30, 28], the approximate
probability of an input $\mathbf{u}\in\mathcal{U}$ given the system state
$\mathbf{x}$ and responsibility allocation $\gamma$ is:

$\displaystyle\mathcal{P}(\mathbf{u}~{}|~{}\mathbf{x},\gamma)=\frac{e^{R(\mathbf{x},\mathbf{u})}}{Z_{\gamma}}\mathds{1}^{\gamma}(\mathbf{x},\mathbf{u}),\>Z_{\gamma}=\sum_{\boldsymbol{\upsilon}\in\textrm{disc}(\mathcal{U})}e^{R(\mathbf{x},\boldsymbol{\upsilon})}\mathds{1}^{\gamma}(\mathbf{x},\boldsymbol{\upsilon})$
(18)

where $Z$ is the partition function,
$R:\mathbb{R}^{n}\times\mathbb{R}^{m}\to\mathbb{R}$ is the reward function,
and $\mathds{1}^{\gamma}(\mathbf{x},\mathbf{u})\mapsto\\{0,1\\}$ indicates
satisfaction of the RA-CBF constraints given $\mathbf{x},\;\mathbf{u},$ and
$\gamma$.

To maximize the likelihood of the demonstration we minimize the number of
feasible inputs while retaining the feasibility of the expert demonstrations.
We note the total number of feasible inputs in $\textrm{disc}(\mathcal{U})$
decreases as $\gamma(i,\mathbf{x})$ increases regardless of $R$, so we can
maximize $\mathcal{P}(\mathbf{u}|\mathbf{x},\gamma)$ without knowledge of the
agents’ reward functions by maximizing $\gamma$ while maintaining feasibility
of the expert demonstrations. This can be expressed as the optimization
problem:

$\displaystyle\gamma_{\textrm{reg}}^{*}\approx\operatorname*{argmax}_{\gamma}\quad$
$\displaystyle\sum_{(\mathbf{x},\mathbf{u})\in\mathcal{D}}\sum_{i=1}^{N}\gamma(i,\mathbf{x})$
(19) s.t. $\displaystyle\textrm{RA-CBF
Constraint}(i,\mathbf{x},\mathbf{u},\gamma)\geq 0,$ $\displaystyle\textrm{for
all }i\in\\{1,\dots,N\\}\textrm{ and }(\mathbf{x},\mathbf{u})\in\mathcal{D}$

Since constraint feasibility on $\mathcal{D}$ is already accounted for in
(15), this regularization can be added to the loss as:

$\displaystyle\mathcal{L}_{\textrm{reg}}(\mathcal{D},\gamma)\triangleq\mathcal{L}(\mathcal{D},\gamma)+\lambda_{3}\sum_{(\mathbf{x},\mathbf{u})\in\mathcal{D}}\sum_{i=1}^{N}-\gamma(i,\mathbf{x})$
(20)

with hyperparamter $\lambda_{3}\in\mathbb{R}_{\geq 0}$ which can be used in
the final regularized optimization problem to estimate $\gamma$:

$\displaystyle\gamma^{*}=\operatorname*{argmin}\;\mathcal{L}_{\textrm{reg}}(\mathcal{D},\gamma)$
(21)

##  V Application to Autonomous Driving

In this section we apply our RA-CBF and responsibility allocation learning
method to urban driving using the Boston Seaport data provided by the nuScenes
dataset [31].

We assume that all agents in the scene are vehicles (i.e. there are no
pedestrians) and we model each agent as:

$\displaystyle\underbrace{\begin{bmatrix}\dot{x}_{i}\\\ \dot{y}_{i}\\\
\dot{v}_{i}\\\
\dot{\theta}\end{bmatrix}}_{\dot{\mathbf{x}}_{i}}=\underbrace{\begin{bmatrix}v_{i}\cos(\theta_{i})\\\
v_{i}\sin(\theta_{i})\\\ 0\\\
0\end{bmatrix}}_{\mathbf{f}_{i}(\mathbf{x}_{i})}+\underbrace{\begin{bmatrix}0&0\\\
0&0\\\ 1&0\\\
0&1\end{bmatrix}}_{\mathbf{g}_{i}(\mathbf{x}_{i})}\underbrace{\begin{bmatrix}a_{i}\\\
\omega_{i}\end{bmatrix}}_{\mathbf{u}_{i}},$ (22)

where $(x_{i},y_{i})\in\mathbb{R}^{2}$,
$v_{i},\theta_{i},a_{i},\omega_{i}\in\mathbb{R}$ represent the position,
velocity, yaw, acceleration, and yaw rate333Bezier curves are fit to position
and yaw data and then differentiated to obtain velocity, acceleration, and yaw
rate. The code repository for learning the responsibility allocation function
$\gamma$ can be found here. of vehicle $i$.

###  V-A Choosing a Safety Metric for Autonomous Driving

To define the safety function $h:\mathbb{R}^{n}\to\mathbb{R}$, we begin by
assuming all vehicles must maintain a minimum inter-vehicle distance
$\underline{d}>0$. With this in mind, let
$d_{\textrm{min}}:\mathbb{R}^{4}\times\mathbb{R}^{4}\to\mathbb{R}$ be the
minimum distance between two agents. We can then define the pairwise safe set
between agents $i$ and $j$ to be:

$\displaystyle\mathcal{C}_{ij}=\big{\\{}\mathbf{x}\in\mathbb{R}^{n}~{}|~{}\underbrace{d_{\textrm{min}}(\mathbf{x}_{i},\mathbf{x}_{j})-\underline{d}}_{h_{ij}(\mathbf{x})\triangleq}\geq
0\big{\\}}.$ (23)

However this function is of relative degree 2 w.r.t. $a_{i}$ (i.e.,
$\frac{dh}{dt}$ is not directly affected by $a_{i}$) and describes safety by
only considering the instantaneous current position.

In order to incorporate a temporal aspect and ensure that the time derivative
of $h_{ij}$ is affected by both vehicles’ acceleration and angle rate we take
inspiration from [32] and forward project the current state using a backup
controller $\mathbf{k}_{B}:\mathbb{R}^{4}\to\mathbb{R}^{2}$ over a time
interval $[0,T]$ for $T\in\mathbb{R}_{>0}$. By assumption, for any
$\mathbf{x}_{i}(t)\in\mathbb{R}^{4}$ there exists a unique solution
$\boldsymbol{\phi}:[0,T]\to\mathbb{R}^{4}$ satisfying:

$\displaystyle\frac{d}{dt}\boldsymbol{\phi}(\tau)$
$\displaystyle=\mathbf{f}_{i}(\boldsymbol{\phi}(\tau))+\mathbf{g}_{i}(\boldsymbol{\phi}(\tau))\mathbf{k}_{B}(\boldsymbol{\phi}(\tau)),$
(24) $\displaystyle\boldsymbol{\phi}(0)$ $\displaystyle=\mathbf{x}_{i}(t)$
(25)

The solution $\boldsymbol{\phi}$ starting at $\mathbf{x}_{i}(t)$ is the flow
under $\mathbf{k}_{B}$, and is denoted as
$\boldsymbol{\varphi}_{\tau}(\mathbf{x}_{i})\triangleq\boldsymbol{\phi}(\tau)$.
Similar ideas of forward projection are also seen in Velocity Obstacles [33],
Safety Force Fields [3], and Responsibility-Sensitive Safety [5].

Using the flow $\boldsymbol{\varphi}$ and the distance function $d_{ij}$ we
can find the minimum distance that would be achieved during the interval
$[t,T+t]$ if $\mathbf{k}_{B}$ were the controller for both vehicles:

$\displaystyle
h^{\boldsymbol{\varphi}}_{ij}(\mathbf{x})=\min_{\tau\in[0,T]}d_{\textrm{min}}(\boldsymbol{\varphi}_{\tau}(\mathbf{x}_{i}),\boldsymbol{\varphi}_{\tau}(\mathbf{x}_{j}))-\underline{d},$
(26)

which has the associated safe set
$\mathcal{C}_{ij}^{\varphi}\subseteq\mathcal{C}_{ij}\subset\mathbb{R}^{n}$

$\displaystyle\mathcal{C}_{ij}^{\boldsymbol{\varphi}}=\left\\{\mathbf{x}\in\mathbb{R}^{n}~{}|~{}h^{\boldsymbol{\varphi}}_{ij}(\mathbf{x})\geq
0\right\\}.$ (27)

To compute $h^{\boldsymbol{\varphi}}_{ij}$ the interval $[0,T]$ was
discretized at 100 Hz as in [32, 34] and soft minimum functions were used to
ensure differentiability. It is shown in [35] that the CBF
$h^{\boldsymbol{\varphi}}_{ij}$ constructed from the backup controller is
guaranteed to be of relative degree 1 under mild assumptions.

For the backup controller we choose
$\mathbf{k}_{B}(\mathbf{x}_{i})=\mathbf{0}$ which approximates idling. Unlike
other methods [36, 5] which assume maximum braking for their predictors, we
choose an idling controller since it better approximates nominal driving
behavior and does not introduce worst-case assumptions.

Given these pairwise safe sets $\mathcal{C}_{ij}^{\varphi}$, we can define the
a global safe set
$\mathcal{C}^{\varphi}\subseteq\mathcal{C}_{ij}^{\varphi}\subseteq\mathcal{C}_{ij}$
for all $i\neq j$ as:

$\displaystyle\mathcal{C}_{ij}^{\varphi}=\bigcap_{i\neq
j}\mathcal{C}_{ij}^{\varphi}\quad\textrm{ with }\quad
h(\mathbf{x})=\min_{i\neq j}h_{ij}^{\boldsymbol{\varphi}}(\mathbf{x}).$ (28)

The intersection of safe sets has been studied in [10] and safety of such sets
can be achieved by enforcing the safety constraint for all $i\neq j$.

Figure 3: Crash scenes for forensic analysis. In both rows the plots from left
to right display: the scene and trajectories, the inputs for agents 1 and 2,
the learned responsibility allocations for each agent, and the values of their
RA-CBF constraints (9).

Closed-Loop Simulation Results  Worst-Case Even-Split Our Method Validation
Constraint Violation 43.99% 8.13% 9.51% Closed-Loop Safety Violation 0.833%
2.50% 0.833% Time Spent Off Road 1.48% 0.59% 0.54% Distance Covered Metric
290.21 309.21 307.84

TABLE I: Results for the closed-loop experiments.

###  V-B Learning Model

The inputs of the responsibility allocation function are a semantic image as
see in Fig. 2 and the relative vehicle states of agents $i$ and $j$. The image
is processed by ResNet-18 [37] and the 256 dimensional output is concatenated
to the vehicle states and processed by a multi-layer perceptron (MLP) with 2
hidden layers of size 128 and a single dimensional output.

The hyperparameters chosen were $\lambda_{1}=1$, $\lambda_{2}=10$,
$\lambda_{3}=0.01$, $\alpha=0.5$, $T=1$, $\underline{d}=0.4$, $\ell_{1}=0.1$,
$\ell_{2}=0.01$ and $\theta_{\textrm{max}}=100^{\circ}$ where $\ell_{1}$ and
$\ell_{2}$ were the negative slopes of the MLP’s leaky ReLU activation
functions and $\theta_{\textrm{max}}$ is used to filter the dataset such that
only interactions between vehicles whose headings are within
$\pm\theta_{\textrm{max}}$ are considered. The parameter
$\theta_{\textrm{max}}$ is necessary since our data does not include lane
direction annotation. We note that this does limit the applicability of this
network and plan to include lane direction information in future work.

The network was trained on the NuScenes Boston Seaport dataset. Example
responsibilities generated by our learned model can be found in Fig. 2. These
figures show that our model conforms to the general intuition that the vehicle
behind is more responsible than the vehicle in front for avoiding collisions
between them, and the vehicle stopped at an intersection is responsible for
not interfering with a vehicle already crossing the intersection.

###  V-C Closed-Loop Testing

We use our RA-CBF framework with a learned responsibility allocation function
as a safety-filter in closed-loop control and simulate human-like driving
using the Bi-Level Imitation for Traffic Simulation (BITS) model [38]. The ego
agent follows (13) where
$Q_{i}(\mathbf{x}_{i},\mathbf{u}_{i})=\|\mathbf{k}_{\textrm{bits}}^{+}(\mathbf{x}_{i})-\mathbf{u}_{i}\|^{2}$
and $\mathbf{k}_{\textrm{bits}}^{+}$ is the BITS controller with an additional
1 $\frac{\textrm{m}}{\textrm{sec}^{2}}$ acceleration added to generate
irresponsible desired behavior that must be filtered to ensure safety. The RA-
CBF constraint is applied for each pairwise vehicle $j$ and slack variables
are used to ensure feasibility. We compare our method to the same controller
with two other baseline constraints: (i) “Worst-Case” constraint (7), and (ii)
“Even-Sharing” constraint which is the RA-CBF constraint with
$\gamma(i,\mathbf{x})\equiv 0$.

The closed-loop system was run in 120 scenarios sampled from NuScenes for 10
seconds at 10 Hz. Table (I) contains metrics comparing the controllers. The
Worst-Case controller has the fewest safety violations (as expected), but
worse compatibility with the expert demonstrations as indicated by the large
constraint violation, smallest distance covered, and significant amount of
time off of drivable surfaces. The Even-Sharing controller has fewer
constraint violations on the validation data and the most distance covered,
but allows for more collisions. Our method has a slightly higher number of
constraint violations, but achieves a better safety-performance trade-off.

###  V-D Forensic Analysis

In addition to closed loop control, the values of $\gamma$ and the RA-CBF
constraint provide useful insight when performing forensic analysis on unsafe
driving behaviors. To demonstrate this, we analyze the two collision scenarios
shown in Fig. 3.

Forensic Scenario 1: In the first scene, agent 1 is crossing an intersection
as agent 2 approaches from behind. For more than 2 seconds preceding the
crash, $\gamma(2,\mathbf{x})>\gamma(1,\mathbf{x})$ indicating that agent 2
should have taken a greater share of the responsibility. However, agent 2 has
several large violations of the RA-CBF constraint while agent 1 generally
satisfies it. From this we interpret that agent 2 is responsible for the crash
which aligns with our intuition.

Forensic Scenario 2:  In the second scene, agent 1 is driving along a road
that agent 2 is turning on to. At first, agent 2 is more responsible for not
entering the lane in front of agent 1
$(i.e.,\gamma(2,\mathbf{x})>\gamma(1,\mathbf{x})$ for $t<2$). However, the
responsibility allocation values then become similar as vehicle 2 enters the
lane and accelerates. Throughout the scene, both agents violate safety, but
agent 1 has much larger constraint violations in the moments preceding the
crash due to its continued acceleration. This can be interpreted to indicate
that although it was irresponsible of agent 2 to merge when it did, agent 1 is
ultimately culpable.

We recognize that this analysis is subjective and one of many possible
interpretations, but we believe that our method can provide useful insight
when performing forensic analysis.

##  VI Conclusion

We have presented Responsibility-Aware Control Barrier Functions (RA-CBFs) as
a framework to learn and synthesize safe and responsible driving behaviors.
RA-CBFs are designed to capture the asymmetric sharing of responsibility
between multiple (human) agents and we present a method to learn context-
dependent responsibility allocations from data. We then demonstrated the
efficacy and utility of our approach using real-world driving data. This work
enables various exciting future directions which include incorporating
explicit traffic rules into our responsibility-learning paradigm, comparing
how responsibility allocations vary across geographical regions, and exploring
other application domains such as crowd navigation.

## References

  * [1] “IEEE Standard for Assumptions in Safety-Related Models for Automated Driving Systems,” _IEEE Std 2846-2022_ , 2022.
  * [2] S. Bansal, M. Chen, S. Herbert, and C. J. Tomlin, “Hamilton-Jacobi reachability: A brief overview and recent advances,” in _IEEE Conference on Decision and Control_. Melbourne, Australia: IEEE, Dec. 2017, pp. 2242–2253. [Online]. Available: http://ieeexplore.ieee.org/document/8263977/
  * [3] D. Nistér, H.-L. Lee, J. Ng, and Y. Wang, “The Safety Force Field,” _NVIDIA White Paper_ , 2019.
  * [4] J. Usevitch and D. Panagou, “Adversarial resilience for sampled-data systems under high-relative-degree safety constraints,” _IEEE Transactions on Automatic Control_ , 2022.
  * [5] S. Shalev-Shwartz, S. Shammah, and A. Shashua, “On a formal model of safe and scalable self-driving cars,” _arXiv preprint arXiv:1708.06374_ , 2017.
  * [6] W. Schwarting, A. Pierson, J. Alonso-Mora, S. Karaman, and D. Rus, “Social behavior for autonomous vehicles,” _Proceedings of the National Academy of Sciences_ , vol. 116, no. 50, pp. 24 972–24 978, 2019.
  * [7] B. Toghi, R. Valiente, D. Sadigh, R. Pedarsani, and Y. P. Fallah, “Cooperative autonomous vehicles that sympathize with human drivers,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2021.
  * [8] L. Sun, W. Zhan, M. Tomizuka, and A. D. Dragan, “Courteous autonomous cars,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2018.
  * [9] U. Borrmann, L. Wang, A. D. Ames, and M. Egerstedt, “Control barrier certificates for safe swarm behavior,” _IFAC-PapersOnLine_ , vol. 48, no. 27, pp. 68–73, 2015.
  * [10] P. Glotfelter, I. Buckley, and M. Egerstedt, “Hybrid Nonsmooth Barrier Functions With Applications to Provably Safe and Composable Collision Avoidance for Robotic Systems,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 2, pp. 1303–1310, Apr. 2019.
  * [11] A. Mustafa and D. Panagou, “Adversary detection and resilient control for multi-agent systems,” _IEEE Transactions on Control of Network Systems_ , 2022.
  * [12] J. van den Berg, M. C. Lin, and D. Manocha, “Reciprocal velocity obstacles for real-time multi-agent navigation,” in _IEEE International Conference on Robotics and Automation_ , 2008.
  * [13] M. Chen, J. C. Shih, and C. J. Tomlin, “Multi-vehicle collision avoidance via hamilton-jacobi reachability and mixed integer programming,” in _IEEE Conference on Decision and Control_ , 2016.
  * [14] X. Wang, K. Leung, and M. Pavone, “Infusing reachability-based safety into planning and control for multi-agent interactions,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2020.
  * [15] Y. Lyu, W. Luo, and J. M. Dolan, “Responsibility-associated multi-agent collision avoidance with social preferences,” _arXiv preprint arXiv:2206.09030_ , 2022.
  * [16] K. Guo, D. Wang, T. Fan, and J. Pan, “VR-ORCA: Variable responsibility optimal reciprocal collision avoidance,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 3, pp. 4520–4527, 2021.
  * [17] Y. Chen, A. Singletary, and A. D. Ames, “Guaranteed obstacle avoidance for multi-robot operations with limited actuation: A control barrier function approach,” _IEEE Control Systems Letters_ , vol. 5, no. 1, pp. 127–132, 2020\.
  * [18] C. Dawson, S. Gao, and C. Fan, “Safe control with learned certificates: A survey of neural lyapunov, barrier, and contraction methods,” _arXiv preprint arXiv:2202.11762_ , 2022.
  * [19] A. Robey, H. Hu, L. Lindemann, H. Zhang, D. V. Dimarogonas, S. Tu, and N. Matni, “Learning control barrier functions from expert demonstrations,” in _IEEE Conference on Decision and Control_ , 2020, pp. 3717–3724.
  * [20] Z. Qin, K. Zhang, Y. Chen, J. Chen, and C. Fan, “Learning safe multi-agent control with decentralized neural barrier certificates,” _International Conference on Learning Representations_ , 2021.
  * [21] Y. Lyu, W. Luo, and J. M. Dolan, “Adaptive safe merging control for heterogeneous autonomous vehicles using parametric control barrier functions,” _arXiv preprint arXiv:2202.09936_ , 2022.
  * [22] A. D. Ames, X. Xu, J. W. Grizzle, and P. Tabuada, “Control Barrier Function Based Quadratic Programs for Safety Critical Systems,” _IEEE Transactions on Automatic Control_ , vol. 62, no. 8, pp. 3861–3876, Aug. 2017. [Online]. Available: http://ieeexplore.ieee.org/document/7782377/
  * [23] A. D. Ames, J. W. Grizzle, and P. Tabuada, “Control barrier function based quadratic programs with application to adaptive cruise control,” in _IEEE Conference on Decision and Control_ , 2014, pp. 6271–6278. [Online]. Available: http://ieeexplore.ieee.org/document/7040372/
  * [24] R. Cosner, M. Tucker, A. Taylor, K. Li, T. Molnar, W. Ubelacker, A. Alan, G. Orosz, Y. Yue, and A. Ames, “Safety-aware preference-based learning for safety-critical control,” in _Learning for Dynamics and Control Conference_. PMLR, 2022, pp. 1020–1033.
  * [25] D. Helbing and P. Molnar, “Social force model for pedestrian dynamics,” _Physical review E_ , vol. 51, no. 5, p. 4282, 1995.
  * [26] P. Glotfelter, J. Cortes, and M. Egerstedt, “A Nonsmooth Approach to Controller Synthesis for Boolean Specifications,” _IEEE Transactions on Automatic Control_ , pp. 1–1, 2020.
  * [27] A. J. Taylor, A. Singletary, Y. Yue, and A. D. Ames, “A control barrier perspective on episodic learning via projection-to-state safety,” _IEEE Control Systems Letters_ , vol. 5, no. 3, pp. 1019–1024, 2020.
  * [28] D. R. Scobee and S. S. Sastry, “Maximum likelihood constraint inference for inverse reinforcement learning,” _arXiv preprint arXiv:1909.05477_ , 2019\.
  * [29] B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, _et al._ , “Maximum entropy inverse reinforcement learning.” in _AAAI Conference on Artificial Intelligence_ , vol. 8, 2008, pp. 1433–1438.
  * [30] N. Aghasadeghi and T. Bretl, “Maximum entropy inverse reinforcement learning in continuous state spaces with path integrals,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2011, pp. 1561–1566.
  * [31] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuScenes: A multimodal dataset for autonomous driving,” in _IEEE / CVF Computer Vision and Pattern Recognition Conference_ , 2020.
  * [32] T. Gurriet, M. Mote, A. Singletary, P. Nilsson, E. Feron, and A. D. Ames, “A scalable safety critical control framework for nonlinear systems,” _IEEE Access_ , vol. 8, pp. 187 249–187 275, 2020.
  * [33] D. Wilkie, J. Van Den Berg, and D. Manocha, “Generalized velocity obstacles,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2009, pp. 5573–5578.
  * [34] R. K. Cosner, A. W. Singletary, A. J. Taylor, T. G. Molnar, K. L. Bouman, and A. D. Ames, “Measurement-Robust Control Barrier Functions: Certainty in Safety with Uncertainty in State,” in _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2021.
  * [35] Y. Chen, M. Jankovic, M. Santillo, and A. D. Ames, “Backup control barrier functions: Formulation and comparative study,” in _IEEE Conference on Decision and Control_ , 2021, pp. 6835–6841.
  * [36] D. Nister, O. Naroditsky, and J. Bergen, “Visual odometry,” in _IEEE / CVF Computer Vision and Pattern Recognition Conference_ , 2004.
  * [37] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” _CoRR_ , vol. abs/1512.03385, 2015. [Online]. Available: http://arxiv.org/abs/1512.03385
  * [38] D. Xu, Y. Chen, B. Ivanovic, and M. Pavone, “BITS: Bi-level imitation for traffic simulation,” _arXiv preprint arXiv:2208.12403_ , 2022.


initial correlations, the hot spin A cools down and the cold spin B heats up
(cyan circles in panel a and b). By contrast, in the presence of initial
quantum correlations, the heat current is reversed as the hot spin A gains and
the cold spin B loses energy (orange squares in panel a and b). This reversal
is made possible by a decrease in the mutual information and the geometric
quantum discord. Figure adapted from the Ref. [19].

Figure 27 shows the experimental result of the reverse in the heat flow using
quantum correlations implemented using NMR techniques [19]. It is observed the
reversal of the heat flow between the two quantum-correlated spins with
different effective temperatures, i.e., heat flows from the colder spin to the
hotter spin. During this reversal behavior, it is also measured the decrease
of the initial quantum correlations during the dynamics [19]. This highlights
that depending on the initial conditions, uncorrelated or correlated in the
experiment, the heat flow direction can be reversed. This experiment
contributes to showing the high degree of control in exploring the quantum
nature of the two-spin system as well as carrying out a test-of-principle
experiment about the role played by non-classical correlations in quantum
thermodynamics.

##  V Conclusion and outlook

We presented a short overview of some approaches to quantum thermodynamics,
highlighting experimental implementations using the NMR platform. One of the
biggest motivations of quantum thermodynamics has been to extend thermodynamic
principles to the quantum world, where energy fluctuations coherence, and
quantum correlations take into play. Addressing these quantum correlations has
made it possible to discuss the role of quantum resources in devices such as
thermal engines, refrigerators, thermal accelerators, and batteries.

Quantum stochastic thermodynamics has brought the possibility to deal with
fluctuation theorems enabling us to consider explicitly energy fluctuations in
non-equilibrium dynamics and settle relations that connect equilibrium
properties of thermodynamics relevance with non-equilibrium features, beyond
dealing with processes involving measurement and feedback control. Once
fluctuations have a pivotal role in small systems, many efforts have been made
to establish a trade-off between fluctuations and dissipation in stochastic
thermodynamics. A set of relations addressing this goal has been collectively
referred to as thermodynamic uncertainty relations (TURs) that can constrain
power, fluctuations, and the entropic cost of quantum tasks. Additionally,
investigating the connection between information and energy quantum
information thermodynamics has aided to solve misinterpretations in famous
conundrums of physics, such as Maxwell’s demon and Szilard’s engine besides
addressing the thermodynamic cost of quantum operations.

This vibrant branch of research is tremendously active and provides the
intersection of several areas, including stochastic thermodynamics, quantum
physics, and condensed matter, beyond having meaningful connections with
computing science and engineering. Owing to the interdisciplinary and broad
nature of this field, many financial efforts have been made to develop quantum
thermodynamic devices and speed up this new industrial revolution on the
tiniest scale (quantum technology 2.0). Although we have focused on the NMR
technique, which significantly contributed to the field due to its precision
in measurement and control, quantum thermodynamics has been widely fueled by
experiments in several physical approaches such as trapped ions, cold atomic,
quantum optics, single electron transistors, cavity QED, Nitrogen-Vacancy
centers, and superconducting circuits.

## Acknowledgments

The authors acknowledge the funding agencies CAPES (Grants No.
88887.354951/2019-00 and No. 88887.499539/2020-00), FAPESP (Grant No.
2019/04184-5), CNPq (Grants No. 310430/2018-6 and No. 142531/2020-0), and the
National Institute for Science and Technology of Quantum Information (CNPq,
Grant No. INCT-IQ 465469/2014-0). The authors are grateful to the Multiuser
Central Facilities (CEM-UFABC) for the experimental support. P.R.D.
acknowledges support by the Foundation for Polish Science (FNP) (IRAP project,
ICTQT, Contract No. MAB/2018/5, co-financed by EU within Smart Growth
Operational Programme). R.M.S. also acknowledges Ministry of Science and
Technology of China, through the High-End Foreign Expert Program (Grant No.
G2021016021L).

Author contributions
All authors discussed the topics and contributed to write the paper.

Competing interests
The authors declare no competing interests.

## References

  * [1] F. Arute, K. Arya, R. Babbush, et al. Quantum supremacy using a programmable superconducting processor, Nature 574, 505-510 (2019).
  * [2] Z. Gedik, I. A. Silva, B. Çakmak, G. Karpat, E. L. G. Vidoto, D. O. Soares-Pinto, E. R. deAzevedo, and F. F. Fanchini, Computational speed-up with a single qudit, Sci. Rep. 5, 14671 (2015).
  * [3] J. A. Jones, Quantum Computing and Nuclear Magnetic Resonance, Phys. Chem. Comm. 11 (2001).
  * [4] J. A. Jones, Quantum computing with NMR, Progress in Nuclear Magnetic Resonance Spectroscopy 59, 91-120 (2011).
  * [5] C. A. Riofrío, D. Gross, S. T. Flammia, T. Monz, D. Nigg, R. Blatt, and J. Eisert, Experimental quantum compressed sensing for a seven-qubit system, Nat Commun 8, 15305 (2017).
  * [6] J. Aasi, J. Abadie, et. al., Enhanced sensitivity of the LIGO gravitational wave detector by using squeezed states of light, Nature Photon 7, 613 (2013).
  * [7] L. Garbe, M. Bina, A. Keller, M. G. A. Paris, and S. Felicetti, Critical Quantum metrology with a finite-component quantum phase transition, Phys. Rev. Lett. 124, 120504 (2020).
  * [8] F. Binder, L. A. Correa, C. Gogolin, J. Anders, and G. Adesso, eds., Thermodynamics in the Quantum Regime: Fundamental Aspects and New Directions (Springer International, Cham, Swtizerland, 2018).
  * [9] S. Deffner and S. Campbell, Quantum Thermodynamics An Introduction to the Thermodynamics of Quantum Information (Morgan & Claypool, San Rafael, CA, 2019).
  * [10] R. Kosloff, Quantum thermodynamics: A dynamical viewpoint, Entropy 15, 2100 (2013).
  * [11] J. Goold, M. Huber, A. Riera, L. del Rio, and P. Skrzypczyk, The role of quantum information in thermodynamics a topical review, J. Phys. A: Math. Theor. 49, (2016) 143001.
  * [12] S. Vinjanampathy and J. Anders, Quantum Thermodynamics, Contemporary Physics 57, 545 (2016).
  * [13] F. Brandão, M. Horodecki, N. Ng, J. Oppenheim, and S. Wehner, The second laws of quantum thermodynamics, Proc. Natl. Acad. Sci. U.S.A. 112, 3275 (2015).
  * [14] J. Millen and A. Xuereb, Perspective on quantum thermodynamics, New J. Phys. 18 011002 (2016).
  * [15] T. B. Batalhão, A. M. Souza, L. Mazzola, R. Auccaise, R. S. Sarthour, I. S. Oliveira, J. Goold, G. De Chiara, M. Paternostro, and R. M. Serra, Experimental reconstruction of work distribution and study of fluctuation relations in a closed quantum system, Phys. Rev. Lett. 113, 140601 (2014).
  * [16] P. A. Camati and R. M. Serra, Verifying Detailed Fluctuation Relations for Discrete Feedback Controlled Quantum Dynamics, Phys. Rev. A 97, 042127 (2018).
  * [17] T. Denzler, J. F. G. Santos, E. Lutz, and R. M. Serra, Nonequilibrium fluctuations of a quantum heat engine, arXiv:2104.13427 (2021).
  * [18] K. Micadei, G. T. Landi, and E. Lutz, Quantum Fluctuation Theorems beyond Two-Point Measurements, Phys. Rev. Lett. 124, 090602 (2020).
  * [19] K. Micadei, J. P. S. Peterson, A. M. Souza, R. S. Sarthour, I. S. Oliveira, G. T. Landi, T. B. Batalhão, R. M. Serra, and E. Lutz, Reversing the direction of heat flow using quantum correlations, Nat. Comm. 10, 2456 (2019).
  * [20] U. Seifert, Stochastic thermodynamics, fluctuation theorems and molecular machines, Rep. Prog. Phys. 75 126001 (2012).
  * [21] R. Dorner, S. R. Clark, L. Heaney, R. Fazio, J. Goold, V. Vedral, Extracting quantum work statistics and fluctuation theorems by single qubit interferometry, Phys. Rev. Lett. 110, 230601 (2013).
  * [22] L. Mazzola, G. De Chiara, and M. Paternostro, Measuring the characteristic function of the work distribution, Phys. Rev. Lett. 110, 230602 (2013).
  * [23] M. Campisi, R. Blattmann, S. Kohler, D. Zueco, and P. Hanggi, Employing circuit QED to measure nonequilibrium work fluctuations, New J. Phys. 15, 105028 (2013).
  * [24] H. Tasaki, Jarzynski Relations for Quantum Systems and Some Applications, arXiv:cond-mat/0009244v2; J. Kurchan, A Quantum Fluctuation Theorem, arXiv:cond-mat/0007360v2; S. Mukamel, Quantum Extension of the Jarzynski Relation: Analogy with Stochastic Dephasing, Phys. Rev. Lett. 90, 170604 (2003).
  * [25] P. Talkner, E. Lutz, and P. Hänggi, Fluctuation theorems: Work is not an observable, Phys. Rev. E 75, 050102(R) (2007).
  * [26] M. Esposito, U. Harbola, and S. Mukamel, Nonequilibrium fluctuations, fluctuation theorems, and counting statistics in quantum systems, Rev. Mod. Phys. 81, 1665 (2009).
  * [27] M. Campisi, P. Hänggi, and P. Talkner, Quantum fluctuation relations: Foundations and applications, Rev. Mod. Phys. 83, 771 (2011).
  * [28] K. Micadei, J. P. S. Peterson, A. M. Souza, R. S. Sarthour, I. S. Oliveira, G. T. Landi, R. M. Serra, and E. Lutz, Experimental Validation of Fully Quantum Fluctuation Theorems Using Dynamic Bayesian Networks, Phys. Rev. Lett. 127, 180603 (2021).
  * [29] C. Jarzynski and D. K. Wöjcik, Classical and quantum fluctuation theorems for heat exchange, Phys. Rev. Lett. 92 230602 (2004).
  * [30] N. A. Sinitsyn, Fluctuation relation for heat engines, J. Phys. A: Math. Theor. 44, 405001 (2011).
  * [31] M. Campisi, J. Pekola, and R. Fazio, Nonequilibrium fluctuations in quantum heat engines: theory, example, and possible solid state experiments, New J. Phys. 17, 035012 (2015).
  * [32] C. I. Henao and R. M. Serra, Role of quantum coherence in the thermodynamics of energy transfer, Physical Review E 97, 062105 (2018).
  * [33] M. Campisi, Fluctuation relation for quantum heat engines and refrigerators, J. Phys. A: Math. Theor. 47, 245001 (2014).
  * [34] H. E. D. Scovil and E. O. Schulz-DuBois, Three-Level Masers as Heat Engines, Phys. Rev. Lett. 2 262 (1959).
  * [35] M. O. Scully, M. Suhail Zubairy, G. S. Agarwal, and H. Walther, Extracting Work from a Single Heat Bath via Vanishing Quantum Coherence, Science 299 5608 (2003).
  * [36] J. L. D. de Oliveira, M. Rojas, and C. Filgueiras, Two coupled double quantum-dot systems as a working substance for heat machines, Physical Review E 104, 014149 (2021).
  * [37] P. A. Camati, J. F. G. Santos, and R. M. Serra, Coherence effects in the performance of the quantum Otto heat engine. Phys. Rev. A 99, 062103 (2019).
  * [38] J. Klatzow, J. N. Becker, P. M. Ledingham, C. Weinzetl, Krzysztof T. Kaczmarek, D. J. Saunders, J. Nunn, I. A. Walmsley, R. Uzdin, and E. Poem, Experimental Demonstration of Quantum Effects in the Operation of Microscopic Heat Engines. Phys. Rev. Lett. 122, 110601 (2019).
  * [39] J. P. S. Peterson, T. B. Batalhão, M. Herrera, A. M. Souza, R. S. Sarthour, I. S. Oliveira, and R. M. Serra, Experimental characterization of a spin quantum heat engine. Phys. Rev. Lett. 123, 240601 (2019).
  * [40] D. Felce and V. Vedral, Quantum refrigeration with indefinite causal order, Phys. Rev. Lett. 125, 070603 (2020).
  * [41] D. Felce, V. Vedral, and F. Tennie, Refrigeration with indefinite causal orders on a cloud quantum computer, arXiv:2107.12413 (2021).
  * [42] J. Yi, P. Talkner and Y. W. Kim, Single-temperature quantum engine without feedback control, Phys. Rev. E 96, 022108 (2017).
  * [43] C. Elouard, D. Herrera-Martí, B. Huard, and A. Auffèves, Extracting Work from Quantum Measurement in Maxwell’s Demon Engines, Phys. Rev. Lett. 118, 260603 (2017).
  * [44] K. Brandner, M. Bauer, M. T. Schmid and U. Seifert, Coherence-enhanced efficiency of feedback-driven quantum engines, New J. Phys. 17, 065006 (2015).
  * [45] Z. Lin, S. Su, J. Chen, J. Chen, and J. F. G. Santos, Suppressing coherence effects in quantum-measurement-based engines, Phys. Rev. A 104, 062210 (2021).
  * [46] L. Bresque, P. A. Camati, S. Rogers, K. Murch, A. N. Jordan, and A. Auffèves, Two-Qubit Engine Fueled by Entanglement and Local Measurements, Phys. Rev. Lett. 126, 120605 (2021).
  * [47] L. Buffoni, A. Solfanelli, P. Verrucchi, A. Cuccoli, and M. Campisi, Quantum Measurement Cooling, Phys. Rev. Lett. 122, 070603 (2019).
  * [48] X. Ding, J. Yi, Y. W. Kim, and P. Talkner, Measurement-driven single temperature engine,  Phys. Rev. E 98, 042122 (2018).
  * [49] A. Jordan, C. Elouard, and A. Auffèves, Quantum measurement engines and their relevance for quantum interpretations, Quantum Stud. Math. Found. 7, 203 (2020).
  * [50] M. H. Mohammady and J. Anders, A quantum Szilard engine without heat from a thermal reservoir, New J. Phys. 19, 113026 (2017).
  * [51] S. Chand and A. Biswas, Single-ion quantum Otto engine with always-on bath interaction, Europhys. Lett. 118, 60003 (2017).
  * [52] S. Chand and A. Biswas, Critical-point behavior of a measurement-based quantum heat engine, Phys. Rev. E 98, 052147 (2018).
  * [53] C. Elouard and A. N. Jordan, Efficient Quantum Measurement Engines, Phys. Rev. Lett. 120, 260601 (2018).
  * [54] M. F. Anka, T. R. de Oliveira, and D. Jonathan, Measurement-based quantum heat engine in a multilevel system, Phys. Rev. E 104, 054128 (2021).
  * [55] C.-K. Hu, A. C. Santos, J.-M. Cui, Y.-F. Huang, D. O. S. Pinto, M. S. Sarandy, C.-F. Li, and G.-C. Guo, Quantum thermodynamics in adiabatic open systems and its trapped-ion experimental realization,  npj Quantum Inf. 6, 73 (2020).
  * [56] P. A.Ivanov, Quantum thermometry with trapped ions, Optics Communications 436, 101-107 (2019).
  * [57] J. Rossnagel, S. T. Dawkins, K. N. Tolazzi, O. Abah, E. Lutz, F. Schmidt-Kaler, and K. Singer, A single-atom heat engine, Science 352, 6283 (2016), 325-329 (2016).
  * [58] M. Cattaneo, G. S. Paraoanu, Engineering Dissipation with Resistive Elements in Circuit Quantum Electrodynamics, Advanced Quantum Technologies, 4, 2100054 (2021).
  * [59] J. Pekola, Towards quantum thermodynamics in electronic circuits, Nature Phys 11, 118123 (2015).
  * [60] C. Elouard, Thermodynamics of Quantum Open Systems: Applications in Quantum Optics and Optomechanics, arXiv:1709.02744
  * [61] G. L. Zanin, T. Häffner, M. A. A. Talarico, E. I. Duzzioni, P. H. Souto Ribeiro, G. T. Landi, and L. C. Céleri, Experimental quantum thermodynamics with linear optics,  Braz. J. Phys. 49, 783798 (2019).
  * [62] M. H. M. Passos, Alan C. Santos, Marcelo S. Sarandy, J. A. O. Huguenin, Optical simulation of a quantum thermal machine, Physical Review A, 100, (2019).
  * [63] M. Brunelli, A. Xuereb, A. Ferraro, G. De Chiara, N. Kiesel, and M. Paternostro, Out-of-equilibrium Thermodynamics of Quantum Optomechanical Systems, New J. Phys. 17, 035016 (2015).
  * [64] S. Pigeon, L. Fusco, A. Xuereb, G. De Chiara, and M. Paternostro, Thermodynamics of trajectories of a quantum harmonic oscillator coupled to N baths, Phys. Rev. A 92, 013844 (2015).
  * [65] Experimental investigation of a quantum heat engine powered by generalized measurements, V. F. Lisboa, P. R. Dieguez, J. R. Guimarães, J. F. G. Santos, and R. M. Serra, Phys. Rev. A 106, 022436 (2022).
  * [66] N. A. Gershenfeld and I. L. Chuang, Bulk Spin-Resonance Quantum Computation, Science, 275, 275-350 (1997).
  * [67] D. G. Cory, A. F. Fahmy, and T. F. Havel, Ensemble quantum computing by NMR spectroscopy, Proceedings of the National Academy of Sciences of the United States of America, 94, 1634 (1997).
  * [68] E. Knill and R. Laflamme, Power of One Bit of Quantum Information, Phys. Rev. Lett. 81, 5672 (1998).
  * [69] L. M. K. Vandersypen and I. L. Chuang, NMR techniques for quantum control and computation, Rev. Mod. Phys. 76, 5672 (2005).
  * [70] S. Pal, T. S. Mahesh, and B. K. Agarwalla, Experimental demonstration of the validity of the quantum heat-exchange fluctuation relation in an NMR setup, Phys. Rev. A 100, 042119 (2019).
  * [71] J. Maziero, R. Auccaise, L. C. Céleri, D. O. Soares-Pinto, E. R. deAzevedo, T. J. Bonagamba, R. S. Sarthour, I. S. Oliveira, and R. M. Serra, Quantum Discord in Nuclear Magnetic Resonance Systems at Room Temperature, Braz. J. Phys. 43, 86 (2013).
  * [72] H. B. Callen, Thermodynamics and Introduction to Thermostatistics (John Wiley & Sons, New York, 1985).
  * [73] N. M. Myers, O. Abah, and S. Deffner, Quantum thermodynamic devices: from theoretical proposals to experimental reality, AVS Quantum Sci. 4, 027101 (2022).
  * [74] A. G. Redfield, The Theory of Relaxation Processes, Adv. Magn. Opt. Reson. 1 (1965).
  * [75] H.-P. Breuer and F. Petruccione, The Theory of Open Quantum Systems (Oxford University Press, 2007).
  * [76] D. O. Soares-Pinto, M. H. Y. Moussa, J. Maziero, E. R. deAzevedo, T. J. Bonagamba, R. M. Serra, L. C. Céleri, Equivalence between Redfield and master equation approaches for a time-dependent quantum system and coherence control, Phys. Rev. A 83, 062336 (2011).
  * [77] M. A. Nielsen and I. L. Chuang, Quantum Computation and Quantum (Cambridge University Press, 2011).
  * [78] J. M. G. Vilar and J. M. Rubi, Failure of the Work-Hamiltonian Connection for Free-Energy Calculations, Phys. Rev. Lett. 100, 020601 (2008).
  * [79] D. Gelbwaser-Klimovsky and A. Aspuru-Guzik, On thermodynamic inconsistencies in several photosynthetic and solar cell models and how to fix them, Chem. Sci. 8, 1008 (2017).
  * [80] W. Niedenzu, M. Huber, and E. Boukobza, Concepts of work in autonomous quantum heat engines, Quantum 3, 195 (2019).
  * [81] R. Sampaio, S. Suomela, T. Ala-Nissila, J. Anders, and T. G. Philbin, Quantum work in the Bohmian framework, Phys. Rev. A 97, 012131 (2018).
  * [82] R. Alicki, The quantum open system as a model of the heat engine, J. Phys. A: Math. Gen. 12, L103 (1979).
  * [83] J. M. R. Parrondo, J. M. Horowitz, and T. Sagawa, Thermodynamics of information, Nature Phys. 11, 131 (2015).
  * [84] Gardas and S. Deffner, Thermodynamic universality of quantum Carnot engines, Phys. Rev. E 92, 042126 (2015).
  * [85] S. Alipour, F. Benatti, F. Bakhshinezhad, M. Afsary, S. Marcantoni, and A. T. Rezakhani, Correlations in quantum thermodynamics: Heat, work, and entropy production, Sci. Rep. 6, 35568 (2016).
  * [86] Y.-H. Shi, H.-L. Shi, X.-H. Wang, M.-L. Hu, S.-Y. Liu, W.-L. Yang, and H. Fan, Quantum coherence in a quantum heat engine, J. Phys. A: Math. Theor. 53, 085301 (2020) .
  * [87] H. Weimer, M. J. Henrich, F. Rempp, H. Schröder, and G. Mahler, Local effective dynamics of quantum systems: A generalized approach to work and heat, Europhys. Lett. 83, 30008 (2008) .
  * [88] J. J. Alonso, E. Lutz, and A. Romito, Thermodynamics of Weakly Measured Quantum Systems, Phys. Rev. Lett. 116, 080403 (2016).
  * [89] R. Dann and R. Kosloff, Unification of the first law of quantum thermodynamics,  arXiv:2208.10561 (2022).
  * [90] H. T. Quan, Yu-xi Liu, C. P. Sun, and Franco Nori, Quantum thermodynamic cycles and quantum heat engines, Phys. Rev. E 76, 031105 (2007).
  * [91] S. Alipour, A. T. Rezakhani, A. Chenu, A. del Campo, and T. Ala-Nissila, Entropy-based formulation of thermodynamics in arbitrary quantum evolution, Phys. Rev. A 105, L040201 (2022) .
  * [92] B. L. Bernardo, Unraveling the role of coherence in the first law of quantum thermodynamics, Phys. Rev. E 102, 062152 (2020).
  * [93] C. Jarzynski, Nonequilibrium Equality for Free Energy Differences, Phys. Rev. Lett. 78, 2690 (1997).
  * [94] G. E. Crooks, Entropy production fluctuation theorem and the nonequilibrium work relation for free-energy differences. Phys. Rev. E 60, 2721 (1999).
  * [95] I. S. Oliveira, T. J. Bonagamba, R. S. Sarthour, J. C. C. Freitas, and R. R. de Azevedo, NMR Quantum Information Processing (Elsevier, Amsterdam, 2007).
  * [96] S. Lahiri, S Rana, and A M Jayannavar, Fluctuation relations for heat engines in time-periodic steady states, J. Phys. A: Math. Theor. 45, 465001 (2012).
  * [97] M. Herrera, J. P. S. Peterson, R. M. Serra, and I. D’Amico, Easy Access to Energy Fluctuations in Nonequilibrium Quantum Many-Body Systems, Phys. Rev. Lett. 127, 030602 (2021).
  * [98] Gabriel T. Landi and Mauro Paternostro, Irreversible entropy production: From classical to quantum, Rev. Mod. Phys. 93, 035008 (2021).
  * [99] R. Kawai, J. M. R. Parrondo, C. Van den Broeck, Dissipation: The Phase-Space Perspective, Physical Review Letters, 98, (2007).
  * [100] Vaikuntanathan, C. Jarzynski, Dissipation and Lag in Irreversible Processes, Europhys. Lett. 87, 60005 (2009).
  * [101] T. B. Batalhao, A. M. Souza, R. S. Sarthour, I. S. Oliveira, M. Paternostro, E. Lutz, and R. M. Serra, Irreversibility and the Arrow of Time in a Quenched Quantum System, Phys. Rev. Lett. 115, 190601 (2015).
  * [102] A. C. Barato and U. Seifert, Thermodynamic Uncertainty Relation for Biomolecular Processes, Phys. Rev. Lett. 114 158101 (2015).
  * [103] P. Pietzonka, A. C. Barato, and U. Seifert, Universal bounds on current fluctuations, Phys. Rev. E 93 052145 (2016).
  * [104] P. Pietzonka, U. Seifert, Universal Trade-Off between Power, Efficiency, and Constancy in Steady-State Heat Engines, Phys. Rev. Lett. 120 190602 (2018).
  * [105] T. R. Gingrich, J. M. Horowitz, N. Perunov, and J. L. England, Dissipation Bounds all Steady State Current Fluctuations, Phys. Rev. Lett. 116 120601 (2016).
  * [106] M. Polettini, A. Lazarescu, and M. Esposito, Tightening the uncertainty principle for stochastic currents, Phys. Rev. E 94 052104 (2016).
  * [107] J. M. Horowitz and T. R. Gingrich, Proof of the finite-time thermodynamic uncertainty relation for steady-state currents, Phys. Rev. E 96 020103(R) (2017).
  * [108] K. Proesmans and C. V. den Broeck, Discrete-time thermodynamic uncertainty relation, Europhys. Lett. 119 20001 (2017).
  * [109] A. C. Barato, R. Chetrite, A. Faggionato, and D. Gabrielli, Bounds on current fluctuations in periodically driven systems, New J. Phys. 20 103023 (2018).
  * [110] T. Koyuk, U. Seifert, and P. Pietzonka, A generalization of the thermodynamic uncertainty relation to periodically driven systems, J. Phys. A 52 02LT02 (2018).
  * [111] A. Dechant, Multidimensional thermodynamic uncertainty relations, J. Phys. A: Math. Theor. 52 035001 (2019).
  * [112] J. P. Garrahan, Simple bounds on fluctuations and uncertainty relations for first-passage times of counting observables, Phys. Rev. E 95 032134 (2017).
  * [113] T. R. Gingrich and J. M. Horowitz, Fundamental Bounds on First Passage Time Fluctuations for Currents, Phys. Rev. Lett. 119 170601 (2017).
  * [114] P. P. Potts and P. Samuelsson, Thermodynamic uncertainty relations including measurement and feedback, Phys. Rev. E 100 052137 (2019).
  * [115] T. Van Vu and Y. Hasegawa, Uncertainty relation under information measurement and feedback control, J. Phys. A: Math. Theor. 53 075001 (2020).
  * [116] G. Paneru, S. Dutta, T. Tlusty, and H. K. Pak, Reaching and violating thermodynamic uncertainty bounds in information engines, Phys. Rev. E 102 032126 (2020).
  * [117] K. Proesmans and J. M. Horowitz, Hysteretic thermodynamic uncertainty relation for systems with broken time-reversal symmetry, J. Stat. Mech. 054005 (2019).
  * [118] H.-M. Chun, L. P. Fischer, and U. Seifert, Effect of a magnetic field on the thermodynamic uncertainty relation, Phys. Rev. E 99 042128 (2019).
  * [119] K. Liu, Z. Gong, and M. Ueda, Thermodynamic Uncertainty Relation for Arbitrary Initial States, Phys. Rev. Lett.125 140602 (2020).
  * [120] Y. Hasegawa and T. Van Vu, Fluctuation Theorem Uncertainty Relation,  Phys. Rev. Lett. 123 110602 (2019).
  * [121] A. M. Timpanaro, G. Guarnieri, J. Goold, and G. T. Landi, Thermodynamic Uncertainty Relations from Exchange Fluctuation Theorems, Phys. Rev. Lett. 123 090604 (2019).
  * [122] S. Pal, S. Saryal, D. Segal, T. S. Mahesh, and B. K. Agarwalla, Experimental study of the thermodynamic uncertainty relation, Phys. Rev. Research. 2 022044(R) (2020).
  * [123] J. Roßnagel, O. Abah, F. Schmidt-Kaler, K. Singer, and E. Lutz, Nanoscale Heat Engine Beyond the Carnot Limit, Physical Review Letters, 112, (2014).
  * [124] de Oliveira Junior, A., de Oliveira, M.C., Unravelling the non-classicality role in Gaussian heat engines, Sci Rep 12, 10412 (2022).
  * [125] J. Klaers, S. Faelt, A. Imamoglu, and E. Togan, Squeezed Thermal Reservoirs as a Resource for a Nanomechanical Engine beyond the Carnot Limit, Physical Review X, 7, (2017).
  * [126] R. J. de Assis, T. M. de Mendonça, C. J. Villas-Boas, A. M. de Souza, R. S. Sarthour, I. S. Oliveira, and N. G. de Almeida, Efficiency of a Quantum Otto Heat Engine Operating under a Reservoir at Effective Negative Temperatures, Phys. Rev. Lett. 122, 240602 (2019).
  * [127] G. Rubino, G. Manzano, and Č. Brukner, Quantum superposition of thermodynamic evolutions with opposing time’s arrows, Commun. Phys. 4 1 (2021).
  * [128] X. Nie, X. Zhu, K. Huang, K. Tang, X. Long, Z. Lin, Y. Tian, C. Qiu, C. Xi, X. Yang, J. Li, Y. Dong, T. Xin, and D. Lu, Experimental Realization of a Quantum Refrigerator Driven by Indefinite Causal Orders, Phys. Rev. Lett. 129, 100603 (2022).
  * [129] P. R. Dieguez, V. F. Lisboa, and R. M. Serra, Thermal devices powered by generalized measurements with indefinite causal order, Phys. Rev. A 107, 012423 (2023).
  * [130] O. Oreshkov, F. Costa, and Č. Brukner, Quantum correlations with no causal order, Nat. Commun. 3, 1 (2012).
  * [131] G. Chiribella, G. M. D’Ariano, P. Perinotti, and B. Valiron, Quantum computations without definite causal structure, Phys. Rev. A 88, 022318 (2013).
  * [132] M. Araújo, C. Branciard, F. Costa, A. Feix, C. Giarmatzi, and Č. Brukner, Witnessing causal nonseparability, New J. Phys. 17, 102001 (2015).
  * [133] L. M. Procopio, A. Moqanaki, M. Araújo, F. Costa, I. A. Calafell, E. G. Dowd, D. R. Hamel, L. A. Rozema, Č. Brukner, and P. Walther, Experimental superposition of orders of quantum gates, Nat. Commun. 6, 7913 (2015).
  * [134] K. Wei, N. Tischler, S.-R. Zhao, Y.-H. Li, J. M. Arrazola, Y. Liu, W. Zhang, H. Li, L. You, Z. Wang, et al., Experimental quantum switching for exponentially superior quantum communication complexity, Phys. Rev. Lett. 122, 120504 (2019).
  * [135] Y. Guo, X.-M. Hu, Z.-B. Hou, H. Cao, J.-M. Cui, B.-H. Liu, Y.-F. Huang, C.-F. Li, G.-C. Guo, and G. Chiribella, Experimental transmission of quantum information using a superposition of causal orders, Phys. Rev. Lett. 124, 030502 (2020).
  * [136] G. Rubino, L. A. Rozema, D. Ebler, H. Kristjáansson, S. Salek, P. A. Guéerin, A. A. Abbott, C. Branciard, Č. Brukner, G. Chiribella, and P. Walther, Experimental quantum communication enhancement by superposing trajectories, Phys. Rev. Res. 3, 013093 (2021).
  * [137] K. Goswami and J. Romero, Experiments on quantum causality, AVS Quantum Sci. 2, 037101 (2020).
  * [138] T. Guha, M. Alimuddin, and P. Parashar, Thermodynamic advancement in the causally inseparable occurrence of thermal maps, Phys. Rev. A 102, 032215 (2020).
  * [139] K. Simonov, G. Francica, G. Guarnieri, and M. Paternostro, Work extraction from coherently activated maps via quantum switch, Phys. Rev. A 105, 032217 (2022).
  * [140] C. Elouard, D. Herrera-Martí, M. Clusel, and A. Auffèves, The role of quantum measurement in stochastic thermodynamics, Nat. Quant. Info. 3, 9 (2017).
  * [141] M. Campisi, J. Pekola, and R. Fazio, Feedback-controlled heat transport in quantum devices: theory and solid-state experimental proposal, New J. Phys. 19, 053027 (2017).
  * [142] S. Chand and A. Biswas, Measurement-induced operation of two-ion quantum heat machines, Phys. Rev. E 95, 032111 (2017).
  * [143] N. Behzadi, Quantum engine based on general measurements, J. Phys. A: Math. Theor. 54 015304 (2020).
  * [144] K. Jacobs, The Second Law of Thermodynamics and Quantum Feedback Control: Maxwell’s Demon with Weak Measurements, Phys. Rev. A 80, 012322 (2009).
  * [145] A. K. Pati, C. Mukhopadhyay, S. Chakraborty, and S. Ghosh, Quantum precision thermometry with weak measurements, Phys. Rev. A 102, 012204 (2020).
  * [146] J. J. Alonso, E. Lutz, and A. Romito, Thermodynamics of Weakly Measured Quantum Systems, Phys. Rev. Lett. 116, 080403 (2016).
  * [147] L. Mancino, M. Sbroscia, E. Roccia, et al, The entropic cost of quantum generalized measurements. npj Quantum Inf 4, 20 (2018).
  * [148] O. Oreshkov and T. Brun, Weak measurements are universal, Phys. Rev. Lett. 95, 1104909 (2005).
  * [149] P. R. Dieguez, R.M. Angelo, Information-reality complementarity, The role of measurements and quantum reference frames, Phys. Rev. A 97, 022107 (2018).
  * [150] Y. Pan, J. Zhang, E. Cohen, et al, Weak-to-strong transition of quantum measurement in a trapped-ion system, Nat. Phys. 16, 12061210 (2020).
  * [151] M. Naghiloo, D. Tan, P. M. Harrington, J. J. Alonso, E. Lutz, A. Romito, and K. W. Murch, Heat and work along individual trajectories of a quantum bit, Phys. Rev. Lett. 124, 110604 (2020).
  * [152] M. Naghiloo, J. J. Alonso, A. Romito, E. Lutz, and K. W. Murch, Information Gain and Loss for a Quantum Maxwell’s Demon, Phys. Rev. Lett. 121, 030604 (2018).
  * [153] S. Hernández-Gómez, S. Gherardini, N. Staudenmaier, F. Poggiali, M. Campisi, A. Trombettoni, F.S. Cataliotti, P. Cappellaro, and N. Fabbri, Autonomous Dissipative Maxwell’s Demon in a Diamond Spin Qutrit, PRX Quantum 3, 020329 (2022).
  * [154] S. Bhattacharjee, A. Dutta, Quantum thermal machines and batteries. Eur. Phys. J. B 94, 239 (2021).
  * [155] F. Campaioli, F. A. Pollock, F. C. Binder, L. Céleri, J. Goold, S. Vinjanampathy, and K. Modi, Enhancing the Charging Power of Quantum Batteries, Phys. Rev. Lett. 118, 150601 (2017).
  * [156] Alicki, Robert and Fannes, Mark, Entanglement boost for extractable work from ensembles of quantum batteries. Phys. Rev. E 87, 042123.
  * [157] K. V. Hovhannisyan, M. Perarnau-Llobet, M. Huber, and A. Acín, Entanglement Generation is Not Necessary for Optimal Work Extraction. Phys. Rev. Lett. 111, 240401 (2013).
  * [158] A. C. Santos, Quantum advantage of two-level batteries in the self-discharging process. Phys. Rev. E 103, 042118 (2021).
  * [159] J. Joshi and T S Mahesh, NMR investigations of quantum battery using star-topology spin systems. Phys. Rev. A 106, 042601 (2022).
  * [160] F. C. Binder, S. Vinjanampathy, K. Modi, and J. Goold, Quantacell: powerful charging of quantum batteries. Phys. 17 075015 (2015).
  * [161] Thao P. Le, Jesper Levinsen, Kavan Modi, Meera M. Parish, and Felix A. Pollock. Spin-chain model of a many-body quantum battery. Phys. Rev. A 97, 022106.
  * [162] T S Mahesh,1, D. Khurana, V R Krithika, G J Sreejith, and C S S. Kumar, Star-topology registers: NMR and quantum information perspectives. J. Phys.: Condens. Matter 33 383002 (2021).
  * [163] Mohammad Mehboudi et al. Thermometry in the quantum regime: recent theoretical progress. J. Phys. A: Math. Theor. 52 303001 (2019).
  * [164] T. M. Stace, Quantum limits of thermometry. Phys. Rev. A 82, 011611(R) (2010).
  * [165] L. A. Correa, M. Mehboudi, G. Adesso, and A. Sanpera, Individual Quantum Probes for Optimal Thermometry. Phys. Rev. Lett. 114, 220405 (2015).
  * [166] C. Raitz, A. M. Souza, R. Auccaise, R. S. Sarthour, and I. S. Oliveira, Experimental implementation of a nonthermalizing quantum thermometer. Quantum Inf. Process. 14, 3746 (2015).
  * [167] C. V. H. B. Uhlig, R. S. Sarthour, I. S. Oliveira, and A. M. Souza, Experimental implementation of an NMR NOON state thermometer. Quantum Inf. Process. 18, 294 (2019).
  * [168] J. C. Maxwell, Theory of Heat (Dover, 1871).
  * [169] Harvey S. Leff, Andrew F. Rex, Maxwell’s Demon 2: Entropy, Classical and Quantum Information, Computing (edited by H. S. Le and A. F. Rex CRC Press, Boca Raton, 2002).
  * [170] Szilard, L. über die Entropieverminderung in einem thermodynamischen System bei Eingriffen intelligenter Wesen, Z. Phys. 53, 840–856 (1929).
  * [171] L. Szilard, On the decrease of entropy in a thermodynamic system by the inter- vention of intelligent beings, Behavioral Science 9, 301 (1964).
  * [172] L. Brillouin , “Maxwell’s Demon Cannot Operate: Information and Entropy. I". Journal of Applied Physics 22, 334-337 (1951).
  * [173] R. Landauer, “Irreversibility and Heat Generation in the Computing Process", IBM Journal of Research and Development, vol. 5, no. 3, pp. 183-191, (1961).
  * [174] R. Landauer, Information is Physical, Physics Today, 44, 23-29 (1991).
  * [175] Bennett, C.H. The thermodynamics of computation a review-, Int J Theor Phys 21, 905-940 (1982).
  * [176] C. Jarzynski, Equalities and inequalities: Irreversibility and the second law of thermodynamics at the nanoscale, Annu. Rev. Condens. Matter Phys. 2, 329 (2011).
  * [177] T. Sagawa, Thermodynamics of Information Processing in Small Systems (Springer Theses, Springer, New York, 2012).
  * [178] T. Sagawa and M. Ueda, Information Thermodynamics: Maxwell’s Demon in: Nonequilibrium Dynamics (Wiley-VCH, Weinheim, 2013).
  * [179] J. V. Koski, A. Kutvonen, I. M. Khaymovich, T. Ala-Nissila, J. P. Pekola, On-Chip Maxwell’s Demon as an Information-Powered Refrigerator, Phys. Rev. Lett. 115, 260602 (2015).
  * [180] P. A. Camati, J. P. S. Peterson, T. B. Batalhão, K. Micadei, A. M. Souza, R. S. Sarthour, I. S. Oliveira, and R. M. Serra, Experimental Rectification of Entropy Production by Maxwell’s Demon in a Quantum System, Phys. Rev. Lett. 117, 240502 (2016).
  * [181] N. Cottet, S. Jezouin, L. Bretheau, P. C.-Ibarcq, Q. Ficheux, J. Anders, A. Auffèves, R. Azouit, P. Rouchon, and B. Huard, Observing a quantum Maxwell demon at work, PNAS 114, 7561 (2017).
  * [182] B. L N.-Santos, P. A. Camati, V. Métillon, M. Brune, J.-M Raimond, A. Auffèves, and I. Dotsenko, Autonomous Maxwell’s demon in a cavity QED system, Phys. Rev. Research 2, 032025(R) (2020).
  * [183] D. Mandal and C. Jarzynski, Work and information processing in a solvable model of Maxwell’s demon, PNAS, 109 (29) 11641-11645 (2012).
  * [184] Z. Lu, D. Mandal, C. Jarzynski, Engineering Maxwell’ s demon, Physics Today 67, 8, 60 (2014).
  * [185] N. Rupprecht and D. Can Vural, Maxwell’s Demons with Finite Size and Response Time, Phys. Rev. Lett. 123, 080603 (2019).
  * [186] H. J. Groenewold, A problem of information gain by quantal measurements, Int. J. Theor. Phys. 4, 327 (1971).
  * [187] G. Lindblad, An entropy inequality for quantum measurements, Commun. Math. Phys. 28, 245 (1972).
  * [188] R. Balian, From Microphysics to Macrophysics: volume 1 (Springer, 2007).
  * [189] S. W. Kim, T. Sagawa, S. D. Liberato, and M. Ueda, Quantum Szilard Engine, Phys. Rev. Lett. 106, 070401 (2011).
  * [190] Kim, KH., Kim, S.W, Szilard’s information heat engines in the deep quantum regime, Journal of the Korean Physical Society 61, 1187-1193 (2012).
  * [191] H. Li, J. Zou, J.-G Li, B. Shao, LAo Wu, Revisiting the quantum Szilard engine with fully quantum considerations, Annals of Physics. 327, 2955-2971 (2012).
  * [192] Z. Zhuang and S.-D. Liang, Quantum Szilard engines with arbitrary spin, Phys. Rev. E 327, 052117 (2014).
  * [193] J. J. Park, K.-H. Kim, T. Sagawa, and S. W. Kim, Quantum Szilard Engine with Attractively Interacting Bosons, Phys. Rev. Lett. 111, 230402 (2013).
  * [194] J. Bengtsson, M. N. Tengstrand, A. Wacker, P. Samuelsson, M. Ueda, H. Linke, and S. M. Reimann, Quantum Szilard Engine with Attractively Interacting Bosons, Phys. Rev. Lett. 120, 100601 (2018).
  * [195] E. Aydiner, Quantum Szilard engine for the fractional power-law potentials,  Sci. Rep. 11, 1576 (2021).
  * [196] J. P. S. Peterson, R. S. Sarthour, and R. Laflamme, Implementation of a Quantum Engine Fuelled by Information, arXiv:2006.10136 (2020).
  * [197] M. Spiecker, P. Paluch, N. Drucker, S. Matityahu, D. Gusenkova, N. Gosling, S. Gunzler, D. Rieger, I. Takmakov, F. Valenti, P. Winkel, R. Gebauer, O. Sander, G. Catelani, A. Shnirman, A. V. Ustinov, W. Wernsdorfer, Y. Cohen, and I. M. Pop, A quantum szilard engine for two-level systems coupled to a qubit, arXiv:2006.10136 (2022).
  * [198] J. Earman and J.D. Norton, Exorcist XIV: The wrath of Maxwell’s demon. Part I. From Maxwell to Szilard, Studies in History and Philosophy of Science Part B 29, 435 (1998).
  * [199] J. Earman and J.D. Norton, Exorcist XIV: The wrath of Maxwell’s demon. Part II. From Maxwell to Szilard, Studies in History and Philosophy of Science Part B 30, 1 (1999).
  * [200] Alexei O. O., Craig S. L., Cameron C. T., Graham P. B. and Gregory L. S., Experimental Test of Landauer’s Principle at the Sub-kBT Level, Jpn. J. Appl. Phys. 51, 06FE10 (2012).
  * [201] S. Toyabe, T. Sagawa, M. Ueda, E. Muneyuki and M. Sano, Experimental demonstration of information-to-energy conversion and validation of the generalized Jarzynski equality, Nature Phys. 6, 988-992 (2010).
  * [202] A. Bérut, A. Arakelyan, A. Petrosyan, S. Ciliberto, R. Dillenschneider and E. Lutz, Experimental verification of Landauer’s principle linking information and thermodynamics, Nature 483, 187-189 (2012).
  * [203] Jonne V. K. , V. F. Maisi, J. P. Pekola and D. V. Averin, Experimental realization of a Szilard engine with a single electron, Proc. Natl Acad. Sci. USA 111, 13786-13789. (2014).
  * [204] Y. Jun, M. Gavrilov, and J. Bechhoefer, High-precision test of Landauer’s principle in a feedback trap, Phys. Rev. Lett. 113, 190601 (2014).
  * [205] L. L. Yan, T. P. Xiong, K. Rehan, F. Zhou, D. F. Liang, L. Chen, J. Q. Zhang, W. L. Yang, Z. H. Ma, and M. Feng, Single-atom demonstration of the quantum Landauer principle, Phys. Rev. Lett. 120, 210601 (2018).
  * [206] J. P. S. Peterson, R. S. Sarthour, A. M. Souza, I. S. Oliveira, J. Goold, K. Modi, D. O. Soares-Pinto and L. C. C leri, Experimental demonstration of information to energy conversion in a quantum system at the Landauer limit, Proc. R. Soc. A 472, 20150813 (2016).
  * [207] D. Reeb and M. M Wolf, An improved Landauer principle with finite-size corrections, New J. Phys. 16 103011 (2014).
  * [208] J. Goold, U. Poschinger e K. Modi, Measuring the heat exchange of a quantum process" Phys. Rev. E 90, 020101(R) (2014).
  * [209] M. Esposito, K. Lindenberg, C. Van den Broeck, Entropy production as correlation between system and reservoir, New J. Phys. 12, 013013 (2010).
  * [210] R. Clausius, The Mechanical Theory of Heat, (MacMillan, London, 1879).
  * [211] L. Boltzmann, On the Relation of a General Mechanical Theorem to the Second Law of Thermodynamics (Sitzungsberichte Akad. Wiss., Vienna, part II 75, 67-73, 1877).
  * [212] Lebowitz, J. L. Boltzmann’s entropy and time’s arrow, Physics today 46, 32-38 (1993).
  * [213] H. D. Zeh,The Physical Basis of the Direction of Time, (Springer, 2007).
  * [214] M. H. Partovi, Entanglement versus stosszahlansatz: disappearance of the thermodynamic arrow in a highcorrelation environment, Phys. Rev. E 77, 021110 (2008).
  * [215] S. Jevtic, D. Jennings, and T. Rudolph, Maximally and minimally correlated states attainable within a closed evolving system, Phys. Rev. Lett. 108, 110403 (2012).
  * [216] D. Jennings and T. Rudolph, Entanglement and the thermodynamic arrow of time, Phys. Rev. E 81, 061130 (2010).
  * [217] M. N. Bera, Riera, A. Lewenstein, and A. M. Winter, Generalized laws of thermodynamics in the presence of correlations, Nature Commun. 8, 2180 (2017).