# A Survey on Physics Informed Reinforcement Learning: Review and Open
Problems

C. Banerjee, K. Nguyen, C. Fookes, M. Raissi, C. Banerjee, K. Nguyen, and C.
Fookes are with Queensland University of Technology, Australia. Maziar Raissi
is with University of Colorado Boulder, USA. E-mail: {c.banerjee,
k.nguyenthanh<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The inclusion of physical information in machine learning frameworks has
revolutionized many application areas. This involves enhancing the learning
process by incorporating physical constraints and adhering to physical laws.
In this work we explore their utility for reinforcement learning applications.
We present a thorough review of the literature on incorporating physics
information, as known as physics priors, in reinforcement learning approaches,
commonly referred to as physics-informed reinforcement learning (PIRL). We
introduce a novel taxonomy with the reinforcement learning pipeline as the
backbone to classify existing works, compare and contrast them, and derive
crucial insights. Existing works are analyzed with regard to the
representation/ form of the governing physics modeled for integration, their
specific contribution to the typical reinforcement learning architecture, and
their connection to the underlying reinforcement learning pipeline stages. We
also identify core learning architectures and physics incorporation biases
(i.e. observational, inductive and learning) of existing PIRL approaches and
use them to further categorize the works for better understanding and
adaptation. By providing a comprehensive perspective on the implementation of
the physics-informed capability, the taxonomy presents a cohesive approach to
PIRL. It identifies the areas where this approach has been applied, as well as
the gaps and opportunities that exist. Additionally, the taxonomy sheds light
on unresolved issues and challenges, which can guide future research. This
nascent field holds great potential for enhancing reinforcement learning
algorithms by increasing their physical plausibility, precision, data
efficiency, and applicability in real-world scenarios.

###### Index Terms:

Physics-informed, Reinforcement Learning, Machine learning, Neural Network,
Deep Learning

##  1 Introduction

Through trial-and-error interactions with the environment, Reinforcement
Learning (RL) offers a promising approach to solving decision-making and
optimization problems. Over the past few years, RL has accomplished impressive
feats in handling difficult tasks, in such domains as autonomous driving [119,
16], locomotion control [99, 129], robotics [71, 94], continuous control [5,
6, 7], and multi-agent systems and control [39, 15]. A majority of these
successful approaches are purely data-driven and leverage trial-and-error to
freely explore the search space. RL methods work well in simulations, but they
struggle with real-world data because of the disconnection between simulated
setups and the complexities of real world systems. Major RL challenges [33],
that are consistently addressed in latest research includes sample efficiency
[91, 9], high dimensional continuous state and action spaces [34, 118], safe
exploration [41, 48], multi-objective and well-defined reward function [65,
10], perfect simulators and learned model [27, 96] and policy transfer from
offline pre-training [72, 131].

When it comes to machine learning, incorporating mathematical physics into the
models can lead to more meaningful solutions. This approach, known as physics-
informed machine learning, helps neural networks learn from incomplete physics
information and imperfect data more efficiently, resulting in faster training
times and better generalization. Additionally, it can assist in tackling high
dimensionality applications and ensure that the resulting solution is
physically sound and follows the underlying physical law [60, 8, 52]. Among
the various sub-fields of ML, RL is the natural candidate for incorporating
physics information since most RL-based solutions deal with real-world
problems and have an explainable physical structure.

Recent research has seen substantial improvement in addressing the RL
challenges by incorporating physics information in the training pipeline. For
example, PIRL approaches seek to use physics to reduce high-dimensional
continuous states with intuitive representations and better simulation. A low-
dimensional representation adhering to physical model PDEs is learned in [45],
while [12] uses features from a supervised surrogate model. Learning a good
world model is a quicker and safer alternative to training RL agents in the
real world. [103] incorporate physics into the network for better world
models, and [128] utilize a high-level specification robot morphology and
physics for rapid model identification.

A well-defined reward function is crucial for successful reinforcement
learning, PIRL approaches also seek to incorporate physical constraints into
the design for safe learning and more efficient reward functions. For example,
in [68] the designed reward incorporates IMU sensor data, imbibing inertial
constraints, while in [75] the physics informed reward is designed to satisfy
explicit operational targets. To ensure safe exploration during training and
deployment, works such as [133, 141] learn a data-driven barrier certificate
based on physical property-based losses and a set of unsafe state vectors.

There are several lines of PIRL research dedicated to exploring more efficient
exploration of the search space and effective policy deployment for real-world
systems. Some approaches were developed to improve simulators for sample
efficiency and better sim to real transfer [1, 81]. Carefully selecting task-
specific state representations [59, 51], reward functions [13, 14], and action
spaces [124, 141] has been shown to improve both the time to convergence and
performance. To sum it up, integrating underlying physics about the learning
task structure has been found to improve performance and accelerate
convergence.

Physics-informed Reinforcement Learning (PIRL) has been a growing trend in the
literature, as demonstrated in the increasing number of papers published in
this area over the past six years, as shown in Figure 1. The bar chart
indicates that this field is gaining more attention, and we can anticipate
even more in the future.

$2023$$2022$$2021$$2020$$2019$$2018$$0$$10$$20$No. of PIRL paper published
Figure 1: PIRL papers published over years. This statistic graph the
exponential growth of PIRL papers over last six years.

Our contributions in this paper are summarized as follows:

  * 1)

Taxonomy: We propose a unified taxonomy to investigate what physics
knowledge/processes are modeled, how they are represented, and the strategies
to incorporate them into RL approaches.

  * 2)

Algorithmic Review: We present state-of-the-art approaches on the physics
information guided/ physics informed RL methods, using unified notations,
simplified functional diagrams and discussion on latest literature.

  * 3)

Training and evaluation benchmark Review: We analyze the evaluation benchmarks
used in the reviewed literature, thus presenting popular evaluation and
benchmark platforms/ suites for understanding the popular trend and also for
easy reference.

  * 4)

Analysis: We delve deep into a wide range of model based and model free RL
applications over diverse domains. We analyze in detail how physics
information is integrated into specific RL approaches, what physical processes
have been modeled and incorporated, and what network architectures or network
augmentations have been utilized to incorporate physics.

  * 5)

Open Problems: We summarize our perspectives on the challenges, open research
questions and directions for future research.

Difference to other survey papers:
George et al. [60] provided one of the most comprehensive reviews on machine
learning (ML) in the context of physics-informed (PI) methods, but approaches
in the RL domain has not been discussed. The work by Hao et al. [52] also
provided an overview of physics-informed machine learning, where the authors
briefly touch upon the topic of PIRL. Another recent study by Eesserer et al.
[35] showcased the use of prior knowledge to guide reinforcement learning (RL)
algorithms, specific to robotic applications. The authors categorize knowledge
into three types: expert knowledge, world knowledge, and scientific knowledge.
Our paper offers a focused and comprehensive review specially on the RL
approaches that utilize the structure, properties, or constraints unique to
the underlying physics of a process/system. Our scope of application domains
is not limited to robotics, but also spanning to motion control, molecular
structure optimization, safe exploration, and robot manipulation.

The rest of this paper is organized as follows. In Section § 2, we provide a
brief overview of the Physics informed ML paradigm. In Section § 3, we present
RL fundamentals/ framework in § 3.1 and provide a definition with an intuitive
introduction to PIRL in § 3.2. Most importantly we introduce a comprehensive
taxonomy in § 3.3 threading together physics information types, PIRL methods
that implement those information and RL pipeline as a backbone. Later in § 3.4
we present and elaborate on two additional categories: Learning architecture
and Bias, through which the implementation side of the literature is explained
more precisely. In Section § 4 we present an elaborate review and analysis of
latest PIRL literature. In Section § 5, we discuss the different open
problems, challenges and research directions that may be addresses in future
works by interested researchers. Finally Section § 6 concludes the paper.

Table I: A list of abbreviations used in this article. Abbreviations
---
FSA | Finite State Automata
FEA | Finite Element Analysis
CFD | Computational Fluid Dynamics
MDP | Markov Decision Process
MBRL | Model based Reinforcement Learning
MFRL | Model Free Reinforcement Learning
CBF | Control Barrier Function
CBC | Control Barrier Certificate
NBC | Neural Barrier Certificate
CLBF | Control Lyapunov Barrier Function
NBC | Neural Barrier Certificate
DFT | Density Functional Theory
AC | Actor Critic
MPC | Model Predictive Control
DDP | Differential Dynamic Programming
NPG | Natural Policy Gradient
TL | Temporal Logic
DMP | Dynamic Movement Primitive
WBTG | Whole Body Trajectory Generator
DPG | Deterministic Policy Gradient
DPPO | Distributed proximal Policy optimization
ABM | Adjoint based method
APG | Analytic Policy Gradient
WBIC | Whole Body Impulse Controller
LNN | Lagrangian Neural Network

##  2 Physics-informed Machine Learning (PIML): an overview

The aim of PIML is to merge mathematical physics models and observational data
seamlessly in the learning process. This helps to guide the process towards
finding a physically consistent solution even in complex scenarios that are
partially observed, uncertain, and high-dimensional [62, 52, 26]. Adding
physics knowledge to machine learning models has numerous benefits, as
discussed in [62, 89]. This information captures the vital physical principles
of the process being modeled and brings following advantages

  1. 1.

Ensures that the ML model is consistent both physically and scientifically.

  2. 2.

Increases data efficiency in model training, meaning that the model can be
trained with fewer data inputs.

  3. 3.

Accelerates the model training process, allowing models to converge faster to
an optimal solution.

  4. 4.

Increases the generalizability of trained models, enabling them to make better
predictions for scenarios that were not seen during the training phase.

  5. 5.

Enhances the transparency and interpretability of models, making them more
trustworthy and explainable.

According to literature, there are three strategies for integrating physics
knowledge or priors into machine learning models: observational bias, learning
bias, and inductive bias.

Observational bias: This approach uses multi-modal data that reflects the
physical principles governing their generation [82, 61, 77, 132]. The deep
neural network (DNN) is trained directly on observed data, with the goal of
capturing the underlying physical process. The training data can come from
various sources such as direct observations, simulation or physical equation-
generated data, maps, or extracted physics data induction.

Learning bias: One way to reinforce prior knowledge of physics is through soft
penalty constraints. This approach involves adding extra terms to the loss
function that are based on the physics of the process, such as momentum or
conservation of mass. An example of this is physics-informed neural networks
(PINN), which combine information from measurements and partial differential
equations (PDEs) by embedding the PDEs into the neural network’s loss function
using automatic differentiation [60]. Some prominent examples of soft penalty
based approaches includes statistically constrained GAN [127], physics-
informed auto-encoders [37] and encoding invariances by soft constraints in
the loss function InvNet [110].

Inductive biases: Custom neural network-induced ’hard’ constraints can
incorporate prior knowledge into models. For instance, Hamiltonian NN [47]
draws inspiration from Hamiltonian mechanics and trains models to respect
exact conservation laws, resulting in better inductive biases. Lagrangian
Neural Networks (LNNs) [25] introduced by Cranmer et al. can parameterize
arbitrary Lagrangians using neural networks, even when canonical momenta are
unknown or difficult to compute. Meng et al. [90] uses a Bayesian framework to
learn functional priors from data and physics with a PI-GAN, followed by
estimating the posterior PI-GAN’s latent space using the Hamiltonian Monte
Carlo (HMC) method. Additionally, DeepONets [82] networks are used in PDE
agnostic physical problems.

##  3 Physics-Informed Reinforcement Learning: Fundamentals, Taxonomy and
Examples

In this section, we will explain how physics information can be integrated
into reinforcement learning applications.

###  3.1 RL fundamentals

Figure 2:  Agent-environment framework, of RL paradigm. Here the reward
generating function and the system/ plant is abstracted as the environment.
And the control policy (e.g. a DNN) and the learning algorithm, forms the RL
agent.

RL algorithms use reward signals from the environment to learn the best
strategy to solve a task through trial and error. They effectively solve
sequential decision-making problems that follow the Markov Decision Process
(MDP) framework. In the RL paradigm, there are two main players: the agent and
the environment. The environment refers to the world where the agent resides
and interacts. Through agent-environment interactions the agent perceives the
state of the world and decides on the appropriate action to take.

The agent-environment RL framework, see Fig. 2, is a large abstraction of the
problem of goal-directed learning from interaction [115]. The details of
control apparatus, sensors, and memory are abstracted into three signals
between the agent and the environment: the control/ action, the state and the
reward. Though typically, the agents computes the rewards, but by the current
convention anything that cannot be changed arbitrarily by the agent is
considered outside of it and hence the reward function is shown as a part of
the environment.

MDP is typically represented by the tuple
$(\mathcal{S},\mathcal{A},R,P,\gamma)$, where $\mathcal{S}$ represents the
states of the environment, $\mathcal{A}$ represents set of actions that the RL
agent can take. Reward function may be typically represented as
$R(s_{t+1},a_{t})$ a function of next state and current action. The function
generates the reward due to action induced state transition from $s_{t}$ to
$s_{t+1}$. $P(s_{t+1}|s_{t},a_{t})$ is the environment model that returns the
probability of transitioning to state $s_{t+1}$ from $s_{t}$. Finally the
discount factor $\gamma\in[0,1]$, determines the amount of emphasis given to
the immediate rewards relative to that of future rewards.

---
(a) Online RL (model-free)

---
(b) Off-policy RL (model-free)

---
(c) Offline RL

---
(d) Off-policy RL (model-based)

Figure 3: Typical RL architectures, based on model use and interaction with
the environment.

The RL framework typically organizes the agent’s interactions with the
environment into episodes. In each episode, the agent starts at a particular
initial state $s_{1}$ sampled from an initial distribution $p(s_{1})$, which
is part of the state space $\mathcal{S}$ of the MDP. At each timestep $t$, the
agent observes the current state $s_{t}\in\mathcal{S}$ and samples an action
$a_{t}\in\mathcal{A}$ from its latest policy $\pi_{\phi}(a_{t}|s_{t})$ based
on the state $s_{t}$, where $\phi$ represents the policy parameters. The
action space of the MDP is denoted by $\mathcal{A}$.

Next, the agent applies the action $a_{t}$ into the environment, which results
in a new state $s_{t+1}$ given by the dynamics of the MDP, i.e., $s_{t+1}\sim
p(s_{t+1}|s_{t},a_{t})$. The agent also receives a reward
$r_{t}=R(s_{t+1},a_{t})$, which can be construed as the desirability of a
certain state transition from the context of the given task. The above process
is repeated up to a certain time horizon $T$, which may also be infinite. The
agent-environment interaction is recorded as a trajectory, and the closed-loop
trajectory distribution for the episode $t=1,\cdots,T$ can be represented by,

$\displaystyle p_{\phi}(\tau)=$
$\displaystyle{p_{\phi}(s_{1},a_{1},s_{2},a_{2},\cdots,s_{T},a_{T},s_{T+1})}$
(1) $\displaystyle=$ $\displaystyle
p(s_{1})\prod_{t=1}^{T}\pi_{\phi}(a_{t}|s_{t})p(s_{t+1}|s_{t},a_{t}),$ (2)

where $\tau=(s_{1},a_{1},s_{2},a_{2},\cdots,s_{T},a_{T},s_{T+1})$ represents
the sequence of states and control actions. The objective is to find an
optimal policy represented by the parameter,

$\displaystyle\phi^{*}=\text{arg
max}_{\phi}\,\underbrace{{\mathbf{E}}_{\tau\sim
p_{\phi}(\tau)}\,\Big{[}\sum_{t=1}^{T}\,\gamma^{t}R(a_{t},s_{t+1})\Big{]}}_{\mathcal{J}(\phi)},$
(3)

which maximizes the objective function $\mathcal{J}(\phi)$, $\gamma$ is a
parameter called discount factor, where $0\leq\gamma\leq 1$. $\gamma$
determines the present value of the future reward, i.e., a reward received at
$k$ timesteps in the future is worth only $\gamma^{k-1}$ times what it would
be worth if received immediately.

Model-free and model-based RL: In RL, algorithms can be classified based on
whether the environment model is available during policy optimization. The
environment dynamics are represented as
$p(s_{t+1},r_{t})=Pr(s_{t+1},r_{t}|s_{t},a_{t})$, which means that given a
state and action, the environment model can predict the state transition and
the corresponding reward. Access to an environment model allows the agent to
plan and choose between options and also improves sample efficiency compared
to model-free approaches. However, the downside is that the environment’s
groundtruth model is typically not available, and learning a perfect model of
the real world is challenging. Additionally, any bias in the learned model can
lead to good performance in the learned model but poor performance in the real
environment.

Online, Off-policy and Offline RL: Online RL algorithms, e.g. PPO, TRPO, and
A3C, optimize policies by using only data collected while following the latest
policy, creating an approximator for the state or action value functions, used
to update the policy. Off-policy RL algorithms, e.g. SAC, TD3 and IPNS,
involve the agent updating its policy and other networks using data collected
at any point during training. This data is stored in a buffer called the
experience replay buffer and is in the form of tuples. Mini-batches are
sampled from the buffer and used for the training process. Offline RL
algorithms use a fixed dataset called $\mathcal{D}$ collected by a policy
$\pi_{\zeta}$ to learn the optimal policy. This allows for the use of large
datasets collected previously.

Combining model-free/model-based with online/off-policy/offline
categorization, typical RL architectures can be presented as Fig. 3.

Figure 4: Map of physics incorporation (PI) in the conventional Reinforcement
Learning (RL) framework.

###  3.2 PIRL: Introduction

####  3.2.1 Definition

The concept of physics-informed RL involves incorporating physics structures,
priors, and real-world physical variables into the policy learning or
optimization process. Physics induction helps improve the effectiveness,
sample efficiency and accelerated training of RL algorithms/ approaches, for
complex problem-solving and real-world deployment. Depending on the specific
problem or scenario, different physics priors can be integrated using various
RL methods at different stages of the RL framework, see Fig. 4.

####  3.2.2 Intuitive introduction to physics priors in RL

Physics priors come in different forms, like intuitive physical rules or
constrains, underlying mathematical/ guiding equations and physics simulators,
to name a few. Here we discuss a couple of intuitive examples. In [128], the
physical characteristics of the system were utilized as priors. The high-level
specifications of a robot’s morphology such as the number and connectivity
structure of links were used as physics priors. This feature based
representation of the system dynamics enabled rapid model identification in
this model based RL setup. In another example, pertaining to adaptive cruise
control problem, [59] (see Fig.5), physics information in the form of jam-
avoiding distance (based on desired physical parameters e.g. velocity and
acceleration constraints, minimum jam avoiding distance etc.) is included in
state space input to the RL agent. Physics info. incorporation results in a RL
controller which performs with less collisions and enables more equidistant
travel.

Figure 5:  An illustrative example of physics incorporation in RL application,
[59]. Here the RL agent is fed with an additional state variable: jam avoiding
distance, which is based on desired physical parameters and primary state
variables.

###  3.3 PIRL Taxonomy

####  3.3.1 Physics information (types): representation of physics priors

There are different types/ forms of physics information, e.g. mathematical
representation of the physical system like PDE/ODE and physics enriched
simulators. Based on the type of the physics information representation, works
can be typically categorized as follows.

  1. 1)

Differential and algebraic equations (DAE): Many works use system dynamics
representations, such as partial/ordinary differential equations (PDE/ ODE)
and boundary conditions (BC), as physics priors primarily through PINN and
other special networks. For example in transient voltage control application
[40], a PINN is trained using PDE of transient process. The PINN learns a
physical constraint which it transfers to the loss term of the RL algorithm.

  2. 2)

Barrier certificate and physical constraints (BPC): It is imperative to
regulate agent exploration in safety-critical applications of reinforcement
learning. One way it is addressed in recent research is through the use of
optimization-based control theoretic constraints. Use of concepts like control
Lyapunov function (CLF)[74, 23], barrier certificate/ barrier function (BF),
control barrier function/ certificate (CBF/ CBC) [19, 11] is made in recent
safety critical RL applications. Barrier certificate is generally used to
establish a safe set of desired states for a system. A control barrier
function is then employed to devise a control law that keeps the states within
the safety set. In certain scenarios barrier functions are represented as NNs
and learned through data driven approaches [141, 140]. In above control
theoretic approaches the system dynamics either partial or learnable and
safety sets represent the primary physical information. For more details on
CBFs refer [2]. Additionally safety in the learning process may also be
ensured by incorporating physical constraints into the RL loss [76, 17].

  3. 3)

Physics parameters, primitives and physical variables (PPV): Physics values
extracted/ derived from the environment or system has been directly used by RL
agents in form of physics parameters [113], dynamic movement (physics)
primitives [3], physical state [59] and physical target [75]. For example in
[75], the reward is created to meet two physical objectives/ targets:
operation cost and self-energy sustainability. In an adaptive cruise control
problem [59], authors use desired physical parameters e.g. velocity and
acceleration constraints and minimum jam avoiding distance, as a state space
input.

  4. 4)

Offline data and representation (ODR): For the improvement simulator based
training, especially during sim-to-real transfer, non-task- specific-policy
data collected from real robot has been used to train RL agents in offline
setting along with simulators [46] and as hardware data to seed simulators
[81].

Another popular way of extracting physics information from environment is
learning physically relevant low dimensional representation from observations
[45, 12]. For example, in [45], PINN is used to extract physically relevant
information about the hidden state of the system, which is further used to
learn a Q-function for policy optimization.

  5. 5)

Physics simulator and model (PS):  Simulators provide a easy way of
experimenting with RL algorithms without exposing the agent e.g. a robot to
the wear and tear of the real environment.

Apart from serving as test-beds for RL algorithms, simulators are also used
alongside RL algorithms to impart physical correctness or physics awareness in
the data or training process. For example in order to improve motion capture
and imitation of given motion clips, [20] have used rigid body physics
simulations to solve the rigid body poses closely following the motion capture
clip. In [42], using a physics simulator, a residual agent is able to learn
how to improve user input in order to achieve a task while staying true to the
original input and expert-recorded trajectories.

In the MBRL setting the system model can be: 1) completely known, 2) partially
known or 3) completely unknown. RL algorithms typically addresses the last two
types, since it deals with environments whose dynamics is complex and
difficult to ascertain through classical approaches. In such cases a DNN based
data-driven approach is generally utilized to learn the system model
completely or enrich the existing partial or basic model of the environment.
In [51] a data driven surrogate traffic flow model is learned that generates
synthetic data. This data is later used by the agent in an offline learning
process, followed by an online control process. In [103] learns environment
and reward models by using Lagrangian NNs [25]. LNNs are models that are able
to Lagrangian functions straight from data gathered from agent-environment
interactions.

  6. 6)

Physical properties (PPR): Fundamental knowledge regarding the physical
structure or properties pertaining to a system has been used in a number of
works. For example system morphology, system symmetry [54]

####  3.3.2 PIRL methods: physics prior augmentations to RL

PIRL methods highlights and discusses about the different components of the
typical RL paradigm e.g. state space, action space, reward function and agent
networks (policy and value function N/W), that has been directly modified/
augmented through the incorporation of physics information.

  * 1)

State design: This category is concerned with the observed state space of the
environment or model. The PIRL approaches, typically modifies or expands the
state representation in order to make it more instructive. Works include state
fusion using additional information from environment [59] and other agents
[112], state as extracted features from robust representation [12], learned
surrogate model generated data as state [51] and state constraints [138].

  * 2)

Action regulation: This pertains to modifying the action value, which is often
achieved through PIRL approaches that impose constraints on the action value
to ensure safety protocols are implemented [76, 19].

  * 3)

Reward design: It concerns approaches that induce physics information through
effective reward design or augmentation of existing reward functions with
bonuses or penalties [28, 83].

  * 4)

Augment policy or value N/W: These PIRL approaches incorporate physics
principles via methods like, adjusting the update rules and losses of the
policy [4, 87], value functions [93, 98] and making direct changes to their
underlying network structure [14]. Works with novel physics based losses [92,
130] and constraints for policy or value function learning [40] are also
included.

  * 5)

Augment simulator or model:  This category encompasses those works that
develops improved simulators through incorporation of underlying physics
knowledge thereby allowing for more accurate simulation of real-world
environments. Works include physics based augmentation of DNN based learnable
models for accurate system model learning [70, 103], improved simulators for
sim-to-real transfer [46, 81] and physics informed learning for partially
known environment model [78].

Figure 6: Deep Reinforcement Learning Pipeline. Here the problem is first
modeled as a MDP, clearly defining the state, action and reward spaces.
Followed by selecting the RL algorithm as Learning strategy and then
selecting/ designing the policy and/or value networks in network design stage.
Finally the agent is trained using default/ custom loss function in training
stage and finally deployed.

####  3.3.3 RL Pipeline

A typical RL pipeline can be represented into four functional stages namely,
the problem representation, learning strategy, network design, training and
trained policy deployment. These stages are elaborated as follows:

  * 1.

Problem Representation: In this stage, a real-world problem is modeled as a
Markov Decision Process (MDP) and thereby described using formal RL terms. The
main challenge is to choose the right observation vector, define the reward
function, and specify the action space for the RL agent so that it can perform
the specified task properly.

  * 2.

Learning strategy: In this stage, the decisions are made regarding the type of
agent-environment interaction e.g. in terms of environment model use, learning
architecture and the choice of RL algorithm.

  * 3.

Network design: Here the finer details of the learning framework are decided
and customized where needed. Decisions are made regarding the type of
constituent units (e.g. layer types, network depth etc.) of underlying Policy
and value function networks.

  * 4.

Training: The policy and allied networks are trained in this stage. It also
represents training augmentation approaches like Sim-to-real, that helps is
reducing discrepancy between simulated and real worlds.

  * 5.

Trained policy deployment: At this stage the policy is completely trained and
is deployed for solving the concerned task.

Figure 7:  PIRL taxonomy and further categories. Physics information (types),
the RL methods that incorporate them and the underlying RL pipeline
constitutes the PIRL-taxonomy, see Fig. 9. bias (sec. 3.4.1) and Learning
architecture (sec. 3.4.2) are two additional categories which has been
introduced to better explain the implementation of PIRL. Figure 8:  Typical RL
architectures with physics information incorporation (a) Safety filter (b) PI-
reward (c) Residual agent (d) Physics embedded network (e) Differentiable
simulator (f) Sim-to-Real (g) Physics variable (h) Hierarchical RL (i) Data
augmentation (j) PI model identification. To keep illustrations simple, we
have not included ancillary networks e.g. value function networks above.

###  3.4 Further categorization

In this section we introduce a couple of additional categorizations: Bias and
Learning architecture. These categories are not part of the taxonomy that we
have discussed in the previous section, see Fig.7. They provide an additional
perspective to the PIRL approaches presented here.

####  3.4.1 Bias

PI approaches in ML paradigm, mentions of different kind of biases or
categories of methods of physics incorporation in ML models. In order to
relate to that existing taxonomy used in PIML methods, in Table II and Table
III., we include corresponding bias categories to each of the PIRL entries.

####  3.4.2 Learning architecture

We also categorize PIRL algorithms based on the alterations that they
introduce to the conventional RL learning architecture to incorporate physics
information/ priors. As listed and discussed below they help us understand the
PIRL methods from an architectural point of view. In the literature review
section we use the aid of such learning architecture categories to group and
discuss the PIRL methods.

  * 1)

Safety filter:  This category includes approaches that has a PI based module
which regulates the agent’s exploration ensuring safety constraints, for
reference see Fig. 8(a). In this typical architecture the safety-filter module
takes action $a_{t}$ from RL agent $\pi_{\varphi}$, and state information
$(s_{t})$ and refines the action, giving $\tilde{a}_{t}$.

  * 2)

PI reward: This category includes approaches where physics information is used
to modify the reward function, see Fig.8(b) for reference. Here the PI-reward
module augments agent’s extrinsic reward $(r_{t})$ with a physics information
based intrinsic component, giving $\tilde{r}_{t}$.

  * 3)

Residual learning: Residual RL is an architecture which typically consists of
two controllers: a human designed controller and a learned policy [58]. In
PIRL setting the architecture consists of a physics informed controller
$\pi_{\psi}$ along with the data-driven DNN based policy $\pi_{\varphi}$,
called residual RL agent, see Fig. 8(c).

  * 4)

Physics embedded network: In this category physics information e.g. system
dynamics is directly incorporated in the policy or value function networks,
see Fig.8(d) for reference.

  * 5)

Differentiable simulator: Here the approaches have use differentiable physics
simulators, which are non-conventional/ or adapted simulators and explicitly
provides loss gradients of simulation outcome w.r.t. control action, see
Fig.8(e) for reference.

  * 6)

Sim-to-Real: In Sim-to-real architecture, the agent is first trained on a
simulator or source domain and is later transferred to a target domain for
deployment. In certain cases the transfer is followed by fine-tuning at the
target domain, see Fig.8(f) for reference.

  * 7)

Physics variable: This architecture encompasses all those approaches where
physical parameters, variables or primitives are introduced to augment
components (e.g. states and reward) of the RL framework. For reference see
Fig.8(g).

  * 8)

Hierarchical RL: This category includes hierarchical and curriculum learning
based approaches, Fig.8(h) for reference. In a hierarchical RL (HRL) setting a
long horizon decision making task is broken into simpler sub-tasks
autonomously. In curriculum learning a complex task is solved by learning to
solve a series of increasingly difficult tasks. In both HRL and CRL physics is
typically incorporated into all the policy (including meta and sub-policies)
and value networks. Approaches here are mostly extensions of physics-embedded
networks (Fig.8(d)), as used in non-HRL/ CRL settings.

  * 9)

Data augmentation: This category includes approaches where the input state is
replaced with a different or augmented form of it, e.g. low dimensional
representation so as to derive special and physically relevant features out of
it. See Fig.8(i) for reference. In this typical architecture, the state vector
$s_{t+1}$ is transformed into an augmented representation $z_{t+1}$.
Physically relevant features are then extracted from it and used by the RL
agent ($\pi_{\varphi}$).

  * 10)

PI model identification:  This architecture represents those PIRL approaches,
especially in data-driven MBRL setting where physics information is directly
incorporated into the model identification process. For reference see
Fig.8(j).

##  4 PIRL: Review and Analysis

In this section we provide a indepth review of latest works in PIRL, followed
by a review of the popular datasets. We also include an analysis of the
algorithms and their derivatives, and discuss crucial insights.

###  4.1 Algorithmic review

We provide a detailed overview of the PIRL approaches as identified by our
literature review in Table II and Table III. We have structured our discussion
according to the methods of the introduced taxonomy (see § 3.3) since they
form a bridge between the physics information sources and practical
applications. We also use learning architecture categories as introduced in
3.4.2, to better explain the PIRL methods.

Figure 9: Taxonomy, the diagram connects PI types with PIRL methods and then
to the RL pipeline backbone. The connection thickness represents the quantity
of work done which corresponds to those components/ categories.

####  4.1.1 State design:

Vehicular traffic control applications have used physics priors to design the
state representations. While controlling connected automated vehicles (CAVs),
[112] proposed the use of surrounding information from downstream vehicles and
roadside geometry, by embedding them in the state representation, see Fig. 10.
The physics-informed state fusion approach integrates received information as
DRL state (input features) i.e. for the $i^{th}$ CAV, DRL state is given as
$s_{i}^{t}=\big{[}e_{i}^{t},\phi_{i}^{t},\delta q_{i}^{-t},\delta
d_{i}^{-t},k_{i}^{t}\big{]}$, which are deviation values, (from left):
lateral, angular, weighed equilibrium spacing and speed, and road curvature
information.

Jurj et al. [59] makes use of physical information like jam-avoiding distance
to train RL agent, in order to improve collision avoidance of vehicles with
adaptive cruise control. In ramp metering control, [51] utilizes an offline-
online policy training process, where the offline training data consists of
historical data and synthetic data generated from a physical traffic flow
model.

Figure 10:  Example of state design, through physics incorporation.
Distributed control framework for connected automated vehicles [112]. Here
information from downstream vehicles and roadway geometry information are
incorporated as physics prior knowledge through state fusion.

In [12] a physics informed graphical representation-enabled, global graph
attention (GGAT) network is trained to model power flow calculation process.
Informative features are then extracted from the GGAT layer (as representation
N/W) and transferred used in the policy training process. While [45], uses
PINNs based on thermal dynamics of buildings for learning better heating
control strategies. Dealing with aircraft conflict resolution problem, [139]
composed intruder’s information e.g. speed and heading angle into an image
state representation. This image now constitutes of the physics prior and
serves as the input feature for RL based learning. In [138], the authors
proposed a safe reinforcement learning algorithm using barrier functions for
distributed MPC nonlinear multi-robot systems, with state constraints. [95],
incorporates trained model alongside control barrier certificates, which
restrict policies and prohibits exploration of the RL agent into certain
undesirable sections of the state space. In case of a safety breach due to
non-stationarity, the Lyapunov stability conditions ensures the re-
establishment of safety.

####  4.1.2 Action regulation:

Figure 11:  Example of action regulation, using physics priors. In [141], a
barrier certification system receives RL control policy generated control
actions and refines them sequentially using a barrier certificate to satisfy
operational constraints.

Many safety critical applications have used physics based constraints and
other information in action regulation. These kind of approaches can be
categorized under shielded RL/ safety filter, where a type of safety shield or
barrier function is employed to check the actions.

For safe power system control [141] proposes a framework for learning a
stabilizing controller that satisfies predefined safety regions, see Fig. 11.
Combining a model-free controller and a barrier-certification system, using a
NN based barrier function, i.e. neural barrier certificate (NBC). Given a
training set they learn a NBC $B_{\epsilon}(x)$ and filtered (regulated)
control action $\mathcal{F}_{u}^{\psi}$, jointly holding the following
condition

$\begin{split}(\forall{x}\in\mathcal{S}_{0},B_{\epsilon}(x)\leq
0)\wedge(\forall{x}\in\mathcal{S}_{u},B_{\epsilon}(x)>0)\\\
\wedge(\forall{x}\in{x|B_{\epsilon}(x)=0},\mathcal{L}_{f(x,u_{RL})}B_{\epsilon}(x)<0)\end{split}$

where $\mathcal{L}_{f(x,u_{RL})}B_{\epsilon}(x)$ is the Lie derivative of
$B_{\epsilon}(x)$, and $\phi,\epsilon$ are NN parameters.
$\mathcal{S}_{0},\mathcal{S}_{u}$ are set of initial states and unsafe states
respectively.

[19] introduces a hybrid approach of MFRL and MBRL using CBF, with provision
of online learning of unknown system dynamics. It assumes availability of a
set of safe states. In a MARL setting, [11] introduced cooperative and non-
cooperative CBFs in a collision-avoid problem, which includes both cooperative
agents and obstacles. Also in MARL setting, [17] proposed efficient active
voltage controller of photovoltaics (PVs) enabled with shielding mechanism.
Which ensures safe actions of battery energy storage systems (BESSs) during
training. [136] deals with controlling a district cooling system (DCS), with
complex thermal dynamic model and uncertainties from regulation signals and
cooling demands. The proposed safe controller a hybrid of barrier function and
DRL and helps avoid unsafe explorations and improves training efficiency. [76]
proposed a safe RL framework for adaptive cruise control, based on a safety-
supervision module. The authors used the underlying system dynamics and
exclusion-zone requirement to construct a safety set, for constraining the
learning exploration.

In a highway motion planning setting for autonomous vehicles[124] proposed a
CBF-DRL hybrid approach. Certain works like [13] and [14] have introduced
multiple physics based artifacts to ensure safe learning in autonomous agents.
Both of them used residual control based architecture merging physical model
and data driven control. Additionally it also leverages physics model guided
reward. [14] extends the work by [13] and introduces physics model guided
policy and value network editing in addition to the physics based reward. In
[32], the authors integrate learning a task space policy with a model based
inverse dynamics controller, which translates task space actions into joint-
level controls. This enables the RL policy to learn actions in task space.

####  4.1.3 Reward design:

Figure 12:  Example of physics incorporation in reward design. In [113] a
reward function design framework was introduced, that describe robot gaits as
a periodic phase sequence such that each of which rewards or penalizes a
particular physical system measurement.

In sim-to-real setting [113] proposed a reward specification framework based
on composing probabilistic periodic costs on basic forces and velocities, see
Fig. 12. The framework defines a parametric reward function for common robotic
(bipedal) gaits. Dealing with periodic robot behavior, the absolute time
reward function is here defined in terms of a cycle time variable $\phi$
(which cycles over time period of $[0,1]$, as $R(s,\phi)$. The updated reward
function as given below, is defined as a biased sum of $n$ reward components
$R_{i}(s,\phi)$, each capturing a desired robot gait characteristic.

$\begin{split}R(s,\phi)=\beta+\Sigma R_{i}(s,\phi),\;\text{where}\\\
R_{i}(s,\phi)=c_{i}\times I_{i}(\phi)\times q_{i}(s)\end{split}$

each $R_{i}(s,\phi)$ is a product of phase-coefficient $c_{i}$, phase
indicator $I_{i}(\phi)$ and phase reward measurement $q_{i}(s)$.

In [18], the authors introduced a RL-PIDL hybrid framework, to learn MFGs,
which generalize well and manage can complex multi-agent systems applications.
The physics based reward component (= evolution of population density/ mean-
field state) is approximated using PINN. To better mimic natural human
locomotion [68], designed reward function based on physical and experimental
information: trajectory optimization rewards, and bio-inspired rewards. In a
similar task of imitation of human motion but from motion clip, [20] proposes
a physics-based controller using DRL. A rigid body physics simulator is used
to solve rigid body poses that closely follows the motion capture (mocap) clip
frames. In a similar work [100], a data driven RL framework was introduced for
training control policies for simulated characters. Refernce motions are used
to define imitation reward and the task goal defines task specific reward.

[75] leverages a federated MADRL approach for energy management in multi-
microgrid settings. The reward is designed to satisfy two physical targets:
operation cost and self energy sufficiency. [135] proposed a DRL based method
for reconstruction of flow fields from noisy data. Physical constraints like
momentum equation, pressure Poisson equation and boundary conditions are used
for designing the reward function. [134] proposed physics based reward shaping
for wireless navigation applications. They used a cost function augmented with
physically motivated costs like costs for link-state monotonicity, for angle
of arrival direction following, and for SNR increasing. In single molecule 3D
structure optimization problem, [22] used physics based DFT calculation is
used as reward function, for physically correct structural prediction. In
[74], the authors used temporal logic through a finite state automata (FSA),
control Lyapunov and barrier function for ensuring effective and safe RL in
complex environments.The FSA simultaneously provides rewards, objectives and
safety constraints to the framework components.

Addressing the problem of dexterous manipulation of objects in virtual
environments, [42] trained the agent in a residual setting using hybrid model-
free RL-IL approach. Using a physics simulator and a pose estimation reward
the agent learns to refine the user input to achieve a task while keeping the
motion close to the input and the expert demonstrations. [83] tackles
physically valid 3D pose estimation from egocentric video. The authors
utilized a combination of kinematics and dynamics approach, whereby the
residual of the action against a learned kinematics model is outputted by the
dynamics-based model. In [56],the authors proposed inclusion of physics based
intrinsic reward for improved policy optimization of RL algorithms.

In the context of predicting interfacial area in two-phase flow, [28]
proposed. The two-phase flow physics information is infused into the
underlying MDP framework, which is then uses RL strategies to describe
behavior of flow dynamics. The work introduces multiple rewards based on
physical interfacial area transport models, other physical parameters and
data. In a work concerning optimization of nuclear fuel assembly [101], the
authors introduce a reward shaping approach in RL optimization, which is based
on physical tactics used by fuel designers. These tactics include moving fuel
rods in assembly to meet certain constraints and objectives.

A number of works have used physics through multiple PIRL methods. Apart from
reward design they have infused physics information through state design [112]
and action regulation [14, 13, 124]. They have been discussed in previous
sections and hence not repeated.

####  4.1.4 Augment simulator or model:

Figure 13: Example, augmentation of learnable model using physics information.
The figure shows system dynamics learning network structured using a LNN [103]
and next state calculations using Ralston’s method. Here the PINN (LNN) based
dynamics model and reward model , are learned via data-driven method.

In MBRL setting, using structure of underlying physics, and building upon
Lagrangian neural network (LLN) [25], Ramesh et al.[103] learned the system
model via data-driven approach, see Fig. 13. Concerning systems obeying
Lagrangian mechanics, the state consists of generalized coordinates $q$ and
velocities $\dot{q}$. Lagrangian, which is a scalar is defined as

$\begin{split}\mathcal{L}(q,\dot{q},t)=\mathcal{T}(q,\dot{q})-\mathcal{V}(q)\end{split}$

where $\mathcal{T}(q,\dot{q})$ is kinetic energy and $\mathcal{V}(q)$ is the
potential energy. And so the Lagrangian equation of motion can be written as

$\begin{split}\tau&=M(q)\ddot{q}+C(q,\dot{q})\dot{q}+G(q),\text{where}\\\
\ddot{q}&=M^{-1}(q)(\tau-C(q,\dot{q})\dot{q}-G(q))\end{split}$

where $C(q,\dot{q})\dot{q}$ is Coriolis term, $G(q)$ is gravitational term and
$\tau$ is motor torque. In the NN implementation, separate networks are used
for learning $\mathcal{V}(q)$ and $L(q)$, leveraging which the acceleration
$(\ddot{q})$ quantity is generated. The output state derivative
$(\dot{q},\ddot{q})$ is then integrated using $2^{nd}$-order Runge-Kutta to
compute next state.

Concerning a sim-to-real setting, in [46] authors train a recurrent neural
network on the differences between robotic trajectories in simulated and
actual environments. This model is further used to improve the simulator. For
improved transfer to real environment, [81] collected hardware data (positions
and calculated system velocities) to seed the simulator, for training control
policies. [1] proposes a framework for autonomous manufacturing of acoustic
meta-material, while leveraging physics informed RL and transfer learning. A
physics guided simulation engine is used to train the agent in source task and
then fine-tuned in a data-driven fashion in the target task.

[88] introduced a PINN based gravity model for training of dynamically
informed RL agents. [106] uses surrogate models that capture primary physics
of the system, as a starting point of training DRL agent. In a curriculum
learning setting, they train an agent to first track limit cycles in a
velocity space for a representative non-holonomic system and then further
trained on a small simulation dataset. [128] combines linear dynamic models of
physical systems with optimism driven exploration. Here the features for the
linear models obtained from robot morphology and the exploration is done using
MPC.

A number of works introduced novel models are better representations of real
world physics and serves as better simulators and ensures effective sim to
real transfers. [108] introduced learnable physics models which supports
accurate predictions and efficient generalization across distinct physical
systems. Concerning dynamic control with partially known underlying physics
(governing laws), [78] proposed a physics informed learning architecture, for
environment model. ODEs and PDEs serves as the primary source of physics for
these models. [121] uses entity abstraction to integrate graphical models,
symbolic computation and NNs in a MBRL agent. The framework presents object-
centric perception, prediction and planning which helps agents to generalize
to physical tasks not encountered before. [70] proposes a context aware
dynamics model which is adaptable to change in dynamics. They break the
problem of learning the environment dynamics model into two stages: learning
context latent vector and predicting next state conditioned on it.

In micro-grid power control problem, [111] combines model-based analytical
proof and reinforcement learning. Here model-based derivations are used to
narrow the learning space of the RL agen, reducing training complexity
significantly. In visual model based RL, [121] models a scene in terms of
entities and their local interactions, thus better generalizing to physical
task the learner has not seen before. Similar to learning entity abstractions,
in [70] the authors tackles the challenge of learning a generalizable global
model through: learning context latent vector, capturing local dynamics and
predicting next state conditioned on the encoded vector. Addressing dynamic
control problem in MBRL setting, [78] leveraged physical laws (in form of
canonical ODE/ PDE) and environmental constraints to mitigate model bias issue
and sample inefficiency. In autonomous driving safe ramp merging problem,
[120] embedded probabilistic CBF in RL policy in order to learn safe policies,
that also optimize the performance of the vehicle. Typically CBFs need good
approximation of car’s model. Here the probabilistic CBF is used as an
estimate of the model uncertainty.

[22] incorporates physics through reward design as well as through simulator
augmentation, and has been discussed in previous section.

####  4.1.5 Augment policy and/or value N/W:

Figure 14: Example, augmentation of policy using physics information. In [4],
given an observation $s_{t}$ from the environment, a neural dynamic policy
generates $w$ i.e. the weights of basis function and $g$ which is a goal for
the robot, for a function $f_{\theta}$. This function is then used by an open
loop controller to generate a set of actions from the robot to execute in the
environment and collect next states and rewards to train the policy.

In [4], Bahl et al. proposes Neural Dynamic Policies (NDP) where they
incorporate dynamical system as a differentiable layer in the policy network,
see Fig. 14. In NDP, a NN $\Phi$ takes an input state $(s_{t})$ and predicts
parameters of the dynamical system (i.e. $(w,g)$ ). Which are then used to
solve second-order differential equation
$\ddot{y}=\alpha(\beta(g-y)-\dot{(}y))+f(x)$, to obtain system states
$(y,\dot{y},\ddot{y}).$, which represents the behavior of the dynamic system,
given a state goal $g$. Here $\alpha,\beta$ are global parameters allowing
critical damping of system and $f$ is a non-linear forcing function which
primarily captures the shape of trajectory. Depending on robot’s coordinate
system an inverse controller may also be used to convert $y$ to $a$, i.e.
$a=\Omega(y,\dot{y},\ddot{y})$ . The NDPs thus can be defined as

$\begin{split}\pi(a|s;\theta)&\triangleq\Omega(DE(\Phi(s;\theta))),\text{where}\\\
DE(w,g)&\rightarrow\\{y,\dot{y},\ddot{y}\\}\end{split}$

here $DE(w,g)$ represents solution of the differential equation.

Extending this work to hierarchical deep policy learning framework, [3]
introduced H-NDP which forms a curriculum by learning local dynamical system-
based policies on small state-space region and then refines them into global
dynamical system based policy. Given the accurate dynamics and constraint of
the system [140] introduces control barrier certificates into actor-critic RL
framework, for learning safe policies in dynamical systems. [87] proposes a
method for generating highly agile and visually guided locomotion behaviors.
They leverage MFRL while using model based optimization of ground reaction
forces, as a behavior regularizer.

In [31] proposes an approach of safe exploration using CLBF without explicitly
employing any dynamic model. The approach approximate the RL critic as a CLBF,
from data samples and parameterized with DNNs. Both the actor and critic
satisfies reachability and safety guarantees. [93] combines PINN with RL,
where the value function is treated as a PINN to solve Hamilton-Jacobi-Bellman
(HJB) PDE. It enables the RL algorithm to exploit the physics of environment
aswell as optimal control to improve learning and convergence.

[98] proposes an optimization method for freeform nanophotonic devices, by
combining adjoint based methods (ABM) and RL. In this work the value network
is initialized with adjoint gradient predicting network during initialization
of RL process. Cao et al. [14] have used physics model to influence reward
function, as well as edit policy and value networks as necessary. The work has
been mentioned before in reward design.

To improve policy optimization, [92] used differentiable simulators to
directly compute the analytic gradient of the policy’s value function w.r.t.
the actions generated by it. This gradient information is used to
monotonically improve the policy’s value function. Gao et al. [40] proposes a
transient voltage control approach, by integrating physical and data-driven
models of power system. They also uses the constraint of the physical model on
the data-driven model to speed up convergence. A PINN trained using PDE of
transient process acts as the physical model and contributes directly to the
loss of the RL algorithm.

Xu et al. [130] presents an efficient differentiable simulator (DS) with a new
policy training algorithm which can effectively leverage simulation gradients.
The learning algorithm alleviates issues inherent in DS while allowing many
physical environments to be run in parallel. [17] incorporates physics through
action regulation and penalty signal to agent, and has been discussed in
previous section.

In MBRL setting, [85] leverage differentiable physics-based simulation and
differentiable rendering. By comparing raw observations between simulated and
real world, the initial learned system model is continually updated, producing
a more physically consistent model. In data center (DC) cooling control
application, [123] proposed a lifelong-RL approach under evolving DC
environment. It leverages physical laws of thermodynamics and the system and
models the DC thermal transition and power usage through data collected
online. Utilizing learned state transition and reward models it accelerates
online adaptation.

Working with a nominal system model, [23] presented an RL framework where the
agent learns model uncertainty in multiple general dynamic constraints, e.g.
CLF and CBF, through data-driven training. A quadratic program then solves for
the control that satisfies the safety constraints under learned model
uncertainty.

---
(a)

---
(b)

---
(c)

---
(d)

Figure 15: Statistical analysis of PIRL literature. (a) Statistic of type of
RL algorithms used, (b) Statistic of PI types used in each PIRL method, (c)
Statistic of PIRL learning architectures and related biases, (d) Statistic of
PIRL applications in different domains. Table II: Summary of PIRL literature -
Model Free.

Ref. | Year | Context/ Application | RL Algorithm | Learning arch. | Bias | Physics information | PIRL methods | RL pipeline
---|---|---|---|---|---|---|---|---
[20] | 2018 | Motion capture | PPO | Physics reward | Learning | Physics simulator | Reward design | Problem representation
[100] | 2018 | Motion control | PPO [109] | Physics reward | Learning | Physics simulator | Reward design | Problem representation
[46] | 2018 | Policy optimization | PPO | Sim-to-Real | Observational | Offline data | Augment simulator | Training
[81] | 2018 | Policy optimization | NPG [126] (C)∗ | Sim-to-Real | Observational | Offline data | Augment simulator | Training
[22] | 2019 | Molecular structure optimization | DDPG | Physics reward | Learning | DFT (PS) | Reward design | Problem representation
|  |  |  |  |  |  | Augment simulator | Training
[74] | 2019 | Safe exploration and control | PPO | Residual RL | Learning | CBF, CLF, FSA/TL (BPC) | Reward design | Problem representation
|  |  |  |  |  |  | Augment policy | Learning strategy
[4] | 2020 | Dynamic system control | PPO | Phy. embed. N/W | Inductive | DMP (PPV) | Augment policy | Network design
[42] | 2020 | Dexterous manipulations | PPO | Residual RL | Observational | Physics simulator | Reward design | Problem representation
[83] | 2020 | 3D Ego pose estimation | PPO | Physics reward | Learning | Physics simulator | State, Reward design | Problem representation
[3] | 2021 | Dynamic system control | PPO | Hierarchical RL | Inductive | DMP (PPV) | Augment policy | Network design
[87] | 2021 | Dynamic system control | PPO | Hierarchical RL | Learning | WBIC (PPV) | Augment policy | Learning strategy
[1] | 2021 | Manufacturing | SARSA [116] | Sim-to-Real | Observational | Physics engine | Augment simulator | Training
[113] | 2021 | Dynamic system control | PPO | Phy. variable | Learning | Physics parameters | Reward design | Problem representation
[76] | 2021 | Safe exploration and control | NFQ [104] | Safety filter | Learning | Physical constraint | Action regulation | Problem representation
[59] | 2021 | Safe cruise control | SAC | Phy. variable | Observational | Physical state (PPV) | State design | Problem representation
[92] | 2021 | Policy optimization | DPG (C) | Diff. Simulator | Learning | Physics simulator | Augment policy | Learning strategy
[101] | 2021 | Optimization, nuclear engineering | DQN, PPO | Physics reward | Learning bias | Physical properties (PPR) | Reward design | Problem representation
[139] | 2021 | Air-traffic control | PPO | Data augmentation | Observational | Representation (ODR) | State design | Problem representation
[124] | 2022 | Motion planner | PPO + AC [67] | Safety filter | Learning | CBF (BPC) | Action regulation | Problem representation
|  |  |  |  |  |  | Reward design |
[17] | 2022 | Active voltage control | TD3 (C) | Safety filter | Learning | Physical constraints | Penalty function | Problem representation
|  |  |  |  |  |  | Action regulation |
[28] | 2022 | Interfacial structure prediction | DDPG | Off-policy | Learning | Physics model | Reward design | Problem representation
[40] | 2022 | Transient voltage control | DQN | PINN loss | Learning | PDE (DAE) | Augment policy | Learning strategy
[45] | 2022 | Building control | Q-learning (C) | Data augment | Observational | Representation (ODR) | State design | Problem representation
[51] | 2022 | Traffic control | Q-Learning | Data augment | Observational | Physics model | State design | Problem representation
[88] | 2022 | Safe exploration and control | SAC | Sim-to-Real | Observational | Physics model | Augment simulator | Training
[56] | 2022 | Dynamic system control | SAC (etc.) | Physics reward | Learning | Barrier function | Reward design | Problem representation
[130] | 2022 | Policy Learning | Actor-critic (C) | Diff. Simulator | Learning | Physics simulator | Augment policy | Learning strategy
[13] | 2023 | Safe exploration and control | DDPG | Residual RL | Learning | Physics model | Reward design | Problem representation
|  |  |  |  |  |  | Action regulation |
[14] | 2023 | Safe exploration and control | DDPG | Residual RL | Inductive | Physics model | Reward design | Problem representation
|  |  |  |  |  |  | Action regulation |
|  |  |  |  | Inductive |  | N/W editing (Aug. pol.) | Network design
[12] | 2023 | Robust voltage control | SAC | Data augment | Observational | Representation (ODR) | State design | Problem representation
[18] | 2023 | Mean field games | DDPG | Physics reward | Learning | Physics model | Reward design | Problem representation
[133] | 2023 | Safe exploration and control | PPO (C) | Safety filter | Learning | NBC (BPC) | Augment policy | Training
[141] | 2023 | Power system stability enhancement | Custom | Safety filter | Learning | NBC (BPC) | Action regulation | Problem representation
[31] | 2023 | Safe exploration and control | AC (C) | Safety filter | Learning | CLBF [107, 29] (BPC) | Augment value N/W | Training
[112] | 2023 | Connected automated vehicles | DPPO | Physics variable | Observational | Physical state (PPV) | State design | Problem representation
|  |  |  |  | Learning |  | Reward design |
[68] | 2023 | Musculoskeletal simulation | SAC (C) | Physics variable | Learning | Physical value | Reward design | Problem representation
[75] | 2023 | Energy management | MADRL(C) | Physics variable | Learning | Physical target | Reward design | Problem representation
[93] | 2023 | Policy optimization | PPO | Phy. embed N/W | Inductive | PDE (DAE) | Augment value N/W | Network design
[135] | 2023 | Flow field reconstruction | A3C | Physics reward | Learning | Physical constraints | Reward design | Problem representation
[98] | 2023 | Freeform nanophotonic devices |  $\epsilon-$ greedy Q | Phy. embed N/W | Inductive | ABM | Augment value N/W | Network design
[106] | 2023 | Dynamic system control | DPG | Curriculum learning | Learning | Physics model | Augment simulator | Training
[111] | 2023 | Energy management | TD3 | Sim-to-Real | Observational | Physics model | Augment simulator | Learning strategy
[134] | 2023 | Robot wireless navigation | PPO | Physics reward | Learning | Physical value | Reward design | Problem representation

C∗ represents custom versions of the adjacent conventional algorithms.

Table III: Summary of PIRL literature - Model based

Ref. | Year | Context/ Application | Algorithm | Learning arch. | Bias | Physics information | PIRL method | RL pipeline
---|---|---|---|---|---|---|---|---
[128] | 2016 | Exploration and control | - | Model learning | Observational | Sys. morphology (PPR) | Augment model | Learning strategy
[108] | 2018 | Dynamic system control | - | Model learning | Inductive | Physics model | Augment model | Learning strategy
[95] | 2019 | Safe navigation | - | Safety filter | Learning | CBC (BPC) | Action regulation | Problem representation
[19] | 2019 | Safe exploration and control | TRPO, DDPG | Residual RL | Learning | CBF (BPC) | Action regulation | Problem representation
[121] | 2020 | Control (visual RL) | - | Model learning | Observational | Entity abstraction (ODR) | Augment model | Learning strategy
[70] | 2020 | Dynamic system control | - | Model learning | Observational | Context encoding (ODR) | Augment model | Learning strategy
[23] | 2020 | Safe exploration and control | DDPG [114] | Safety filter | Learning | CBF, CLF, QP (BPC) | augment policy | Learning strategy
[78] | 2021 | Dynamic system control | Dyna + TD3(C)∗ | Model identification | Learning | PDE/ ODE, BC (DAE) | Augment model | Learning strategy
[32] | 2021 | Dynamic system control | PPO | Residual-RL | Learning | Physics model | Action regulation | Problem representation
[11] | 2021 | Multi agent collision avoidance | MADDPG (C) | Safety filter | Learning | CBF (BPC) | Action regulation | Problem representation
[85] | 2022 | Dynamic system control | TD3(C) | Sim-to-Real | Learning | Physics simulator | Augment policy | Learning strategy
[120] | 2022 | Traffic control | AC[86] | Safety filter | Learning | CBF (BPC) | Augment model | Learning strategy
[140] | 2022 | Safe exploration and control | DDPG | Safety filter | Learning | CBC (BPC) | Augment policy | Learning strategy
[138] | 2022 | Distributed MPC | AC [57] | Safety filter | Learning | CBF (BPC) | State design | Problem representation
[103] | 2023 | Dynamic system control | Dreamer [50] | Phy. embed. N/W | Inductive | Physics model | Augment model | Network design
[24] | 2023 | Safe exploration and control | - | Safety filter | Learning | CBF (BPC) | Augment model | Learning strategy
[54] | 2023 | Attitude control | - | Phy. embed N/W | Inductive | System symmetry (PPR) | Augment model | Network design
[123] | 2023 | Data center cooling | SAC | Model identification | Learning | Physics laws (PPR) | Augment model | Learning strategy
[136] | 2023 | Cooling system control | DDPG | Residual RL | Learning | CBF (BPC) | Action regulation | Problem representation

C∗ represents custom versions of the adjacent conventional algorithms.

###  4.2 Review of simulation/ evaluation benchmarks

In Table IV, we present the different training and evaluation benchmarks that
has been used in the reviewed PIRL literature. We list the important insights
from the table:

  * 1.

A majority works dealing with dynamic control have used OpenAI Gym [128], Safe
Gym [133], MuJoCo [121, 142], Pybullet [31] and Deep mind control suite
environments [108, 103], which are standard benchmarks in RL . Works dealing
specifically with traffic management have used platforms like SUMO [124] and
CARLA [120].

  * 2.

Works dealing with power and voltage management problems have used IEEE
distribution system benchmarks [17, 40] to evaluate proposed algorithms.
Alternatively in some works MATLAB/ SIMULINK platform is also used for
training or evaluating RL agents [111]

  * 3.

One crucial observation is that a huge number of work have used customized or
adapted environments for training and evaluation and have not used
conventional environments [74, 24, 84].

###  4.3 Analysis

####  4.3.1 Research trend and statistics

Use of RL algorithms:  As is evident from Fig.15 (a), PPO[109] and its
variants are the most preferred RL algorithm, followed by DDPG [114]. Among
the comparatively new algorithms SAC[49] is preferred over TD3[38].

Types of physics priors used:  In Fig.15 (b), we can see that physics
information takes the form of physics simulator, system models, barrier
certificates and physical constraints, in a majority of works. PI types
Barrier certificate constraints and physical constraint and Physics simulator
and models dominates in more that $60\%$ of works in Action regulation and
Augment policy and value N/W PIRL methods.

Learning architecture and bias:  In Fig.15 (c) we visualize the relationship
between PIRL learning architectures (sec: 3.4.2) and the three biases through
which physics is typically incorporated in PIML approaches. In architectures
PI reward and safety filter, physics is incorporated strictly through learning
bias, signifying the heavy use of constraints, regularizers and specialized
loss functions. While Physics embedded network incorporates physics
information through inductive bias, i.e. through imposition of hard
constraints through use specialized and custom physics embodied networks.

Application domains:  In Fig.15 (d) almost $85\%$ of the application problem
dealt with PIRL approaches relates to controller or policy design.
Miscellaneous control includes optimal policy/ controller learning approaches
for different application sectors like energy management [75, 111] and data-
center cooling [123], and accounts to majority of applications. Safe control
and exploration, includes those works concerning with safety critical systems,
ensuring safe exploration and policy learning, accounts for $25\%$. Dynamic
control, includes control of dynamic systems, including robot systems and
amounts to about $23\%$ of all works surveyed. Other specific applications
include optimization/ prediction [22, 28], motion capture/simulation [124, 20]
and improvement of general policy optimization approaches [46, 81] through
physics incorporation.

####  4.3.2 RL challenges addressed

In this section we will discuss and elaborate on how recent physics
incorporation in RL algorithms have addressed certain open problems of the RL
paradigm.

  * 1)

Sample efficiency: RL approaches need a huge number of agent-environment
interaction and related data to work. One effective way of dealing with this
problem is to use a surrogate for the real environment in the form of a
simulator or learned model via data-driven approaches.

PIRL approaches incorporate physics to augment simulators thus reducing the
sim-to-real gap, thereby bringing down online evaluation cycles [85, 1]. Also
physics incorporation during system identification or model learning phase in
MBRL help reduce sample efficiency through learning a truer to real
environment using lesser training samples [108, 121].

  * 2)

Curse of dimensionality: RL algorithms become less efficient both in training
and performing on environment defined with high-dimensional and continuous
state and action spaces, known as the ’curse of dimensionality’. Typically
dimensionality reduction techniques are used to encode the large state or
action vectors into low dimensional representations. The RL algorithm is then
trained in this low dimensional setting.

PIRL approaches extract underlying physics information from environment
through learning physically relevant low dimensional representation from high
dimensional observation or state space [45, 12]. In [45], a PINN is utilized
to extract physically relevant information about the system’s hidden state,
which is then used to learn a Q-function for policy optimization.

  * 3)

Safety exploration: Safe reinforcement learning involves learning control
policies that guarantee system performance and respect safety constraints
during both exploration and policy deployment.

In safety-critical applications using reinforcement learning, it’s crucial to
regulate agent exploration. Control Lyapunov function (CLF)[74, 23], barrier
certificate/ barrier function (BF), control barrier function/ certificate
(CBF/ CBC) [19, 11] are commonly used concepts. Barrier certificates define
safe states, while control barrier functions ensure states stay in the safety
set. These approaches are typically used for systems with partial or learnable
dynamics model and generally a known set of safe states/ actions.

  * 4)

Partial observability or imperfect measurement: Partial observability is a
setting where due to noise, missing information, or outside interference, an
RL agent is unable to obtain the complete states needed to understand the
environment.

PIRL approaches modify or enhance the state representation to provide more
useful information, in cases of missing or inadequate information. This may
involve state fusion, which incorporates additional physics or geographical
information from the environment [59] or other agents [112].

  * 5)

Under-defined reward function: Defining the reward function is critical in
creating MDPs and ensuring the effectiveness and efficiency of RL algorithms.
However, since they are created by humans, there is a risk of them being
under-defined and not guiding the RL algorithm effectively in policy
optimization.

PIRL approaches introduce physics information through effective reward design
or augmentation of existing reward functions with bonuses or penalties [28,
83, 42, 113]. For example, in a sim-to-real setting, [113] proposed a
framework for specifying rewards that combines probabilistic costs associated
with primary forces and velocities. The framework creates a parametric reward
function for common robotic gaits, in biped robots.

Table IV: Summary of PIRL training/ evaluation benchmarks.

Simulator/ platform | Specific environment/ system name | Reference
---|---|---
OpenAI Gym | Pusher, Striker, ErgoReacher | [46]
OpenAI Gym | Mountain Car, Lunar Lander $(continuous)$ | [56]
OpenAI Gym | Cart-Pole, Pendulum (simple and double) | [128]
OpenAI Gym | Cart-pole | [13]
OpenAI Gym | Cart-pole and Quadruped robot | [14]
OpenAI Gym | CartPole, Pendulum | [78]
OpenAI Gym | Inverted Pendulum $(pendulum-v0)$, | [19]
OpenAI Gym | Mountain car $(cont.)$, Pendulum, Cart pole | [140]
OpenAI Gym | Simulated car following [53] |
MuJoCo | Ant, HalfCheetah, Humanoid, Walker2d | [93]
| Humanoid standup, Swimmer, Hopper |
| Inverted and Inverted Double Pendulum $(v4)$ |
MuJoCo | Cassie-MuJoCo-sim [105] | [113, 32]
| 6 DoF Kinova Jaco [44] | [4, 3]
MuJoCo | HalfCheetah, Ant, | [70]
| CrippledHalfCheetah, and SlimHumanoid [142] |
MuJoCo | Block stacking task [55] | [121]
OpenAI Gym | CartPole, Pendulum |
OpenSim-RL[64] | L2M2019 environment | [68]
Safety gym [137] | Point, car and Doggo goal | [133]
- | Cart pole swing up, Ant | [130]
- | Humanoid, Humanoid MTU |
- | Autonomous driving system | [18]
Deep control suite [117] | Pendulum, Cartpole, Walker2d | [108]
| Acrobot, Swimmer, Cheetah |
- | JACO arm (real world) |
Deep control suite | Reacher, Pendulum, Cartpole, | [103]
| Cart-2-pole, Acrobot, |
| Cart-3-pole and Acro-3-bot |
- | Rabbit[21] | [23]
MARL env. [80] | Multi-agent particle env. | [11]
ADROIT[102] | Shadow dexterous hand | [42]
- | First-Person Hand Action Benchmark[43] |
MuJoCo | Door opening, in-hand manipulation, |
| tool use and object relocation |
SUMO[79], METANET[69] | - | [51]
SUMO | - | [124]
CARLA[30] | - | [120]
Gazebo[66] | Quadrotor $(IF750A)$ | [54]
IEEE Distribution | IEEE 33-bus and 141-bus distribution networks | [17]
system benchmarks | IEEE 33-node system | [12, 17]
| IEEE 9-bus standard system | [40]
- | Custom (COMSOL based) | [1]
- | Custom (DFT based) | [22]
- | Custom (based on [122]) | [45]
- | Custom (based on [63]) | [59]
- | Custom | [75, 88, 28]
- | Custom | [98, 112, 134]
- | Custom | [135, 136, 141]
- | Custom | [74, 24, 84]
- | Custom | [123, 18]
Open AI Gym | Custom (based on geometries of Nuclear reactor) | [101]
MATLAB-Simulink | Custom | [111, 138]
- | Custom [143] | [92]
MATLAB | Cruise control | [76]
Pygame | Custom | [139]
- | Custom (Unicycle, Car-following) | [36]
- | Brushbot, Quadrotor (sim) | [95]
| Phantom manipulation platform | [81]
Pybullet | 2 finger gripper |
| gym-pybullet-drones[97] | [31]
Pybullet | Franka Panda, Flexiv Rizon (also real world robots) | [85]
NimblePhysics[125], |  |
Redner[73] (Differentiable sim.) |  |
- | Custom MOCAP | [20, 100, 83]

##  5 Open Challenges and Research Directions

###  5.1 High Dimensional Spaces

A large number of real world tasks deals with high dimensional and continuous
state and action spaces. One popular method to address this high
dimensionality issue is to compress the state space (or action space) vectors
into low dimensional vectors. A PI based approach may learn high quality
environment representations using deep networks and extract physically
relevant low dimensional features from them.

But learning a compressed and informative latent space from high dimensional
continuous state (or action) space still remains a hurdle. Also learning
physically relevant representation is still an open problem. Future research
should address this issue and try to devise approaches that helps to
incorporate or take guidance of underlying physics during representation
learning or feature extraction, so as to make them both informative and
physically pertinent.

###  5.2 Safety in Complex and Uncertain Environments

In the realm of safe reinforcement learning, striking a balance between the
complexity of the environment and ensuring safety is always a challenge.
Current physics informed approaches uses different control theoretic concepts
e.g. CBFs to ensure safe exploration and learning of the RL agent. But these
approaches are limited by the approximated model of the system and the prior
knowledge about safe state sets. There has been a lot of research for better
system identification or model learning through physics incorporation. But
most works do not generalize well to different tasks and environments. To
summarize, future works should address these crucial research goals: 1) model
agnostic safe exploration and control using RL agents in complex and uncertain
environments and 2) devise generalized approach of incorporating physics in
data-driven Model learning.

###  5.3 Choice of physics prior

Choice of the physics prior is very crucial for the PIRL algorithm. But such
choice is difficult and requires extensive study of the system and may vary
extensively from one case to another even in same domains. To enhance
efficacy, devising a comprehensive framework with physics information to
manage novel physical tasks is preferable rather than dealing with tasks
individually.

###  5.4 Evaluation and bench-marking platform

Currently, PIRL doesn’t have comprehensive benchmarking and evaluation
environments to test and compare new physics approaches before induction. This
limitation makes it challenging to assess the quality and uniqueness of new
works.

Additionally, most PIRL works rely on customized environments related to a
particular domain, making it difficult to compare PIRL algorithms fairly.
Moreover, PIRL application cases are diverse, and the physics information
chosen is specific to a domain, requiring extensive study and domain expertise
to understand and compare such works.

##  6 Conclusions

This paper presents a state-of-the-art reinforcement learning paradigm, known
as physics-informed reinforcement learning (PIRL). By leveraging both data-
driven techniques and knowledge of underlying physical principles, PIRL is
capable of improving the effectiveness, sample efficiency and accelerated
training of RL algorithms/ approaches, for complex problem-solving and real-
world deployment. We have created two taxonomies that categorize conventional
PIRL methods based on physics prior/information type and physics prior
induction (RL methods), providing a framework for understanding this approach.
To help readers comprehend the physics involved in solving RL tasks, we have
included various explanatory images from recent papers and summarized their
characteristics in Tables II and III. Additionally, we have provided a
benchmark-summary table IV detailing the training and evaluation benchmarks
used for PIRL evaluation. Our objective is to simplify the complex concepts of
existing PIRL approaches, making them more accessible for use in various
domains. Finally, we discuss the limitations and unanswered questions of
current PIRL work, encouraging further research in this area.

##  7 Acknowledgment

This research was partly supported by the Advance Queensland Industry Research
Fellowship AQIRF024-2021RD4.

## References

  * [1] Md Ferdous Alam, Max Shtein, Kira Barton and David J Hoelzle  “A physics-guided reinforcement learning framework for an autonomous manufacturing system with expensive data”  In _2021 American Control Conference (ACC)_ , 2021, pp. 484–490
  * [2] Aaron D Ames et al.  “Control barrier functions: Theory and applications”  In _2019 18th European control conference (ECC)_ , 2019, pp. 3420–3431
  * [3] Shikhar Bahl, Abhinav Gupta and Deepak Pathak  “Hierarchical neural dynamic policies”  In _arXiv preprint arXiv:2107.05627_ , 2021
  * [4] Shikhar Bahl, Mustafa Mukadam, Abhinav Gupta and Deepak Pathak  “Neural dynamic policies for end-to-end sensorimotor learning”  In _Advances in Neural Information Processing Systems_ 33, 2020, pp. 5058–5069
  * [5] Chayan Banerjee, Zhiyong Chen and Nasimul Noman  “Boosting Exploration in Actor-Critic Algorithms by Incentivizing Plausible Novel States”  In _arXiv preprint arXiv:2210.00211_ , 2022
  * [6] Chayan Banerjee, Zhiyong Chen and Nasimul Noman  “Improved soft actor-critic: Mixing prioritized off-policy samples with on-policy experiences”  In _IEEE Transactions on Neural Networks and Learning Systems_ IEEE, 2022
  * [7] Chayan Banerjee, Zhiyong Chen, Nasimul Noman and Mohsen Zamani  “Optimal Actor-Critic Policy With Optimized Training Datasets”  In _IEEE Transactions on Emerging Topics in Computational Intelligence_ 6.6 IEEE, 2022, pp. 1324–1334
  * [8] Chayan Banerjee, Kien Nguyen, Clinton Fookes and George Karniadakis  “Physics-Informed Computer Vision: A Review and Perspectives”, 2023  arXiv:2305.18035 [eess.IV]
  * [9] Gabriel Barth-Maron et al.  “Distributed distributional deterministic policy gradients”  In _arXiv preprint arXiv:1804.08617_ , 2018
  * [10] Serena Booth et al.  “The perils of trial-and-error reward design: misdesign through overfitting and invalid task specifications”  In _AAAI Conference on Artificial Intelligence_ 37.5, 2023, pp. 5920–5929
  * [11] Zhiyuan Cai et al.  “Safe multi-agent reinforcement learning through decentralized multiple control barrier functions”  In _arXiv preprint arXiv:2103.12553_ , 2021
  * [12] Di Cao et al.  “Physics-informed Graphical Representation-enabled Deep Reinforcement Learning for Robust Distribution System Voltage Control”  In _IEEE Transactions on Smart Grid_ IEEE, 2023
  * [13] Hongpeng Cao, Yanbing Mao, Lui Sha and Marco Caccamo  “Physical Deep Reinforcement Learning Towards Safety Guarantee”  In _arXiv preprint arXiv:2303.16860_ , 2023
  * [14] Hongpeng Cao, Yanbing Mao, Lui Sha and Marco Caccamo  “Physical Deep Reinforcement Learning: Safety and Unknown Unknowns”  In _arXiv preprint arXiv:2305.16614_ , 2023
  * [15] Ci Chen et al.  “Off-policy learning for adaptive optimal output synchronization of heterogeneous multi-agent systems”  In _Automatica_ 119, 2020, pp. 109081
  * [16] Jianyu Chen, Bodi Yuan and Masayoshi Tomizuka  “Model-free deep reinforcement learning for urban autonomous driving”  In _2019 IEEE intelligent transportation systems conference (ITSC)_ , 2019, pp. 2765–2771
  * [17] Pengcheng Chen, Shichao Liu, Xiaozhe Wang and Innocent Kamwa  “Physics-Shielded Multi-Agent Deep Reinforcement Learning for Safe Active Voltage Control with Photovoltaic/Battery Energy Storage Systems”  In _IEEE Transactions on Smart Grid_ IEEE, 2022
  * [18] Xu Chen, Shuo Liu and Xuan Di  “A Hybrid Framework of Reinforcement Learning and Physics-Informed Deep Learning for Spatiotemporal Mean Field Games”  In _2023 International Conference on Autonomous Agents and Multiagent Systems_ , 2023, pp. 1079–1087
  * [19] Richard Cheng, Gábor Orosz, Richard M Murray and Joel W Burdick  “End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks”  In _AAAI conference on artificial intelligence_ 33.01, 2019, pp. 3387–3395
  * [20] Nuttapong Chentanez et al.  “Physics-based motion capture imitation with deep reinforcement learning”  In _11th ACM SIGGRAPH Conference on Motion, Interaction and Games_ , 2018, pp. 1–10
  * [21] Christine Chevallereau et al.  “Rabbit: A testbed for advanced control theory”  In _IEEE Control Systems Magazine_ 23.5, 2003, pp. 57–79
  * [22] Youngwoo Cho et al.  “Physics-guided reinforcement learning for 3D molecular structures”  In _Workshop at the 33rd Conference on Neural Information Processing Systems (NeurIPS)_ , 2019
  * [23] Jason Choi, Fernando Castaneda, Claire J Tomlin and Koushil Sreenath  “Reinforcement learning for safety-critical control under model uncertainty, using control lyapunov functions and control barrier functions”  In _arXiv preprint arXiv:2004.07584_ , 2020
  * [24] Max H Cohen and Calin Belta  “Safe exploration in model-based reinforcement learning using control barrier functions”  In _Automatica_ 147 Elsevier, 2023, pp. 110684
  * [25] Miles Cranmer et al.  “Lagrangian neural networks”  In _arXiv preprint arXiv:2003.04630_ , 2020
  * [26] Salvatore Cuomo et al.  “Scientific Machine Learning through Physics-Informed Neural Networks: Where we are and What’s next”  In _arXiv preprint arXiv:2201.05624_ , 2022
  * [27] Mark Cutler, Thomas J Walsh and Jonathan P How  “Real-world reinforcement learning via multifidelity simulators”  In _IEEE Transactions on Robotics_ 31.3 IEEE, 2015, pp. 655–671
  * [28] Zhuoran Dang and Mamoru Ishii  “Towards stochastic modeling for two-phase flow interfacial area predictions: A physics-informed reinforcement learning approach”  In _International Journal of Heat and Mass Transfer_ 192 Elsevier, 2022, pp. 122919
  * [29] Charles Dawson, Zengyi Qin, Sicun Gao and Chuchu Fan  “Safe nonlinear control using robust neural lyapunov-barrier functions”  In _Conference on Robot Learning_ , 2022, pp. 1724–1735  PMLR
  * [30] Alexey Dosovitskiy et al.  “CARLA: An open urban driving simulator”  In _Conference on robot learning_ , 2017, pp. 1–16  PMLR
  * [31] Desong Du et al.  “Reinforcement Learning for Safe Robot Control using Control Lyapunov Barrier Functions”  In _arXiv preprint arXiv:2305.09793_ , 2023
  * [32] Helei Duan et al.  “Learning task space actions for bipedal locomotion”  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 1276–1282
  * [33] Gabriel Dulac-Arnold et al.  “Challenges of real-world reinforcement learning: definitions, benchmarks and analysis”  In _Machine Learning_ 110.9 Springer, 2021, pp. 2419–2468
  * [34] Gabriel Dulac-Arnold et al.  “Deep reinforcement learning in large discrete action spaces”  In _arXiv preprint arXiv:1512.07679_ , 2015
  * [35] Julian EEßerer et al.  “Guided Reinforcement Learning: A Review and Evaluation for Efficient and Effective Real-World Robotics”  In _IEEE Robotics & Automation Magazine_ IEEE, 2022
  * [36] Yousef Emam et al.  “Safe reinforcement learning using robust control barrier functions”  In _IEEE Robotics and Automation Letters_ IEEE, 2022, pp. 1–8
  * [37] N Benjamin Erichson, Michael Muehlebach and Michael W Mahoney  “Physics-informed autoencoders for Lyapunov-stable fluid flow prediction”  In _arXiv preprint arXiv:1905.10866_ , 2019
  * [38] Scott Fujimoto, Herke Hoof and David Meger  “Addressing function approximation error in actor-critic methods”  In _International conference on machine learning_ , 2018, pp. 1587–1596  PMLR
  * [39] Bolin Gao and Lacra Pavel  “On Passivity, Reinforcement Learning, and Higher Order Learning in Multiagent Finite Games”  In _IEEE Transactions on Automatic Control_ 66.1, 2021, pp. 121–136
  * [40] Jiemai Gao, Siyuan Chen, Xiang Li and Jun Zhang  “Transient Voltage Control Based on Physics-Informed Reinforcement Learning”  In _IEEE Journal of Radio Frequency Identification_ 6 IEEE, 2022, pp. 905–910
  * [41] Javier Garcia and Fernando Fernández  “Safe exploration of state and action spaces in reinforcement learning”  In _Journal of Artificial Intelligence Research_ 45, 2012, pp. 515–564
  * [42] Guillermo Garcia-Hernando, Edward Johns and Tae-Kyun Kim  “Physics-based dexterous manipulations with estimated hand poses and residual reinforcement learning”  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2020, pp. 9561–9568
  * [43] Guillermo Garcia-Hernando, Shanxin Yuan, Seungryul Baek and Tae-Kyun Kim  “First-person hand action benchmark with rgb-d videos and 3d hand pose annotations”  In _IEEE conference on computer vision and pattern recognition_ , 2018, pp. 409–419
  * [44] Dibya Ghosh et al.  “Divide-and-conquer reinforcement learning”  In _arXiv preprint arXiv:1711.09874_ , 2017
  * [45] Gargya Gokhale, Bert Claessens and Chris Develder  “PhysQ: A Physics Informed Reinforcement Learning Framework for Building Control”  In _arXiv preprint arXiv:2211.11830_ , 2022
  * [46] Florian Golemo, Adrien Ali Taiga, Aaron Courville and Pierre-Yves Oudeyer  “Sim-to-real transfer with neural-augmented robot simulation”  In _Conference on Robot Learning_ , 2018, pp. 817–828  PMLR
  * [47] Samuel Greydanus, Misko Dzamba and Jason Yosinski  “Hamiltonian neural networks”  In _Advances in neural information processing systems_ 32, 2019
  * [48] Shangding Gu et al.  “A review of safe reinforcement learning: Methods, theory and applications”  In _arXiv preprint arXiv:2205.10330_ , 2022
  * [49] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel and Sergey Levine  “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor”  In _International conference on machine learning_ , 2018, pp. 1861–1870  PMLR
  * [50] Danijar Hafner, Timothy Lillicrap, Jimmy Ba and Mohammad Norouzi  “Dream to control: Learning behaviors by latent imagination”  In _arXiv preprint arXiv:1912.01603_ , 2019
  * [51] Yu Han et al.  “A physics-informed reinforcement learning-based strategy for local and coordinated ramp metering”  In _Transportation Research Part C: Emerging Technologies_ 137 Elsevier, 2022, pp. 103584
  * [52] Zhongkai Hao et al.  “Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications”  In _arXiv preprint arXiv:2211.08064_ , 2022
  * [53] Chaozhe R He, I Ge Jin and Gábor Orosz  “Data-based fuel-economy optimization of connected automated trucks in traffic”  In _2018 Annual American Control Conference (ACC)_ , 2018, pp. 5576–5581
  * [54] Junchang Huang et al.  “Symmetry-informed Reinforcement Learning and Its Application to the Attitude Control of Quadrotors”  In _IEEE Transactions on Artificial Intelligence_ IEEE, 2023
  * [55] Michael Janner et al.  “Reasoning about physical interactions with object-oriented prediction and planning”  In _arXiv preprint arXiv:1812.10972_ , 2018
  * [56] Jiazhou Jiang, Minyue Fu and Zhiyong Chen  “Physics informed intrinsic rewards in reinforcement learning”  In _2022 Australian & New Zealand Control Conference (ANZCC)_, 2022, pp. 74–69
  * [57] Yi Jiang et al.  “Cooperative adaptive optimal output regulation of nonlinear discrete-time multi-agent systems”  In _Automatica_ 121 Elsevier, 2020, pp. 109149
  * [58] Tobias Johannink et al.  “Residual reinforcement learning for robot control”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 6023–6029
  * [59] Sorin Liviu Jurj et al.  “Increasing the safety of adaptive cruise control using physics-guided reinforcement learning”  In _Energies_ 14.22 MDPI, 2021, pp. 7572
  * [60] George Em Karniadakis et al.  “Physics-informed machine learning”  In _Nature Reviews Physics_ 3.6 Nature Publishing Group, 2021, pp. 422–440
  * [61] Ali Kashefi, Davis Rempe and Leonidas J Guibas  “A point-cloud deep learning framework for prediction of fluid flow fields on irregular geometries”  In _Physics of Fluids_ 33.2 AIP Publishing LLC, 2021, pp. 027104
  * [62] K Kashinath et al.  “Physics-informed machine learning: case studies for weather and climate modelling”  In _Philosophical Transactions of the Royal Society A_ 379 The Royal Society Publishing, 2021, pp. 1–36
  * [63] Arne Kesting et al.  “Jam-avoiding adaptive cruise control (ACC) and its impact on traffic dynamics”  In _Traffic and Granular Flow’05_ , 2007, pp. 633–643  Springer
  * [64] Łukasz Kidziński et al.  “Learning to run challenge: Synthesizing physiologically accurate motion using deep reinforcement learning”  In _The NIPS’17 Competition: Building Intelligent Systems_ , 2018, pp. 101–120  Springer
  * [65] W Bradley Knox et al.  “Reward (mis) design for autonomous driving”  In _Artificial Intelligence_ 316 Elsevier, 2023, pp. 103829
  * [66] Nathan Koenig and Andrew Howard  “Design and use paradigms for gazebo, an open-source multi-robot simulator”  In _2004 IEEE/RSJ international conference on intelligent robots and systems (IROS)(IEEE Cat. No. 04CH37566)_ 3, 2004, pp. 2149–2154
  * [67] Vijay Konda and John Tsitsiklis  “Actor-critic algorithms”  In _Advances in neural information processing systems_ 12, 1999
  * [68] Soroush Korivand, Nader Jalili and Jiaqi Gong  “Inertia-Constrained Reinforcement Learning to Enhance Human Motor Control Modeling”  In _Sensors_ 23.5 MDPI, 2023, pp. 2698
  * [69] Apostolos Kotsialos et al.  “Traffic flow modeling of large-scale motorway networks using the macroscopic modeling tool METANET”  In _IEEE Transactions on intelligent transportation systems_ 3.4 IEEE, 2002, pp. 282–292
  * [70] Kimin Lee et al.  “Context-aware dynamics model for generalization in model-based reinforcement learning”  In _International Conference on Machine Learning_ , 2020, pp. 5757–5766  PMLR
  * [71] Sergey Levine, Chelsea Finn, Trevor Darrell and Pieter Abbeel  “End-to-end training of deep visuomotor policies”  In _The Journal of Machine Learning Research_ 17.1 JMLR. org, 2016, pp. 1334–1373
  * [72] Sergey Levine, Aviral Kumar, George Tucker and Justin Fu  “Offline reinforcement learning: Tutorial, review, and perspectives on open problems”  In _arXiv preprint arXiv:2005.01643_ , 2020
  * [73] Tzu-Mao Li, Miika Aittala, Frédo Durand and Jaakko Lehtinen  “Differentiable monte carlo ray tracing through edge sampling”  In _ACM Transactions on Graphics (TOG)_ 37.6 ACM New York, NY, USA, 2018, pp. 1–11
  * [74] Xiao Li and Calin Belta  “Temporal logic guided safe reinforcement learning using control barrier functions”  In _arXiv preprint arXiv:1903.09885_ , 2019
  * [75] Yuanzheng Li et al.  “Federated multiagent deep reinforcement learning approach via physics-informed reward for multimicrogrid energy management”  In _IEEE Transactions on Neural Networks and Learning Systems_ IEEE, 2023
  * [76] Yutong Li et al.  “Safe reinforcement learning using robust action governor”  In _Learning for Dynamics and Control_ , 2021, pp. 1093–1104  PMLR
  * [77] Zongyi Li et al.  “Fourier neural operator for parametric partial differential equations”  In _arXiv preprint arXiv:2010.08895_ , 2020
  * [78] Xin-Yang Liu and Jian-Xun Wang  “Physics-informed Dyna-style model-based deep reinforcement learning for dynamic control”  In _Royal Society A_ 477.2255 The Royal Society, 2021, pp. 20210618
  * [79] Pablo Alvarez Lopez et al.  “Microscopic traffic simulation using sumo”  In _2018 21st international conference on intelligent transportation systems (ITSC)_ , 2018, pp. 2575–2582
  * [80] Ryan Lowe et al.  “Multi-agent actor-critic for mixed cooperative-competitive environments”  In _Advances in neural information processing systems_ 30, 2017
  * [81] Kendall Lowrey et al.  “Reinforcement learning for non-prehensile manipulation: Transfer from simulation to physical system”  In _2018 IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots (SIMPAR)_ , 2018, pp. 35–42
  * [82] Lu Lu et al.  “Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators”  In _Nature Machine Intelligence_ 3.3 Nature Publishing Group, 2021, pp. 218–229
  * [83] Zhengyi Luo et al.  “Kinematics-guided reinforcement learning for object-aware 3d ego-pose estimation”  In _arXiv preprint arXiv:2011.04837_ , 2020
  * [84] Michael Lutter, Johannes Silberbauer, Joe Watson and Jan Peters  “Differentiable physics models for real-world offline model-based reinforcement learning”  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 4163–4170
  * [85] Jun Lv et al.  “SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Rendering”  In _arXiv preprint arXiv:2210.15185_ , 2022
  * [86] Haitong Ma et al.  “Model-based constrained reinforcement learning using generalized control barrier function”  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2021, pp. 4552–4559
  * [87] Gabriel B Margolis et al.  “Learning to jump from pixels”  In _arXiv preprint arXiv:2110.15344_ , 2021
  * [88] John Martin and Hanspeter Schaub  “Reinforcement learning and orbit-discovery enhanced by small-body physics-informed neural network gravity models”  In _AIAA SCITECH 2022 Forum_ , 2022, pp. 2272
  * [89] Chuizheng Meng et al.  “When Physics Meets Machine Learning: A Survey of Physics-Informed Machine Learning”  In _arXiv preprint arXiv:2203.16797_ , 2022
  * [90] Xuhui Meng et al.  “Learning functional priors and posteriors from data and physics”  In _Journal of Computational Physics_ 457 Elsevier, 2022, pp. 111073
  * [91] Volodymyr Mnih et al.  “Playing atari with deep reinforcement learning”  In _arXiv preprint arXiv:1312.5602_ , 2013
  * [92] Miguel Angel Zamora Mora et al.  “Pods: Policy optimization via differentiable simulation”  In _International Conference on Machine Learning_ , 2021, pp. 7805–7817  PMLR
  * [93] Amartya Mukherjee and Jun Liu  “Bridging Physics-Informed Neural Networks with Reinforcement Learning: Hamilton-Jacobi-Bellman Proximal Policy Optimization (HJBPPO)”  In _arXiv preprint arXiv:2302.00237_ , 2023
  * [94] Michael Neunert et al.  “Continuous-discrete reinforcement learning for hybrid control in robotics”  In _Conference on Robot Learning_ , 2020, pp. 735–751  PMLR
  * [95] Motoya Ohnishi, Li Wang, Gennaro Notomista and Magnus Egerstedt  “Barrier-certified adaptive reinforcement learning with applications to brushbot navigation”  In _IEEE Transactions on robotics_ 35.5 IEEE, 2019, pp. 1186–1205
  * [96] Błażej Osiński et al.  “Simulation-based reinforcement learning for real-world autonomous driving”  In _2020 IEEE international conference on robotics and automation (ICRA)_ , 2020, pp. 6411–6418
  * [97] Jacopo Panerati et al.  “Learning to fly—a gym environment with pybullet physics for reinforcement learning of multi-agent quadcopter control”  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2021, pp. 7512–7519
  * [98] Chaejin Park et al.  “Physics-informed reinforcement learning for sample-efficient optimization of freeform nanophotonic devices”  In _arXiv preprint arXiv:2306.04108_ , 2023
  * [99] Xue Bin Peng, Glen Berseth, KangKang Yin and Michiel Van De Panne  “DeepLoco: Dynamic locomotion skills using hierarchical deep reinforcement learning”  In _ACM Transactions on Graphics (TOG)_ 36.4 ACM New York, NY, USA, 2017, pp. 1–13
  * [100] Xue Bin Peng, Pieter Abbeel, Sergey Levine and Michiel Van de Panne  “Deepmimic: Example-guided deep reinforcement learning of physics-based character skills”  In _ACM Transactions On Graphics (TOG)_ 37.4 ACM New York, NY, USA, 2018, pp. 1–14
  * [101] Majdi I Radaideh et al.  “Physics-informed reinforcement learning optimization of nuclear assembly design”  In _Nuclear Engineering and Design_ 372 Elsevier, 2021, pp. 110966
  * [102] Aravind Rajeswaran et al.  “Learning complex dexterous manipulation with deep reinforcement learning and demonstrations”  In _arXiv preprint arXiv:1709.10087_ , 2017
  * [103] Adithya Ramesh and Balaraman Ravindran  “Physics-Informed Model-Based Reinforcement Learning”  In _Learning for Dynamics and Control Conference_ , 2023, pp. 26–37  PMLR
  * [104] Martin Riedmiller  “Neural fitted Q iteration–first experiences with a data efficient neural reinforcement learning method”  In _16th European Conference on Machine Learning_ , 2005, pp. 317–328  Springer
  * [105] Agility robotics  “Cassie-mujoco-sim”, Year Published/ Last Updated  URL: https://github.com/osudrl/cassie-mujoco-sim
  * [106] Colin Rodwell and Phanindra Tallapragada  “Physics-informed reinforcement learning for motion control of a fish-like swimming robot”  In _Scientific Reports_ 13.1 Nature Publishing Group, 2023, pp. 1–17
  * [107] Muhammad Zakiyullah Romdlony and Bayu Jayawardhana  “Stabilization with guaranteed safety using control Lyapunov–barrier function”  In _Automatica_ 66 Elsevier, 2016, pp. 39–47
  * [108] Alvaro Sanchez-Gonzalez et al.  “Graph networks as learnable physics engines for inference and control”  In _International Conference on Machine Learning_ , 2018, pp. 4470–4479  PMLR
  * [109] John Schulman et al.  “Proximal policy optimization algorithms”  In _arXiv preprint arXiv:1707.06347_ , 2017
  * [110] Viraj Shah et al.  “Encoding invariances in deep generative models”  In _arXiv preprint arXiv:1906.01626_ , 2019
  * [111] Buxin She et al.  “Inverter PQ Control with Trajectory Tracking Capability for Microgrids Based on Physics-informed Reinforcement Learning”  In _IEEE Transactions on Smart Grid_ IEEE, 2023
  * [112] Haotian Shi et al.  “Physics-informed deep reinforcement learning-based integrated two-dimensional car-following control strategy for connected automated vehicles”  In _Knowledge-Based Systems_ 269 Elsevier, 2023, pp. 110485
  * [113] Jonah Siekmann, Yesh Godse, Alan Fern and Jonathan Hurst  “Sim-to-real learning of all common bipedal gaits via periodic reward composition”  In _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 7309–7315
  * [114] David Silver et al.  “Deterministic policy gradient algorithms”  In _International conference on machine learning_ , 2014, pp. 387–395  Pmlr
  * [115] Richard S Sutton and Andrew G Barto  “Reinforcement learning: An introduction”  MIT press, 2018
  * [116] Richard S Sutton and Andrew G Barto  “Reinforcement learning: an introduction MIT Press”  In _Cambridge, MA_ 22447, 1998
  * [117] Yuval Tassa et al.  “Deepmind control suite”  In _arXiv preprint arXiv:1801.00690_ , 2018
  * [118] Chen Tessler et al.  “Action assembly: Sparse imitation learning for text based games with combinatorial action spaces”  In _arXiv preprint arXiv:1905.09700_ , 2019
  * [119] Marin Toromanoff, Emilie Wirbel and Fabien Moutarde  “End-to-end model-free reinforcement learning for urban driving using implicit affordances”  In _IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 7153–7162
  * [120] Soumith Udatha, Yiwei Lyu and John Dolan  “Safe Reinforcement Learning with Probabilistic Control Barrier Functions for Ramp Merging”  In _arXiv preprint arXiv:2212.00618_ , 2022
  * [121] Rishi Veerapaneni et al.  “Entity abstraction in visual model-based reinforcement learning”  In _Conference on Robot Learning_ , 2020, pp. 1439–1456  PMLR
  * [122] Evangelos Vrettos et al.  “Experimental demonstration of frequency regulation by commercial buildings—Part I: Modeling and hierarchical control design”  In _IEEE Transactions on Smart Grid_ 9.4 IEEE, 2016, pp. 3213–3223
  * [123] Ruihang Wang et al.  “Phyllis: Physics-Informed Lifelong Reinforcement Learning for Data Center Cooling Control”  In _14th ACM International Conference on Future Energy Systems_ , 2023, pp. 114–126
  * [124] Xiao Wang  “Ensuring safety of learning-based motion planners using control barrier functions”  In _IEEE Robotics and Automation Letters_ 7.2 IEEE, 2022, pp. 4773–4780
  * [125] Keenon Werling et al.  “Fast and feature-complete differentiable physics for articulated rigid bodies with contact”  In _arXiv preprint arXiv:2103.16021_ , 2021
  * [126] Ronald J Williams  “Simple statistical gradient-following algorithms for connectionist reinforcement learning”  In _Machine learning_ 8 Springer, 1992, pp. 229–256
  * [127] Jin-Long Wu et al.  “Enforcing statistical constraints in generative adversarial networks for modeling chaotic dynamical systems”  In _Journal of Computational Physics_ 406 Elsevier, 2020, pp. 109209
  * [128] Chris Xie et al.  “Model-based reinforcement learning with parametrized physical models and optimism-driven exploration”  In _2016 IEEE international conference on robotics and automation (ICRA)_ , 2016, pp. 504–511
  * [129] Zhaoming Xie et al.  “Feedback control for cassie with deep reinforcement learning”  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2018, pp. 1241–1246
  * [130] Jie Xu et al.  “Accelerated policy learning with parallel differentiable simulation”  In _arXiv preprint arXiv:2204.07137_ , 2022
  * [131] Mengjiao Yang and Ofir Nachum  “Representation matters: Offline pretraining for sequential decision making”  In _International Conference on Machine Learning_ , 2021, pp. 11784–11794  PMLR
  * [132] Yibo Yang and Paris Perdikaris  “Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems”  In _Computational Mechanics_ 64.2 Springer, 2019, pp. 417–434
  * [133] Yujie Yang et al.  “Model-Free Safe Reinforcement Learning through Neural Barrier Certificate”  In _IEEE Robotics and Automation Letters_ IEEE, 2023
  * [134] Mingsheng Yin et al.  “Generalizable Wireless Navigation through Physics-Informed Reinforcement Learning in Wireless Digital Twin”  In _arXiv preprint arXiv:2306.06766_ , 2023
  * [135] Mustafa Z Yousif et al.  “Physics-guided deep reinforcement learning for flow field denoising”  In _arXiv preprint arXiv:2302.09559_ , 2023
  * [136] Peipei Yu, Hongcai Zhang and Yonghua Song  “District cooling system control for providing regulation services based on safe reinforcement learning with barrier functions”  In _Applied Energy_ 347 Elsevier, 2023, pp. 121396
  * [137] Zhaocong Yuan et al.  “Safe-control-gym: A unified benchmark suite for safe learning-based control and reinforcement learning”  In _arXiv preprint arXiv:2109.06325_ , 2021
  * [138] Xinglong Zhang et al.  “Barrier Function-based Safe Reinforcement Learning for Formation Control of Mobile Robots”  In _2022 International Conference on Robotics and Automation (ICRA)_ , 2022, pp. 5532–5538
  * [139] Peng Zhao and Yongming Liu  “Physics informed deep reinforcement learning for aircraft conflict resolution”  In _IEEE Transactions on Intelligent Transportation Systems_ 23.7 IEEE, 2021, pp. 8288–8301
  * [140] Qingye Zhao, Yi Zhang and Xuandong Li  “Safe reinforcement learning for dynamical systems using barrier certificates”  In _Connection Science_ 34.1 Taylor & Francis, 2022, pp. 2822–2844
  * [141] Tianqiao Zhao, Jianhui Wang and Meng Yue  “A Barrier-Certificated Reinforcement Learning Approach for Enhancing Power System Transient Stability”  In _IEEE Transactions on Power Systems_ IEEE, 2023
  * [142] Wenxuan Zhou, Lerrel Pinto and Abhinav Gupta  “Environment Probing Interaction Policies”  In _International Conference on Learning Representations_ , 2018
  * [143] Simon Zimmermann, Roi Poranne, James M Bern and Stelian Coros  “Puppetmaster: robotic animation of marionettes”  In _ACM Transactions on Graphics (TOG)_ 38.4 ACM New York, NY, USA, 2019, pp. 1–11