# Safe Reinforcement Learning via Confidence-Based Filters

Sebastian Curi1∗, Armin Lederer2∗, Sandra Hirche2, Andreas Krause1 This work
was supported by the European Research Council (ERC) Consolidator Grant ”Safe
data-driven control for human-centric systems (CO-MAN)” under grant agreement
number 864686 and under the European Unions Horizon 2020 research and
innovation program grant agreement No 815943\. A. L. gratefully acknowledges
financial support from the German Academic Scholarship Foundation. 1Learning &
Adaptive Systems Group, Department of Computer Science, ETH Zurich,
Switzerland. Email<EMAIL_ADDRESS>of Information-
oriented Control, Department of Electrical and Computer Engineering, Technical
University of Munich, Germany. Email: [armin.lederer<EMAIL_ADDRESS>authors contributed equally.

###### Abstract

Ensuring safety is a crucial challenge when deploying reinforcement learning
(RL) to real-world systems. We develop confidence-based safety filters, a
control-theoretic approach for certifying state safety constraints for nominal
policies learnt via standard RL techniques, based on probabilistic dynamics
models. Our approach is based on a reformulation of state constraints in terms
of cost functions, reducing safety verification to a standard RL task. By
exploiting the concept of hallucinating inputs, we extend this formulation to
determine a “backup” policy which is safe for the unknown system with high
probability. Finally, the nominal policy is minimally adjusted at every time
step during a roll-out towards the backup policy, such that safe recovery can
be guaranteed afterwards. We provide formal safety guarantees, and empirically
demonstrate the effectiveness of our approach.

##  I INTRODUCTION

When agents operate autonomously in unknown environments, they need the
ability to adapt to new situations. This adaptiveness can be achieved using
reinforcement learning [1], which allows autonomous agents to modify their
behavior according to observations of the environment. Such reinforcement
learning approaches have been demonstrated to achieve state-of-the-art
performance on various problems where high-fidelity simulations are available
[2], but they cannot be directly applied to real-world autonomous systems
because their safe operation must always be guaranteed. This has lead to
safety being a major hurdle for the application of reinforcement learning in
real-world applications [3].

#### Related work

Due to this high relevance of safety in reinforcement learning, it has been
the focus of a variety of recent approaches (see [4, 5] for surveys). A common
framework for safe reinforcement learning are constrained Markov decision
processes (CMDP) [6]. In the CMDP setting, constraints are posed on the
expected cumulative cost along roll-outs of a policy. This allows to treat the
cumulative cost analogously to rewards, such that methods such as trust region
policy optimization can be adapted to maintain constraint satisfaction when
initialized with a safe policy [7]. When no initially safe policy is known, a
Lagrangian relaxation can be used to asymptotically find safe policies [8].
This dual representation of the constrained optimization problem can also be
combined with techniques such as upper confidence reinforcement learning to
guarantee learning rates, e.g., for linear CMDPs [9]. However, constraint
violation during training cannot be excluded in general, which often prevents
the usage in real-world applications.

Control theoretic methods consider safety through constraints on the system
states, but often severely restrict the allowed policy and system classes. For
example, linear quadratic regulators can be learned efficiently under
polytopic constraints on the system states as shown by [10]. The limitation to
linear dynamics can be relaxed in the case of deterministic systems by
employing model predictive control (MPC) techniques, such that the performance
can be iteratively improved [11]. To allow active exploration, MPC can be
combined with reinforcement learning ideas by ensuring safety using a suitable
MPC parameterization [12]. Moreover, Monte-Carlo approximations can be used to
deal with the stochastic dynamics commonly found in reinforcement learning
problems [13], although this comes at the price of a prohibitive computational
complexity.

To achieve the beneficial properties of both control theoretical and
reinforcement learning based approaches, it has recently been proposed to
employ reinforcement learning for finding the optimal policy, while in a
secondary step a control method is used to certify safety and, if necessary,
adapt the applied action [14]. This approach can be realized using control
barrier functions [15], or a ”backup” policy, which is locally safe in some
region of the state space [16]. While designing control barrier functions is
challenging in general, determining locally safe policies often requires
solving computationally expensive optimization problems on-line [16] or can
only be applied to linearized systems [17]. Therefore, the practical
applicability of such safety filters in combination with highly flexible RL
techniques is currently limited.

#### Our contributions

We mitigate these weaknesses by proposing hallucinating upper confidence
safety filters for ensuring safety of arbitrary policies applied to
stochastic, nonlinear systems for which merely a model with high probability
error bounds is known. To this end, we first establish a relationship between
state constraints and level sets of value functions. Using the concept of
hallucinating inputs [18], we show that these value functions can be
efficiently estimated with standard reinforcement learning methods. Our
approach can be naturally extended to finding safe policies, by formulating it
as robust reinforcement learning problem. These safe policies can then be used
for computationally efficient on-line safety adaptation of arbitrary
reinforcement learning policies. We demonstrate the effectiveness of the
proposed method on deep RL benchmark tasks.

The remainder of this paper is structured as follows. In Section II, we
formalize the problem setting. The hallucinating upper confidence filter is
explained and shown to yield safe policies in Section IV. Finally, the
performance of the safety filter is demonstrated in simulations in Section V,
before the paper is concluded in Section VI.

##  II PROBLEM STATEMENT AND BACKGROUND

We consider a discrete-time dynamical system

$\displaystyle\bm{x}_{k+1}=\bm{f}^{\star}(\bm{x}_{k},\bm{u}_{k})+\bm{\omega}_{k},$
(1)

where $\bm{x}_{k}\in\mathcal{X}\subset\mathbb{R}^{d_{x}}$ are states,
$\bm{u}_{k}\in\mathcal{U}\subset\mathbb{R}^{d_{u}}$ control actions,
$\bm{\omega}_{k}\\!\sim\\!\rho$ is process noise sampled from a zero-mean
probability distribution $\rho$111We consider constant noise distributions
$\rho$ for notational simplicity, but our approach directly extends to state
and action dependent distributions., and
$\bm{f}^{\star}\\!:\mathcal{X}\times\mathcal{U}\rightarrow\mathcal{X}$ denotes
the unknown deterministic transition function. The control actions
$\bm{u}_{k}$ are determined using a policy
$\bm{\pi}\\!:\mathcal{X}\rightarrow\mathcal{U}$, with the goal to maximize an
expected cumulative return

$\displaystyle\\!\\!\\!R(\bm{f}^{\star},\bm{\pi};\bm{x})\\!=$
$\displaystyle\mathbf{E}_{\bm{\omega}}\\!\\!\left[\sum\limits_{k=0}^{\infty}\gamma^{k}r(\bm{x}_{k},\bm{\pi}(\bm{x}_{k}))\right],$
(2a) $\displaystyle\text{s.t.
}\bm{x}_{k+1}\\!=\\!\bm{f}^{\star}(\bm{x}_{k},\bm{\pi}(\bm{x}_{k}))\\!+\\!\bm{\omega}_{k},~{}\bm{\omega}_{k}\\!\sim\\!\rho,\\!\\!\\!$
(2b) $\displaystyle\quad~{}~{}\bm{x}_{0}=\bm{x},$ (2c)

where $r:\mathcal{X}\times\mathcal{U}\rightarrow\mathbb{R}$ is a known
immediate reward function and $\gamma\in(0,1)$ is a discount factor.

In practice, the policy $\bm{\pi}$ must additionally ensure safety of the
closed-loop dynamical system, e.g., because damage to the system described by
$\bm{f}^{\star}$ must be avoided. In the RL literature, this is typically
addressed through constrained Markov decision processes, which additionally
consider a constraint on a cumulative cost function

$\displaystyle\\!C(\bm{f}^{\star},\bm{\pi};\bm{x})\\!=\\!\mathbf{E}_{\bm{\omega}}\\!\\!\left[\sum_{k=0}^{\infty}\gamma^{k}c(\bm{x}_{k})\\!\right]\\!<\\!\xi,$
(3)

where $c:\mathcal{X}\rightarrow\mathbb{R}$ is an immediate cost,
$\xi\in\mathbb{R}$ is a constant specifying the constraint, and $\bm{x}_{k}$
is defined iteratively through (1) with actions
$\bm{u}_{k}=\bm{\pi}(\bm{x}_{k})$ and initial state $\bm{x}_{0}=\bm{x}$.
Therefore, an optimization problem of the form

$\displaystyle\bm{\pi}^{*}=$
$\displaystyle\operatorname*{arg\,max}\limits_{\bm{\pi}}R(\bm{f}^{\star},\bm{\pi};\bm{x})$
(4a) $\displaystyle~{}\text{s.t. }C(\bm{f}^{\star},\bm{\pi};\bm{x})<\xi$ (4b)

is usually solved to determine safe policies.

While this problem can be directly solved by adapting standard RL algorithms
with techniques akin to Lagrangian relaxation [8], this approach generally
cannot ensure safety during training. Moreover, it does not reflect the fact
that the safety of many systems is defined in terms of safe and unsafe states
classified into a set of safe states
$\mathcal{X}_{\mathrm{safe}}\subset\mathcal{X}$ and its complement
$\mathcal{X}_{\mathrm{unsafe}}=\mathcal{X}\setminus\mathcal{X}_{\mathrm{safe}}$.
For example, an autonomously driving car should not leave the road, which
directly defines the road as $\mathcal{X}_{\mathrm{safe}}$. When using the
natural indicator $\bm{1}_{\bm{x}\in\mathcal{X}_{\mathrm{unsafe}}}$ as cost
function, satisfying Equation 4b bounds the discounted probability of
violating the constraints by $\xi$. Nonetheless, this does not guarantee that
constraints will not be violated when deploying $\bm{\pi}^{*}$.

Therefore, we consider safety in terms of state constraints
$\bm{x}_{k}\in\mathcal{X}_{\mathrm{safe}}$, which we require to hold with high
probability, since the process noise $\bm{\omega}$ generally prevents
deterministic guarantees. This leads to the following definition of safety.

###### Definition II.1

A policy $\bm{\pi}$ is $K$-step $\delta$-safe for a state
$\bm{x}\in\mathcal{X}$ if it holds that
$\mathbb{P}\left(\bm{x}_{k}\in\mathcal{X}_{\mathrm{safe}}~{}\forall
k=0,\ldots,K|\bm{x}_{0}=\bm{x}\right)\\!\geq\\!1-\delta$, where states
$\bm{x}_{k}$ are defined in (1).

The concept of $K$-step $\delta$-safety is commonly found in stochastic model
predictive control, where it is typically referred to as joint chance
constraint [19].

###### Remark II.1

We consider finite values of $K$ because ensuring $\delta$-safety over an
infinite horizon, i.e., $K=\infty$, is not possible for unbounded process
noise $\bm{\omega}$ in general. This can be easily seen for a system with
$\bm{f}^{\star}=\bm{0}$ and i.i.d. zero mean Gaussian noise $\bm{\omega}$,
which almost surely leaves any compact safe set $\mathcal{X}_{\mathrm{safe}}$
eventually.

In order to obtain the optimal policy $\bm{\pi}_{\mathrm{safe}}^{*}$ ensuring
$\delta$-safety, we generally need to consider the optimization problem

$\displaystyle\\!\\!\bm{\pi}_{\mathrm{safe}}^{*}=$
$\displaystyle\operatorname*{arg\,max}\limits_{\bm{\pi}}~{}R(\bm{f}^{\star},\bm{\pi};\bm{x})$
(5a) $\displaystyle~{}\text{s.t.
}\mathbb{P}\left(\\!\bm{x}_{k}\\!\\!\in\\!\mathcal{X}_{\mathrm{safe}}~{}\forall
k\\!=\\!0,\ldots,K|\bm{x}_{0}\\!=\\!\bm{x}\\!\right)\\!\geq\\!1\\!-\\!\delta.\\!$
(5b)

Solving this optimization problem is challenging since there usually exists no
closed-form expression for the probability (5b), such that computationally
expensive uncertainty propagation methods have to be employed, e.g.,
generalized polynomial chaos expansions [20].

In order to efficiently determine approximate solutions for (5), we follow the
idea of [17] and separate it into two phases: an initial phase for determining
a nominal policy $\bm{\pi}^{*}$ using an arbitrary method, followed by an on-
line phase in which a safety filter is employed to adapt the policy
$\bm{\pi}^{*}$ to ensure $K$-step $\delta$-safety. Since we cannot ensure
safety without any knowledge about $\bm{f}^{\star}$, we assume to have access
to a set of plausible models
$\mathcal{M}=\left\\{\bm{f}\mid|\bm{f}-\bm{\mu}|\leq\beta\bm{\sigma}\right\\}$
described by a nominal model
$\bm{\mu}\\!:\mathcal{X}\\!\times\mathcal{U}\\!\rightarrow\\!\mathbb{R}^{d_{x}}$,
the state-action dependent uncertainty about the model
$\bm{\sigma}\\!:\mathcal{X}\\!\times\\!\mathcal{U}\\!\rightarrow\\!\mathbb{R}^{d_{x}}$,
and a constant scaling factor $\beta\\!\in\\!\mathbb{R}_{+}$. We assume that
this set of models is well-calibrated, i.e.,
$\bm{f}^{\star}\\!\in\\!\mathcal{M}$ with high probability, as formalized in
the following.

######  Assumption II.1 ([18])

The statistical model is calibrated with respect to $\bm{f}^{\star}$, i.e.,
there exists a $\beta\in\mathbb{R}_{+}$ such that, with probability at least
$1-\delta_{f}$, it holds jointly for all
$\bm{x},\bm{u}\in\mathcal{X}\times\mathcal{U}$ that
$|\bm{f}^{\star}(\bm{x},\bm{u})-\bm{\mu}(\bm{x},\bm{u})|\leq\beta\bm{\sigma}(\bm{x},\bm{u})$,
element-wise.

Since the statistical model is often obtained by applying supervised machine
learning to data obtained from policy roll-outs [18], the uncertainty usually
decreases with the number of roll-outs. Thereby, this assumption typically
enables less conservative and higher performant policies over time.

Using II.1, we investigate the following sub-problems for the derivation of
the safety filter.

#### State Constraints as Cumulative Cost

In order to enable the application of reinforcement learning methods, we
consider the problem of converting the $K$-step $\delta$-safety constraint
(5b) for known dynamics $\bm{f}^{\star}$ into a constraint on an expected
cumulative cost function. We show that this can be achieved by deriving a
condition of the form

$\displaystyle\\!\mathbf{E}_{\bm{\omega}}\\!\\!\left[C(\bm{f}^{\star},\bm{\pi};\bm{f}^{\star}(\bm{x},\bm{\pi})+\bm{\omega})\right]\\!<\\!\xi.$
(6)

for suitably chosen immediate costs $c$, cf. Section III.

#### Safety Filter

Using this condition, we derive a novel approach for computing safe policies
$\bm{\pi}_{\mathrm{safe}}$ for systems with unknown dynamics $\bm{f}^{\star}$.
This allows us to address the problem of ensuring the safety of a possibly
unsafe nominal policy $\bm{\pi}^{*}$ on-line using a confidence-based filter

$\displaystyle\hat{\bm{\pi}}(\bm{x})=$
$\displaystyle\operatorname*{arg\,min}\limits_{\bm{u}\in\mathcal{U}}\|\bm{\pi}^{*}(\bm{x})-\bm{u}\|,\qquad$
(7a) $\displaystyle~{}\text{s.t.
}\max_{\bm{f}\in\mathcal{M}}\mathbf{E}_{\bm{\omega}}\\!\\!\left[C(\bm{f},\bm{\pi}_{\mathrm{safe}};\bm{x}^{\prime})\right]<\xi,$
(7b)
$\displaystyle\qquad\bm{x}^{\prime}=\bm{f}(\bm{x},\bm{u})+\bm{\omega},~{}\bm{\omega}\sim\rho,$
(7c)

which outputs the closest safe action to $\bm{\pi}^{*}$. We derive tractable
formulations for this filter in Section IV.

##  III EXPRESSING STATE CONSTRAINTS THROUGH COST FUNCTIONS

To reformulate the $\delta$-safety constraint into a constraint on cumulative
costs, we first show in Section III-A that sub-level sets of $C$ contained in
$\mathcal{X}_{\mathrm{safe}}$ can be easily defined. Based on this result, we
derive sufficient conditions on the cost function, which allow to conclude
safety from cumulative cost constraints in Section III-B, providing useful
design freedom.

###  III-A Safe Sub-Level Sets of the Cumulative Cost

For deriving the sub-level set
$\mathcal{C}_{\bar{\xi}}^{\bm{\pi}}\\!=\\!\\{\bm{x}\\!\in\\!\mathcal{X}\\!:C(\bm{f}^{\star},\bm{\pi};\bm{x})\\!<\\!\bar{\xi}\\}$,
$\bar{\xi}\\!\in\\!\mathbb{R}$, which is contained in the set of safe states
$\mathcal{X}_{\mathrm{safe}}$, we consider an immediate cost function
$c:\mathcal{X}\rightarrow\mathbb{R}$ satisfying

$\displaystyle\underline{c}\leq
c(\bm{x})\leq\bar{c}\quad\forall\bm{x}\in\mathcal{X},\quad$ $\displaystyle
c(\bm{x})\geq\hat{c}\quad\text{if }\bm{x}\in\mathcal{X}_{\mathrm{unsafe}}$ (8)

for constants $\underline{c},\bar{c},\hat{c}\\!\in\\!\mathbb{R}$. For example,
using the indicator function $\bm{1}_{\bm{x}\in\mathcal{X}_{\mathrm{unsafe}}}$
as cost, which equals $1$ for $\bm{x}\in\mathcal{X}_{\mathrm{unsafe}}$ and $0$
otherwise, implies $\underline{c}\\!=\\!0$ and
$\bar{c}\\!=\\!\hat{c}\\!=\\!1$. Using this definition, we can define an
inner-approximation of the safe set of states $\mathcal{X}_{\mathrm{safe}}$
through the $\hat{c}$ sub-level set of the immediate cost $c$, which becomes
exact if $c(\bm{x})\\!<\\!\hat{c}$ for all
$\bm{x}\\!\in\\!\mathcal{X}_{\mathrm{safe}}$. Moreover, we can define the
expected cumulative cost using (3).

$\mathcal{X}_{\mathrm{unsafe}}$$c(\bm{x})=\hat{c}$$c(\bm{x})=\hat{c}-1$$C(\bm{f}^{\star},\bm{\pi},\bm{x})=\hat{c}$$C(\bm{f}^{\star},\bm{\pi},\bm{x})=\bar{\xi}$$\mathcal{C}_{\bar{\xi}}^{\bm{\pi}}$

Figure 1: The expected cumulative cost can be $0$ even if the immediate cost
$c$ at the first state is greater than $0$, as this positive cost can be
compensated by negative costs afterwards (red trajectory). Therefore,
$\mathcal{C}_{\hat{c}}^{\bm{\pi}}\not\subset\mathcal{X}_{\mathrm{safe}}$, such
that we have to consider the tightened threshold $\bar{\xi}$, which ensures
that states $\bm{x}\in\mathcal{C}_{\bar{\xi}}^{\bm{\pi}}$ start with immediate
cost $c(\bm{x})\leq\hat{c}$ (blue trajectory).

While one might think that the definition of the immediate cost $c$ in (8)
ensures that the $\hat{c}$ sub-level set $\mathcal{C}_{\hat{c}}^{\bm{\pi}}$ of
$C$ is also contained in the safe set of states $\mathcal{X}_{\mathrm{safe}}$,
this is not true in general. As illustrated by the red trajectory in Fig. 1,
the cumulative cost $C$ can equal $\hat{c}$ even if the immediate cost $c$ in
the initial state is greater than $\hat{c}$, since negative costs of following
states along the trajectory can compensate it. Therefore, the sub-level set
$\mathcal{C}_{\hat{c}}^{\bm{\pi}}$ is generally not completely contained in
the set of safe states $\mathcal{X}_{\mathrm{safe}}$, such that we must
consider a tightened threshold $\bar{\xi}$. Due to the lower bound
$\underline{c}$ of the cost $c$, this constant $\bar{\xi}$ can be determined
using the following lemma.

###### Lemma III.1

Consider an immediate cost function $c\\!:\mathcal{X}\rightarrow\mathbb{R}$
satisfying (8). Then, it holds that
$\mathcal{C}_{\bar{\xi}}^{\bm{\pi}}\subset\mathcal{X}_{\mathrm{safe}}$, where

$\displaystyle\bar{\xi}=\gamma\min\limits_{\bm{x}\in\mathcal{X}}C(\bm{f}^{\star},\bm{\pi};\bm{x})+\hat{c}.$
(9)

###### Proof:

Due to the lower bound for $c$, $C$ is lower bounded, such that we obtain
$C(\bm{f}^{\star},\bm{\pi},\bm{x})\geq c(\bm{x})+\gamma C_{\min}$ from Bellman
equation, where
$C_{\min}=\min_{\bm{x}\in\mathcal{X}}C(\bm{f}^{\star},\bm{\pi};\bm{x})\geq\frac{\underline{c}}{1-\gamma}$.
Moreover, due to condition (8) we have $c(\bm{x})>\hat{c}$ for
$\mathcal{X}_{\mathrm{unsafe}}$, which yields
$C(\bm{f}^{\star},\bm{\pi},\bm{x})>\hat{c}+\gamma C_{\min}$ for all
$\bm{x}\in\mathcal{X}_{\mathrm{unsafe}}$. Therefore, the level set
$\mathcal{C}_{\bar{\xi}}^{\bm{\pi}}$ is completely contained in
$\mathcal{X}_{\mathrm{safe}}$, i.e.,
$\mathcal{C}_{\bar{\xi}}^{\bm{\pi}}\subset\mathcal{X}_{\mathrm{safe}}$, which
concludes the proof. ∎

This lemma relies on the idea that the cumulative cost can be lower bounded by
$\min_{\bm{x}\in\mathcal{X}}C(\bm{f}^{\star},\bm{\pi};\bm{x})$, such that any
state with immediate cost $c$ greater than $\hat{c}$ also must have an
expected cumulative cost greater than $\bar{\xi}$. For the example of the
indicator cost, $\bar{\xi}$ can be straightforwardly computed as
$\bar{\xi}\\!=\\!1$ since $C$ is trivially lower bounded by $0$. It is
straightforward to see that this choice of cost function generally allows to
accurately approximate $\mathcal{X}_{\mathrm{safe}}$ using
$\mathcal{C}_{\bar{\xi}}^{\bm{\pi}}$, and indeed
$\mathcal{X}_{\mathrm{safe}}\\!=\\!\mathcal{C}_{\bar{\xi}}^{\bm{\pi}}$ is
possible for deterministic dynamics with $\bm{\omega}\\!=\\!\bm{0}$. However,
Lemma III.1 is not limited to indicator type cost functions, but applies to
arbitrary costs $c$ satisfying (8). This is particularly beneficial for
computing optimal policies using $C$, where informative gradients may aid the
convergence of common RL techniques. Thus, Lemma III.1 allows a flexible
approximation of the safe set $\mathcal{X}_{\mathrm{safe}}$ suitable for the
optimization-based approaches employed in the following sections.

###  III-B Cumulative Cost Safety Conditions

In order to express $K$-step $\delta$-safety through expected cumulative costs
$C$, it remains to derive conditions which ensure that the system state
$\bm{x}_{k}$ stays inside the sub-level set
$\mathcal{C}_{\bar{\xi}}^{\bm{\pi}}$ for all $k=1,\ldots,K$ with probability
$\delta$. For this purpose, we employ techniques from stochastic stability
analysis [21]. In particular, we define the cost-value function of a policy at
a given state $\bm{x}$ as $V_{\bm{\pi}}(\bm{x})\equiv
C(\bm{f}^{\star}\\!,\bm{\pi};\bm{x})$.

###### Theorem III.1

Consider an immediate cost function $c:\mathcal{X}\rightarrow\mathbb{R}$,
which satisfies (8). Assume there exists a class $\mathcal{K}$ function222A
function $\alpha:\mathbb{R}_{0,+}\rightarrow\mathbb{R}_{0,+}$ is a class
$\mathcal{K}$ function, if it is monotonically increasing and $\alpha(0)=0$.
$\alpha:\mathbb{R}\rightarrow\mathbb{R}_{0,+}$, such that

$\displaystyle\mathbf{E}_{\bm{\omega}}\\!\\!\left[V_{\bm{\pi}}(\bm{x}^{\prime})\right]\leq
V_{\bm{\pi}}(\bm{x})\\!-\\!\alpha(V_{\bm{\pi}}(\bm{x})\\!-\\!C_{\min})$ (10)

holds for all $\bm{x}\in\mathcal{X}_{\mathrm{safe}}$ with
$\bm{x}^{\prime}=\bm{f}^{\star}(\bm{x},\bm{\pi}(\bm{x}))+\bm{\omega})$. Then,

$\displaystyle\mathbf{E}_{\bm{\omega}}\\!\\!\left[V_{\bm{\pi}}(\bm{x}^{\prime})\right]\leq\xi<\bar{\xi}$
(11)

guarantees that the policy $\bm{\pi}$ is $K$-step $\delta(\xi)$-safe.

###### Proof:

The result directly follows from Lemma III.1 and Proposition A.1, which ensure
$\delta$-safety with $\delta\\!=\\!\delta_{\mathrm{FL}}(\xi)$. ∎

Condition (10) effectively resembles a Foster-Lyapunov drift condition, which
is satisfied if stochastic stability can be shown with $V_{\bm{\pi}}$ as a
Lyapunov function [22]. Since stability is a well-studied problem, it has been
shown that this condition can be satisfied for many dynamics $\bm{f}^{\star}$,
e.g., systems which are asymptotically controllable with respect to the
immediate cost $c$ [23]. In contrast to stability theory, Theorem III.1 does
require $C$ to be positive definite or the existence of a class $\mathcal{K}$
function lower bounding $C$. Therefore, the conditions of Theorem III.1 are
slightly weaker than for stability.

Due to the close relationship to stability, it is straightforward to see that
the increase rate of $\alpha$ determines the convergence rate of the system.
If $\alpha$ is only slowly growing, a relatively small noise realization can
cause an increase in the expected cumulative cost, and thereby, increases the
probability $\delta$ of leaving the safe set. This can be compensated by
choosing a smaller value of $\xi$, such that there essentially is a larger
margin between the safe initial states $\bm{x}_{0}$ and the unsafe set
$\mathcal{X}_{\mathrm{unsafe}}$. Note that the noise distribution also affects
the probability $\delta$ through (10), since flat distributions with heavy
tails generally cause higher values of
$\mathbf{E}_{\bm{\omega}}\\!\\!\left[V_{\bm{\pi}}(\bm{x})\right]$ leading to
smaller increase rates of $\alpha$.

##  IV HALLUCINATING UPPER CONFIDENCE SAFETY FILTERS

We now derive a tractable safety filter for unknown dynamics, for which merely
a set of plausible models is available. To this end, we first show how the
safety analysis of Section III can be extended to unknown dynamics by
reformulating it as reinforcement learning problem in Section IV-A. Based on
this reformulation, we present a natural approach to obtain backup policies
for the safety filter by computing a safe policy through robust reinforcement
learning in Section IV-B. Finally, the pre-computed backup policy is employed
in a confidence-based safety filter for ensuring constraint satisfaction in
Section IV-C.

###  IV-A Safety Certification with Unknown Dynamics

Since we assume only the availability of a set of plausible models
$\mathcal{M}$, but not the true dynamics $\bm{f}^{\star}$, we cannot determine
$C$ and consequently cannot directly exploit Theorem III.1 for determining a
safe policy. To overcome this issue, we must be _pessimistic_ about the
dynamics. In particular, we define the _pessimistic cost-value_ as

$\displaystyle
V_{\bm{\pi}}^{(p)}(\bm{x})\equiv\max_{\bm{f}\in\mathcal{M}}C(\bm{f},\bm{\pi};\bm{x}).$
(12)

It is straightforward to see that
$V_{\bm{\pi}}^{(p)}(\bm{x})\leq\xi\Rightarrow V_{\bm{\pi}}(\bm{x})\leq\xi$ due
to II.1. Hence, the technique for unknown models reduces to solving (12) and
verifying if its value satisfies the conditions of Theorem III.1. To solve the
pessimistic verification problem (12) we reparameterize the functions
$\bm{f}\in\mathcal{M}$ following [18] as

$\displaystyle\bm{f}(\bm{x},\bm{u})=\bm{\mu}(\bm{x},\bm{u})+\beta\bm{\Sigma}(\bm{x},\bm{u})\bm{\eta}(\bm{x},\bm{u}),$
(13)

where
$\bm{\eta}:\mathcal{X}\times\mathcal{U}\\!\rightarrow\\![-1,1]^{d_{x}}\\!$,
$\bm{\Sigma}(\bm{x},\bm{u})\\!=\\!\mathrm{diag}(\bm{\sigma}(\bm{x},\bm{u}))$
and $\bm{\mu}$ is the model mean function. We refer to $\bm{\eta}$ as the
hallucinating policy in the sequel since it acts on the outputs of the
dynamics similarly as a policy acts on its inputs. Moreover, the set of
admissible functions $\bm{\eta}$ is defined via

$\displaystyle\mathcal{N}$
$\displaystyle=\\{\bm{\eta}:~{}-1\leq\bm{\eta}(\bm{x},\bm{u})\leq
1~{}~{}\forall\bm{x},\bm{u}\in\mathcal{X}\times\mathcal{U}\\}.$ (14)

To determine that a policy is safe, we must verify that the value of the
pessimistic estimate $V_{\bm{\pi}}^{(p)}$ defined as

$\displaystyle\\!\\!\\!V_{\bm{\pi}}^{(p)}(\bm{x})=$
$\displaystyle\max_{\bm{\eta}\in\mathcal{N}}\mathbb{E}_{\bm{\omega}}\Bigg{[}\sum_{k=0}^{\infty}\gamma^{k}c(\bm{x}_{k})\Bigg{]}$
(15a) $\displaystyle\text{s.t.
}\bm{x}_{k+1}=\bm{f}(\bm{x}_{k},\bm{\pi}(\bm{x}_{k}))+\bm{\omega}_{k},~{}\bm{\omega}_{k}\sim\rho,$
(15b)
$\displaystyle\quad~{}~{}\bm{f}(\bm{x},\bm{u})\\!=\\!\bm{\mu}(\bm{x},\bm{u})\\!+\\!\beta\bm{\Sigma}(\bm{x},\bm{u})\bm{\eta}(\bm{x},\bm{u}),\\!\\!$
(15c) $\displaystyle\quad~{}~{}\bm{x}_{0}\\!=\\!\bm{x}$ (15d)

satisfies the conditions of Theorem III.1. Computing $V_{\bm{\pi}}^{(p)}$ is
equivalent to an RL problem that can be solved using standard RL algorithms,
where the policy is $\bm{\eta}$ and the dynamics is given by (13), i.e., it
can be done purely in simulation. Using this formulation as an optimization of
the hallucinating policy $\bm{\eta}$, it is straightforward to extend Theorem
III.1 to unknown dynamics $\bm{f}^{\star}$ as shown in the following
proposition.

###### Proposition IV.1

Consider a set of plausible models $\mathcal{M}$ satisfying II.1 and an
immediate cost $c$, which satisfies (8). If $V_{\bm{\pi}}^{(p)}(\bm{x})$
satisfies

$\displaystyle\max_{\bm{\eta}\in\mathcal{N}}\mathbf{E}_{\bm{\omega}}\\!\\!\left[V_{\bm{\pi}}^{(p)}(\bm{x}^{\prime})\right]\leq
V_{\bm{\pi}}^{(p)}(\bm{x})\\!-\\!\alpha(V_{\bm{\pi}}^{(p)}(\bm{x})\\!-\\!C_{\min})$
(16)

for all $\bm{x}\\!\in\\!\mathcal{X}_{\mathrm{safe}}$, where $\bm{x}^{\prime}$
is the next state defined through the reparameterized dynamics (15b), (15c),
for all $\bm{x}\\!\in\\!\mathcal{X}_{\mathrm{safe}}$, then,
$\max_{\bm{\eta}\in\mathcal{N}}\\!\mathbf{E}_{\bm{\omega}}[V_{\bm{\pi}}^{(p)}(\bm{x}^{\prime})]\\!\leq\\!\xi\\!<\\!\bar{\xi}$
guarantees $K$-step $\delta$-safety of $\bm{\pi}$.

###### Proof:

Due to Assumption II.1, $V_{\bm{\pi}}^{(p)}(\bm{x})\\!\leq\\!\bar{\xi}$
implies $V_{\bm{\pi}}(\bm{x})\\!\leq\\!\bar{\xi}$ with probability at least
$1-\delta_{f}$. Therefore,
$\max_{\bm{\eta}\in\mathcal{N}}\hat{V}(\bm{x}^{\prime})\leq\bar{\xi}$ implies
that $\bm{x}\in\mathcal{X}_{\mathrm{safe}}$ due to Lemma III.1. Moreover, it
can be directly shown that

$\displaystyle\max_{\bm{\eta}\in\mathcal{N}}\mathbf{E}_{\bm{\omega}}\\!\\!\left[V_{\bm{\pi}}^{(p)}(\bm{f}^{\star}(\bm{x},\bm{\pi}(\bm{x}))+\bm{\omega})\right]\leq\max_{\bm{\eta}\in\mathcal{N}}\mathbf{E}_{\bm{\omega}}\\!\\!\left[V_{\bm{\pi}}^{(p)}(\bm{x}^{\prime})\right]$

for $\bm{x}^{\prime}$ the next state defined through the reparameterized
dynamics (15b), (15c) with probability at least $(1-\delta_{f})$ due to
Assumption II.1. Due to (16), this yields

$\displaystyle\max_{\bm{\eta}\in\mathcal{N}}$
$\displaystyle\mathbf{E}_{\bm{\omega}}\\!\\!\left[V_{\bm{\pi}}^{(p)}(\bm{f}^{\star}(\bm{x},\bm{\pi}(\bm{x}))+\bm{\omega})\right]\leq$
$\displaystyle\qquad\qquad\qquad\qquad\quad
V_{\bm{\pi}}^{(p)}(\bm{x})-\alpha(V_{\bm{\pi}}^{(p)}(\bm{x})-C_{\min}),$

such that we can apply Proposition A.1. Hence safety follows with
$\delta=\delta_{\mathrm{FL}}(\xi)+\delta_{f}-\delta_{\mathrm{FL}}(\xi)\delta_{f}$.
∎

When the model is known accurately, i.e., $\bm{\sigma}(\bm{x},\bm{u})=\bm{0}$
for all $\bm{x},\bm{u}\in\mathcal{X}\times\mathcal{U}$, the conditions of
Proposition IV.1 intuitively reduce to the conditions of Theorem III.1.

###  IV-B Robust Reinforcement Learning of Safe Policies

Based on the formulation of the $K$-step $\delta$-safety as an optimization
problem in Proposition IV.1, it is natural to augment the optimization problem
to directly find $\delta$-safe policies by finding the policy that minimizes
the pessimistic cost estimate $V_{\bm{\pi}}^{(p)}$. Namely, we propose to
obtain safe policies via

$\displaystyle\bm{\pi}_{\mathrm{safe}}\coloneqq\operatorname*{arg\,min}\limits_{\bm{\pi}\in\Pi}V_{\bm{\pi}}^{(p)}=\operatorname*{arg\,min}\limits_{\bm{\pi}\in\Pi}\max_{\bm{\eta}\in\mathcal{N}}\mathbf{E}_{\bm{x}}\\!\\!\left[C(\bm{f},\bm{\pi};\bm{x})\right]$
(17)

where the dynamics $\bm{f}$ is the reparameterized dynamics in (13). As (17)
can be seen as a robust RL problem, we refer to $\bm{\pi}_{\mathrm{safe}}$ as
the _learned safe policy_ in the following. To solve (17) we use standard
robust RL techniques such as those in [24, 25], which perform gradient descent
for $\bm{\pi}$ and gradient ascent for $\bm{\eta}$. Moreover, if the cost $c$
and the discount $\gamma$ allow to establish the safety of this system for
some policy $\bm{\pi}$, it is straightforward to show that (17) yields a
$\delta$-safe policy.

###### Proposition IV.2

Assume the learned safe policy $\bm{\pi}_{\mathrm{safe}}$ is obtained using
(17) with sufficiently expressive function classes $\Pi$, $\mathcal{N}$, and
has a unique solution. If there exists a policy $\bm{\pi}$ and a class
$\mathcal{K}$ function $\alpha$ such that $V_{\bm{\pi}}^{(p)}$ satisfies (16),
$\bm{\pi}_{\mathrm{safe}}$ is $K$-step $\delta$-safe for all
$\bm{x}\\!\in\\!\mathcal{X}$ with
$V_{\bm{\pi}_{\mathrm{safe}}}^{(p)}(\bm{x})\\!\leq\\!\xi\\!<\\!\bar{\xi}$.

###### Proof:

Given the policy $\bm{\pi}_{\bm{x}}$ defined as

$\displaystyle\bm{\pi}_{\bm{x}}$
$\displaystyle=\operatorname*{arg\,min}\limits_{\bm{\pi}\in\Pi}\max_{\bm{\eta}\in\mathcal{N}}C(\bm{f},\bm{\pi};\bm{x}),$

we can lower bound the expected cost for arbitrary $\bm{\pi}\in\Pi$ by
$\mathbf{E}_{\bm{x}}[V_{\bm{\pi}}^{(p)}]\geq\mathbf{E}_{\bm{x}}[V_{\bm{\pi}_{x}}^{(p)}]$.
Therefore, for a sufficiently expressive policy class $\Pi$, it must follow
that
$\bm{\pi}_{x}=\operatorname*{arg\,min}_{\bm{\pi}\in\Pi}\mathbf{E}_{\bm{x}}[V_{\bm{\pi}}^{(p)}]$
holds. Moreover, we have
$\mathbf{E}_{\bm{x}}[V_{\bm{\pi}_{x}}^{(p)}]\geq\mathbf{E}_{\bm{x}}[C(\bm{f},\bm{\pi}_{x};\bm{x})]$
for arbitrary $\bm{\eta}\in\mathcal{N}$. Therefore, for a sufficiently
expressive function class $\mathcal{N}$, it must hold that
$\max_{\bm{\eta}\in\mathcal{N}}\mathbf{E}_{\bm{x}}[C(\bm{f},\bm{\pi}_{x};\bm{x})]=\mathbf{E}_{\bm{x}}[V_{\bm{\pi}_{x}}^{(p)}]$,
such that $\bm{\pi}_{\mathrm{safe}}=\bm{\pi}_{x}$ is ensured due to uniqueness
of the solution. Since $V_{\bm{\pi}_{\mathrm{safe}}}^{(p)}(\bm{x})\leq
V_{\bm{\pi}}^{(p)}(\bm{x})$ is satisfied due to point-wise optimality of
$\bm{\pi}_{x}$ and (16) can be straightforwardly reformulated to

$\displaystyle(1-\gamma)V_{\bm{\pi}}^{(p)}(\bm{x})+\gamma\alpha(V_{\bm{\pi}}^{(p)}(\bm{x})-C_{\min})\leq
c(\bm{x}),$

safety of $\bm{\pi}_{\mathrm{safe}}$ immediately follows from the existence of
a $\delta$-safe policy and Proposition IV.1. ∎

Since there exist combinations of dynamics $\bm{f}^{\star}$ and safe sets
$\mathcal{X}_{\mathrm{safe}}$ for which safety cannot be ensured, Proposition
IV.2 cannot guarantee the $\bm{\pi}_{\mathrm{safe}}$ to be always $K$-step
$\delta$-safe. However, as discussed in Section III-B, there exist system
classes for which (10) can be satisfied. Moreover, the necessary function
classes for (17) are well-known for many systems, e.g., it is straightforward
to see that continuous systems and costs require piece-wise continuous
policies in general.

###  IV-C Ensuring Constraint Satisfaction with Safety Filters

While the learned safe policy $\bm{\pi}_{\mathrm{safe}}$ is safe during a
policy roll-out under certain assumptions, it can possibly result in bad
performance since it does not consider the reward function $r$. In contrast,
the nominal policy $\bm{\pi}^{*}$ (4a) results in a high reward of generated
trajectories, but can possibly lead to unsafe states. Therefore, we ideally
want to maintain the beneficial properties of both policies, while avoiding
their shortcomings. The core idea for achieving this relies on a continuous
monitoring of every nominal action $\bm{\pi}^{*}(\bm{x})$, such that they can
be adapted to ensure a safe roll-out of $\bm{\pi}_{\mathrm{safe}}$ afterwards.
Using the reparameterization (13) of the set of plausible models
$\mathcal{M}$, this yields our confidence-based safety filter

$\displaystyle\\!\\!\\!\hat{\bm{\pi}}(\bm{x})=$
$\displaystyle\operatorname*{arg\,min}\limits_{\bm{u}\in\mathcal{U}}\|\bm{\pi}(\bm{x})-\bm{u}\|,\qquad$
(18a) $\displaystyle\text{s.t.
}\max\limits_{\bm{n}\in[-1,1]^{d_{x}}}\mathbf{E}_{\bm{\omega}}\\!\\!\left[V_{\bm{\pi}_{\mathrm{safe}}}^{(p)}(\bm{x}^{\prime})\right]\\!\leq\\!\xi,\\!\\!$
(18b)
$\displaystyle\quad~{}~{}~{}\bm{x}^{\prime}=\bm{\mu}(\bm{x},\bm{u})\\!+\\!\beta\bm{\Sigma}(\bm{x},\bm{u})\bm{n}\\!+\\!\bm{\omega},~{}\bm{\omega}\\!\sim\\!\rho.\\!$
(18c)

Since it cannot be ensured that the state $\bm{x}_{k}$ satisfies
$V_{\bm{\pi}_{\mathrm{safe}}}^{(p)}(\bm{x}_{k})\leq\xi$ for all $k$, a
recovery mechanism steering the system back into this sub-level set is
required. This can be straightforwardly achieved using the learned safe policy
$\bm{\pi}_{\mathrm{safe}}$, resulting in the overall roll-out policy

$\displaystyle\tilde{\bm{\pi}}(\bm{x})=\begin{cases}\hat{\bm{\pi}}(\bm{x})&\text{if
}V_{\bm{\pi}_{\mathrm{safe}}}^{(p)}(\bm{x})\leq\xi\\\
\bm{\pi}_{\mathrm{safe}}(\bm{x})&\text{if
}V_{\bm{\pi}_{\mathrm{safe}}}^{(p)}(\bm{x})>\xi\end{cases}.$ (19)

Due to its strong foundation on the learned safe policy
$\bm{\pi}_{\mathrm{safe}}$, the roll-out policy $\tilde{\bm{\pi}}$ inherits
its theoretical safety guarantees as shown in the following theorem.

###### Theorem IV.1

Consider a set of plausible models $\mathcal{M}$ satisfying II.1 and assume
that the learned safe policy $\bm{\pi}_{\mathrm{safe}}$ satisfies the
conditions of Proposition IV.1. Then, the confidence-based safety filtered
policy (19) is $K$-step $\delta$-safe for all states $\bm{x}\in\mathcal{X}$
with $V_{\bm{\pi}_{\mathrm{safe}}}^{(p)}(\bm{x})\leq\xi<\bar{\xi}$.

###### Proof:

Since $\bm{\pi}_{\mathrm{safe}}$ satisfies the conditions of Proposition IV.1,
the trivial solution $\bm{u}=\bm{\pi}_{\mathrm{safe}}(\bm{x})$ is guaranteed
to ensure (18b). Therefore, (18) is feasible for all states
$\bm{x}\in\mathcal{X}$ with
$V_{\bm{\pi}_{\mathrm{safe}}}^{(p)}(\bm{x})\leq\xi$, such that the $K$-step
$\delta$-safety of $\tilde{\bm{\pi}}$ follows directly from Proposition IV.1.
∎

While the safety filter problem (18) is not compatible with standard
reinforcement learning methods, it can easily be solved on-line in the fashion
of model predictive control. In order to see this, note that the pessimistic
cost-value $V_{\bm{\pi}_{\mathrm{safe}}}^{(p)}(\bm{x})$ can be efficiently
obtained offline using actor-critic methods for reinforcement learning similar
to $V_{\bm{\pi}}^{(p)}$ in Section IV-A. Hence, (18) requires optimization
merely for one time step and consequently only for a single actual and
hallucinating adversarial action in contrast to similar predictive safety
filter approaches [16, 17], which require optimization over a sequence of
actions. Therefore, (18) can be solved with comparatively low computational
complexity using numerical optimization schemes, which allows a
straightforward on-line application as safety filter.

###### Remark IV.1

For practical implementation of (18), $\xi$ can be considered a tuning
parameter. The smaller its value is, the higher the probability of safety is.
However, a small $\xi$ will lead to more conservatism of the safety filter,
such that it must be carefully chosen to trade-off safety and performance.

##  V EXPERIMENTAL RESULTS

In this section, we evaluate the safety filter and compare it with three
competing algorithms: the constraint-free model-free algorithm SAC [26], a
Lagrangian primal-dual approach with SAC as the base algorithm, which we call
CMDP [8], and the model-based alternative Safe-CEM [27]. We consider two
widely used environments to test our approach. First, we test it on an
airplane pitch control [28], where the pitch angle $\theta$ starts at $-0.2$
radians and the constraint function is simply $c_{t}=\theta_{t}$ such that the
angle should never exceed $0$. The reward is given by
$r_{t}=-2\theta^{2}+0.02u^{2}$, where $u$ is the control input. Second, we use
the Mujoco Half-Cheetah environment with the default reward function [29]. The
constraint is that the forward speed is less than $2$. Due to the Cheetah’s
trot, the penalty is on the average forward speed, calculated as
$\bar{v}_{t}=0.1v_{t}+0.9\bar{v}_{t-1}$, $\bar{v}_{0}=0$, where $v_{t}$ is the
instantaneous speed and $\bar{v}_{t}$ is the average speed. Thus we use
$c_{t}=\bar{v}_{t}-2$. We run each environment for $100$ episodes, each
episode for $1000$ time steps, using $\gamma=0.99$ as a discount factor.

To learn the model, we use deterministic ensembles of five members following
[18]. Each member is a neural network with 3 fully connected layers of width
$200$ and Swish non-linearities. For the first ten episodes, data is collected
using a random policy. Such random policy was safe in these environments but
only at the given initial conditions, i.e., it is not the learned safe policy
used by the safety filter. After the initial exploration phase, the model is
pre-trained for $100$ iterations using Adam with learning rate $0.0005$ and
weight decay $0.0001$. Then, after each subsequent episode, the model is
updated using the additional data collected during the episode. We store the
data using an experience replay buffer of at most $100000$ transitions.
Finally, to solve the safety filter problem (18) we use the cross-entropy
method [30] with 1000 particles and 5 iterations per time-step.

In Figure 2, we show the results in the pitch control environment. In this
setting, only the Safety Filter algorithm avoids any constraint violation
while achieving comparable performance in terms of returns and costs. In
Figure 3, we show the results for the Half-Cheetah. Here, both Safe-CEM and
the safety filter avoid any constraint violations. However, the safety filter
achieves higher returns than Safe-CEM. The main difference between these two
environments relies on the backup policy. While in the Cheetah it is enough to
_do nothing_ in order to stop it, in the Pitch Control environment this is not
the case and the _learned_ safe backup policy is crucial to ensure safety.
Thus, with these two environments we demonstrate the scalability of our method
in the Half Cheetah environment as well as the ability to satisfy constraints
in the Pitch Control environment.

Figure 2: Total returns, costs, and constraint violation in the Pitch Control
environment. Only the safety filter attains no constraint violations and
achieves comparable performance to the benchmarks.  Figure 3:  Total returns,
costs, and constraint violation in the Half-Cheetah environment. The safety
filter and Safe-CEM achieve no constraint violation. These two algorithms
perform slightly worse than the benchmarks in terms of returns, but the safety
filter performs better than Safe-CEM.

##  VI CONCLUSION

In this paper, we introduced confidence-based safety filters, a novel approach
for ensuring the safety of nominal policies learnt via standard reinforcement
learning techniques. The approach relies on a reformulation of state
constraints as cumulative costs, such that safety conditions can be expressed
as cost constraints. This allows us to obtain safe policies via robust
reinforcement learning, which can be used as “backup” policy in a safety
filter. We demonstrated the effectiveness and scalability of our approach in
simulations.

###### Lemma A.1

If there exists a function $V\\!:\mathcal{X}\\!\rightarrow\\!\mathbb{R}$ such
that

$\displaystyle\mathbf{E}_{\omega}\\!\\!\left[V(\bm{f}^{\star}(\bm{x},\bm{\pi}(\bm{x}))+\omega)\right]\leq\theta_{1}$
(20)

for $\theta_{1}\in\mathbb{R}$, then, it holds that

$\displaystyle\frac{\theta_{2}-\theta_{1}}{\theta_{2}\\!-\\!\underline{V}}\leq\mathbb{P}\left(\bm{x}_{k+1}\in\mathcal{V}_{\theta_{2}}|\bm{x}_{k}=\bm{x}\right)\leq\frac{\theta_{2}-\theta_{1}}{\bar{V}\\!-\\!\theta_{2}}$
(21)

for every $\underline{V}\leq\theta_{1}<\theta_{2}<\bar{V}$, where
$\underline{V}=\min_{\bm{x}\in\mathcal{X}}V(\bm{x})$ and
$\bar{V}=\max_{\bm{x}\in\mathcal{X}}V(\bm{x})$.

###### Proof:

In order to prove this lemma, we follow the ideas of [21]. It is
straightforward to see that

$\displaystyle(\theta_{2}-\underline{V})\mathbb{P}\left(\bm{x}_{k+1}\notin\mathcal{V}_{\theta_{2}}|~{}\bm{x}_{k}=\bm{x}\right)\leq$
(22)
$\displaystyle\qquad\qquad\quad\mathbf{E}_{\bm{\omega}_{k}}\\!\\!\left[\mathbb{I}_{\bm{x}_{k+1}\notin\mathcal{V}_{\theta_{2}}}(V(\bm{x}_{k+1})-\underline{V})|~{}\bm{x}_{k}=\bm{x}\right],$

where
$\mathcal{V}_{\theta_{2}}=\\{\bm{x}\in\mathcal{X}:V(\bm{x})<\theta_{2}\\}$,
and $\mathbb{I}_{\bm{x}_{k+1}\notin\mathcal{V}_{\theta_{2}}}=1$ if
$\bm{x}_{k+1}\notin\mathcal{V}_{\theta_{2}}$ and $0$ otherwise, since
$\mathbb{I}_{\bm{x}_{k+1}\notin\mathcal{V}_{\theta_{2}}}V(\bm{x}_{k+1})\geq\mathbb{I}_{\bm{x}_{k+1}\notin\mathcal{V}_{\theta_{2}}}\theta_{2}$.
Moreover, we trivially have

$\displaystyle\mathbf{E}_{\bm{\omega}_{k}}\\!\\!\left[\mathbb{I}_{\bm{x}_{k+1}\notin\mathcal{V}_{\theta_{2}}}(V(\bm{x}_{k+1})-\underline{V})|~{}\bm{x}_{k}=\bm{x}\right]\leq\\!$
(23)
$\displaystyle\qquad\qquad\qquad\qquad\qquad\mathbf{E}_{\bm{\omega}_{k}}\\!\\!\left[(V(\bm{x}_{k+1})-\underline{V})|~{}\bm{x}_{k}=\bm{x}\right].$

By combining (20), (22) and (23), we therefore obtain
$(\theta_{2}-\underline{V})\mathbb{P}\left(\bm{x}_{k+1}\notin\mathcal{V}_{\theta_{2}}|~{}\bm{x}_{k}=\bm{x}\right)\leq-\underline{V}+\theta_{1},$
which results in

$\displaystyle\mathbb{P}\left(\bm{x}_{k+1}\notin\mathcal{V}_{\theta_{2}}|~{}\bm{x}_{k}=\bm{x}\right)\leq\frac{\theta_{1}-\underline{V}}{\theta_{2}-\underline{V}}.$
(24)

The proof for the upper bound is analogous. ∎

###### Proposition A.1

Assume there exists a function
$V\\!\\!:\\!\mathcal{X}\\!\rightarrow\\!\mathbb{R}$ and a class $\mathcal{K}$
function $\alpha:\mathbb{R}\rightarrow\mathbb{R}_{0,+}$, such that

$\displaystyle\mathbf{E}_{\omega}\\!\\!\left[V(\bm{f}^{\star}(\bm{x},\bm{\pi}(\bm{x}))+\omega)\right]-V(\bm{x})\leq-\alpha(V(\bm{x}))$
(25)

holds for all $\bm{x}\in\mathcal{V}_{\bar{\xi}}$ for $\bar{\xi}\in\mathbb{R}$.
Then,
$\mathbf{E}_{\omega}\\!\\!\left[V(\bm{f}^{\star}(\bm{x},\bm{\pi}(\bm{x}))+\omega)\right]\leq\xi$
with $\xi<\bar{\xi}$ ensures

$\displaystyle\mathbb{P}\left(V(\bm{x}_{k})\leq\bar{\xi}~{}\forall
k=1,\ldots,K|\bm{x}_{0}=\bm{x}\right)\geq 1-\delta_{\mathrm{FL}}(\xi)$ (26)

with

$\displaystyle\delta_{\mathrm{FL}}(\xi)=\begin{bmatrix}1&\cdots&0\end{bmatrix}\\!\begin{bmatrix}1&\bm{1}^{T}(\bm{I}-[\bm{P}]_{+})\\\
\bm{0}&[\bm{P}]_{+}\end{bmatrix}^{\\!K}\\!\begin{bmatrix}0\vspace{-0.2cm}\\\
\vdots\vspace{-0.05cm}\\\ 1\end{bmatrix}\\!\\!,\\!$ (27)

where the elements of $\bm{P}$ are defined as

$\displaystyle\\!p_{i,j}\\!=\\!\\!\begin{cases}\\!\frac{\theta^{i}\\!-\theta^{j}\\!+\alpha(\theta^{j\\!+\\!1}\\!+\\!\underline{V})}{\theta^{i}\\!-\underline{V}}\\!-\\!\frac{\theta^{i\\!+\\!1}\\!-\theta^{j}\\!+\alpha(\theta^{j\\!+\\!1}\\!+\underline{V})}{\bar{V}\\!-\theta^{j}\\!+\alpha(\theta^{j\\!+\\!1}\\!+\underline{V})}&\\!\\!\text{if
}i\\!\leq\\!j\\\
\\!\frac{(1-\vartheta)\alpha(\theta^{j\\!+\\!1}+\underline{V})}{\theta^{j}-\alpha(\theta^{j\\!+\\!1}+\underline{V})-\underline{V}}&\\!\\!\text{if
}i\\!=\\!j\\!+\\!1.\\!\\!\\\ \\!0&\\!\\!\text{if
}i\\!>\\!j\\!+\\!1\end{cases}\\!$ (28)

and $M$ is the largest integer such that $\theta^{1}\leq\bar{\xi}$ for
$\theta^{i}$ recursively defined by
$\theta^{i-1}=\theta^{i}+\vartheta\alpha(\theta^{i}+\underline{V})$ with
$\theta^{M+1}$ implicitly defined via
$\theta^{M+1}+(\vartheta-1)\alpha(\theta^{M+1}+\underline{V})=\xi$ and
sufficiently small $\vartheta\\!\in\\!(0,1)$.

###### Proof:

For proving this proposition, we construct a sequence of sub-level sets
$\mathcal{V}_{\theta^{j}}$ as illustrated in Fig. 4 and bound the transition
probabilities between them using Lemma A.1. Given a sub-level set
$\mathcal{V}_{\theta^{j}}$, the probability of transitioning into sub-level
set $\mathcal{V}_{\theta^{j+1}}$ can be lower bounded using
$\theta_{2}=\theta^{j}-\vartheta\alpha(\theta^{j+1}-\underline{V})$,
$\theta_{1}=\theta^{j}-\alpha(\theta^{j+1}-\underline{V})$, which yields

$\displaystyle
p_{j+1,j}=\frac{(1-\vartheta)\alpha(\theta^{j+1}+\underline{V})}{\theta^{j}-\alpha(\theta^{j+1}+\underline{V})-C_{\min}}.$

For transitioning from the sub-level set $\mathcal{V}_{\theta^{j}}$ to a sub-
level set $\mathcal{V}_{\theta^{i}}$, $i\leq j$, we have

$\displaystyle\mathbb{P}\left(\bm{x}_{t+1}\in\mathcal{V}_{\theta^{i}}\setminus\mathcal{V}_{\theta^{i+1}}|\bm{x}_{t}\in\mathcal{V}_{\theta^{j}}\right)=$
$\displaystyle\quad~{}\mathbb{P}\left(\bm{x}_{t+1}\in\mathcal{V}_{\theta^{i}}|\bm{x}_{t}\in\mathcal{V}_{\theta^{j}}\right)-\mathbb{P}\left(\bm{x}_{t+1}\in\mathcal{V}_{\theta^{i+1}}|\bm{x}_{t}\in\mathcal{V}_{\theta^{j}}\right)\\!,$

such that applying Lemma A.1 to both summands with $\theta_{2}=\theta^{i}$,
$\theta_{1}=\theta^{j}-\alpha(\theta^{j+1}-\underline{V})$ and
$\theta_{2}=\theta^{i+1}$,
$\theta_{1}=\theta^{j}-\alpha(\theta^{j+1}-\underline{V})$, respectively,
yields

$\displaystyle
p_{i,j}\\!=\\!\frac{\theta^{i}\\!-\\!\theta^{j}\\!+\\!\alpha(\theta^{j\\!+\\!1}\\!+\\!\underline{V})}{\theta^{i}\\!-\underline{V}}\\!-\\!\frac{\theta^{i\\!+\\!1}\\!-\\!\theta^{j}\\!+\\!\alpha(\theta^{j\\!+\\!1}\\!+\\!\underline{V})}{\bar{V}\\!-\theta^{j}\\!+\\!\alpha(\theta^{j\\!+\\!1}\\!+\underline{V})}.$

Note that for $i=M$ we have $\theta_{1}=\xi$. Since we cannot guarantee to
directly transition from sub-level sets $\mathcal{V}_{\theta^{j}}$ to sub-
level sets $\mathcal{V}_{\theta^{i}}$ with $i\geq j+2$, we obtain the trivial
bound $p_{i,j}=0$ in this case, which results in (28). Based on the bounds
$p_{i,j}$, we can construct a left stochastic matrix similar to the transition
matrix of a Markov chain, whose first row corresponds to an absorbing state as
shown in (27). Since the first state is absorbing and the transition
probabilities to all other states are lower bounds, multiplying this matrix
$K$ times with itself and multiplying the initial probability distribution
from the right yields the upper bound $\delta$ for leaving the sub-level set
$\mathcal{V}_{\bar{\xi}}$ within $K$ time steps. ∎

$\mathcal{C}_{\bar{\xi}}^{\bm{\pi}}$$\mathcal{C}_{\theta^{1}}^{\bm{\pi}}$$\mathcal{C}_{\theta^{2}}^{\bm{\pi}}$$\mathcal{C}_{\xi}^{\bm{\pi}}$$1\\!-\\!\sum\limits_{i=1}^{3}p_{i,2}$$p_{3,2}$$p_{1,2}$$p_{2,2}$

Figure 4: In order to certify the $K$-step $\delta$-safety of a policy
$\bm{\pi}$, we define a sequence of sub-level sets
$\mathcal{C}_{\theta^{i}}^{\bm{\pi}}$ with decreasing thresholds $\theta^{i}$.
We can bound the probabilities for transitioning to other sub-level sets in
each time step using Lemma A.1 as exemplarily illustrated for
$\mathcal{C}_{\epsilon^{2}}^{\bm{\pi}}$, such that the probability of leaving
the $\mathcal{C}_{\bar{\xi}}^{\bm{\pi}}$ can be bounded using methods for
Markov chains.

## References

  * [1] R. S. Sutton and A. G. Barto, _Reinforcement Learning: An Introduction_ , 2nd ed. The MIT Press, 2017.
  * [2] T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” in _International Conference on Learning Representations_ , 2016\.
  * [3] G. Dulac-Arnold, D. Mankowitz, and T. Hester, “Challenges of real-world reinforcement learning,” in _ICML Workshop on Real-Life Reinforcement Learning_ , 2019.
  * [4] J. García and F. Fernández, “A comprehensive survey on safe reinforcement learning,” _Journal of Machine Learning Research_ , vol. 16, pp. 1437–1480, 2015.
  * [5] L. Brunke, M. Greeff, A. W. Hall, Z. Yuan, S. Zhou, J. Panerati, and A. P. Schoellig, “Safe learning in robotics: From learning-based control to safe reinforcement learning,” 2021. [Online]. Available: http://arxiv.org/abs/2108.06266
  * [6] E. Altman, _Constrained Markov Decision Processes_. CRC Press, 1999.
  * [7] J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” in _International Conference on Machine Learning_ , 2017, pp. 30–47.
  * [8] S. Paternain, L. F. Chamon, M. Calvo-Fullana, and A. Ribeiro, “Constrained reinforcement learning has zero duality gap,” in _Advances in Neural Information Processing Systems_ , vol. 32, 2019, pp. 7555–7565.
  * [9] D. Ding, X. Wei, Z. Yang, Z. Wang, and M. Jovanović, “Provably efficient safe exploration via primal-dual policy optimization,” in _International Conference on Artificial Intelligence and Statistics_ , 2021, pp. 3304–3312.
  * [10] S. Dean, S. Tu, N. Matni, and B. Recht, “Safely learning to control the constrained linear quadratic regulator,” in _American Control Conference_ , 2019, pp. 5582–5588.
  * [11] U. Rosolia and F. Borrelli, “Learning model predictive control for iterative tasks. A data-driven control framework,” _IEEE Transactions on Automatic Control_ , vol. 63, no. 7, pp. 1883–1896, 2018.
  * [12] T. Koller, F. Berkenkamp, M. Turchetta, and A. Krause, “Learning-based model predictive control for safe exploration,” in _IEEE Conference on Decision and Control_ , 2018, pp. 6059–6066.
  * [13] A. Capone, A. Lederer, J. Umlauft, and S. Hirche, “Data selection for multi-task learning under dynamic constraints,” _IEEE Control Systems Letters_ , vol. 5, no. 3, pp. 959–964, 2021.
  * [14] J. F. Fisac, A. K. Akametalu, M. N. Zeilinger, S. Kaynama, J. Gillula, and C. J. Tomlin, “A general safety framework for learning-based control in uncertain robotic systems,” _IEEE Transactions on Automatic Control_ , vol. 64, no. 7, pp. 2737–2752, 2019.
  * [15] A. Taylor, A. Singletary, Y. Yue, and A. Ames, “Learning for safety-critical control with control barrier functions,” in _Learning for Dynamics & Control_, 2019, pp. 708–717.
  * [16] O. Bastani, “Safe reinforcement learning with nonlinear dynamics via model predictive shielding,” in _American Control Conference_ , 2021, pp. 3488–3494.
  * [17] K. Wabersich, L. Hewing, A. Carron, and M. Zeilinger, “Probabilistic model predictive safety certification for learning-based control,” _IEEE Transactions on Automatic Control_ , vol. 76, no. 1, pp. 176–188, 2021.
  * [18] S. Curi, F. Berkenkamp, and A. Krause, “Efficient model-based reinforcement learning through optimistic policy search and planning,” in _Advances in Neural Information Processing Systems_ , 2020.
  * [19] A. Mesbah, “Stochastic model predictive control: An overview and perspectives for future research,” _IEEE Control Systems_ , vol. 36, no. 6, pp. 30–44, 2016.
  * [20] K. Kim and R. Braatz, “Generalised polynomial chaos expansion approaches to approximate stochastic model predictive control,” _International Journal of Control_ , vol. 86, no. 8, pp. 1324–1337, 2013.
  * [21] Y. Li, W. Zhang, and X. Liu, “Stability of nonlinear stochastic discrete-time systems,” _Journal of Applied Mathematics_ , vol. 2013, no. 2, 2013.
  * [22] S. Meyn and R. Tweedie, _Markov Chains and Stochastic Stability_ , 1993\.
  * [23] V. Gaitsgory, L. Grüne, M. Höger, C. M. Kellett, and S. R. Weller, “Stabilization of strictly dissipative discrete time systems with discounted optimal control,” _Automatica_ , vol. 93, pp. 311–320, 2018\.
  * [24] S. Curi, I. Bogunovic, and A. Krause, “Combining pessimism with optimism for robust and efficient model-based deep reinforcement learning,” in _International Conference on Machine Learning_ , 2021, pp. 2254–2264.
  * [25] L. Pinto, J. Davidson, R. Sukthankar, and A. Gupta, “Robust adversarial reinforcement learning,” in _International Conference on Machine Learning_ , 2017, pp. 4310–4319.
  * [26] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in _International Conference on Machine Learning_ , 2018, pp. 1861–1870.
  * [27] Z. Liu, H. Zhou, B. Chen, S. Zhong, M. Hebert, and D. Zhao, “Constrained model-based reinforcement learning with robust cross-entropy method,” _arXiv preprint arXiv:2010.07968_ , 2020.
  * [28] R. Hafner and M. Riedmiller, “Reinforcement learning in feedback control,” _Machine learning_ , vol. 84, no. 1, pp. 137–169, 2011.
  * [29] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for model-based control,” in _IEEE/RSJ international conference on intelligent robots and systems_ , 2012, pp. 5026–5033.
  * [30] Z. I. Botev, D. P. Kroese, R. Y. Rubinstein, and P. L’Ecuyer, “The cross-entropy method for optimization,” in _Handbook of Statistics_. Elsevier, 2013, vol. 31, pp. 35–59.


††thanks:  ©Copyright R J Ellis, 2021-22.

# I: Evidence for Phenomena Beyond 4-D Space-Time and Theory Thereof

R J Ellis<EMAIL_ADDRESS>Corpus Christi College, Oxford OX1 4JF, UK

###### Abstract

The mismatch between theory, which can predict other dimensions or spaces, and
experimental physics, which is currently limited to determining objective
facts in four-dimensional (4-D) space-time, is examined. There may be
phenomena in other dimensions/spaces, which cannot be determined completely
objectively. However, if results are reproducible, then these should be
considered physically real. Magnetic monopoles are an example, because several
experimentalists have shown that they only exist in the presence of intense
illumination and so are not objective. Mikhailov made repeated measurements
and determined that the monopole charge is quantized ($g=ng_{D},n$=1-5);
($\bar{g}$ = (0.99 $\pm$ 0.05)$g_{D}$). We conclude Dirac monopoles exist, not
in 4-D, but in another space-time. In the second part we bring together two
theories to explain how light can switch (via the Brittin and Gamow effect)
into another space (predicted by chronometric invariant general relativity:
CIGR) to reveal phenomena there. Light lowers the entropy level and so
reverses the arrow of time into the mirror world of CIGR, where time flows
from the future to the past, and reveals monopoles there. We call this the
photo-mirror effect. This explains the infinite length of Dirac’s string.
These results are evidence for unification.

experimental method; Dirac magnetic monopoles; intense illumination; mirror
space-time; photo-mirror effect; Brittin and Gamow; Chronometric invariants

##  I Introduction

The paper is within the framework of the low energy frontier of particle
physics, albeit with a new approach. A review of this field is given in the
following reference.LowE  The historical backgrounds of the other sections,
where they exist, are given in each section. In the case of the mismatch
between the methods of theory and experiment, the author knows of none which
are relevant. In the case of magnetic monopoles, there are two histories: that
of the searches for objective monopoles, which is given in the surveys of the
same; and that of the more subjective monopoles which exist only when
illuminated, which we present in more detail below.

We begin by examining some of the assumptions of physics. Physics is based on
reason (usually defined mathematically) and fact (usually determined by
controlled experiment). There are two problems with facts determined by
controlled experiment. Firstly, if one relies upon objective facts only, this
assumes that the Universe can be reduced to objective facts, or at least if
there are any subjective or non-objective aspects, they can be ignored. There
is no proof of this, and it could lead to an infinite regression. (For
example, if the Universe is made from some fundamental objective substance SA,
then what is this made of? Either it is something subjective, or it is another
objective substance SB, and so on.)

Secondly, there is an imbalance here because a theoretician can define a
theory in multiple dimensions (e.g. string theory) whilst an experimentalist
can currently only do controlled experiments in four-dimensional (4-D) space-
time. If one wants to make measurements in other spaces or dimensions, then
assuming it is possible, there is inevitably some reduction in control and/or
objectivity. Currently such results tend to be rejected because they are not
completely objective (i.e. not in 4-D space time). So we make the assumption
that there maybe phenomena which cannot be detected completely objectively,
but nevertheless they should be considered physically real if the results can
be reproduced.  We have examined the literature and find that magnetic
monopoles are an example of this.

##  II Magnetic Monopoles

We present experimental evidence from the literature, for real ($\nabla\cdot$B
$\neq$ 0) magnetic monopoles, as opposed to the pseudo-monopoles
($\nabla\cdot$H $\neq$ 0) sometimes observed in spin ices or other solid-state
phenomena.

Over the last 70 years there have been numerous searches for real magnetic
monopoles with mostly negative results. Compilations of these searches
conclude that there is no reproducible evidence for magnetic monopoles.SurveyA
; SurveyB  But there is an assumption behind this conclusion, namely that
magnetic monopoles must be particles which can be detected objectively in 4-D
space-time. The problem is that in Dirac’s theory there is a line connecting
two monopoles which has to be infinitely long, and yet the universe is
finite.DiracA  This infinite length of the Dirac string is normally explained
away as an artefact of the calculation. However, if we take this seriously,
then it implies that both monopoles are outside 4-D space-time. (In principle
one could be inside, the other outside, but that would require preferential
treatment for one monopole over another, which the theory does not provide. So
we reject this.) In which case it will not be possible to detect them
objectively by the normal methods of experimental physics. Therefore the
conclusion of the above compilations is not strictly correct. It should read
“there is no reproducible evidence for magnetic monopoles in 4-D space-time”.
If we take the infinite length of the Dirac string seriously, then this is not
evidence for or against magnetic monopoles because they are not predicted to
be in 4-D space-time.

Furthermore, if a phenomenon is not objective, then it is currently rejected
by physicists as not being physically real. Therefore, the above monopole
surveys usually omit most, if not all, of the references to the following
experiments which provide reproducible evidence for magnetic monopoles, but of
a subjective nature. They are subjective because these monopoles are only
visible under intense illumination. When the intense illumination is turned
off, they disappear, in the sense that the particle being observed ceases to
move as a monopole, and moves as a neutral particle or dipole. Thus these
monopoles do not seem to exist in their own right. However, these results are
reproducible, and so we argue they are physically real. Here is a summary of
the published evidence.

###  II.1 Ehrenhaft

Ehrenhaft first reported observation of single magnetic charges, which were
only detectable under intense illumination, in 1930,Ehrenhaft1  before Dirac’s
paper in 1931.DiracA  However, Dirac did not recognise Ehrenhaft’s
results.DiracB ; DiracC  (Not only were Ehrenhaft’s results subjective, but
they were obtained at very low energies. So Dirac rejected them. We discuss
this below.) Curiously more physicists know of Dirac’s theory than Ehrenhaft’s
experiments. The essence of Ehrenhaft’s observations is that when
microparticles of ferromagnetic substances (such as iron, nickel or cobalt)
are suspended in a gas atmosphere and subjected simultaneously to a uniform
magnetic field and to intense illumination by light, they move as objects
carrying single magnetic charges. If the magnetic field H is reversed, then
the direction of motion of the single magnetic charges are reversed (magnetic
dipoles would not do this). This effect was confirmed by Benedikt and
Leng.Benedikt

Ehrenhaft did a number of experiments,Ehrenhaft2  and when he did not get the
recognition he felt he deserved, he made more extreme claims, such as that
“light magnetises matter”.Ehrenhaft3  He was convinced that he had discovered
free magnetic charges and should get the kind of recognition of someone such
as Ampère or Faraday. He claimed he had created a magnetic current by causing
the monopoles to move.Ehrenhaft4  He also claimed to have discovered
“magnetolysis”, being the magnetic equivalent of electrolysis.Ehrenhaft5  Many
physicists were unconvinced that ”light makes magnetism“, found the effect not
objectively real, and so tended to ridicule the results.Telegdi  Einstein took
the observations seriously, but wanted a better explanation.Einstein2

Kemple made a review of experimental searches for monopoles up to 1961,
including not only the work of Ehrenhaft, but also by his contemporaries. He
noted that other experimenters could not reproduce some of these results, and
therefore concluded that this work is not evidence for magnetic
monopoles.Kemple  However, this is not strictly correct, because even though
some of the experiments may not have been confirmed, the basic observation of
magnetic monopoles under intense illumination, was confirmed by Benedikt and
Leng.Benedikt

###  II.2 Mikhailov

There the matter might have rested, had it not been that Mikhailov repeated
Ehrenhaft’s magnetic charge experiment with better technique, and confirmed
the result.Mikhailov1A ; Mikhailov1B ; Mikhailov1C  In his first experiment,
he used iron microparticles suspended in an atmosphere of argon, illuminated
by a laser with power up to 1 kW/cm2, and in the presence of crossed uniform
electric and magnetic fields, which were switched by a square wave-form with a
frequency of a few Hertz. The particles were observed with a microscope, and
moved under the influence of the crossed electric and magnetic fields (E and
H). By observing their motion, one could select the signs of the electric and
magnetic charges of the particles being observed.

The observed microparticles had a mass M $\leq 10^{-14}$ gram and size r $\leq
10^{-5}$ cm, and their motion was governed by Stokes’ law. By making
measurements on particles carrying both an electric and a magnetic charge, it
was possible to measure the ratio g/q independently of the Stokes’
coefficient, and hence of the size of the particle. From observations of 1200
such particles, Mikhailov found that the magnetic charge is quantized. But
this initial value of g disagreed with Dirac’s prediction. However, Akers
pointed out that Mikhailov had ignored components of the particle’s velocity
orthogonal to E and H, and so this interpretation of the result could be
incorrect.Akers

Mikhailov reanalysed his experiment and found that the magnetic charge in this
experiment, is in fact the solution of a quadratic equation and so gives two
possible values. One value is the one he had previously reported, the other
being that predicted by Dirac. In order to distinguish between these two
values, Mikhailov redesigned the experiment to remove this ambiguity and also
possible surface effects.

He condensed super-saturated vapour onto solid ferromagnetic particles in a
diffusion chamber, which created a smooth surface round each particle and so
eliminated surface effects. These ferromagnetic particles, surrounded by
fluid, were allowed to drop through a beam of light, under the force of
gravity in a magnetic field H, which was periodically inverted. Under these
conditions, particles exhibiting the magnetic charge effect, fall in a zig-zag
path. He observed 428 such tracks with a mean magnetic charge of $\bar{g}$ =
(2.5${}_{-1.3}^{+1.6}$).10-8 gauss.cm2, which agrees with the value predicted
by Dirac of gD = 3.29 10-8 gauss.cm2 within the errors. In this way, Mikhailov
showed unambiguously that he was observing Dirac “monopoles”, and furthermore,
these were not due to surface effects on the particles.Mikhailov2

He also re-ran his previous experiment, choosing the correct root, and found
that the ferromagnetic particles carried from 1 to 5 magnetic charges. The
histogram of magnetic charges clearly shows 5 separate peaks corresponding
$g=ng_{D}$ where n = 1 to 5, with the peaks being gaussian-like with some gaps
in between.Mikhailov3  This confirms that the magnetic charge is quantised,
and rules out Schwinger monopoles which have twice the magnetic charge
($g_{S}=2g_{D}$).Schwinger

He also reanalysed his previous experiments, dividing the data by n, and
obtained a narrow bell-shaped distribution centred on $\bar{g}$ = (3.27 $\pm$
0.16) x 10-8 gauss.cm2 = 0.99 gD with an accuracy of $\pm$5%.Mikhailov4
Therefore, by these ingenious experiments, Mikhailov has observed Dirac
monopoles, but only when illuminated by light. There is no satisfactory
explanation for this.

###  II.3 Discussion

In the main, these subjective results have been rejected or ignored. However
they are reproducible, because several experimentalists have observed more
than one thousand single magnetic charges. Therefore these single magnetic
charges should be considered a real physical phenomena. However the objective
methods of physics have failed to detect them, and there is no evidence for
them in 4-D space-time, as shown above. Therefore the monopoles observed must
be in another space.

Nevertheless, this is not a complete explanation. We also need a theory which
predicts the existence of this second space, together with a mechanism which
enables light to switch space-time into this second space. We now present such
a combined theory.

##  III Sunlight Shining on the Earth’s Surface

We start with an existing theory of an unexpected property of light which does
the switching, and then introduce a version of general relativity which
predicts a more complex structure to space-time. The basic idea is that light
switches the direction of the flow of time into that of another space-time.

###  III.1 Brittin and Gamow’s Theory

In a simple theory, Brittin and Gamow have suggested that sunlight shining on
the earth, pumps entropy out into space, thereby allowing negentropy to
accumulate on the earth’s surface. The sun’s radiation consists of high
temperature photons coming from the surface at Ts $\simeq$ 5,900∘ K, which
spreads out in space and becomes diluted. By the time it reaches the earth’s
surface, it’s energy density corresponds to a temperature of the earth (Te
$\simeq$ 300∘ K), so these photons are not in thermodynamic equilibrium.
Brittin and Gamow use the quantum theory of radiation to show that the net
entropy change when sunlight interacts with the earth’s surface is:Gamow61

$\Delta S~{}=~{}\Delta S_{s}\,\,-\,\,\Delta
S_{e}~{}=~{}{\frac{4}{3}}\,\,\Delta
Q\,\,\,{\left(\,{\frac{1}{T}}_{s}\,\,-\,\,{\frac{1}{T}}_{e}\,\right)}$ (1)

which is negative because Ts $>$ Te. So the entropy at the earth’s surface is
reduced. They reason that this is not contrary to the second law of
thermodynamics because it is simply due to the temperature gradient Ts $>$ Te
$>$ Tspace, but see below. (Note this effect is not limited to sunlight, but
also occurs with light from an artificial source. For example, light from a
halogen lamp can also lower entropy levels, because Th $\simeq$ 3000∘ K and so
a temperature gradient similar to sunlight, applies Th $>$ Te $>$ Tspace.)
However, there is a hidden complication, whether the source is natural or
artificial.
This mechanism enables negative entropy to build up on the earth’s surface,
provided it can be stored. In the case of sunlight, they calculate that
photosynthesis has an efficiency of about 10% for capturing this negative
entropy. Brittin and Gamow suggest that this is the source of order for the
food chain, which Schrödinger proposed to be a current of negative
entropy.SchrodingerA ; SchrodingerB  As a result, this is not a purely
physical theory because it relies upon plants (and hence biochemistry) to
capture the negentropy. (This may explain why there is not much published by
physicists about this.) This theory would be more interesting to physics if
there is a physical mechanism to store the negentropy produced.

###  III.2 Discussion of Brittin and Gamow Effect

There is also a contradiction in this theory. On the one hand, it predicts
that solar photons lower the entropy level on the earth’s surface, and this
logically follows from the temperature gradient. However the reduction in
entropy level of a closed system is contrary to the second law of
thermodynamics. In order to remove this contradiction, we conclude that there
is something missing from Brittin and Gamow’s theory, or the second law of
thermodynamics, or both.

In classical thermodynamics, the entropy increases with the arrow of
time.Eddington  Quite a lot has been written about time in recent decades, but
mainly from a theoretical or philosophical point of view.TimeA ; TimeB ; TimeC
However, we are concerned with a very specific problem. What happens to time
when a solar photon interacts with the earth’s surface, thereby lowering its
entropy level? Is the direction of time reversed (e.g. locally), either
momentarily or more persistently, when the photon lowers the entropy level? We
conclude that it logically must, because otherwise Eddington’s arrow of time
would be violated, and the second law of thermodynamics also. Therefore what
is missing from Brittin and Gamow’s theory, is a theory of space-time with a
second time dimension which flows from the future to the past. (Experimental
support for this reasoning is given in the following reference.Ellis1 )

There are several theories with two time dimensions. For example,
11-dimensional extended supersymmetry in M-theory is really a 12-dimensional
SUSY with an SO(10,2) symmetry.Bars1A ; Bars1B ; Bars1C ; Bars1D  F-theory in
twelve dimensions (12-D) is similar.Vafa  But these second time dimensions are
compactified. Köhn has taken a different approach and added a second time
dimension to the Einstein-Friedmann equations.Kohn  This enables him to solve
the cosmological constant problem, but the second time dimension, whilst not
compactified, is on the spacial scale of the Planck length. Bars and his
colleagues have done the most work in this area of two-time physics.Bars2
However, neither of his two time dimensions help explain the phenomena which
interest us. Furthermore, there does not appear to be any GUT which predicts a
second time dimension which flows from the future to the past.

There is, however, a version of the theory of general relativity which
predicts this. This extension is not well known, and so we present a summary
of it below.

##  IV General Relativity: Chronometric Invariants

One important development in general relativity was the correct introduction
of the Observer by Abraham Zelmanov. Einstein’s original theory was of space-
time and matter-fields, and did not include the Observer. However, each
Observer has a specific reference frame where he or she is located, for
example, at particular coordinates on the earth with its own gravitational
field, which is also rotating. As a result, what is actually observed there is
not well-defined by the original theory. The values of quantities predicted by
general relativity are in effect theoretical, because in practice they are not
corrected for the Observer’s reference frame. So without the Observer, general
relativity is incomplete. The case for including the Observer is thus
compelling. This problem was recognised and worked on in the 1930s by Landau
and others. Some progress was made to solve it for specific cases by Landau
and Lifshitz.Landau  What was required was a strict mathematical formalism to
calculate the observable values for any tensor quantity. Zelmanov correctly
developed this general theory in 1944. However, it was not published until
1956.ZelmanovA ; ZelmanovB  The mathematical details of the theory are given
in the references. We just present a short summary of the main points here.

Physically observable quantities are obtained by projecting four-dimensional
quantities onto the time lines and three-dimensional space of the Observer’s
reference frame. Physically observable quantities must be invariant with
respect to transformations of time, and so they are chronometrically invariant
quantities. Thus the general case of the Observer was incorporated into
general relativity in Russia in the era of the Soviet Union. Cattaneo later
obtained similar results.CattaneoA ; CattaneoB ; CattaneoC

Table 1: Summary of Spacial Properties of Chronometric Invariant General
Relativity

mass |  Particles |  Energies |  Class of motion |  Area |  Time |  Entropy
---|---|---|---|---|---|---
m $>$ 0 |  massive particles |  E $>$ 0 |  move at sub-light speeds |  our world |  dt $>$ 0 |  $\Delta$ S $>$ 0
m = 0 |  massless particles (photons) |  E $>$ 0 |  move at speed of light |  our world |  |
m = 0 |  light-like vortices |  E = 0 |  moving and rotating at speed of light |  the membrane |  dt = 0 |
m = 0 |  massless particles (photons) |  E $<$ 0 |  move at speed of light |  the mirror world |  |
m $<$ 0 |  massive particles |  E $<$ 0 |  move at sub-light speeds |  the mirror world |  dt $<$ 0 |  $\Delta$ S $<$ 0

Borissova and Rabounski have developed this theory further. They find that the
chronometric invariant equations of motion for mass-bearing particles into the
past and into the future are asymmetric in time. They conclude there is a
fundamental asymmetry of the directions of time in the in-homogeneous space-
time of general relativity. They hold up a ”mirror“ to time and find that it
does not reflect completely, and that there is a different world ”beyond the
mirror“. The four-dimensional momentum vector for a particle with non-zero
rest mass, $m_{0}$ is:

${P^{\alpha}=m_{0}}{\frac{dx^{\alpha}}{ds}},~{}~{}~{}{P_{\alpha}P^{\alpha}}=1,~{}~{}~{}\alpha=0,1,2,3.$
(2)

When a vector (or tensor) is projected onto the time line and spacial section
of an observer, these projections give the physically observable quantities
for that observer.ZelmanovA  Using the properly observable time interval
($d\tau$ = $\sqrt{g_{00}}$ $dt+$ $\frac{g_{0i}}{c\sqrt{g_{00}}}$
$dx^{i}$),ZelmanovA ; Landau  the above four-dimensional momentum vector has
two projections onto the time line, namely:Borissova ; Rabounski

$\frac{P_{0}}{\sqrt{g_{00}}}=\pm
m,~{}~{}~{}~{}~{}\text{where}~{}m=\frac{m_{0}}{\sqrt{1-v^{2}/c^{2}}}$ (3)

whereas it has only one spacial projection:

$P^{i}=\frac{m}{c}v^{i}=\frac{1}{c}p^{i}~{}~{}~{}~{}\text{where}~{}v^{i}=\frac{dx^{i}}{d\tau},~{}~{}i=1,2,3.$
(4)

where $p_{i}$ is the three-dimensional observable momentum. They conclude that
any massive particle, having two time projections, exists in two observable
states, entangled to each other: the positive mass state is in our world,
while the negatively charged mass state is in the mirror world. Using the
techniques of chronometric invariants, they find that there are three separate
areas: our world (i.e. normal space-time), the mirror world, and a membrane
which separates the two.Borissova  A summary of their results is shown in
table 1.Table

The flow of time is well defined mathematically in general relativity. It is
determined by the sign of the derivative of the coordinate time t with respect
to the proper time ($dt/d\tau$). Using $w=c^{2}(1-\sqrt{g_{00}}$) and
$\nu_{i}=-c\frac{g_{0i}}{\sqrt{g_{00}}}$ Borissova and Rabounski derive the
following quadratic equation:

$\left(\frac{dt}{d\tau}\right)^{2}-\frac{2\nu_{i}v^{i}}{c^{2}\left(1-\frac{w}{c^{2}}\right)}\frac{dt}{d\tau}+\frac{1}{\left(1-\frac{w}{c^{2}}\right)^{2}}\left(\frac{1}{c^{4}}\nu_{i}\nu_{k}v^{i}v^{k}-1\right)=0$
(5)

the two roots of which are:Rabounski

$\left(\frac{dt}{d\tau}\right)_{1,2}=\frac{1}{1-\frac{w}{c^{2}}}\left(\frac{1}{c^{2}}\nu_{i}v^{i}\pm
1\right)$ (6)

This equation has three possible solutions $dt/d\tau>0$, $dt/d\tau<0$, and
$dt/d\tau=0$. In our world, $dt/d\tau>0$ and time flows from the past to the
future . In the mirror world $dt/d\tau<0$ and so time flows in the opposite
direction. Between the two is a membrane where time has stopped $dt/d\tau=0$.
Thus the two worlds are separate, because of the membrane, but equal. So that
to an Observer (in our world), time in the mirror world flows from the future
to the past.

The membrane which separates the two worlds, has its own unique three-fold
structure. On our world side and the mirror world side, are streams of light-
like particles (photons), moving at the speed of light, but with opposite
energies and frequencies. Between the two in the membrane, time has stopped
because $dt/d\tau=0$, and so this region is a void which is purely spacial.
However, in this void there are light-like vortices, previously unknown, which
have zero relativistic masses (unlike photons which, although massless, have
non-zero relativistic masses). These light-like vortices move and rotate at
the speed of light, but have no energy because for them time has stopped -
they are purely spacial.

Any mass-bearing particle has two time projections, one in each world, and
exists in two observable states. Each particle is in effect a four dimensional
dipole object, which exists in two states: in our world with positive mass and
energy; in the mirror world with negative mass and energy (NB this negative
mass state is not anti-matter, because the inertial mass of anti-matter is
positive). However, they cannot ”annihilate“ or rather ”nullify“ (since the
net energy is zero) because they are separated by the membrane. Furthermore
our world and the mirror world have the same background space, and the three-
dimensional momentum remains positive in both sectors. More details are given
in the references above.

We refer to this theory of physically observable quantities, as ”chronometric
invariant General Relativity“ or CIGR.

###  IV.1 Extensions of CIGR and Thermodynamics

Despite its history, chronometric invariants seems to be almost
unknown.stringA ; stringB  One possible explanation is the timing of its
publication (1956 and 1958), after Einstein’s death and before major
observational discoveries in astronomy (X-ray sources in 1962; quasars in
1963; the 3∘ K background in 1965; pulsars in 1967; and the black hole Cygnus
X-1 in 1972). As a result, general relativity has been widely applied to these
large-scale, extreme phenomena, which makes it difficult to unify it with the
small-scale phenomena of quantum mechanics. This has been further reinforced
by the discovery of gravitational waves.

We note that in CIGR, our world and the mirror world have the same background
space. We make the following deductions from this.

1\. Time in the mirror world is a macroscopic time dimension.

Furthermore, the two worlds or sectors in this theory are separated by the
triple-layer membrane, with time flowing in opposite directions on either
side.

2\. Photons are trapped on either side of this membrane because there is no
time dimension in the void in between. Therefore photons cannot traverse this
membrane, and so it will reflect or scatter light like a mirror.

3\. The space-time structure of this theory has thermodynamic consequences.
Given that entropy in our world increases with our time, it seems reasonable
to assume that entropy in the mirror world also increases with that arrow of
time. However, mirror-world time flows from our future to the past, so it
logically follows that an Observer in our world would see the entropy in the
mirror world to be constant or decrease with our time. Thus in the
chronometrically invariant form of general relativity, the second law of
thermodynamics, in its present form, only applies in our world. When the
mirror sector is included, the second law would appear to be dual, with
respect to our time.

##  V Photo Mirror Hypothesis

We now combine together Brittin and Gamow’s theory with CIGR. We make the
hypothesis that under certain circumstances, light can switch space-time into
the mirror world state, by means of the Brittin and Gamow effect, because this
reduces the entropy level which reverses the direction of time. We predict
this will occur locally where each photon interacts. This reversal could be
momentary or persistent depending on the phenomenon being observed.

Note that this is a low energy effect for two reasons. Firstly according to
CIGR, any massive particle exists in a 4-dimensional dipole state with
positive mass and energy in our world and negative mass and energy in the
mirror world. Since the mirror world state already exists, it does not require
any energy to produce it. All that is required is the reversal of the
direction of time to reveal it. Secondly the reversal of the flow of time
simply requires photons with energies of a few electron volts to lower the
entropy level according to the Brittin and Gamow effect (equation 1).

The reader may wonder why, if photons can switch matter into the mirror world
state, it has not been observed before. Firstly, the effect is subtle and
occurs at very low energies. Secondly, physicists are so convinced that the
second law of thermodynamics is absolute, that few have looked for the
creation of order. Thirdly, this effect has been observed in experiments using
light to reveal magnetic charges in mirror space-time, but these have been
rejected or ignored, because they are at low energies and subjective (i.e. not
objective).

###  V.1 Explanation of Magnetic Monopoles

In his second paper on monopoles, Dirac states that Ehrenhaft’s observation of
magnetic poles ”is not a confirmation of the present theory, since Ehrenhaft
does not use high energies“.DiracB  It seems that Dirac was convinced that
magnetic monopoles would only be observed at high energies because of the very
strong force between pairs of monopoles. (High energies also imply
objectiveness.) However, in this paper Dirac also concludes that not only may
two monopoles be connected by an infinite string, but alternatively a monopole
may be connected to a string extending to infinity. If he had taken this more
seriously, he would have concluded that the monopoles could be in a separate
space, and so not necessarily a high energy phenomenon.

We have concluded above that under intense illumination, observation of
magnetic monopoles is reproducible, so they are physically real, but do not
appear to exist in 4-D space-time. We suggest that photons of the intense
illumination switch the direction of time in the space-time containing the
ferromagnetic particles (via the Brittin and Gamow effect), into the mirror
world space-time, where the magnetic monopoles exist and are observed.
Therefore the intense illumination does not ”make magnetism“ as Ehrenhaft
claimed, but ”reveals magnetic monopoles“ in this other space-time.

The microparticles measured by Mikhailov were composite (M $\leq 10^{-14}$
gram), so the monopoles could be composite pseudo-particles (instantons).
However, the charge of these pseudo-particles would not then be quantised, and
certainly not with the monopole charge of Dirac which Mikhailov
observed.Mikhailov4

Furthermore, Dirac’s theory provides us with a topological picture of this
phenomenon. If the monopoles are in one space, and the dipole is in another,
then the Dirac string between a monopole and the corresponding pole of the
dipole, is naturally infinitely long. Therefore observation of monopoles in
mirror space-time and magnetic dipoles in normal 4-D space-time, provides a
natural physical explanation of the infinite length of the Dirac string, and
so it confirms this aspect of his theory. In view of these results, Ehrenhaft,
Benedikt and Leng, and Mikhailov really did observe Dirac monopoles at these
very low energies.

##  VI Predictions

By incorporating the Observer into general relativity, CIGR includes
thermodynamics and makes predictions about smaller scale phenomena (as opposed
to black holes), such as low entropy states in the mirror world with its
second time dimension. We predict that this inclusion of thermodynamics makes
it possible to unify CIGR with quantum mechanics.

We predict that mirror space-time will became the basis for the hidden sector
predicted by most grand unified theories. We predict that any new force(s) in
this mirror sector will lower entropy levels, since mirror time flows from the
future to the past.

We predict that phenomena in this second sector will be different from those
in the normal sector currently studied by high energy physics. For example,
they will tend to be subjective, collective (low entropy), low energy
phenomena, rather than objective, separate particles (high entropy) currently
observed at high energies. Therefore experimental methods to detect phenomena
in the mirror sector, will be different from those currently used in high
energy physics.

##  VII Conclusions

We have presented reproducible evidence for magnetic monopoles which appear to
exist outside 4-D space-time. We conclude that the current method of
experimental physics is flawed, because it limits observations to 4-D space
time. Phenomena beyond 4-D space-time currently are rejected. The solution is
to relax the criterion of objectivity, and recognise reproducible phenomena as
being physically real. This is especially the case if there is a theory for
that phenomenon.

On this basis, magnetic monopoles have been observed which have the charge of
Dirac monopoles ($\bar{g}$ = (3.27 $\pm$ 0.16) x 10-8 gauss.cm2 = 0.99 gD with
an accuracy of $\pm$5%). Furthermore, this charge is quantised ($g=ng_{D}$
with n = 1 to 5). Therefore Dirac monopoles have been observed. However,
whilst this phenomenon is not objective, and therefore outside 4-D space-time,
nevertheless it is reproducible and therefore physically real.

We then combine the Brittin and Gamow effect and CIGR to make the photo-mirror
hypothesis, namely that light lowers the entropy level and reverses the
direction of time, thereby switching matter into mirror space-time of CIGR,
where time flows from the future to the past. Therefore the photons of the
intense illumination switch the ferromagnetic particles, via the photo-mirror
hypothesis, into the mirror world space-time, where the magnetic monopoles are
observed. In this way, the intense illumination reveals magnetic monopoles in
mirror space-time. In effect, under certain circumstances, light gives us a
window into another world.

We conclude that observation of magnetic monopoles only in mirror space-time
and dipoles only in normal 4-D space-time, provides a natural physical
explanation for the infinite length of the Dirac string, and confirms this
aspect of his theory. Furthermore, this is evidence for phenomena beyond 4-D
space-time.

The author has obtained independent experimental evidence for the photo-mirror
hypothesis, which will be published separately.Ellis1 ; Ellis2  This justifies
its use above to explain the magnetic monopole data. The photo-mirror
hypothesis links a quantum mechanical effect (Brittin and Gamow) with general
relativity (CIGR), which implies unification. Furthermore, magnetic monopoles
are predicted by many unified theories. Therefore evidence for these phenomena
suggests unification.

###  VII.1 Limitations

The author is an experimentalist. The above theory (CIGR) has limitations,
because it has not yet been properly unified with quantum mechanics, nor had
the standard model embedded within it. Therefore, some of the details and
predictions of CIGR may change.

##  VIII Acknowledgements

The author thanks Dmitri Rabounski for helpful communications.

## References

  * (1) J. Jaeckel and A. Ringwald, Annu. Rev. Nucl. Part. Sci. 60, 405 (2010).
  * (2) M. Tanabashi et al.(Particle Data Group), Phys. Rev. D98, 030001 (2018).
  * (3) G. Giacomelli et al, DFUB 2000-9 Bologna, [arXiv:hep-ex/0005041].
  * (4) P.A.M. Dirac, Proc. Roy. Soc. A 133, 60 (1931).
  * (5) F. Ehrenhaft, Phys. Zeitschr. 31, 478 (1930).
  * (6) P.A.M. Dirac, Phys. Rev. 74, 817 (1948).
  * (7) H. Kragh, Dirac: A Scientific Biography (Cambridge University Press, 1990), pp. 216-217.
  * (8) E.T. Benedikt and H.R. Leng, Phys. Rev. 71, 454 (1947).
  * (9) F. Ehrenhaft, Ann. de Physique (Paris), 11/13, 151 (1940).
  * (10) F. Ehrenhaft and L. Banet, Nature, 147, 297 (1941).
  * (11) F. Ehrenhaft, Nature, 154, 426 (1944).
  * (12) F. Ehrenhaft, Phys. Rev. 65, 287 (1944)
  * (13) The late V.L. Telegdi, Private communication.
  * (14) Dibner Library, Smithsonian, Washington DC, MSS 2898, Einstein to Ehrenhaft, 20 Feb. 1940.
  * (15) T.E. Kemple, The Magnetic Monopole, MSc Thesis, Univ. Missouri, School of Mines and Metallurgy, (1961) pp 36-44 and references therein; scholarsmine.mst.edu/masters_theses/2765/
  * (16) V.F. Mikhailov, Phys. Lett. B 130, 331 (1983).
  * (17) V.F. Mikhailov, J. Phys. A: Math. Gen. 18 L903 (1985).
  * (18) V.F. Mikhailov, An. de la Fond. Louis de Broglie 12, 491 (1987).
  * (19) D. Akers, Int. J. Th. Phys. 27, 1019 (1988).
  * (20) V.F. Mikhailov, J. Phys. A: Math. Gen. 24 53 (1991).
  * (21) V.F. Mikhailov, ”On Interpretation of the Magnetic Charge Effect on Ferromagnetic Aerosols“, Preprint HEPI 88-05, Kazakh Academy of Sciences, Alma-Ata, USSR (1988) see figure 2.
  * (22) J. Schwinger, Phys. Rev. 144 (1966) 1087-1093.
  * (23) V.F. Mikhailov, ”Experimental Detection of Dirac’s Magnetic Charge? “, J. Phys. D: Applied Physics, 29 (1996) 801-804.
  * (24) W. Brittin and G. Gamow, Negative Entropy and Photosynthesis. Proc. of the Nat. Acad. of Sciences, 47 (1961) 724.
  * (25) E. Schrödinger, What is Life? (Cambridge University Press, 1944).
  * (26) I. Prigogine, Bull. Acad. Roy. Belg. Cl. Sci. 31 (1945) 600.
  * (27) A. S. Eddington, The Nature of the Physical World (Cambridge University Press, 1928).
  * (28) G. F. R. Ellis, The arrow of time and the nature of spacetime“, [arXiv:1302.7291v2].
  * (29) M. Cortes and L. Smolin, ”Reversing the Irreversible: from limit cycles to emergent time symmetry“, Phys. Rev. D 97, 026004 (2018).
  * (30) C. Rovelli, ”Space and Time in Loop Quantum Gravity“, [arXiv:1802.02382v1] and references therein
  * (31) R.J. Ellis, ”Preliminary Evidence for a Second Time Dimension Directed from the Future to the Past“, submitted for publication.
  * (32) I. Bars, ”Duality and hidden dimensions“, Frontiers in Quantum Field Theory (World Scientific, Singapore, 1996), p 52, [arXiv:hep-th/9604200].
  * (33) I. Bars, Phys. Rev. D54 (1996) 5203, [arXiv:hep-th/9604139].
  * (34) I. Bars, Phys. Rev. D55 (1997) 2373, [arXiv:hep-th/9607112].
  * (35) I. Bars, [arXiv:hep-th/9608061].
  * (36) C. Vafa, Evidence for F-Theory, Nucl. Phys. B469 (1996) 403, [arXiv:hep-th/9602022].
  * (37) C. Köhn, ”A Solution to the Cosmological Constant Problem in Two Time Dimensions“, J. HEP, Grav. and Cosm., 6, (2020) 640-655.
  * (38) I. Bars, Gauge Symmetry in Phase Space Consequences for Physics and Space-Time, Int. J. Mod. Phys. A 25 (2010), pp 5235-5252.
  * (39) Landau L. D. and Lifshitz E. M. The classical theory of fields, (GITTL, Moscow, 1939, see 4th final edition, Butterworth-Heinemann, 1980).
  * (40) A.L. Zelmanov, ”Chronometric invariants and the accompanying frames of reference in the General Theory of Relativity“, Soviet Physics Doklady, (1956) vol 1, 227-230.
  * (41) A.L. Zelmanov, Chronometric Invariants, Translation of 1944 Dissertation, CERN, EXT-2004-117.
  * (42) C. Cattaneo, Nuov. Cim. 10 (1958) pp 318-337.
  * (43) C. Cattaneo, Nuov. Cim. 11 (1959) pp 733-735.
  * (44) C. Cattaneo, Nuov. Cim. 13 (1959) pp 237-240.
  * (45) L. Borissova and D. Rabounski, Fields, Vacuum and the Mirror Universe (Svenska Fysikarkivet, 2009) Chapters 1 and 6; CERN, EXT-2003-025.
  * (46) D. Rabounski and L. Borissova, Particles here and beyond the Mirror, 3rd Edition (American Research Press, NM, USA, 2012); or 2nd Edition (Svenska Fysikarkivet, 2009); or 1st Edition [arXiv:gr-qc/0304018v4].
  * (47) L. Borissova and F. Smarandache, ”Positive, Neutral and Negative Mass-Charges in General Relativity“, Progress in Physics, 3 (2006) 51-54.
  * (48) M. Pollock, ”Chronometric Invariance and String Theory“, Mod. Phys. Letts. A 23 (2008) pp 797-813.4
  * (49) M. Pollock, ”Quantum gravity in the chronometrically invariant formalism of Zel’manov“, IAEA-ICTP-90-69 (1990).
  * (50) R.J. Ellis, ”Preliminary Evidence for a New Type of Radiation in Sunlight“, submitted for publication.