# Refractive index for the mechanical refraction of a relativistic particle

Bikram Keshari Behera  Surendra Kumar Gour111Present address: Department of
Physics, Indian Institute of Technology Guwahati, Assam-781039, India Shyamal
Biswas  sbsp [at] uohyd.ac.in School of Physics, University of Hyderabad, C.R.
Rao Road, Gachibowli, Hyderabad-500046, India

###### Abstract

We have analytically determined the refractive index for the mechanical
refraction of a relativistic particle for its all possible speeds. We have
critically analysed the importance of Descartes’ metaphysical theory and
extended it in this regard. We have considered the conservation of the
tangential component of the relativistic momentum and the relativistic energy
of the particle in the process of the mechanical refraction within the
optical-mechanical analogy. Our result for the mechanical refractive index
exactly matches with the forms of both the Fermat’s result on Snell’s law of
optical refraction at the ultra-relativistic limit and the Descartes’
metaphysical result on the pseudo-Snell law of optical refraction at the non-
relativistic limit.

###### pacs:

42.25.Gy (Edge and boundary effects; reflection and refraction), 01.55.+b
(General physics), 45.20.dh (Energy conservation), 03.30.+p (Special
relativity)

##  I Introduction

Historically, the law of refraction of light was discovered from the
experimental observations by Ibn Sahl in the year 984 Snell1  and Snellius in
the year 1621 Snell2 . The law of optical refraction, however, is known as
Snell’s law: $\frac{\sin(\theta_{i})}{\sin(\theta_{r})}=n_{ri}$, according to
which - the ratio of the sine of the angle of incidence ($\theta_{i}$) to the
sine of the angle of refraction ($\theta_{r}$) is a non-universal constant
($n_{ri}$) which depends on the optical properties of the two media Snell2 .
Later the non-universal constant $n_{ri}$ was equated with the ratio of the
speed of light in the medium of incidence ($c_{i}$) to that in the refracting
medium ($c_{r}$) by Huygens within the geometrical optics limit222In the
limiting case of the geometrical optics the wavelength ($\lambda$) of a
monochromatic wave in an optical medium is so small compared to the lowest
dimension ($d$) of an aperture in the medium that it satisfies
$\lambda/d\rightarrow 0$. of Huygens-Fresnel principle in 1678 Huygens ;
Fresnel  soon after Rømer’s discovery of the finiteness of the speed of light
Roemer . The phase speed of light, however, was not known this time and the
speed of light had been treated like the speed of a particle even before and
after this work. Later in 1704 Newton defined the non-universal constant
($n_{ri}$) as the relative refractive index of the refracting medium with
respect to the medium of incidence in connection with the colour theory and
the corpuscular theory Newton . The refractive index of an optical medium
($n$) was defined much later in 1807 by Young as a ratio of the speed of light
in free space ($c$) to the (phase) speed of light in the refracting medium
($c_{r}$) Young . The wave theory of light significantly progressed at around
and after this time especially after Fresnel’s 1818’s work on the diffraction
of light Fresnel . The phase (angle) and the phase velocity of light were
realized by Fresnel in connection with this work. Since then the speed of
light in a medium had been treated as its phase speed Fresnel2  until its
distinction from the group speed was made by Hamilton in 1839 Hamilton ;
Rayleigh . The modern form of Snell’s law is expressed as
$n_{r}\sin(\theta_{r})=n_{i}\sin(\theta_{i})$ where $n_{i}$ is the refractive
index of the medium of incidence and $n_{r}$ is the refractive index of the
refracting medium which are expressed in terms of the permittivities and
permeabilities of the respective optical media Born  after the formulation of
Maxwell’s theory of electromagnetism in 1865 Maxwell . The refractive index of
an optical medium, of course, is considered after Maxwell’s theory as the
ratio of the phase speed of light (electromagnetic wave) to the speed of light
in the free space Born . Surprisingly, the precession measurement of the
wavelength dependency of the refractive index of an optical medium is still of
experimental interest Hansinger . However, there had been an important
progress in the determination of the relative refractive index (i.e. non-
universal constant) in terms of the ratio of the phase speeds of light in the
two media during the time duration between Snell’s discovery (in 1621 Snell2 )
and Rømer’s discovery (in 1676 Roemer ). This progress was made by Descartes
Descartes  and Fermat Fermat  by considering the speed of light to be finite
in the optical media.

Descartes obtained pseudo-Snell law Descartes

$\displaystyle\frac{\sin(\theta_{i})}{\sin(\theta_{r})}=\frac{v_{r}}{v_{i}}$
(1)

for the refraction of light from theoretical (metaphysical) point of view in
1637 by considering the conservation of the tangential333Here tangential
means, tangential to the interface of the two media. component of the velocity
of a particle in the process of mechanical refraction and further considering
a metaphysical form of the optical-mechanical analogy which now-a-days maps
the group speed of light in the refracting medium ($v_{r}$) and that in the
medium of incidence ($v_{i}$) to the respective particle velocities in the two
media. The modern form of the optical-mechanical analogy, however, was
proposed long after by Hamilton in 1834 Hamilton2 . The phase speed and group
speed though are the same in a non-dispersive optical medium (free space),
they are different in a dispersive optical medium and still is a subject of
experimental study Danielmeyer ; Hansinger ; Babicz . Descartes’ derivation of
the pseudo-Snell law was the first attempt for the determination of the
relative refractive index ($n_{ri}=\frac{v_{r}}{v_{i}}=\frac{c_{r}}{c_{i}}$)
in terms of the optical properties of the two non-dispersive media. The
pseudo-Snell law, however, was rejected by Fermat in 1662 by introducing the
principle of least “time”444Here, the time is truly not least. Maupertuis’
least abbreviated action principle ($\delta\int_{q_{1}}^{q_{2}}pdq=0$
Maupertuis ; Landau ) can, however, be applied to a photon of (relativistic)
energy $E=\hbar\omega$ in an optical medium of refracting index $n$. The
magnitude of the (relativistic) momentum of the photon is given by $p=E/v_{p}$
where $v_{p}$ is the phase speed of the photon in the medium. Since the energy
is conserved in the process of refraction, we can recast Maupertuis’ least
abbreviated action principle for the photon as
$\delta\int_{q_{1}}^{q_{2}}\frac{dq}{v_{p}}=0$. The infinitesimal displacement
of the photon ($dq$) over the infinitesimal time $dt$ is not same as $v_{p}dt$
because the phase speed does not represent the actual speed of a photon
(particle) in a medium. The difference of these two, of course, was not known
during Fermat’s time. However, we can define $d``t"=\frac{dq}{v_{p}}$ as an
infinitesimal abbreviated time. Thus we have
$\delta\int_{q_{1}}^{q_{2}}d``t"=0$. This is Fermat’s least “time” principle.
By multiplying $c$ both the sides, it can be further recast as
$\delta\int_{q_{1}}^{q_{2}}d``q"=0$ where $d``q"=\frac{c}{v_{p}}dq=ndq$ is an
infinitesimal optical path-length. Thus the least“time” principle can also be
called as the least optical path-length principle. for light Fermat . After
the principle of least “time”, Snell’s law of refraction takes the form Fermat

$\displaystyle\frac{\sin(\theta_{i})}{\sin(\theta_{r})}=\frac{c_{i}}{c_{r}}.$
(2)

This form of Snell’s law correctly determines the relative refractive index
($n_{ri}=\frac{c_{i}}{c_{r}}=\frac{c/c_{r}}{c/c_{i}}=\frac{n_{r}}{n_{i}}$) for
a dispersive optical medium. The same form of Snell’s law was also obtained
later within Huygens’ principle Huygens  and it takes the modern form of
Snell’s law ($n_{r}\sin(\theta_{r})=n_{i}\sin(\theta_{i})$ Born ).
Eventuality, Descartes’ derivation of the pseudo-Snell law is hardly given any
importance now-a-days. However, Descartes’ idea was brilliant. Unfortunately,
the application of this idea went towards a wrong direction due to the non-
existence of the special theory of relativity which, however, was introduced
much later by Einstein in 1905 Einstein . A careful extension of Descartes’
method of deriving the pseudo-Snell law, however, can still be useful for the
derivation of the refractive index within a relativistic treatment not only
for the mechanical refraction but also for the optical refraction. This work
was not done so far though an attempt in arriving at Eqns. (1) and (2) was
made with the consideration of the conservations of the tangential component
of the momentum and energy of a particle and further consideration of the
dressing of the mass of the particle in the process of the mechanical
refraction in the line of thoughts of aether drag Joyce . We are overruling
the idea of the dressing of the mass in the refracting media as the existence
of aether has been made redundant after the introduction of the special theory
of relativity Einstein . Snell’s law for the optical refraction has been
obtained by Drosdoff-Widom within a particle point of view by considering
conservations of the tangential component of the momentum and energy of a
photon Drosdoff . Maupertuis’ least abbreviated action principle Maupertuis ;
Landau  has recently been used by Luca et al to arrive only at Eqns. (1) and
(2) for the mechanical refraction of a particle Luca . They did not, however,
determine the refractive index for all possible values of the speed of the
particle. We dedicate this article for a careful extension of Descartes’
method to derive the refractive index for the mechanical refraction of a
relativistic particle having an arbitrary speed and reach both the results
Eqn. (1) and Eqn. (2) at the non-relativistic and ultra-relativistic limits,
respectively.

Our article begins with the optical-mechanical analogy for a relativistic
particle. In connection with the same we consider the analogy of Helmholtz’s
wave equation for light in an optical medium and the time-independent Klein-
Gordon equation for a particle in a potential. We consider the conservation of
the tangential component of the relativistic momentum and relativistic energy
in the process of mechanical refraction of a relativistic particle. We define
the refractive index for the mechanical refraction according to the optical-
mechanical analogy from the relativistic point of view for all possible values
of the speed of the particle. We show that the ultra-relativistic limit of the
mechanical refractive index obtained by us matches with the form of Fermat’s
result on Snell’s law Fermat  and the non-relativistic limit of the same
matches with the form of Descartes’ metaphysical result on pseudo-Snell law
Descartes .

##  II Refractive index for mechanical refraction

The time-independent form of the scalar wave equation, called as Helmholtz’s
wave equation, for the propagation of light in a medium of refractive index
$n$ is given by Born2

$\displaystyle\bigg{[}\nabla^{2}+\frac{n^{2}\omega^{2}}{c^{2}}\bigg{]}\psi(\vec{r})=0$
(3)

where $\omega$ is the (angular) frequency of the scalar field $\psi(\vec{r})$
which is the spatial part of one of the components of the electromagnetic
field at the position $\vec{r}$ in the optical medium. Since the energy-
momentum relation for a relativistic particle of energy $E$, momentum
$\vec{p}$, and rest mass $m_{0}$ in a potential $V(\vec{r})$ takes the form
$(E-V)^{2}=p^{2}c^{2}+m_{0}c^{2}$, it follows the time-independent Klein-
Gordon equation for the potential $V$ as Klein-Gordon

$\displaystyle\bigg{[}\nabla^{2}+\frac{[E-V(\vec{r})]^{2}-m_{0}^{2}c^{4}}{\hbar^{2}c^{2}}\bigg{]}\psi(\vec{r})=0$
(4)

where $\psi(\vec{r})$ is the spatial part of the wave function at the position
$\vec{r}$ for the case of spin 0555With spin 0, we are considering the
particle to be a scalar particle.. While the refraction of light at the
interface of two optical media indicates a sudden change in the wave-vector
($\vec{k}$), the refraction of a particle at the interface of two media of
different potential energy would indicate a sudden change in relativistic
momentum ($\vec{p}$) according to the optical-mechanical analogy Hamilton2 .
Since the potential energy is time-independent, the relativistic energy of the
particle would be conserved in the process of the mechanical refraction. Thus
the sudden change in the relativistic momentum would be caused by a finite
discontinuity of the potential energy across the interface. We consider the
case of two constant potentials across the interface as shown in figure 1 so
that there would be no potential gradient towards the direction tangential to
the interface. This causes the conservation of the component of the
relativistic momentum tangential to the interface.

Figure 1: Schematic diagram for mechanical refraction from medium-1 to
medium-2 for $U_{2}>U_{1}$.

However, according to the optical-mechanical analogy Hamilton2 ; Cohen-
Tannoudji ; Dutta-Gupta , the wave function for the free particle in the
medium with either of the potential energy ($V(\vec{r})=U_{1}$ or
$V(\vec{r})=U_{2}$) follows Helmholtz’s wave equation

$\displaystyle\bigg{[}\nabla^{2}+\frac{n^{2}[E^{2}-m_{0}^{2}c^{4}]}{c^{2}\hbar^{2}}\bigg{]}\psi(\vec{r})=0$
(5)

with the magnitude of the momentum of a photon ($\hbar\omega/c$) in the free
space is replaced with the magnitude of the relativistic momentum
($+\sqrt{E^{2}-m_{0}^{2}c^{4}}$) of the particle in the free space
($V(\vec{r})=0$). Now comparing Eqns. (4) and (5), we get the refractive index
for the mechanical refraction as

$\displaystyle n(\vec{r})$ $\displaystyle=$
$\displaystyle+\frac{\sqrt{[E-V(\vec{r})]^{2}-m_{0}^{2}c^{4}}}{\sqrt{E^{2}-m_{0}^{2}c^{4}}}$
(6) $\displaystyle=$
$\displaystyle+\frac{\sqrt{(E-V(\vec{r})-m_{0}c^{2})(E-V(\vec{r})+m_{0}c^{2})}}{\sqrt{E^{2}-m_{0}^{2}c^{4}}}~{}~{}$

where $V(\vec{r})=U_{1}$ $\forall~{}\vec{r}\in$ medium-1 (medium of incidence)
and $V(\vec{r})=U_{2}$ $\forall~{}\vec{r}\in$ medium-2 (medium of refraction).

Eqn. (6) is our desired result for the refractive index for the mechanical
refraction. This can be called as mechanical refractive index. It is
interesting to note that the mechanical refractive index of the medium, as
defined in Eqn. (6), also takes the simpler form
$n(\vec{r})=+\frac{pc}{\sqrt{E^{2}-m_{0}^{2}c^{4}}}$ in terms of the magnitude
of the relativistic momentum of the particle in the medium. It is clear from
Eqn. (6) that the mechanical refractive index decreases if the potential
energy increases and consequently, the magnitude of the relativistic momentum
of the particle also decreases. This means that when moving from free space
into a medium with positive potential energy, the angle of refraction is
greater than the angle of incidence as depicted in figure 1. Such a decrement
of the refractive index corresponding to the increment of the potential energy
of a particle is a well-known phenomenon in neutron interferometry Rauch .

The result obtained in Eqn. (6) though is a relativistic one, it is a purely
classical result as because the optical-mechanical analogy is valid only in
the domains of the geometrical optics (wavelength $\lambda\rightarrow 0$) and
the classical mechanics ($\hbar\rightarrow 0$). The time-independent Klein-
Gordon equation and Helmholtz’s wave equation have been used just to define
the refractive index (in the limiting case of $\hbar\rightarrow 0$) not to
investigate any quantum mechanical effect. The refractive index as obtained in
Eqn. (6), however, is nothing but the ratio of the magnitude of relativistic
momentum of a particle in a medium to the magnitude of the relativistic
momentum of the particle in the free space ($V(\vec{r})=0$). We plot Eqn. (6)
in figure 2 to show the potential energy dependence of the refractive index
(solid line) for the mechanical refraction. We also plot the potential energy
dependence of the relativistic momentum of the particle (dashed line) in the
same figure. It is clear from figure 2 that both the mechanical refractive
index and the relativistic momentum reach $0$ as $E-V-m_{0}c^{2}\rightarrow
0$. This is the non-relativistic limit for the refractive index.

The refractive index $n$ can be greater than $1$, as shown in figure 2, if the
potential energy $V$ is negative which is possible for a particle in a
potential well. Eventually, at the ultra-relativistic limit
($p/m_{0}c\rightarrow\infty$) which would be achieved by sending
$V\rightarrow-\infty$ keeping $E$ fixed, the refractive index tends to
infinity as indicated in figure 2.

In the non-relativistic regime ($p/m_{0}c\ll 1$) Eqn. (6), however, takes the
form

$\displaystyle
n(\vec{r})\simeq+\frac{1}{\sqrt{E^{2}-m_{0}^{2}c^{4}}}\sqrt{2m_{0}c^{2}[E_{nr}-V(\vec{r})]}$
(7)

where $E_{nr}=\frac{1}{2}m_{0}v^{2}+V(\vec{r})$ is the non-relativistic energy
of the particle and $v$ is the speed of the particle. It should be mentioned
in this regard that a similar form of the refractive index
($n(x)=\frac{1}{\hbar\omega}\sqrt{2m_{0}c^{2}[E_{nr}-V(x)]}$666Here
$E_{nr}=\frac{\hbar^{2}[\vec{k}\cdot{\hat{x}}]^{2}}{2m_{0}}$ and
$V(x)=-\frac{k^{2}}{2m_{0}}[n^{2}(x)-1]$.) for light was obtained by Kay-Moses
recasting Helmholtz’s wave equation in the form of time-independent
Schrödinger equation to determine the reflection and transmission coefficients
in the language of quantum mechanics for the 1-D scattering of light (of wave
vector $\vec{k}=\frac{\omega n}{c}\hat{k}$) in an engineered dielectric medium
of refractive index $n=n(x)$ Kay-Moses . This form of the refractive index is
still relevant for the theoretical Yang ; Horsley  and experimental Szameit
study of the optical tunnelling of an evanescent wave in an engineered
dielectric medium Cohen-Tannoudji ; Dutta-Gupta . However, the non-
relativistic form of the refractive index obtained in Eqn. (7) is not useful
for a highly dispersive medium where the speed of the particle (as well as the
group speed of a wave-packet) is not close to $c$. For such a case, one needs
the relativistic form of the refractive index as obtained in Eqn. (6).

###  II.1 Law of mechanical refraction

Figure 2: The solid line indicates the plot of the refractive index with
respect to the potential energy for mechanical refraction and the dashed line
indicates the plot of the magnitude of the relativistic momentum with respect
to the potential energy. While the solid line follows Eqn. (6), the dashed
line follows the relativistic energy-momentum relation
$p=+\sqrt{(E-V)^{2}-m_{0}^{2}c^{4}}$.

The mechanical refraction as described in figure 1 must follow the
conservation of relativistic energy and the tangential component of
relativistic momentum as mentioned above. Let us now explore the conservation
of these two physical quantities to find the law of mechanical refraction. For
the mechanical refraction too, we take the angle of incidence to be as
$\theta_{i}$ and the angle of refraction to be as $\theta_{r}$ as described in
figure 1. While the magnitude of the relativistic momentum ($\vec{p}_{i}$) of
the scalar particle of rest mass $m_{0}$ and relativistic energy $E$ in the
medium of incidence (medium-1) is
$p_{i}=+\frac{1}{c}\sqrt{(E-U_{1})^{2}-m_{0}^{2}c^{4}}$, the magnitude of the
relativistic momentum ($\vec{p}_{r}$) of the same particle in the medium of
refraction (medium-2) is
$p_{r}=+\frac{1}{c}\sqrt{(E-U_{2})^{2}-m_{0}^{2}c^{4}}$ according to the
conservation of the relativistic energy. Now, according to the conservation of
the component of the relativistic momenta tangential to the interface
($p_{i}\sin(\theta_{i})=p_{r}\sin(\theta_{r})=constant$), we have

$\displaystyle\frac{\sqrt{(E-U_{1})^{2}-m_{0}^{2}c^{4}}}{c}\sin(\theta_{i})=\frac{\sqrt{(E-U_{2})^{2}-m_{0}^{2}c^{4}}}{c}\sin(\theta_{r})$

in the process of the mechanical refraction as described in figure 1. By
multiplying the denominators of both the sides of above equation by the
constant $\sqrt{E^{2}-m_{0}^{2}c^{4}}/c$ we can recast it to the form

$\displaystyle n_{i}\sin(\theta_{i})=n_{r}\sin(\theta_{r})$ (9)

where
$n_{i}=+\frac{\sqrt{[E-U_{1}]^{2}-m_{0}^{2}c^{4}}}{\sqrt{E^{2}-m_{0}^{2}c^{4}}}$
and
$n_{r}=+\frac{\sqrt{[E-U_{2}]^{2}-m_{0}^{2}c^{4}}}{\sqrt{E^{2}-m_{0}^{2}c^{4}}}$
are the refractive indices of medium-1 and medium-2, respectively, and follow
Eqn. (6) for the mechanical refraction. Eqn. (9) can be called as the law of
mechanical refraction.

###  II.2 Law of mechanical refraction at ultra-relativistic and non-
relativistic limits

The velocity of the particle, can however, be determined from the Hamiltonian
$H=+\sqrt{p^{2}c^{2}+m_{0}^{2}c^{4}}+V(\vec{r})$. The velocity of the particle
can be determined from this Hamiltonian as
$\vec{v}=\dot{\vec{r}}=\frac{\partial
H}{\partial\vec{p}}=\frac{\vec{p}c}{\sqrt{p^{2}+m_{0}^{2}c^{2}}}$. From this
expression we can easily show
$\vec{p}=\frac{m_{0}\vec{v}}{\sqrt{1-v^{2}/c^{2}}}$ for all values of $v$ and
$v\rightarrow c$ in the limiting case of $p/m_{0}c\rightarrow\infty$. Such a
limiting case is referred to as the ultra-relativistic limit. Now we need to
define the phase speed of the particle, according to the optical-mechanical
analogy, as
$v_{p}=\frac{H}{p}=c^{2}/v+[V(\vec{r})\sqrt{1-v^{2}/c^{2}}]/m_{0}v$. The group
speed, on the other hand, would be the speed of the particle
$v=\frac{pc}{\sqrt{p^{2}c^{2}+m_{0}^{2}c^{2}}}$ according to the optical-
mechanical analogy. We can define the non-dispersive medium for the mechanical
refraction, according to the optical-mechanical analogy, as a medium with the
potential energy $V(\vec{r})$ such that $v_{p}=v$. This is possible only if
$V(\vec{r})=-m_{0}c^{2}\sqrt{1-v^{2}/c^{2}}$. For the refraction of a mass-
less particle (e.g. a photon) in a non-dispersive medium, we have
$V(\vec{r})=0$. However, In the case of the ultra-relativistic limit, the
phase speed of the particle takes the form $v_{p}\rightarrow
c+\frac{c}{E/V(\vec{r})-1}$. It is interesting to note that the mechanical
refractive index of the medium
$n(\vec{r})=+\frac{pc}{\sqrt{E^{2}-m_{0}^{2}c^{4}}}$, as obtained in Eqn. (6),
takes the form $n\rightarrow\frac{c}{v_{p}}$ only at the ultra-relativistic
limit.

####  II.2.1 Snell’s law

The mechanical refractive index of the medium-1 can be written from Eqn. (6)
as $n_{i}=+\frac{p_{i}c}{\sqrt{E^{2}-m_{0}^{2}c^{4}}}$ where $p_{i}$ is the
magnitude of the relativistic momentum of the particle in the medium-1. At the
ultra-relativistic limit ($p_{i}/m_{0}c\rightarrow\infty$), this expression of
the mechanical refractive index takes the form
$n_{i}\rightarrow\frac{c}{c_{i}}$ where $c_{i}=c\frac{E}{E-U_{1}}$ is the
phase speed of the particle at the ultra-relativistic limit. Similarly, from
Eqn. (6) we have the mechanical refractive index in medium-2 as
$n_{r}=+\frac{p_{r}c}{\sqrt{E^{2}-m_{0}^{2}c^{4}}}$ where $p_{r}$ is the
magnitude of the relativistic momentum of the particle in the medium-2. At the
ultra-relativistic limit ($p_{r}/m_{0}c\rightarrow\infty$), it takes the form
$n_{r}\rightarrow\frac{c}{c_{r}}$ where $c_{r}=c\frac{E}{E-U_{2}}$ is the
phase speed of the particle in medium-2. Thus combining the above two results
for the particle refractive indices of the two media, we get the law of
mechanical refraction from Eqn. (9) at ultra-relativistic limit
($p/m_{0}c\rightarrow\infty$) as

$\displaystyle\frac{\sin(\theta_{i})}{\sin(\theta_{r})}\rightarrow\frac{c_{i}}{c_{r}}.$
(10)

This ultra-relativistic result for the mechanical refraction takes the form of
Fermat’s result on Snell’s law Fermat  as mentioned in Eqn. (2).

In reality, a photon having zero rest mass ($m_{0}\rightarrow 0$) is an ultra-
relativistic particle though its relativistic momentum is finite
($p=n\hbar\omega/c$). So, a photon with a single spin component must follow
Eqn. (10) as expected from Snell’s law for the refraction of light. However,
it does not necessarily mean that the group speed of light (electromagnetic
waves) has to take the form $v\rightarrow c$, like that for an ordinary
particle with $m_{0}\neq 0$, in order to reach the ultra-relativistic limit.
The wavelength of the wave associated with the photon, however, takes the form
$\lambda\sim 2\pi\hbar/p$. Incidentally, the wavelength takes a non-zero
finite value which takes the case away from the geometrical optics limit where
the optical-mechanical analogy does not hold. Hence, the group speed of light
is not necessarily same as the speed ($v$) of a photon rather the signal speed
of light is same as the speed of a photon in an optical medium Sommerfeld . In
order to hold the optical-mechanical analogy for a photon, we need to consider
$p\rightarrow\infty$. In such a case, the group speed of light takes the form
$v\rightarrow c$ even in a dispersive medium.

####  II.2.2 Pseudo-Snell law

On the other hand, at the non-relativistic limit ($p_{i}/m_{0}c\rightarrow
0$), the mechanical refractive index of medium-1 as expressed in Eqn. (6)
takes the form
$n_{i}=\frac{p_{i}c}{\sqrt{E^{2}-m_{0}^{2}c^{4}}}\rightarrow\frac{v_{i}}{\sqrt{E^{2}-m_{0}^{2}c^{4}}/m_{0}c}$
where $v_{i}$ is the speed of the particle in medium-1. Here we are not taking
the non-relativistic limit of $\sqrt{E^{2}-m_{0}^{2}c^{4}}$ as it is a
constant in the process of refraction. Similarly, we have the mechanical
refractive index of medium-2 at the non-relativistic limit
($p_{r}/m_{0}c\rightarrow 0$) as
$n_{r}=\frac{p_{r}c}{\sqrt{E_{0}^{2}-m_{0}^{2}c^{4}}}\rightarrow\frac{v_{r}}{\sqrt{E^{2}-m_{0}^{2}c^{4}}/m_{0}c}$
where $v_{r}$ is the speed of the particle in medium-2. Thus combining the
above two non-relativistic results for the two media we get the law of
mechanical refraction from Eqn. (9) as

$\displaystyle\frac{\sin(\theta_{i})}{\sin(\theta_{r})}\rightarrow\frac{v_{r}}{v_{i}}.$
(11)

This non-relativistic result for the mechanical refraction takes the form of
Descartes’ metaphysical result on pseudo-Snell law Descartes  as mentioned in
Eqn. (1).

##  III Conclusion

To conclude, we have analytically determined the mechanical refractive index
for the refraction of a particle from the relativistic point of view. Eqn. (6)
is our key result for the same. A few other interesting results for the law of
mechanical refraction are obtained in Eqns. (9), (11), and (10). We have
employed the optical-mechanical analogy with the correspondence of Helmholtz’s
wave equation for the propagation of light in an optical medium to the time-
independent Klein-Gordon equation for the motion of a particle in a potential
for the derivation of the mechanical refractive index. The mechanical
refractive index obtained by us though is relativistic, it is purely a
classical result.

We have considered a scalar particle (spin 0) for the derivation of the
mechanical refractive index. A photon, however, is a vector particle (spin 1).
Since the refractive index of an optical medium is a property of geometrical
optics which is analogous to classical mechanics, any particle with a specific
component of spin will also have the same mechanical refractive index.
Similarly, a photon with a specific component of spin will also have the same
refractive index in the limiting case of $p/m_{0}c\rightarrow\infty$.

The optical-mechanical analogy with Helmholtz’s wave equation and Klein-Gordon
equation also mathematically allows us to take the negative square root of
$[E-V(\vec{r})]^{2}-m_{0}^{2}c^{4}$ in Eqn. (6) to get a negative refractive
index
$n(\vec{r})=-\frac{\sqrt{[E-V(\vec{r})]^{2}-m_{0}^{2}c^{4}}}{\sqrt{E^{2}-m_{0}^{2}c^{4}}}$
for the mechanical refraction. In reality, the negative refractive index is a
property of a metamaterial Bose  under an optical refraction for a certain
frequency range ($\omega\sim$ GHz - THz) for which both the permittivity and
the permeability of the medium are negative Veselago ; Shelby . Such a
property, however, is not seen for a very large frequency. The optical-
mechanical analogy we have adopted here is applicable for a very large
frequency (ideally $\omega\rightarrow\infty$). This analogy, however, does not
allow any one-to-one correspondence of the negative permittivity and the
negative permeability to the mechanical quantities ($E,V(\vec{r})$, and
$m_{0}$). Thus we don’t have an analogy of the negative mechanical refractive
index with the negative optical refractive index.

The motion of the particle in the process of refraction though follows
Hamilton’s least action principle
($\delta\int_{t_{1}}^{t_{2}}L(q,\dot{q},t)=0$ Hamilton3 ; Landau2 ; Rojo )
when the time is taken into account for the path of the particle and
Maupertuis’ least abbreviated action principle
($\delta\int_{q_{1}}^{q_{2}}pdq=0$ Maupertuis ; Landau ) when the time is not
taken into account for the trajectory of the particle Landau , the refraction
is a consequence of the conservation of the relativistic energy and the
tangential component of the relativistic momentum. The conservation of the
tangential component of the wave vector of the electromagnetic field (which is
analogous to the momentum of a photon) is also used as boundary conditions to
obtain Snell’s law from Maxwell’s equations Maxwell  for the optical
refraction across two linear media with no free charge and free current at the
interface in the limiting case of the geometrical optics Born ; Griffiths . In
general, the normal component of the electric field and the tangential
component of the magnetic field are discontinuous at the interface for an
optical refraction Griffiths . The normal component of the electric
displacement vector, however, is continuous for the optical refraction across
two linear dielectric media Griffiths .

In optics, we usually work with angular frequency $\omega$ and the wave-vector
$\vec{k}$ of an electromagnetic wave rather than the energy ($E=\hbar\omega$)
and momentum ($\vec{p}=\hbar\vec{k}$) of a photon. While the angular frequency
of the incident electromagnetic wave is unaltered in the process of optical
refraction, the relativistic energy of the particle is conserved in the
process of the mechanical refraction. The wavenumber ($k=n\omega/c$) and the
magnitude of the relativistic momentum, on the other hand, on the other hand,
are not conserved in the process of the optical refraction and the mechanical
refraction, respectively. However, the electromagnetic field also has a
momentum called either Minkowski momentum
($\vec{p}_{\text{Min}}=\int\text{d}^{3}\vec{r}\vec{D}\times\vec{B}$) or
Abraham momentum
($\vec{p}_{\text{Abr}}=\int\text{d}^{3}\vec{r}\frac{\vec{E}\times\vec{H}}{c^{2}}$)
Barnett . On the other hand, the medium also has a momentum due to the
coupling of the electromagnetic field with the electric and magnetic dipoles
in it. While the Abraham momentum is of kinetic momentum type, the Minkowski
momentum is of canonical momentum type. For a photon, while the canonical
momentum in a medium takes the form $\vec{p}=\frac{n\hbar\omega}{c}\hat{k}$,
the kinetic momentum in the same medium takes the form
$\vec{p}_{\text{kin}}=\frac{\hbar\omega}{cn}\hat{k}$. The total (kinetic or
canonical) momentum i.e. the field momentum and the momentum of the medium,
however, is a conserved quantity in the process of the optical refraction
Barnett . We, of course, don’t have such a vector field description for the
mechanical refraction of a scalar particle.

For a photon with a single spin component in a non-dispersive medium, the
least action principle, however, becomes the least “time” principle Luca .
Hence Fermat’s theory of the least “time” Fermat  is a relativistic theory
though the actual 4-vector formulation of the theory of relativity Einstein ;
Resnick  was done long after his death. Huygens-Fresnel principle Huygens ;
Fresnel ; Born3 , of course, goes one step further for the wave optics. All
the theories of light which are found correct, are by default relativistic
theory as because photon is an ultra-relativistic particle. So, the derivation
of Snell’s law of refraction needs a relativistic treatment.

Descartes’ metaphysical theory for the refraction of light was essentially a
non-relativistic theory and was inconsistent with the phase speed of light in
a denser or rarer medium Descartes . Incidentally this metaphysical theory was
rejected by Fermat introducing the principle of least “time” Fermat . Since
then Descartes’ metaphysical theory of the refractive index has not been
respected well. Descartes considered the conservation of the component of the
ordinary velocity (of both the particle and light) tangential to the interface
of the two media for the process of refraction Descartes . This idea, however,
was brilliant. We have extended this idea and considered the conservation of
the tangential component of the relativistic momentum along with the
relativistic energy for the derivation of the mechanical refractive index.
While the ultra-relativistic limit of the mechanical refractive index obtained
by us matches with the form of Fermat’s result on Snell’s law Fermat , the
non-relativistic limit of the same matches with the form of Descartes’
metaphysical result on pseudo-Snell law Descartes .

The mechanical refractive index obtained by us may find potential applications
towards the transmission and reflection of particles by engineered
(mechanical) media in analogy with the optical transmission and reflection by
engineered dielectric media as mentioned in the introductory section.
Semiclassical study of the transmission and reflection coefficients for a
particle in an engineered media can be studied both in non-relativistic regime
and ultra-relativistic regime by using the mechanical refractive index (Eqn.
(6)) obtained by us. A classical path integral formulation was considered to
determine the refractive index of an optical medium in terms of multiple
scattering of a photon due to the atoms/molecules in the medium Grooth . A
quantum mechanical study Mead  of the same with the path integral approach
(beyond the optical-mechanical analogy) for the mechanical refractive index is
kept as an open problem.

### Acknowledgement

S. Biswas acknowledges partial financial support of the SERB, DST, Govt. of
India under the EMEQ Scheme [No. EEQ/2023/000788].

## References

  * (1) R. Rashed, A pioneer in anaclastics: Ibn Sahl on burning mirrors and lenses, ISIS 81, 464 (1990); A. Kwan, J. Dudley, and E. Lantz, Who really discovered Snell’s law?, Phys. World 15, 64 (2002)
  * (2) C. Huygens, Dioptrica, ed. B. de Volder and B. Fullenius (Opuscula Posthuma, Leiden, 1703)
  * (3) C. Huygens, Traité de la Lumière, (Van der Aa, Leiden 1690); English Translation: S. P. Thompson, Treatise on Light (Macmillan, London, 1912)
  * (4) A. Fresnel, Mémoire sur la diffraction de la lumière, Mémoires de l’Académie Royale des Sciences de l’Institut de France (Paris) 5, 339 (1826)
  * (5) O. Römer, Démonstration touchant le mouvement de la lumière trouvé par, J. des Sçavans, 276 (Paris, 7 Dec. 1676); English Translation: A demonstration concerning the motion of light, Philos. Trans. R. Soc. (London) 12, 893 (1677)
  * (6) I. Newton, Opticks: or, A treatise of the reflexions, refractions, inflexions and colours of light. Also two treatises of the species and magnitude of curvilinear figures, (Printed for S. Smith and B. Walford, London, 1704)
  * (7) T. Young, A course of lectures on natural philosophy and the mechanical arts, vol. 1, lec. 35: on the theory of optics, p. 413 (Printed for J. Johnson, London, 1807)
  * (8) A. Fresnel, Mémoire sur la Loi des modifications que la réflexion imprime à la lumière polarisée, Mémoires de l’Académie Royale des Sciences de l’Institut de France (Paris) 11, 393 (1832)
  * (9) W. R. Hamilton, Researches respecting vibration, connected with the theory of light, Proc. Royal Irish Academy 1, 341 (1841). See Ref. Rayleigh  for the group speed of sound waves.
  * (10) J. W. Strutt (B. Rayleigh), The Theory of Sound, vol. 1, sec. 190 & 191, pp. 244-249 (Macmillan, London, 1877); vol. 2, pp. 297-302 (Macmillan, London, 1878); The vol. 1 and vol. 2 are reprinted (Cambridge University Press, Cambridge, 2011).
  * (11) M. Born and E. Wolf, Principles of Optics, 7th ed., sec. 1.5.1, pp. 38-40 and sec. 3.3.2, pp. 132-133 (Cambridge University Press, Cambridge, 1999)
  * (12) J. C. Maxwell, A dynamical theory of the electromagnetic field, Philos. Trans. R. Soc. (London) 155, 459 (1865)
  * (13) P. Hansinger, P. Töpfer, N. Dimitrov, D. Adolph, D. Hoff, T. Rathje, A. M. Sayler, A. Dreischuh, and G. G. Paulus, Refractive index dispersion measurement using carrier-envelope phasemeters, New J. Phys. 19, 023040 (2017)
  * (14) R. Descartes, La Dioptrique, Discours II: de la Réfraction (I. Maire, Leiden, 1637). Reprinted in Oeuvres de Descartes: Discours de la Méthode & Essais VI, ed. C. Adam and P. Tannery, pp. 93-105 (Léopold Cerf, Paris, 1902)
  * (15) P. de Fermat, A letter on réfraction to C. de la Chambre (Paris, 1 Jan. 1662). Reprinted in Oeuvres de Fermat, ed. P. Tannery and C. Henry, vol. 2, pp. 457-463 (Gauthier-Villars, Paris, 1894)
  * (16) W. R. Hamilton, On the application to dynamics of a general mathematical method previously applied to optics, British Association Report 4777Here 4 means the 4th meeting., 513 (1834). A. S. Kompaneyets, A Course of Theoretical Physics, vol. 1 (Fundamental Laws), sec. 21, pp. 269-278 (Mir, Moscow, 1978). Also see Ref. Hamilton3  for the same.
  * (17) H. G. Danielmeyer and H. P. Weber, Direst measurement of the group velocity of light, Phys. Rev. A 3, 1708 (1970)
  * (18) M. Babicz, S. Bordoni, A. Fava, U. Kose, M. Nessi, F. Pietropaolo, G. L. Raselli, F. Resnati, M. Rossella, P. Sala, F. Stockera, and A. Zani, A measurement of the group velocity of scintillation light in liquid argon, JINST 15, P09009 (2020)
  * (19) A. Einstein, Zur elektrodynamik bewegter körper, Annalen der Physik 17, 891 (1905); Ist die trägheit eines körpers von seinem energieinhalt abhängig?, Annalen der Physik 18, 639 (1905)
  * (20) W. B. Joyce and A. Joyce, Descartes, Newton, and Snell’s law, J. Opt. Soc. Am. 66, 1 (1976)
  * (21) D. Drosdoff and A. Widom, Snell’s law from an elementary particle viewpoint, Am. J. Phys. 73, 973 (2005)
  * (22) P. L. M. de Maupertuis, Accord de différentes loix de la nature qui avoient jusqu’ici paru incompatibles, Mémoires de l’Académie Royale des Sciences de l’Institut de France (Paris), 417 (15 Apr. 1744); Reprinted in Essay de Cosmologie, pp. 154-173 (Mens Agitat Molem. Æneid. Lib. VI. 1750)
  * (23) L. D. Landau and E. M. Lifshitz, Mechanics, 3rd edn., sec. 44, pp. 140-143 (Butterworth-Heinemann, Oxford, 1976)
  * (24) R. D. Luca, M. Di Mauro, A. Naddeo, Revista Brasileira de Ensino de Física 42, e20190339 (2020)
  * (25) See sec. 1.3, pp. 14-15 of Ref.Born .
  * (26) O. Klein, Quantentheorie und fünfdimensionale relativitätstheorie, Z. Physik 37, 895 (1926); W. Gordon, Der comptoneffekt nach der Schrödingerschen theorie, Z. Physik 40, 117 (1926); V. Fock, Zur Schrödingerschen wellenmechanik, Z. Physik 38, 242 (1926)
  * (27) C. Cohen-Tannoudji, B. Diu, and F. Laloë, Quantum Mechanics, 2nd edn., vol. 1, ch. 1, sec. D-2-b & D-2-c, pp. 27-30 (Wiley-VCH, Weinheim, 2020)
  * (28) S. Dutta Gupta, A. Banerjee, and N. Ghosh, Wave Optics: Basic Concepts and Contemporary Trends, sec. 9.5, pp. 183-185 (Taylor & Francis, New York, 2016)
  * (29) H. Rauch and S. A. Werner, Neutron Interferometry, 2nd edn, sec. 1.1, pp. 9-9 (Oxford University Press, Oxford, 2015)
  * (30) I. Kay and H. E. Moses, Reflectionless transmission through dielectrics and scattering potentials, J. Appl. Phys. 27, 1503 (1956)
  * (31) E. Yang, Y. Lu, Y. Wang, Y. Dai, and P. Wang, Unidirectional reflectionless phenomenon in periodic ternary layered material, Opt. Express 24, 14311 (2016)
  * (32) S. A. R. Horsley and S. Longhi, Spatiotemporal deformations of reflectionless potentials, Phys. Rev. A 96, 023841 (2017)
  * (33) A. Szameit, F. Dreisow, M. Heinrich, S. Nolte, and A. A. Sukhorukov, Realization of reflectionless potentials in photonic lattices, Phys. Rev. Lett. 106, 193903 (2011)
  * (34) A. Sommerfeld, Über die fortpflanzung des lichtes in dispergierenden medien, Annalen der Physik 44, 177 (1914); L. Brillouin, Über die fortpflanzung des lichtes in dispergierenden medien, Annalen der Physik 44, 203 (1914); A. Sommerfeld, Optics (Lectures on Theoretical Physics Vol. IV), (Academic Press, New York, 1954); Also see sec. 22, pp. 114-135 of the Indian reprint of the book (Levant Books, Kolkata, 2006)
  * (35) J. C. Bose, On the rotation of plane of polarisation of electric waves by a twisted structure, Proc. R. Soc. Lond. 63, 146 (1898)
  * (36) V. G. Veselago, The electrodynamics of substances with simultaneously negative values of $\epsilon$ and $\mu$, Sov. Phys. Usp. 10, 509 (1967)
  * (37) R. A. Shelby, D. R. Smith, and S. Shultz, Experimental verification of a negative index of refraction, Science 292, 77 (2001)
  * (38) W. R. Hamilton, On a general method in dynamics; by which the study of the motions of all free systems of attracting or repelling points is reduced to the search and differentiation of one central relation, or characteristic function, Philos. Trans. R. Soc. (London) 124, 247 (1834); W. R. Hamilton, Second essay on a general method in dynamics, Philos. Trans. R. Soc. (London) 125, 95 (1835)
  * (39) See sec. 2, pp. 2-4 of Ref.Landau .
  * (40) A. Rojo and A. Bloch, The Principle of Least Action: History and Physics, (Cambridge University Press, Cambridge, 2018)
  * (41) D. J. Griffiths, Introduction to Electrodynamics, 4th Indian edn, sec. 9.3.1, pp. 407-407, (Pearson, Noida, 2015)
  * (42) S. M. Barnett, Resolution of the Abraham-Minkowski dilemma, Phys. Rev. Lett. 104, 070401 (2010)
  * (43) R. Resnick, Introduction to Special Relativity, (Wiley, New York, (1968)
  * (44) See sec. 8.2, pp. 413-417 of Ref.Born .
  * (45) D. G. de Grooth, Why is the propagation velocity of a photon in a transparent medium reduced? Am. J. Phys. 65, 1156 (1997)
  * (46) C. A. Mead, Quantum theory of the refractive index, Phys. Rev. 110, 359 (1958)


# Mesh Based Analysis of Low Fractal Dimension Reinforcement Learning Policies

Sean Gillen1 and Katie Byl2 Sean Gillen and Katie Byl are with the Electrical
and Computer Engineering Department at the University of California, Santa
Barbara CA 93106<EMAIL_ADDRESS><EMAIL_ADDRESS>
Code hosted at: github.com/sgillen/fractal_mesh

###### Abstract

In previous work, using a process we call meshing, the reachable state spaces
for various continuous and hybrid systems were approximated as a discrete set
of states which can then be synthesized into a Markov chain. One of the
applications for this approach has been to analyze locomotion policies
obtained by reinforcement learning, in a step towards making empirical
guarantees about the stability properties of the resulting system. In a
separate line of research, we introduced a modified reward function for on-
policy reinforcement learning algorithms that utilizes a ”fractal dimension”
of rollout trajectories. This reward was shown to encourage policies that
induce individual trajectories which can be more compactly represented as a
discrete mesh. In this work, we combine these two threads of research by
building meshes of the reachable state space of a system subject to
disturbances and controlled by policies obtained with the modified reward. Our
analysis shows that the modified policies do produce much smaller reachable
meshes. This shows that agents trained with the fractal dimension reward
transfer their desirable quality of having a more compact state space to a
setting with external disturbances. The results also suggest that the previous
work using mesh based tools to analyze RL policies may be extended to higher
dimensional systems or to higher resolution meshes than would have otherwise
been possible.

##  I INTRODUCTION

Legged robots have clear potential to play an important role in our society in
the near future. Examples include contact-free delivery during a pandemic,
emergency work after an environmental disaster, or as a logistical tool for
the military. Legged robots simply expand the reach of robotics when compared
to wheeled systems. However, compared to wheeled systems, designing control
policies for legged systems is a much more complex task, especially in the
presence of disturbances, noise, and unstructured environments.

The increasing availability of massive quantities of computation has led to a
resurgence of reinforcement learning (RL) in recent years. RL provides a
promising approach for complex, under-actuated, hybrid control problems, such
as those involved in designing control for legged locomotion. Recent examples
in the context of robotics include controlling a 47 DOF humanoid to navigate a
variety of obstacles [6], dexterously manipulating objects with a 24 DOF
robotic hand [11], and allowing a physical quadruped robot to run [7], and
recover from falls [8].

Despite the obvious promise of RL approaches, several problems need to be
resolved before these systems are ready for real world applications. One of
the biggest problems is that the resulting policy is typically a complete
black box, there are no good ways to make theoretical, or even empirical
guarantees about the resulting policies. Prior work has used so called mesh
based techniques to this end [13]. Broadly, these techniques take a continuous
system and approximate it with a discrete set of states. This allows us to
model the system as a Markov chain, these systems are arguably easier to
reason about, and it opens up a new box of tools we can bring to bear on the
problem. For example we can use value iteration to switch between several
controllers to improve the robustness [12] or agility [3] of the system. We
can also perform eigen-analysis on the Markov chain’s transition matrix, which
provides us insights on the stability of the system [4]. These techniques
could both be used for policy refinement, and/or for verification and analysis
of existing policies.

However, these methods suffer from the ”curse of dimensionality”, because the
number of possible states in our mesh grows exponentially with the degrees of
freedom in our system. That is, if we make a change to the volume of
continuous space represented by each discrete state, the number of states in
our new mesh will grow exponentially with respect to the change in discrete
state size. However for virtually all plausible walking controllers, the
reachable state space is a small fraction of the total state space. Although
the scaling for meshes of the reachable state space also scale exponentially
as we increase the mesh resolution, the rate of scaling is typically much
smaller. The scaling factor for the reachable mesh can be seen as a fractal
dimension, which is elaborated upon in section II-A.

In previous work [5], we introduced a modified reward function for on-policy
reinforcement learning algorithms. The reward explicitly encourages policies
which induce trajectories which have a smaller fractal dimension. It’s worth
noting that although each individual trajectory was encouraged to have a
smaller fractal dimension, this does not obviously extend to properties of the
entire reachable state space for the system, which is what was used for the
previous mesh based analysis of RL policies.

In this work we take the next step and construct reachable state space meshes
of agents trained with and without our modified reward. Our primary
contribution is showing that these modified policies result in significantly
smaller reachable meshes for a given box size, and in smaller fractal
dimensions for the reachable state space. We then use the modified policies to
construct a much finer mesh than would be possible otherwise. We use this mesh
to compute a quantity called the mean first passage time (MFPT), and validate
the obtained MFPT with Monte Carlo trials. Finally we use our mesh to produce
interesting visualizations of failure states, which motivates future work.

##  II BACKGROUND

In this section we introduce fractional dimensions, meshing, reinforcement
learning, and our test environment. The environment is a hopping robot,
coupled with a specific reward function. Reinforcement learning is used to
train a control policy for this system which attempts to maximize the given
reward function. In previous work we used a fractional dimension to modify the
reward function, this modified reward results in policies that can be meshed
significantly more efficiently.

###  II-A Meshing and Fractal Dimensions

(a) Scaling in different dimensions

(b) A non uniform mesh

Figure 1: Image credit: [1], [12].

Let’s say we have a continuous set S that we want to approximate by selecting
a discrete set M composed of regions in S. We will call this set M a mesh of
our space. Figure 1(a) shows some examples of this: a line is broken into
segments, a square into grid spaces, and so on. The question is: as we
increase the resolution of these regions, how many more regions N do we need?
Again, Figure 1(a) shows us some very simple examples. For a D dimensional
system, if we go from regions of size d to d/k, then we would expect the
number of mesh points to scale as $N\propto k^{D}$. But not all systems will
scale like this, as Figure 1(b) illustrates. Figure 1(b) is an example of a
curve embedded in a two dimensional space. The question of how many mesh
points are required must be answered empirically. Going backwards, we can use
this relationship to assign a notion of ”dimension” to the curve.

$D_{f}=-\lim_{k\rightarrow 0}\frac{\log N(k)}{\log k}.$ (1)

This quantity is known as the Minkowski–Bouligand dimension, also called the
box counting dimension. This dimension need not be an integer, hence the name
”fractional” or ”fractal” dimension. This is one of many measures of
fractional dimensionality that emerged from the study of fractal geometry.
Although these measures were invented to study fractals, they can still be
usefully applied to non-fractal sets. For non fractal sets, we use the slope
of the log-log relation of mesh sizes to d to compute the dimension, rather
than taking a limit.

###  II-B Box Meshing

In this work, we identify any state s with a key obtained by:

$\displaystyle s_{k}=\frac{s-\mu_{s}}{\sigma_{s}}$
$\displaystyle\text{key}=\text{round}(\frac{s_{k}}{d_{thr}})d_{thr}.$ (2)

where $\mu_{s}$ and $\sigma_{s}$ are the mean and standard deviation of all
the states seen by the policy of interest during training. The round function
here performs an element-wise rounding to the nearest integer. We can then use
these keys to store mesh points in a hash table. Using this data structure, we
can store the mesh compactly, only keeping the points we come across, and
lookups are done in constant time. The parameter $d_{thr}$ is called the box
size. Geometrically we can think of this operation as dividing the state space
into a uniform grid of hypercubes, each with a side length of $d_{thr}$

###  II-C Reinforcement Learning

The goal of reinforcement learning is to train an agent acting in an
environment to maximize some reward function. At every timestep
$t\in\mathbb{Z}$, the agent receives the current state $s_{t}\in R^{n}$, uses
that to compute an action $a_{t}\in\mathbb{R}^{b}$, and receives the next
state $s_{t+1}$, which is used to calculate a reward
$r:\mathbb{R}^{n}\times\mathbb{R}^{m}\times\mathbb{R}^{n}\rightarrow\mathbb{R}$.
The objective is to find a policy
$\pi_{\theta}:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$ that satisfies:

$\operatorname*{arg\,max}_{\theta}\mathop{\mathbb{E}}_{\eta}\left[\sum_{t=0}^{T}r(s_{t},a_{t},s_{t+1})\right].$
(3)

Where $\theta\in\mathbb{R}^{d}$ is a set that parameterizes the policy, and
$\eta$ is a parameter representing the randomness in the environment. This
includes the random initial conditions for episodes.

In [5], we introduced a modified reward function:

$\operatorname*{arg\,max}_{\theta}\mathop{\mathbb{E}}_{\eta}\left[\frac{1}{D_{m}(s)}\sum_{t=0}^{T}r(s_{t},a_{t},s_{t+1})\right]$
(4)

where $D_{m}$ is the ”lower mesh dimension” explained in detail in [5], which
is an estimate of equation 1 dimension for our non fractal set.

###  II-D Environment

Our model system is openAI gym’s Hopper-v2 environment introduced in [2]. This
environment is part of a popular and standardized set of benchmarking tasks
for reinforcement learning algorithms. The system is a 4 link, 6 DOF hopper
constrained to travel in the XZ plane, seen in Figure 2. The observation space
for the agent has 11 states, the position in the direction of motion is held
out, since we seek a policy that is invariant to forward progress. The actions
in this case are commanded joint torques. The reward function for this
environment is simply forward velocity minus a small penalty to actions.
Successful controllers in this environment must execute a dynamic hopping
motion to move robot along the x axis as quickly as possible. This is clearly
a toy problem, but it captures many of the challenges of legged locomotion.
The system is highly non-linear, under-actuated, and must interact with
friction and ground contacts to maximize it’s reward.

Figure 2: A render of the hopper system studied in this work.

##  III MESHING

In [5] we used meshes of individual trajectories to calculate fractional
dimensions. However the previous work that has used meshing for analysis of RL
policies [13] instead examines meshes of the reachable state space of a
system. In particular [13] examines the reachable state space with a fixed
control policy subject to a given set of disturbances. We will now outline the
process for this style of meshing.

We are interested in the set of states that our system can transition to with
a fixed policy and a given set of push disturbances. We first introduce a
failure state to the mesh. The failure state is assumed to be absorbing, once
the robot falls it is assumed to stay that way. For our hopper, any state
where the COM falls below .7m is considered to have failed, which works well
in practice. This is also the failure condition of the environment during
training, and therefore the agent is never trained in regions of the state
space that satisfy the failure condition.

In addition to the reachable set of states, we want to construct a state to
state transition map. That is, for a given initial state, we wish to know
which state we transition to for every disturbance in our disturbance set.
It’s worth emphasizing that this map is completely deterministic.

To make this concrete, recall that we manifest our mesh as a hash table. The
key for any given state is obtained by 2. When we insert a new key into our
hash table, the value we place is a pair with a unique state ID (which is
simply the number of keys in the table at the time of insertion), and an
initially empty list of all mesh states which are reachable after one step
from the key state. This data structure will provide both the reachable set,
and the transition mapping.

For the hopper in particular, the system transitions from its initial standing
position to a stable long term hopping gait. After letting the system enter
its gait, we start detecting states on the Poincaré section by selecting the
state corresponding to the peak of the base link’s height in every ballistic
phase. These states are then collected as the initial states to seed the mesh
with. Throughout this paper, we seed the mesh with trajectories from 10
initial conditions.

For each snapshot, we initialize the system in the snap-shotted state. For
each disturbance in our fixed disturbance set, we simulate the system forward
subject to that disturbance. If the system does not fail, then the next
Poincaré snapshot is captured, this state is then checked for membership in
our mesh. If the new state is already in our mesh, then we simply append the
new state to the list of states that the initial state can transition to. If
the new state is not already in our mesh, then we expand our mesh to include
the new state, and append this new state to the transition list of the initial
state. If the system does fail, then we simply append the failure state to the
transition list of the initial state, and no new state is added to the mesh.

For every new state added to the mesh, we repeat this process until every
state has been explored. Algorithm 1 details this process in pseudo code.

Input: Initial states $S_{i}$, Disturbance set $D$

Output: Mesh M.

Q $\leftarrow S_{i}$ (excluding the failure state)

while  _Q not empty_ do

pop q from Q

for  _d $\in$ D_ do

Initialize system in state q

Run system for one step subject to disturbance d

Obtain final state x

if  _x $\notin$ M_ then

M[x] = List()

Push x onto Q

end if

Append x to M[q]

end for

end while

Return: M

Algorithm 1 createMesh

###  III-A Stochastic Transition Matrix

The stochastic transition matrix $\mathbf{T}$ is defined as follows:

$\mathbf{T}_{ij}=\text{Pr}(id[n+1]=j\ |\ id[n]=i)$ (5)

where $id[n]$ is the index in our mesh data structure of the state at step n.
For some intuition, consider the transition matrix as the adjacency matrix for
a graph. There is one row/column for every state in our mesh, for a given row
i, each entry j is the probability of transitioning from state i to state j.
Every row will sum to one, but the sum for each column has no such constraint.
After constructing a mesh using algorithm 1, it is straightforward to create
the stochastic transition matrix by iterating through every transition list in
our mesh.

###  III-B Mean First Passage Time

We wish to use our mesh based methods to quantify the stability of our system.
To do this we estimate the average number of steps the agent will take before
falling, subject to a given distribution of disturbances. To do this we will
use the so called Mean First Passage Time (MFPT) which in this case will
describe expected number of footsteps, rather than the number of timesteps to
failure. First recall that our assumption is that our failure state is an
absorbing state in our Markov chain approximation, and this implies that the
largest eigenvalue of T will always be $\lambda_{1}=1$. In [4] Byl showed that
when the second largest eigenvalue $\lambda_{2}$ is close to unity, the MFPT
is approximately equal to:

$MFPT\approx\frac{1}{(1-\lambda_{2}).}$ (6)

##  IV TRAINING

In [9] Mania et al introduce Augmented Random Search (ARS) which proved to be
efficient and effective on the locomotion tasks. Rather than a neural network,
ARS used static linear policies, and compared to most modern reinforcement
learning, the algorithm is very straightforward. The algorithm is known to
have high variance; not all seeds obtain high rewards, but to our knowledge
their work in many ways represents the state of the art on the Mujoco
benchmarks. Mania et al introduce several small modifications of the algorithm
in their paper, our implementation corresponds to the version they call
ARS-V2t, hyper parameters are provided in the appendix.

The training process is done in episodes, each episode corresponds to 1000
policy evaluations played out in the simulator. At the start of each episode,
the system is initialized in a nominal initial condition offset by a small
amount of noise added to each state. During each episode we fix a static
policy to let the the system evolve under, we collect the observed state, the
resulting action, and the resulting reward at each timestep. This information
is then used to update the policy for the next episode.

We compare four different sets of agents trained in different conditions, for
each training condition we use training runs across 10 different random seeds.
As mentioned ARS is a very high variance algorithm, so a common practice is to
run many seeds in parallel and choose the highest performing one. The standard
environment has two sources of randomness which are set by the random seed.
The first is a small amount of noise added to a the nominal initial condition
at the beginning of each episode. The second is noise added to the policy
parameters as part of the normal ARS training procedure. Using ARS in the
unmodified Hopper-v2 environment will be called the standard training
procedure. In addition to this, we have a second set of agents which are
initialized with the standard training, and then trained for another 250
epochs with the fractal reward function used in equation 4, these are called
the fractal agents. Using the standard training agents as the initial policies
for the fractal reward was also used in [5], please see that manuscript for
more details.

In addition to standard training, we repeat this standard / fractal setup but
with the addition of a small amount of zero mean Gaussian noise added to both
the states and actions at training time. For brevity we will call these the
Standard noise and Fractal noise scenarios. Hyper parameters for ARS and noise
values are reported in the appendix.

##  V Results

###  V-A Mesh Sizes Across All Seeds

First we wish to compare the reachable state space mesh sizes obtained for
these four different training regiments. For this we assume a disturbance
profile consisting of 25 pushes equally spaced between -15 and 15 Newtons,
applied for 0.01 seconds along the x axis at the apex of each jump. The goal
for this particular exercise is to get an idea of the relative mesh sizes
among the different agents across box sizes. Table I shows these results. We
can see that across all box sizes, adding noise at run time decreases the mesh
sizes slightly, and that adding the fractal reward training decreases the mesh
size even further. The combination of adding noise and the fractal reward
seems to perform best at reducing the mesh size.

Training | $d_{thr}=.4$ | $d_{thr}=.3$ | $d_{thr}=.2$ | $d_{thr}=.1$
---|---|---|---|---
Standard | 64.9 | 129.0 | 289.2 | 2975.2
Standard Noise | 40.7 | 73.3 | 231.6 | 2133.3
Fractal | 26.0 | 41.8 | 67.7 | 684.4
Fractal Noise | 15.1 | 24.6 | 45.1 | 297.2
TABLE I: Mesh sizes across all seeds for a disturbance profile of 25 pushes.
All values are the average mesh size across 10 agents trained with different
seeds.

###  V-B Larger Meshes

With the general trend established, we now take the best performing seed from
the noisy training for further study. We chose the seed that had the smallest
mesh size from both the standard noise and fractal noise agents.

For this next experiment, we consider a richer distribution of 100 randomly
generated push disturbances. These disturbances have a magnitude drawn from a
uniform distribution between 5-15 Newtons. This force is applied in the xz
plane with an angle drawn from a uniform distribution between 0 and 2$\pi$.
The number of forces was chosen by increasing the number of forces sampled
until the mesh sizes between two random sets did not change. The magnitude of
the pushes was chosen arbitrarily, in principle one can use these methods for
any distribution of disturbance they expect their robot to encounter during
operation.

We then construct meshes for different box sizes. For each agent we construct
10 meshes. We vary the box size between 0.1 and 0.01 for the fractal noise
agent. For the standard noise agent we instead vary the box size between 0.1
and 0.02 because the mesh sizes for the standard agent were proving to be too
large at the smaller box sizes. Figure 3 shows the comparison, We can see
clearly that at the very least, the exponential blowup in mesh size starts at
much more accurate mesh resolutions for the fractal agent.

Figure 3: Mesh sizes for the top performing standard noise and fractal noise
agents.

We are also interested in the exponential scaling factor in the mesh size as
the box gets smaller, which is captured by the fractal dimension discussed in
section II-A. As mentioned before, in previous work our modified reward signal
resulted in agents with a smaller fractal dimension with respect to individual
trajectories. We now ask if this carries over to meshes of the reachable state
space obtained by the procedure from algorithm 1. Table II shows the results,
we can see that indeed, the fractal training does seem to reduce the mesh
dimensionality for the reachable state space meshes.

Training | Trajectory Mesh Dim. | Reachable Mesh Dim.
---|---|---
Standard Noise | 1.38 | 3.83
Fractal Noise | 1.16 | 3.16
TABLE II: Mesh dimensions for the best performing seed from the standard with
noise training, and the fractal with noise training, given the same
disturbance profile of 100 pushes. For reference the state space for our
system has 12 dimensions.

###  V-C Validating the Mean First Passage Time

We emphasize that the reward function for the hopper environment is simply to
move forward with the highest velocity possible, no attempts were made to make
the system robust to disturbances. Perhaps because of this, the mean first
passage time for these systems are relatively small, on the order of 100 foot
steps. For this small number of steps, we can validate the mean first passage
time with Monte Carlo trials. It’s worth noting that the eigen estimate of the
mean first passage time is much more valuable for more robust systems. This is
because this estimate becomes more accurate as the system becomes more stable,
and because the cost of calculating the MFPT with Monte Carlo trials grows
much more expensive for more stable systems. In previous works [10] it was
used to quantify robustness for systems with a MFPT as high as $10^{15}$.

To do this, we compare the mean first passage time as estimated by equation 6
to the value computed by looking at many Monte Carlo rollouts. For the
rollouts we apply a random action drawn from the same distribution described
above. Instead of sampling 100 pushes though we sample a new push every time
we need a new disturbance. During the rollouts we still apply the push at the
apex height of the ballistic phase.

Figure 4 shows the convergence of the MFPT as we expand the size of the mesh,
and compares it to the mean steps to failure obtained with Monte Carlo trials.
We can see that it does look like the MFPT is converging to the Monte Carlo
result. Although at the largest mesh we tried, the eigen analysis gives an
estimate of 110.2 steps to failure, while the Monte Carlo trials tell us that
an average of 85 steps are taken before failure. It’s worth noting that the
distribution of failure times has a large variance with a standard deviation
of 80 steps.

Figure 4: Estimated mean first passage time computed from 6 compared to a
Monte Carlo estimate. The blue dashed line and shaded region are the mean and
standard deviation of the steps to failure for 2500 Monte Carlo rollouts.

###  V-D High Resolution Mesh

We now use the fractal agent and construct an even more accurate mesh. Figure
5 show the sparsity pattern for the state transition matrix for the fractal
noise agent with a box size of 0.005. Recall that in the process for creating
the mesh, we start with a small number initial seed states. After that every
new state that we add is added in order we find them to the mesh. So if we are
expanding state # 2, and there are currently 100 states in the mesh, if we
transition to an unseen state, that state will be labeled # 101. So although
it may seem like it is not possible for states in the top right quadrant to
visit states later in the mesh, this is really an artifact of how we construct
our mesh and label our points.

Figure 5: Visualization of the stochastic transition matrix for the top
performing fractal noise agent. All non zero values are shown with equal size
and coloration. Recall that each entry in $T_{ij}$ tell us the probability of
transitioning to state j after one step if we start in state i.

We note that there are a smaller set of states that make up most of the
transitions. In fact we can see from Figure 6 that 20% of the states in our
mesh account for about 90% of all transitions seen during the mesh
construction.

Figure 6: Cumulative sum of probability mass excluding the failure state. We
take the sum of each column of T, and sort it in descending order, then report
the cumulative sum of probability. Each point on the curve tells us that x% of
states make up y% of all state transitions.

One of the advantages of having a discrete set of states is that it opens up
new tools and visualizations, for example we can apply Principle Component
Analysis (PCA). Figure 7 shows a projection of our mesh states on the top 3
principle components. We note that these three states account for more than
97% of the variance, we also note that our analysis reveals that states in red
are where 99% of all failures occur. The visualization reveals that at least
in PCA space, all the trouble states are clustered in one spot. A promising
direction for future work is to introduce a policy refinement step that
attempts to avoid these states. Additionally, if we were designing a real
robot this may give us insights into design changes that could be made.

Figure 7: View of the first 3 principle components of the mesh for a fractal
noise policy.

##  VI CONCLUSIONS

In this work, we apply previously developed tools that create discrete meshes
for the reachable state space of a system. These tools were applied to
policies obtained with a modified reinforcement learning reward function which
was previously shown to encourage small mesh dimensions for individual
trajectories not subject to any disturbances. We showed that these modified
policies have a smaller average reachable mesh size across all random seeds
for coarse meshes and a small number of disturbances. We then showed a clear
difference in mesh sizes and mesh dimensions for the top performing seeds on a
richer set of disturbances and finer mesh sizes. We also validated our use of
the MFPT as a tool by comparing it to Monte Carlo trials. Finally, we
constructed a high fidelity mesh at a resolution that would not have been
feasible with standard ARS policies. In addition, we created visualizations
with this mesh that revealed insights about the contracting nature of the
policy, and which point to future applications of this approach. Taken
together, these results show two things. First, it further validates the
utility of the fractal dimension reward, which we have shown transfers it’s
desirable quality of having a more compact state space to a setting with
external disturbances. These results are also a credit to the mesh based
tools, because it shows that the fractal training can be used to extend the
reach of these tools to higher dimensional systems or higher resolution meshes
than would have otherwise been possible.

## APPENDIX

### Hyper Parameters

ARS: (from [9]) $\alpha=0.02$, $\sigma=0.025$, $N=50$, $b=20$.
MeshDim: (from [5]) f = 1.5, $d_{0}$ = 1e-2

### Noise During Training

Zero mean Gaussian noise with std = 0.01 added to policy actions before being
passed to the environment, for reference all actions from the policy are
between -1 and 1. Zero mean Gaussian noise with std = 0.001 added to
observations before being passed to the policy.

## References

  * [1] Brendan Ryan / Public domain  “Fractal Dimension Example”  In _Wikapedia_ , 2020
  * [2] Greg Brockman et al.  “OpenAI Gym”, 2016  eprint: arXiv:1606.01540
  * [3] Katie Byl, Tom Strizic and Jason Pusey  “Mesh-based switching control for robust and agile dynamic gaits”  In _Proceedings of the American Control Conference_ , 2017, pp. 5449–5455  DOI: 10.23919/ACC.2017.7963802
  * [4] Katie Byl and Russ Tedrake  “Metastable walking machines”  In _International Journal of Robotics Research_ 28.8, 2009, pp. 1040–1064  DOI: 10.1177/0278364909340446
  * [5] S. Gillen and K. Byl  “Explicitly Encouraging Low Fractional Dimensional Trajectories Via Reinforcement Learning”  In _2020 4th Conference Of Robot Learning (CORL)_ , 2020
  * [6] Nicolas Heess et al.  “Emergence of Locomotion Behaviours in Rich Environments” arXiv: 1707.02286  In _arXiv:1707.02286 [cs]_ , 2017  URL: http://arxiv.org/abs/1707.02286
  * [7] Jemin Hwangbo et al.  “Learning agile and dynamic motor skills for legged robots”  In _Science Robotics_ 4.26, 2019, pp. eaau5872  DOI: 10.1126/scirobotics.aau5872
  * [8] Joonho Lee, Jemin Hwangbo and Marco Hutter  “Robust Recovery Controller for a Quadrupedal Robot using Deep Reinforcement Learning” arXiv: 1901.07517  In _arXiv:1901.07517 [cs]_ , 2019  URL: http://arxiv.org/abs/1901.07517
  * [9] Horia Mania, Aurelia Guy and Benjamin Recht  “Simple random search of static linear policies is competitive for reinforcement learning”  In _Advances in Neural Information Processing Systems_ 2018-December.NeurIPS, 2018, pp. 1800–1809
  * [10] Cenk Oguz Saglam and Katie Byl  “Robust Policies via Meshing for Metastable Rough Terrain Walking”  In _Robotics Science and Systems_ , 2015  DOI: 10.15607/rss.2014.x.049
  * [11] OpenAI et al.  “Learning Dexterous In-Hand Manipulation” arXiv: 1808.00177  In _arXiv:1808.00177 [cs, stat]_ , 2018f  URL: http://arxiv.org/abs/1808.00177
  * [12] Nihar Talele and Katie Byl  “Mesh-based methods for quantifying and improving robustness of a planar biped model to random push disturbances”  In _Proceedings of the American Control Conference_ 2019-July, 2019, pp. 1860–1866  DOI: 10.23919/acc.2019.8815226
  * [13] Nihar Talele and Katie Byl  “Mesh-based Tools to Analyze Deep Reinforcement Learning Policies for Underactuated Biped Locomotion”, 2019  arXiv: http://arxiv.org/abs/1903.12311