# Particle acceleration at magnetized, relativistic turbulent shock fronts

Virginia Bresci  Institut d’Astrophysique de Paris, CNRS – Sorbonne
Université, 98 bis boulevard Arago, F-75014 Paris, France  CEA, DAM, DIF,
F-91297 Arpajon, France  Leibniz-Institut für Astrophysik Potsdam (AIP), An
der Sternwarte 16, 14482 Potsdam, Germany  Martin Lemoine  Institut
d’Astrophysique de Paris, CNRS – Sorbonne Université, 98 bis boulevard Arago,
F-75014 Paris, France  Laurent Gremillet  CEA, DAM, DIF, F-91297 Arpajon,
France  Université Paris-Saclay, CEA, LMCE, 91680 Bruyères-le-Châtel, France

###### Abstract

The efficiency of particle acceleration at shock waves in relativistic,
magnetized astrophysical outflows is a debated topic with far-reaching
implications. Here, for the first time, we study the impact of turbulence in
the pre-shock plasma. Our simulations demonstrate that, for a mildly
relativistic, magnetized pair shock (Lorentz factor $\gamma_{\rm sh}\simeq
2.7$, magnetization level $\sigma\simeq 0.01$), strong turbulence can revive
particle acceleration in a superluminal configuration that otherwise prohibits
it. Depending on the initial plasma temperature and magnetization, stochastic-
shock-drift or diffusive-type acceleration governs particle energization,
producing powerlaw spectra $\mathrm{d}N/\mathrm{d}\gamma\propto\gamma^{-s}$
with $s\sim 2.5-3.5$. At larger magnetization levels, stochastic acceleration
within the pre-shock turbulence becomes competitive and can even take over
shock acceleration.

Motivation– The nonthermal radiative spectra observed from high-energy,
relativistic astrophysical sources point to a bulk energy reservoir being
dissipated into accelerated particles through, e.g., magnetic reconnection
(Guo  _et al._ , 2014; Kagan  _et al._ , 2015; Werner and Uzdensky, 2017;
Petropoulou  _et al._ , 2019), shock acceleration Spitkovsky (2008); Sironi
_et al._ (2015a), or turbulent Fermi processes Zhdankin  _et al._ (2017);
Comisso and Sironi (2018); Zhdankin  _et al._ (2019); Bresci  _et al._ (2022).
As direct offsprings of the powerful outflows associated with those sources,
collisionless shock waves emerge as natural dissipation agents Bykov  _et al._
(2012). Yet, in relativistic and magnetized plasmas, particle acceleration
appears inhibited by the generic superluminal nature of the shock (Begelman
and Kirk, 1990; Lemoine  _et al._ , 2006; Niemiec  _et al._ , 2006; Pelletier
_et al._ , 2009; Sironi and Spitkovsky, 2009; Lemoine and Pelletier, 2010;
Sironi  _et al._ , 2013). Specifically, _ab initio_ particle-in-cell (PIC)
numerical simulations have revealed a powerlaw tail of nonthermal particles
increasingly shrinking as the magnetization parameter $\sigma$ 111We define
the magnetization level as $\sigma\equiv\langle
B^{2}\rangle/4\pi\langle\epsilon\rangle$, in terms of mean-squared magnetic
field $\langle B^{2}\rangle$ and the energy density
$\langle\epsilon\rangle=n\langle\gamma\rangle m_{\rm e}c^{2}$ as measured in
the simulation (downstream) frame; $n$ represents the total apparent density.
When measured in the comoving turbulence frame, the resulting value of
$\sigma$ may be a factor $\sim 2$ larger. rises above $\sim 10^{-4}$, until it
vanishes at $\sigma\sim 10^{-2}$ (Sironi  _et al._ , 2013; Plotnikov  _et al._
, 2018). Given that a significant magnetization is expected in a wide class of
high-energy astrophysical jets, e.g. gamma-ray bursts, pulsar wind nebulae or
blazars, this result challenges the role of shocks as sources of high-energy
particles (Sironi  _et al._ , 2015b).

One limitation of previous studies, though, is to systematically consider
laminar in-flow conditions, _i.e._ , nonturbulent, homogenous background
plasmas of uniform magnetization. Now, the presence of a strong turbulence
upstream of a fast shock may change the picture in various ways: it may
preaccelerate the plasma particles via a stochastic Fermi process (Zhdankin
_et al._ , 2017; Comisso and Sironi, 2018; Zhdankin  _et al._ , 2019; Bresci
_et al._ , 2022; Lemoine, 2022), just as it may corrugate the shock front so
that the turbulence does not transform trivially through the shock (Zank  _et
al._ , 2002; Mizuno  _et al._ , 2011; Lemoine, 2016; Lemoine  _et al._ , 2016;
Demidem  _et al._ , 2018; Trotta  _et al._ , 2021; Nakanotani  _et al._ ,
2022), possibly unlocking particles from the field lines and enabling their
acceleration. In light of these considerations, the paradigm of inefficient
relativistic magnetized shocks as particle accelerators needs to be revisited
in the likely common case of turbulent environments.

To this goal, we here report on the first PIC simulations of relativistic
shocks propagating in turbulent, magnetized pair plasmas. We demonstrate that,
despite a substantial magnetization ($\sigma\sim 0.01$), shock acceleration is
manifest, and that the particle spectrum develops a powerlaw tail extending in
time, which is absent without turbulence. This drastic change involves a
significant ($\delta B>B_{0}$), but not too strong upstream turbulence,
otherwise its own contribution to particle acceleration can supersede that of
the shock.

Numerical method– We perform such simulations using the fully electromagnetic
and relativistic calder code (Lefebvre  _et al._ , 2003; Lemoine  _et al._ ,
2019; Vanthieghem  _et al._ , 2022). Turbulence is excited close to the right-
hand side of the domain, in a pair plasma continually injected along
$-\bm{\hat{x}}$ at a relativistic velocity $v_{\infty}=-0.87\,c$ (Lorentz
factor $\gamma_{\infty}=2$). The flow is left to propagate across the domain
until the turbulence hits its left-hand side. Switching the local boundary
condition from open to reflective at that time triggers a rightward
propagating shock wave Spitkovsky (2008), which sweeps the incoming turbulent
plasma. The simulation frame thus corresponds to the downstream rest frame of
the shock. Due to physical constraints discussed thereafter, we restrict
ourselves to 2D3V geometry (2D in space, 3D in momentum). A uniform magnetic
guide field $\bm{B_{0}}$ is applied along the (out-of-plane) $z$ direction
with corresponding magnetization level $\sigma_{0}$.

Figure 1: Spatial distributions of (a) apparent positron density $n_{p}$
(normalized to the apparent far-upstream positron density $n_{0}$), (b) mean
kinetic energy per positron $\langle\gamma-1\rangle$, and (c) squared
turbulent magnetic field $\delta B_{x}^{2}+\delta B_{y}^{2}$ (normalized to
$4\pi n_{0}m_{\rm e}c^{2}$) for simulation (S2) at $t\simeq
12\,400\,\omega_{p}^{-1}$. In panel (d), the longitudinal profile of the
turbulent magnetization $\sigma_{\delta B}=(\delta B_{x}^{2}+\delta
B_{y}^{2})/(4\pi\sum_{\alpha}n_{\alpha}\langle\gamma\rangle_{\alpha}m_{\rm
e}c^{2})$, with $n_{\alpha}$ the total (apparent) density of the plasma,
averaged over the transverse dimension ($y$). The left and right vertical
lines locate the shock front and the boundary of forced turbulence.

The turbulence is driven in the rest frame of the drifting plasma, in a finite
region covering a few stirring length scales $\ell_{\rm c}$. Elsewhere, the
energy injection in the system is halted and the turbulence is let to develop
freely, thereby initiating the cascade before it interacts with the shock.
Following Zhdankin  _et al._ (2017), we aim at exciting turbulence on a scale
$\ell_{\rm c}$ as large as possible compared to the kinetic scale
$c/\omega_{\rm p}$, in order to simulate an inertial range under near-
magnetohydrodynamics conditions where the turbulence cascades down to the
dissipative range. Those simulations are demanding, because the interaction
between the turbulence and the shock must be followed over a long enough
timespan, the transverse dimension must accommodate $\gtrsim 1-2\,\ell_{\rm
c}$, and also because the need to stir turbulence in the plasma rest frame
brings in further constraints due to time dilation effects.

In detail, the simulation domain contains $48\,000\times 6\,000$ cells along
the $x$ and $y$ axes and the simulation is run over up to $120\,000$ time
steps. The mesh size is $\Delta x=\Delta y=0.1\,c/\omega_{\rm p}$
222$\omega_{\rm p}\equiv(4\pi n_{\infty}e^{2}/m_{\rm e})^{1/2}$ denotes the
nonrelativistic plasma frequency of the far upstream plasma, with
$n_{\infty}=2n_{0}/\gamma_{\infty}$ the (total) proper density, $n_{0}$ the
apparent density of one species. and the time step is $\Delta t=0.99\,\Delta
x/c$. Periodic boundary conditions are used for particles and fields in the
transverse direction. Turbulence is excited in the interval
$4200\leq\omega_{\rm p}x/c\leq 4800$ through external magnetic perturbations
$(\delta B_{x},\delta B_{y})$ implemented as plane waves. Those are seeded
following a Langevin antenna scheme (TenBarge  _et al._ , 2014; Zhdankin  _et
al._ , 2017; Bresci  _et al._ , 2022), which acts in the proper plasma frame
(Sup, ). The mean wavenumber $\langle k^{\prime}\rangle\simeq 2.9\times
2\pi/L_{y}$ ($L_{y}=600\,c/\omega_{\rm p}$ the transverse box size) implies a
comoving coherence length $\ell_{\rm c}=2\pi/\langle k^{\prime}\rangle\simeq
200\,c/\omega_{\rm p}$ (primed quantities are evaluated in the local plasma
frame). At $x\leq 4200\,c/\omega_{\rm p}$, stirring is halted, hence
fluctuations evolve on a (proper) nonlinear timescale $\tau_{\rm
nl}^{\prime}\equiv\ell_{\rm c}/v_{\rm A}\sim 1\,200\,\omega_{\rm p}^{-1}$
($v_{\rm A}\simeq c\sqrt{\sigma_{\delta B}}\simeq 0.17\,c$ is the Alfvén
velocity), corresponding to $\tau_{\rm nl}=\gamma_{\infty}\ell_{\rm c}/v_{\rm
A}\sim 2\,400\,\omega_{\rm p}^{-1}$ in the simulation frame.

We report here on three main simulations exploring different initial
background magnetization levels $\sigma_{0}\sim 10^{-4}\rightarrow 10^{-3}$,
turbulent magnetization $\sigma_{\delta B}\sim 10^{-2}\rightarrow 10^{-1}$ and
initial proper plasma temperatures $T$, from subrelativistic to relativistic
(Sup, ). In detail, $\left\\{\sigma_{0},\,\,\sigma_{\delta B},\,\,k_{\rm
B}T/m_{e}c^{2}\right\\}=\left\\{0.2\times 10^{-3},\,\,2\times
10^{-2},\,\,0.1\right\\}$ [hereafter (S1)], $\left\\{0.6\times
10^{-3},\,\,1\times 10^{-2},\,\,4.\right\\}$ [(S2)], $\left\\{0.6\times
10^{-3},\,\,10^{-1},\,\,4.\right\\}$ [(S3)]. A relativistically hot initial
plasma as in (S2) and (S3) could describe internal shocks inside a strongly
turbulent jet. We have run ancillary simulations, in particular (S2a), similar
to (S2) albeit deprived of turbulence, (S2b) which retains open boundary
conditions and thus models drifting turbulence without a shock, and finally
(S4), for which $k_{\rm B}T/m_{e}c^{2}=0.1$ as in (S1), but with larger
$\sigma_{0}$ and $\sigma_{\delta B}$, as in (S2).

Results– Figure 1 displays (from top to bottom) the spatial distributions of
plasma positron density, mean Lorentz factor and magnetization level at the
final simulation time $t\simeq 12\,400\,\omega_{\rm p}^{-1}$ for simulation
(S2). The rightward-moving shock front has then reached $x\simeq
2\,400\,c/\omega_{\rm p}$. Once swept up by the shock, the plasma is
compressed by a factor of $\simeq 3.5$ and the mean kinetic energy per
particle slightly increases, in good agreement with the shock-crossing
conditions Kirk and Duffy (1999) [Figs. 1(a,b)]. Magnetic fluctuations are at
their highest near the right boundary where turbulence is continually excited
[Fig. 1(c)]. The $\sim 2\,000\,c/\omega_{\rm p}$ distance between the shock
and the boundary of forced turbulence is then just below the minimum distance
$c\tau_{\rm nl}$ needed for nonlinear evolution of the turbulence. This
guarantees that the shock-turbulence interaction is not affected by the
stirring procedure in the right part of the domain.

As shown in Fig. 1(d), the turbulence profile reaches an approximately steady
state by the time it encounters the shock. We have checked that the spatial
power spectrum of magnetic fluctuations in the transverse $y$ direction, which
extends over three orders of magnitude, shows a general scaling $\propto
k_{y}^{-5/3}$ at large scales $k_{y}\lesssim 15\ell_{c}^{-1}$, and a steeper
behavior at kinetic scales, consistent with previous PIC studies of
nondrifting decaying turbulence (Comisso and Sironi, 2018).

Further upstream, corresponding to an earlier stage in the turbulence
evolution, stronger fluctuations are observed, as expected. The eddies are
compressed when transiting across the shock and continue interacting until the
turbulence eventually relaxes further downstream. This general picture
resembles that observed in MHD simulations of the interaction of a
monochromatic, linear plasma eigenmode with a relativistic shock front
(Demidem  _et al._ , 2018).

Figure 2: Particle energy spectra $\gamma^{2}{\rm d}N/{\rm d}\gamma$ at
different times (from light to dark solid blue) in simulations (S1), (S2) and
(S3), from top to bottom. Middle panel: in dotted line, spectrum from (S2a),
i.e., a shock interacting with a non-turbulent plasma, and in dashed line,
spectrum from (S2b), i.e., a drifting turbulence without a shock, both in
conditions otherwise similar to (S2). In the lower panel, the light orange
band delineates the range of spectra measured in a simulation similar to (S3)
albeit without a shock, as extracted at various places and times.

The shock moves at velocity $\simeq 0.4\,c$ in the simulation (downstream)
frame ($0.36\,c$ is predicted by the shock-crossing conditions (Kirk and
Duffy, 1999)), and consequently at $v_{\rm sh}\simeq 0.93\,c$ in the upstream
frame, corresponding to a shock Lorentz factor $\gamma_{\rm sh}\simeq 2.7$.
Ahead of the shock, the transversely averaged magnetic field strength is
$\delta B/(m_{\rm e}\omega_{\rm p}c/e)\simeq 0.8$, so that particles with
Lorentz factor $\gamma\simeq 100-300$ have a gyroradius $r_{\rm g}\simeq
120-360\,c/\omega_{\rm p}$.

Figure 2 plots the time evolution of the particle energy spectra
$\gamma^{2}{\rm d}N/{\rm d}\gamma$ (per log-interval of energy) in each of the
three simulations, as integrated over a moving window centered on the shock
front position $x_{\rm sh}$ (located from the plasma density map) and with a
$200\,c/\omega_{\rm p}$ half width along $x$. Those spectra peak at energies
consistent with shock-heating of the incoming plasma. Remarkably, a
suprathermal tail develops in all cases, with approximate powerlaw index
$s\simeq 2.5\rightarrow 3.5$ (as defined through ${\rm d}N/{\rm
d}\gamma\propto\gamma^{-s}$), providing manifest evidence of particle
acceleration. In (S1) and (S2), and unlike in (S3), the maximal energy is seen
to increase with time.

Figure 3: Position $x$ along the shock normal (a) and Lorentz factor $\gamma$
(b) versus time for $5$ particles with initial Lorentz factor $30<\gamma<100$
in simulation (S2). The dashed curve in (a) indicates the shock front position
$x_{\rm sh}(t)$.

The middle panel of Fig. 2 presents additional spectra from the turbulence-
free simulation (S2a), thus where the shock forms immediately in the external
field $\bm{B_{0}}$ (dotted line), and from shock-free simulation (S2b), which
contains only drifting turbulence (dashed line). Clearly, the suprathermal
tail only arises when the shock interacts with the turbulence. The absence of
particle acceleration in (S2a) can be attributed to the background
magnetization level $\sigma\equiv\sigma_{0}\simeq 10^{-3}$, large enough to
inhibit Fermi cycles around the shock. In (S2b), the magnetic fluctuations are
too slow to accelerate particles, as the characteristic acceleration timescale
$t_{\rm acc}\sim\gamma_{\infty}\,c\ell_{\rm c}/v_{\rm A}^{2}\sim
10^{4}\,\omega_{\rm p}^{-1}$ indeed exceeds the time needed for the plasma to
cross the domain. Note that the spectrum of (S2b) coincides with that of (S2)
at $t=5940\,\omega_{\rm p}^{-1}$, because the plasma has then just hit the
reflective wall, and the shock has not formed yet.

Particle acceleration– Standard theory depicts acceleration at a magnetized
shock front as the result of pitch-angle diffusion back and forth across the
shock, or of shock-drift motion along the mean advected electric field
(Blandford and Eichler, 1987). In the relativistic limit, shock acceleration
is ineffective Begelman and Kirk (1990); Lemoine  _et al._ (2006) unless
intense turbulence can unlock particles off the field lines Lemoine and
Pelletier (2010), and thereby trigger diffusive-type acceleration Kirk  _et
al._ (2000); Achterberg  _et al._ (2001), a form of stochastic-shock-drift
process Takamoto and Kirk (2015); Matsumoto  _et al._ (2017), or a combination
of both, meaning orbits in the regular field upstream of the shock, diffusive
orbits downstream Pelletier  _et al._ (2009).

To probe the acceleration process at work, we have tracked a large number
$\sim\mathcal{O}(10^{5})$ of particles sampled in various (initial) energy
intervals. Figure 3 shows the trajectories and energy histories of four
particles in (S2), representative of the population able to circulate around
the shock for an extended period of time. The Lorentz factor of some particles
(e.g. orange and cyan in that figure) undergoes sizable oscillations before
they encounter the shock; this results from their gyromotion along the fast-
moving magnetic field lines Wong  _et al._ (2020), not from acceleration _per
se_. Notwithstanding this effect, the energization of the particles traveling
in the vicinity of the shock is evident.

We discriminate the acceleration process in simulations (S1) and (S2) using
the following argument. In a shock-drift process, the energy gained by a
particle of velocity $\bm{v}$ equals the amount of work performed by the mean
motional electric field $\bm{E_{0}}=v_{\infty}B_{0}\,\bm{\hat{y}}$, _i.e._ ,
$W(E_{0})=q\int{\rm d}t\,\bm{v}\cdot\bm{E_{0}}$, while for diffusive-type
acceleration, the energy gain is rather related to the work $W(\delta
E_{z})=q\int{\rm d}t\,v_{z}\delta E_{z}$ performed by the motional turbulent
electric field. The latter is mostly directed along $\bm{\hat{z}}$ because the
plasma, which flows along $-\bm{\hat{x}}$, carries essentially $(\delta
B_{x},\,\delta B_{y})$ magnetic fluctuations; hence $\delta E_{z}\simeq-
v_{\infty}\delta B_{y}$. For each tracked particle with initial Lorentz factor
at the onset of the powerlaw tail, i.e. $\gamma\geq 20$ [in (S1)] and
$\gamma\geq 200$ [(S2)], we have thus recorded $W(E_{0})$ and $W(\delta
E_{z})$ during the time interval $\Delta t_{\rm sh}$ between the first and
last encounters of the particle with the shock front.

Figure 4: Correlation between the measured variation in Lorentz factor
$\Delta\gamma$ and that predicted by shock-drift acceleration
[$W(E_{0})/m_{\rm e}c^{2}$, top row (a) and (c)] or diffusive-type
acceleration [$W(\delta E_{z})/m_{\rm e}c^{2}$, bottom row (b) and (d)] in
simulations (S1) [left column, (a) and (b)] and (S2) [right column, (c) and
(d)]. Orange dots represent individual measurements which are interpolated by
the pseudocolor density plot.)

Figure 4 shows the correlations between the observed variation in Lorentz
factor $\Delta\gamma$ during $\Delta t_{\rm sh}$ and $W(E_{0})/m_{\rm e}c^{2}$
[top row, (a) and (c)], as well as $W(\delta E_{z})/m_{e}c^{2}$ [bottom row,
(b) and (d)], for simulations (S1) [left column, (a) and (b)] and (S2) [right
column, (c) and (d)]. In these plots, the dashed red line indicates the
expected level of contribution from either shock-drift or diffusive-type
acceleration. Figures 4 (c) and (d) reveal that the shock-drift process nicely
accounts for particle energization in (S2), while in (S1), acceleration
appears dominated by diffusive-type acceleration.

This general picture is further supported by the angular distribution of the
suprathermal particle momenta (Sup, ). In (S2), this angular map presents a
clear asymmetry between positrons and electrons, roughly polarized along
$\bm{E_{0}}$, such that positrons (resp. electrons) appear to drift with
negative (resp. positive) $p_{y}$, as expected for $E_{0}<0$. By contrast, the
angular map is significantly more isotropic in (S1), as expected if diffusive-
type acceleration dominates. Additionally, simulation (S2) displays a net
linear correlation between $\Delta\gamma$ and $\Delta t_{\rm sh}$, from which
one can infer an acceleration timescale, $t_{\rm
acc}\equiv|\langle\Delta\gamma/\gamma\rangle|^{-1}\Delta t_{\rm sh}\sim
2-3\,p/(eE_{0})$, again consistent with that expected for particles drifting
along $\bm{E_{0}}$ at mildly relativistic speeds.

The difference in spectral index observed between simulations (S1) ($s\sim
2.5$) and (S2) ($s\sim 3.5$) likely results from the distinct acceleration
mechanisms at play. In particular, the spectral index for stochastic-shock-
drift acceleration – at least, in subrelativistic shocks – depends sensitively
on the shock speed and on the ratio of the scattering frequency of particles
to their gyrofrequency in the mean field (Takamoto and Kirk, 2015). At large
scattering frequencies, the index approaches the canonical value of $2$,
associated with diffusive shock acceleration, while at small scattering
frequencies, the spectrum steepens significantly, encompassing the value
measured in (S2).

Regarding the different acceleration processes in (S1) and (S2), we observe
that the $r_{\rm g}$ vs $\ell_{\rm c}$ ordering, which controls the scattering
rate of particles, varies between those two simulations because of different
initial temperatures and magnetizations: $r_{\rm g}/\ell_{\rm c}\sim 0.1$ at
the onset of the powerlaw tail for (S1), while $r_{\rm g}/\ell_{\rm c}\sim 1$
for (S2). A detailed study of the ancillary simulation (S4), which shares the
same initial temperature as (S1) and same magnetization as (S2), reveals,
however, that although a powerlaw index similar to that in (S2) is obtained,
shock-drift and diffusive-type processes now contribute in about equal amounts
to particle energization; accordingly, the angular map is less anisotropic
than in (S2), more than in (S1). Overall, this suggests that both plasma
temperature and magnetization may influence the prevalence of one mechanism
over the other. While a comprehensive explanation for this change of regime
certainly deserves further investigation, we emphasize that the main result of
the present work, namely, the formation of a nonthermal spectrum at
relativistic, magnetized turbulent shocks, is a robust feature.

Let us finally address simulation (S3), characterized by a relativistically
hot initial plasma and a substantial magnetization $\sigma_{\delta B}\sim 0.1$
Sup . This simulation probes a new regime in which stochastic Fermi
acceleration inside the pre-shock turbulence controls the acceleration
process, because the (stochastic) acceleration timescale $t_{\rm
acc}\sim\gamma_{\infty}\ell_{\rm c}/\sigma_{\delta B}c$ now becomes short
enough ($\simeq 10^{3}\,\omega_{\rm p}^{-1}$) to energize the freshly injected
particles before they attain the shock. Accordingly, the spectrum shown in
Fig. 2 (lower panel) does not vary with time because the turbulence inside the
simulation box is stationary, up to its fluctuations. This figure also reveals
that the particle distribution has undergone significant heating beyond the
simple shock-crossing conditions, as can be seen by direct comparison to
simulation (S2) (middle panel). Furthermore, we have verified that the same
simulation, albeit with open boundary conditions to prevent shock formation,
yields a similar spectrum Sup . Finally, the spectral index $s\sim 3.5$ falls
in line with that observed in PIC simulations of turbulence in the semi-
relativistic regime Zhdankin  _et al._ (2017); Comisso and Sironi (2018);
Bresci  _et al._ (2022).

Discussion– Interestingly, the range of magnetizations that we consider here,
$\sigma\sim 0.01\rightarrow 0.1$, and the range of spectral indices that we
measure, $s\sim 2.5\rightarrow 3.5$, appear rather typical of what is inferred
from one-zone models of blazars (Celotti and Ghisellini, 2008) and gamma-ray
bursts (Burgess  _et al._ , 2020). This supports the idea that mildly
relativistic shocks interacting with magnetized turbulence can play a leading
role in dissipation and particle acceleration in a broad range of relativistic
high-energy sources, up to moderate magnetizations. Our study thus
significantly extends the realm where relativistic shock acceleration can
operate without turbulence (_i.e._ $\sigma\ll 10^{-4}$). As stochastic
turbulent acceleration is observed to take over shock acceleration at
$\sigma\gtrsim 0.1$, one is tempted to sketch a picture in which, as the
magnetization level rises, a shock, or a shock plus turbulence, then
turbulence and eventually magnetic reconnection control dissipation and
acceleration.

As noted, the present simulations are computationally expensive, which limits
a broad parameter study. Future works should explore a larger parameter range,
in particular larger dimensions (and dimensionalities) in order to examine how
the particle spectrum changes with increasing $\ell_{\rm c}$, to make better
contact with phenomenology.

###### Acknowledgements.

This work has been supported by the Sorbonne Université DIWINE Emergence-2019
program and by the ANR (UnRIP project, Grant No. ANR-20-CE30-0030). V. B.
acknowledges support by the European Research Council under ERC-AdG Grant
PICOGAL-101019746. This work was granted access to the HPC resources of TGCC
under the allocations 2019-A0050407666, 2020-A0080411422, 2021-A0080411422 and
2022-A0130512993 made by GENCI. We thank X. Davoine for his assistance on
particle-tracking diagnostics.

After this manuscript was submitted, Ref. Demidem  _et al._ (2022) appeared,
which shows that the interaction of a shock with a monochromatic linear
eigenmode of the upstream plasma leads to particle acceleration in the
resultant downstream turbulence.

## References

  * Guo  _et al._ (2014) Fan Guo, Hui Li, William Daughton, and Yi-Hsin Liu, “Formation of hard power laws in the energetic particle spectra resulting from relativistic magnetic reconnection,” Phys. Rev. Lett. 113, 155005 (2014).
  * Kagan  _et al._ (2015) D. Kagan, L. Sironi, B. Cerutti, and D. Giannios, “Relativistic magnetic reconnection in pair plasmas and its astrophysical applications,” Space Sci. Rev. 191, 545–573 (2015).
  * Werner and Uzdensky (2017) Gregory R. Werner and Dmitri A. Uzdensky, “Nonthermal Particle Acceleration in 3D Relativistic Magnetic Reconnection in Pair Plasma,” Astrophys. J. Lett. 843, L27 (2017).
  * Petropoulou  _et al._ (2019) Maria Petropoulou, Lorenzo Sironi, Anatoly Spitkovsky, and Dimitrios Giannios, “Relativistic Magnetic Reconnection in Electron-Positron-Proton Plasmas: Implications for Jets of Active Galactic Nuclei,” Astrophys. J. Lett. 880, 37 (2019).
  * Spitkovsky (2008) Anatoly Spitkovsky, “Particle Acceleration in Relativistic Collisionless Shocks: Fermi Process at Last?” Astrophys. J. Lett. 682, L5 (2008).
  * Sironi  _et al._ (2015a) L. Sironi, U. Keshet, and M. Lemoine, “Relativistic Shocks: Particle Acceleration and Magnetization,” Space Sci. Rev. 191, 519–544 (2015a).
  * Zhdankin  _et al._ (2017) Vladimir Zhdankin, Gregory R. Werner, Dmitri A. Uzdensky, and Mitchell C. Begelman, “Kinetic Turbulence in Relativistic Plasma: From Thermal Bath to Nonthermal Continuum,” Phys.Rev. Lett. 118, 055103 (2017).
  * Comisso and Sironi (2018) Luca Comisso and Lorenzo Sironi, “Particle acceleration in relativistic plasma turbulence,” Phys. Rev. Lett. 121, 255101 (2018).
  * Zhdankin  _et al._ (2019) Vladimir Zhdankin, Dmitri A. Uzdensky, Gregory R. Werner, and Mitchell C. Begelman, “Electron and ion energization in relativistic plasma turbulence,” Phys. Rev. Lett. 122, 055101 (2019).
  * Bresci  _et al._ (2022) Virginia Bresci, Martin Lemoine, Laurent Gremillet, Luca Comisso, Lorenzo Sironi, and Camilia Demidem, “Nonresonant particle acceleration in strong turbulence: Comparison to kinetic and MHD simulations,” Phys. Rev. D 106, 023028 (2022).
  * Bykov  _et al._ (2012) A. Bykov, N. Gehrels, H. Krawczynski, M. Lemoine, G. Pelletier, and M. Pohl, “Particle Acceleration in Relativistic Outflows,” Sp. Sc. Rev. 173, 309–339 (2012).
  * Begelman and Kirk (1990) Mitchell C. Begelman and John G. Kirk, “Shock-Drift Particle Acceleration in Superluminal Shocks: A Model for Hot Spots in Extragalactic Radio Sources,” Astrophys. J. 353, 66 (1990).
  * Lemoine  _et al._ (2006) M. Lemoine, G. Pelletier, and B. Revenu, “On the Efficiency of Fermi Acceleration at Relativistic Shocks,” Astrophys. J. Lett. 645, L129–L132 (2006).
  * Niemiec  _et al._ (2006) Jacek Niemiec, Michał Ostrowski, and Martin Pohl, “Cosmic-Ray Acceleration at Ultrarelativistic Shock Waves: Effects of Downstream Short-Wave Turbulence,” Astrophys. J 650, 1020–1027 (2006).
  * Pelletier  _et al._ (2009) Guy Pelletier, Martin Lemoine, and Alexandre Marcowith, “On Fermi acceleration and magnetohydrodynamic instabilities at ultra-relativistic magnetized shock waves,” Mon. Not. R. Astron. Soc. 393, 587–597 (2009).
  * Sironi and Spitkovsky (2009) Lorenzo Sironi and Anatoly Spitkovsky, “Particle Acceleration in Relativistic Magnetized Collisionless Pair Shocks: Dependence of Shock Acceleration on Magnetic Obliquity,” Astrophys. J. 698, 1523–1549 (2009).
  * Lemoine and Pelletier (2010) M. Lemoine and G. Pelletier, “On electromagnetic instabilities at ultra-relativistic shock waves,” Mon. Not. Roy. Astron. Soc. 402, 321–334 (2010).
  * Sironi  _et al._ (2013) L. Sironi, A. Spitkovsky, and J. Arons, “The Maximum Energy of Accelerated Particles in Relativistic Collisionless Shocks,” Astrophys. J. 771, 54 (2013).
  * Note (1) We define the magnetization level as $\sigma\equiv\langle B^{2}\rangle/4\pi\langle\epsilon\rangle$, in terms of mean-squared magnetic field $\langle B^{2}\rangle$ and the energy density $\langle\epsilon\rangle=n\langle\gamma\rangle m_{\rm e}c^{2}$ as measured in the simulation (downstream) frame; $n$ represents the total apparent density. When measured in the comoving turbulence frame, the resulting value of $\sigma$ may be a factor $\sim 2$ larger.
  * Plotnikov  _et al._ (2018) Illya Plotnikov, Anna Grassi, and Mickael Grech, “Perpendicular relativistic shocks in magnetized pair plasma,” Mon. Not. R. Astron. Soc. 477, 5238–5260 (2018).
  * Sironi  _et al._ (2015b) Lorenzo Sironi, Maria Petropoulou, and Dimitrios Giannios, “Relativistic jets shine through shocks or magnetic reconnection?” Mon. Not. R. Astron. Soc. 450, 183–191 (2015b).
  * Lemoine (2022) Martin Lemoine, “First-Principles Fermi Acceleration in Magnetized Turbulence,” Phys. Rev. Lett. 129, 215101 (2022), arXiv:2210.01038 [astro-ph.HE] .
  * Zank  _et al._ (2002) G. P. Zank, Ye Zhou, W. H. Matthaeus, and W. K. M. Rice, “The interaction of turbulence with shock waves: A basic model,” Phys. Fluids 14, 3766–3774 (2002).
  * Mizuno  _et al._ (2011) Yosuke Mizuno, Martin Pohl, Jacek Niemiec, Bing Zhang, Ken-Ichi Nishikawa, and Philip E. Hardee, “Magnetic-field Amplification by Turbulence in a Relativistic Shock Propagating Through an Inhomogeneous Medium,” Astrophys. J. 726, 62 (2011).
  * Lemoine (2016) Martin Lemoine, “A corrugated termination shock in pulsar wind nebulae?” J. Plasma Phys. 82, 635820401 (2016).
  * Lemoine  _et al._ (2016) Martin Lemoine, Oscar Ramos, and Laurent Gremillet, “Corrugation of Relativistic Magnetized Shock Waves,” Astrophys. J. 827, 44 (2016).
  * Demidem  _et al._ (2018) Camilia Demidem, Martin Lemoine, and Fabien Casse, “Relativistic magnetohydrodynamical simulations of the resonant corrugation of a fast shock front,” Mon. Not. R. Astron. Soc. 475, 2713–2723 (2018).
  * Trotta  _et al._ (2021) Domenico Trotta, Francesco Valentini, David Burgess, and Sergio Servidio, “Phase space transport in the interaction between shocks and plasma turbulence,” Proc. Natl. Acad. Sci. USA 118, 2026764118 (2021).
  * Nakanotani  _et al._ (2022) M. Nakanotani, G. P. Zank, and L. L. Zhao, “Turbulence-dominated Shock Waves: 2D Hybrid Kinetic Simulations,” Astrophys. J. 926, 109 (2022).
  * Lefebvre  _et al._ (2003) E. Lefebvre, N. Cochet, S. Fritzler, V. Malka, M. M. Aléonard, J. F. Chemin, S. Darbon, L. Disdier, J. Faure, A. Fedotoff, O. Landoas, G. Malka, V. Méot, P. Morel, M. Rabec LeGloahec, A. Rouyer, Ch. Rubbelynck, V. Tikhonchuk, R. Wrobel, P. Audebert, and C. Rousseaux, “Electron and photon production from relativistic laser plasma interactions,” Nucl. Fusion 43, 629–633 (2003).
  * Lemoine  _et al._ (2019) Martin Lemoine, Laurent Gremillet, Guy Pelletier, and Arno Vanthieghem, “Physics of Weibel-Mediated Relativistic Collisionless Shocks,” Phys. Rev. Lett. 123, 035101 (2019).
  * Vanthieghem  _et al._ (2022) Arno Vanthieghem, Martin Lemoine, and Laurent Gremillet, “Origin of Intense Electron Heating in Relativistic Blast Waves,” Astrophys. J. Lett. 930, L8 (2022).
  * Note (2) $\omega_{\rm p}\equiv(4\pi n_{\infty}e^{2}/m_{\rm e})^{1/2}$ denotes the nonrelativistic plasma frequency of the far upstream plasma, with $n_{\infty}=2n_{0}/\gamma_{\infty}$ the (total) proper density, $n_{0}$ the apparent density of one species.
  * TenBarge  _et al._ (2014) J.M. TenBarge, G.G. Howes, W. Dorland, and G.W. Hammett, “An oscillating Langevin antenna for driving plasma turbulence simulations,” Comp. Phys. Commun. 185, 578–589 (2014).
  * (35) Specific technical details on the numerical methods and additional simulation results can be found in the Supplemental Material.
  * Kirk and Duffy (1999) J. G. Kirk and P. Duffy, “Particle acceleration and relativistic shocks,” Journal of Physics G Nuclear Physics 25, R165–R194 (1999).
  * Blandford and Eichler (1987) R. Blandford and D. Eichler, “Particle acceleration at astrophysical shocks: A theory of cosmic ray origin,” Phys. Rep. 154, 1–75 (1987).
  * Kirk  _et al._ (2000) J. G. Kirk, A. W. Guthmann, Y. A. Gallant, and A. Achterberg, “Particle Acceleration at Ultrarelativistic Shocks: An Eigenfunction Method,” Astrophys. J. 542, 235–242 (2000).
  * Achterberg  _et al._ (2001) Abraham Achterberg, Yves A. Gallant, John G. Kirk, and Axel W. Guthmann, “Particle acceleration by ultrarelativistic shocks: theory and simulations,” Mon. Not. R. Astron. Soc. 328, 393–408 (2001).
  * Takamoto and Kirk (2015) Makoto Takamoto and John G. Kirk, “Rapid Cosmic-ray Acceleration at Perpendicular Shocks in Supernova Remnants,” Astrophys. J. 809, 29 (2015).
  * Matsumoto  _et al._ (2017) Yosuke Matsumoto, Takanobu Amano, Tsunehiko N. Kato, and Masahiro Hoshino, “Electron Surfing and Drift Accelerations in a Weibel-Dominated High-Mach-Number Shock,” Phys. Rev. Lett. 119, 105101 (2017).
  * Wong  _et al._ (2020) Kai Wong, Vladimir Zhdankin, Dmitri A. Uzdensky, Gregory R. Werner, and Mitchell C. Begelman, “First-principles Demonstration of Diffusive-advective Particle Acceleration in Kinetic Simulations of Relativistic Plasma Turbulence,” Astrophys. J. Lett. 893, L7 (2020).
  * Celotti and Ghisellini (2008) Annalisa Celotti and Gabriele Ghisellini, “The power of blazar jets,” Mon. Not. R. Astron. Soc. 385, 283–300 (2008).
  * Burgess  _et al._ (2020) J. Michael Burgess, Damien Bégué, Jochen Greiner, Dimitrios Giannios, Ana Bacelj, and Francesco Berlato, “Gamma-ray bursts as cool synchrotron sources,” Nat. Astron. 4, 174–179 (2020).
  * Demidem  _et al._ (2022) Camilia Demidem, Joonas Nättilä, and Alexandra Veledina, “Relativistic Collisionless Shocks in Inhomogeneous Magnetized Plasmas,” arXiv e-prints , arXiv:2212.06053 (2022), arXiv:2212.06053 [astro-ph.HE] .


# Co-learning Planning and Control Policies Constrained by Differentiable
Logic Specifications

Zikang Xiong, Daniel Lawson, Joe Eappen, Ahmed H. Qureshi, and Suresh
Jagannathan Authors affiliate to Computer Science Department, Purdue
University, IN 47906, USA. (E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Synthesizing planning and control policies in robotics is a fundamental task,
further complicated by factors such as complex logic specifications and high-
dimensional robot dynamics. This paper presents a novel reinforcement learning
approach to solving high-dimensional robot navigation tasks with complex logic
specifications by co-learning planning and control policies. Notably, this
approach significantly reduces the sample complexity in training, allowing us
to train high-quality policies with much fewer samples compared to existing
reinforcement learning algorithms. In addition, our methodology streamlines
complex specification extraction from map images and enables the efficient
generation of long-horizon robot motion paths across different map layouts.
Moreover, our approach also demonstrates capabilities for high-dimensional
control and avoiding suboptimal policies via policy alignment. The efficacy of
our approach is demonstrated through experiments involving simulated high-
dimensional quadruped robot dynamics and a real-world differential drive robot
(TurtleBot3) under different types of task specifications.

##  I Introduction

Figure 1:  The differentiable specification encodes a task visiting location
$A,B,C$ repeatedly, and the predicate $AvoidObstacles$ is a Signed Distance
Field (SDF) directly specified from a map image. _Differentiable_ symbolic
specifications constrain policy updates and notably reduce the _sample
complexity_ of training policies. Based on a different initial position and
map layouts, the planning policy crafts paths in compliance with the given
specification. The control policy is a goal-conditioned policy, following the
paths laid out by the planner. It learns to meet goals generated by the
planning policy. The planning policy and control policy are aligned with each
other during training.

Synthesizing planning and control policies is among the core tasks of
robotics. The planning policy determines a path comprising a sequence of robot
configurations between the given start and goal. In contrast, the control
policy helps robots interact with the physical environment allowing them to
follow the given plan to reach a goal. However, synthesizing planning and
control policies is challenging since the high-dimensional robot dynamics
introduce a vast space of possible movements and reactions, making it
difficult to predict and control every potential outcome. In addition, the
planning policy may be subject to complex logic specifications, which dictate
how a robot should behave under certain conditions or constraints. These
specifications can range from simple commands like “reaching A”, and “avoiding
B” to intricate sets of rules like “surveilling crucial spots and stopping
after a certain condition is triggered”. Merging planning and control while
adhering to these logic rules and managing the inherent complexities of robot
dynamics presents a significant challenge, even for state-of-the-art
approaches.

We leverage Reinforcement Learning (RL) for high-dimensional robot planning
and control under complex dynamics and sensor observation. To ensure that
robots adhere to behavioral rules, temporal logic can be incorporated into RL
[1, 2, 3, 4, 5]. This offers a structured way to define and check a robot’s
behavior over time, ensuring tasks are completed in order. However,
incorporating temporal logic into RL presents its own challenges. For example,
previous approaches [1, 2, 3, 4, 6, 5, 7, 8, 9, 10] that have used temporal
logic for reward shaping, involves the construction of complicated reward
functions that require large amounts of samples [11]. Furthermore,
handcrafting these temporal logic specifications can be challenging. For
instance, situations where map layouts are represented as images or have
irregular obstacle shapes present numerous challenges in defining precise,
actionable logical rules. Although existing solvers [12] can find robot paths
under a given logical specification, they lack scalability and can only handle
linear and, to some extent, quadratic constraints, which is not ideal for
practical scenarios, as highlighted in our experiments. Furthermore, these
methods only account for path planning without considering errors from the
underlying controller or vice versa, which often leads to failure in
executions.

Therefore, we propose a novel approach, called Differentiable Specifications
Constrained Reinforcement Learning (DSCRL), to address the above challenges
using the following key features:

Lower Sample Complexity: We introduce a new methodology that integrates
differentiable specifications into constrained RL, which significantly lowers
the sample complexity of training.

Control and Planning Alignment: Learning both planning and control policies
separately faces the challenges of suboptimal policies due to lack of
alignment. We analyze this issue and demonstrate that our approach allows for
better alignment between planning and control policies.

Efficient Planning: Our approach generates long horizon plans in cluttered
environments, which is typically hard for other RL algorithms as well as
existing Signal Temporal Logic (STL) solvers to handle.

Specifications from Image: Our approach can extract irregular obstacle
specifications directly from different map images, and solve them efficiently
with a novel neural planning policy built upon pre-trained backbones.

High-dimensional Policy: Our approach has been effectively tested on simulated
high-dimensional quadruped robot dynamics and a real-world differential drive
robot (TurtleBot3) with LiDAR observation.

##  II Related Work

Temporal Logic and RL:  This work considers symbolic specifications written in
temporal logic [13, 14], which are widely used in specifying various planning
and control tasks[15, 16, 17, 18]. Along with the advances of RL in
stochastic, model-free settings, a line of work [1, 2, 3, 4, 6, 5, 7, 8, 9,
10] have considered combining LTL-specified tasks with reinforcement learning.
However, these approaches formulate such integration using temporal logic as
rewards, leading to inefficient or even intractable algorithms [19].

Constrained RL: Constrained reinforcement learning [20, 21, 22] provides an
efficient approach to enforce constraints on policies. Given a differentiable
specification, these methods can enforce constraints only on the policy
without shaping rewards. However, prior work did not consider how to
synergistically take advantage of policies with differentiable formal
constraints [23].

Gradient and Neural Path Planning: Our work also relates to gradient-based
motion planning [23, 24, 25, 26, 27]. While these techniques are typically
only applicable when the dynamics are known and deterministic, our focus is on
planning in the face of unknown and stochastic dynamics. Our high-level
planning policy is related to planning networks [28]. However, learning a
planning neural network in an RL loop with task specifications, which aligns
both the planning policy and control policy, has not been explored previously.

##  III Background

###  III-A Policy Gradient

Consider a trajectory $\tau$ sampled from policy $\pi_{\theta}$. The expected
return $R(\tau)$ for a policy parameterized by $\theta$ is given by
$J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}[R(\tau)]$. The policy gradient
theorem [29] states that the gradient of the expected return is given by
$\nabla_{\theta}J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}[\nabla_{\theta}\mathcal{P}(\pi_{\theta},\tau)R(\tau)]$,
where $\mathcal{P}(\pi_{\theta},\tau)$ is the log probability of trajectory
$\tau$ under policy $\pi_{\theta}$. This implies that the probability of
generating a trajectory $\tau$ with a high return is maximized compared to
those with a lower return. In practice, a baseline $b(\tau)$ is also
subtracted from the reward to reduce the variance of the gradient estimator
[30].

###  III-B Signal Temporal Logic

The syntax of STL contains both first-order logic operators $\land$ (and),
$\lnot$ (not), $\lor$ (or), $\Rightarrow$ (implies), and temporal operators
$\bigcirc$ (next), $\Diamond_{[a,b]}$ (eventually), $\square_{[a,b]}$
(globally), $\ \mathcal{U}_{[a,b]}$ (until). The initial time $a$ and end time
$b$ “truncate” a path. For example, $\square_{[a,b]}$ qualifies property that
globally holds during time $a$ and $b$. The syntax of STL is recursively
defined via the following grammar:

$\displaystyle\phi:=$
$\displaystyle\top\mid\bot\mid\mathcal{F}\mid\lnot\phi\mid\phi\land\psi\mid\phi\lor\psi\mid\phi\Rightarrow\psi$
(1)
$\displaystyle\mid\bigcirc\phi\mid\Diamond_{[a,b]}\phi\mid\square_{[a,b]}\phi\mid\phi\
\mathcal{U}_{[a,b]}\psi,$

where $\mathcal{F}:\mathbb{R}^{n}\rightarrow\mathbb{R}$ is a predicate
function mapping a state to a real value (e.g., a function computes the
distance to an obstacle surface). The STL semantics
$\rho(\tau,\phi):\mathbb{R}^{T\times n}\rightarrow\mathbb{R}$ is defined over
a formula $\phi$ and a trajectory $\tau$ with length $T$, and maps them to a
real value [23]. For convenience, we denote $\phi(\tau):=\rho(\tau,\phi)$. As
an example, the semantics of $\square_{[a,b]}\phi$ is:

$\displaystyle\rho(\tau,\square_{[a,b]}\phi)=\min_{t\in[a,b]}\rho(\tau_{[t:b]},\phi),$
(2)

meaning that the formula $\square_{[a,b]}\phi$ holds if and only if $\phi$
holds at every time step ($\forall t\in[a,b],\phi(\tau_{[t:b]})>0$).
$\tau_{[t:b]}$ is a path slice from $t$ to $b$. The full semantics can be
found in previous works such as [23, 31]. In practice, $\min,\max$ in STL
semantics can cause gradient vanishing, which can be alleviated by using
soften techniques [31].

##  IV Approach

This section presents the key components of our method for solving complex
robot navigation tasks. In Sec. IV-A, we will introduce a novel planning
policy $\pi^{h}_{\phi}$ which generates paths based on task specification
$\phi$ and map images, while also aligns with the control policy $\pi^{l}$.
Then, we introduce the control policy $\pi^{l}$ which follows paths and aligns
itself with the planning policy $\pi^{h}_{\phi}$ in Sec. IV-B.

###  IV-A Planning Policy

The planning policy $\pi^{h}_{\phi}$ is trained to meet a task specification
$\phi$. Given an initial position $x_{0}$ and an arbitrary map image $I$, the
planned path $\tau$ is sampled from the distribution predicted by the planning
policy, where $\tau=(g_{0},g_{1},\dots,g_{T})$, $g_{0}=x_{0}$, and $T$ is the
max time horizon, i.e.,$\tau\sim\pi^{h}_{\phi}(x_{0},I)$. STL has the
following definition:

$\phi(\tau)=\begin{cases}>0,&\text{if }\tau\text{ satisfies }\phi,\\\
<0,&\text{if }\tau\text{ does not satisfy }\phi.\end{cases}$ (3)

####  IV-A1 Differentiable STL and Control Feedback

The quantitative semantics of $\phi$ is provided by STL [14]. Because $\phi$
is differentiable [23], we can directly use $\phi$ as a part of the training
objective, and turn the policy search into a gradient optimization problem.
Meanwhile, to align the planning policy $\pi^{h}_{\phi}$ with the feedback
from the control policy $\pi^{l}$, while also satisfying $\phi$, we formulate
the problem by minimizing the following loss function:

$\displaystyle\mathcal{L}_{\pi^{h}_{\phi}}=-\mathop{\mathbb{E}}_{\begin{subarray}{c}I\sim\mathcal{I},x_{0}\sim\mathcal{X}_{0},\\\
\tau\sim\pi^{h}_{\phi}(x_{0},I)\end{subarray}}\biggl{(}\underbrace{\mathcal{P}(\tau,\pi^{h}_{\phi})\cdot
r^{h}(\tau,\pi^{l})}_{\text{PG}}+\lambda\phi(\tau)\biggr{)}.$ (4)

The reward $r^{h}(\tau,\pi^{l})$ here is the negative of the steps to reach
the goal $g_{T}$. When following the path $\tau$ with the control policy
$\pi^{l}$, fewer steps mean a higher reward. The
$\mathcal{P}(\tau,\pi^{h}_{\phi})$ is the log probability of the path $\tau$
under policy $\pi^{h}_{\phi}$. The task specification $\phi$ is formulated as
a Lagrangian constraint with a multiplier $\lambda$. Optimizing (4) will
maximize the expectation of reward, which is positively correlated to the
expectation of $\mathcal{P}(\tau,\pi^{h}_{\phi})\cdot r^{h}(\tau,\pi^{l})$,
and generate paths satisfied $\phi$ (i.e., $\phi(\tau)>0$). In practice,
$r^{h}(\tau,\pi^{l})$ will subtract a baseline to reduce the policy gradient
variance [30]. The Lagrangian multiplier $\lambda$ can be dynamically updated
for better performance [32].

One may only train the planning policy $\pi^{h}_{\phi}$ with the task
specification $\phi$ without control feedback (i.e., ignoring
$\mathcal{P}(\tau,\pi^{h}_{\phi})\cdot r^{h}(\tau,\pi^{l})$). However, due to
the complexity of the high-dimensional dynamics we are handling, it is
impractical to encode dynamics constraints in STL. Thus, despite the fact that
the planner trained with $\phi(\tau)$ can generate paths satisfying $\phi$,
these paths are not necessary to be easy to follow by the control policy. For
example, it may generate a path with lots of sharp turns. More importantly,
choosing the goal for a high-dimensional, non-linear goal-conditioned control
policy $\pi^{l}$ is non-trivial. Naively choosing a path based on Euclidean
distance does not necessarily mean the minimum in the steps. Thus, optimizing
the $\pi^{h}_{\phi}$ with control feedback is necessary to avoid suboptimal
policies.

Previous RL approaches [1, 2, 3, 6, 4, 7, 8, 9, 5, 10] have investigated
training policies with STL. However, these approaches are based on reward
shaping, which is different from our constraint-based formalization. The
reward-shaping formulation minimizes the following loss function:

$\displaystyle\mathcal{L}_{RS}=-\mathop{\mathbb{E}}_{\begin{subarray}{c}I\sim\mathcal{I},x_{0}\sim\mathcal{X}_{0},\\\
\tau\sim\pi^{h}_{\phi}(x_{0},I)\end{subarray}}\underbrace{\mathcal{P}(\tau,\pi^{h}_{\phi})\cdot\biggl{(}r^{h}(\tau,\pi^{l})+\lambda\phi(\tau)\biggr{)}}_{\text{PG}}.$
(5)

The policy gradient part of (4) and (5) is marked as PG. Getting an accurate
policy gradient typically requires many samples, and a more complex reward
function typically requires more samples to estimate the policy gradient.
Thus, as illustrated in (4), decoupling the complex STL specification $\phi$
from the policy gradient part and directly backpropagating through $\phi$ is
expected to reduce the sample complexity in training (i.e., converge faster).
In addition, the estimated policy gradients are only meaningful around
existing samples [20, 22]. However, during iterative updates, the policy can
quickly shift to a region with few samples and make the policy gradient
updates random. In contrast, when backpropagating through $\phi$, as the
gradient is not estimated from “local” samples, but computed from $\phi$, our
objective (4) also demonstrates more stable performance during training.

####  IV-A2 Model Design of Planning Policy

Figure 2: The planning policy model has three parts. The Mobile-SAM [33],
whose input is an arbitrary map image, outputs a mask of obstacles. The mask
image is then fed into a Mobile-ViT encoder [34] that outputs the map
embedding. The transformer path decoder takes the map embedding and the
initial position as input and outputs the deviation between subgoals. Mobile-
SAM and Mobile-ViT are pre-trained models with frozen weights (denoted by ❄),
and the decoder is trained with the loss function in (4).

Given an arbitrary map layout image $I$, the planning policy model processes
it in stages to make path planning efficient. Initially, to reduce the noise
inherent in real images, the MobileSAM [33] extracts an obstacle mask $M$ from
the map image $I$. Subsequently, the MobileViT encoder [34] maps this mask
image $M$ into a map embedding $E$. Lastly, $Decoder_{\theta}$, parameterized
by $\theta$ (the component we exclusively trained) computes subgoal
deviations, denoted as $\frac{d\tau}{dt}$, using the map embedding $E$ and the
robot’s initial position $x_{0}$:

$\displaystyle\frac{d\tau}{dt}=tanh(Decoder_{\theta}(E,x_{0}))\times\Delta g.$
(6)

Here, $\Delta g$ is the maximum deviation between subgoals. The deviation is
scaled by a $tanh$ function to ensure that the subgoals are within a
reasonable range. The subgoal in a planned path
$\tau=[g_{0},g_{1},\dots,g_{T}]$ is generated by integrating the deviation
with the initial position $x_{0}$:
$g_{t}=x_{0}+\int_{0}^{t}\frac{d\tau}{dt}dt.$ The decoder is a transformer
decoder [35] which will be trained with the loss function in (4).

####  IV-A3 Extract Obstacle Specification from Map Image

One limitation of STL is that all specifications have to be encoded by users,
including all obstacles on the map. This can be challenging as the map layout
becomes complicated and implicit (e.g., the map is given as an image). In
addition, state-of-the-art STL solvers, such as [12], can only encode
obstacles with linear or limited quadratic constraints, making it challenging
to handle real-world obstacles with irregular shapes.

Our approach extends STL with specifications directly generated from images,
by converting them into a Signed Distance Field (SDF) [36]. The SDF is a
function that returns the distance to the nearest obstacle surface for any
given point in an obstacle mask $M$:

$\displaystyle SDF(g,M)=sign(g)\cdot\min_{p\in M}\|g-p\|_{2},$ (7)

Here, $sign(g)$ is negative if $g$ is inside an obstacle, and positive if $g$
is outside an obstacle. We directly use $SDF$ as an obstacle avoidance
predicate in STL, which is consistent with the semantics of STL (3). As an
example, the predicate $AvoidObstacles$ in Fig. 1 is defined as:

$\displaystyle AvoidObstacles=\square_{[0,T]}SDF(\cdot,M).$ (8)

The $\square_{[0,T]}$ is a $\min$ operator defined in the path
$\tau=[g_{0},g_{1},\dots,g_{T}]$, which returns the minimum value of
$SDF(g_{t},M)$ for all $g_{t}$. In other words, $AvoidObstacles(\tau)$ is
positive if and only if $\forall g_{t}\in\tau,SDF(g_{t},M)>0$, which means
that all $g_{t}$ are outside the obstacles.

The mask $M$ is stored as a binary image, where the obstacle pixels are 1 and
the free space pixels are 0. We can extract an SDF from the mask image $M$
with a KD-Tree [37]. Given a mask image with $W\times H$ pixels, we build a
KD-Tree with the outline points of the obstacles, where each point is a 2D
index of a pixel in the mask image. A $\mathit{KDTree_{M}}$ constructed
specifically for a mask $M$, when input a pixel index, returns the nearest
pixel that is part of an obstacle outline within the mask $M$. If the field
size is $m\times n$ meters in the real world, we can compute a transformation
$\mathcal{T}$ from the pixel index to the real-world coordinate, and its
inversion $\mathcal{T}^{-1}$, with $W,H,m,n$. An SDF is implemented as:

$\displaystyle
SDF(g,M)=sign(g)\|\mathcal{T}\cdot\mathit{KDTree_{M}}(\mathcal{T}^{-1}\cdot
g)-g\|_{2}.$ (9)

The gradient can be backpropagated through (9) to update the planning policy.

###  IV-B Control Policy

The low-level control policy $\pi^{l}(o_{t}\mid g)$ is a goal-conditioned
policy [38] trained with PPO [22], and $o_{t}$ is the robot observation at
time $t$. The control reward $r^{l}_{t}$ at time $t$ is defined as

$\displaystyle
r^{l}(o_{t},o_{t+1},g)=\|g-pos(o_{t})\|_{2}-\|g-pos(o_{t+1})\|_{2},$ (10)

where $pos(o_{t})$ and $pos(o_{t+1})$ are the positions of the robot at time
$t$ and $t+1$, respectively. The reward is positive if and only if the robot
gets closer to the planned goal $g$.

The planning policy $\pi^{h}_{\phi}$ generates the goal distribution
$\mathcal{G}$ for the control policy. As the planning policy is updated,
$\mathcal{G}$ will change. Thus, the control policy $\pi^{l}$ needs to be
updated to follow the new goal distribution. Formally, the general goal for
the goal-conditioned RL is to find an optimal policy $\pi^{*}$ with

$\displaystyle\pi^{*}=\mathop{\mathrm{argmax}}_{\pi}\mathop{\mathbb{E}}_{\begin{subarray}{c}g\sim\mathcal{G}\end{subarray}}\mathop{\mathbb{E}}_{\begin{subarray}{c}o_{0}\sim\mathcal{O}_{0},a_{t}\sim\pi(o_{t}\mid
g),\\\ o_{t+1}\sim P(o_{t+1}\mid
o_{t},a_{t})\end{subarray}}\biggl{(}\sum_{t=0}^{T}\gamma^{t}r^{l}(o_{t},o_{t+1},g)\biggr{)},$
(11)

where $\mathcal{O}_{0}$ is the initial observation distribution, and
$P(o_{t+1}\mid o_{t},a_{t})$ is the transition function. The $\gamma$ is the
discount factor. Let
$V^{\pi}(g)=\mathop{\mathbb{E}}_{\begin{subarray}{c}o_{0}\sim\mathcal{O}_{0},a_{t}\sim\pi(o_{t}\mid
g),\\\ o_{t+1}\sim P(o_{t+1}\mid
o_{t},a_{t})\end{subarray}}\biggl{(}\sum_{t=0}^{T}\gamma^{t}r^{l}(o_{t},o_{t+1},g)\biggr{)}$
be the value function of the control policy $\pi$ with goal $g$, the objective
function in (11) can be rewritten as:

$\displaystyle\pi^{*}=\mathop{\mathrm{argmax}}_{\pi}\sum_{g}P_{\mathcal{G}}(g)V^{\pi}(g).$
(12)

The $P_{\mathcal{G}}(g)$ is the probability of the goal $g$ under the goal
distribution $\mathcal{G}$. The change of the goal distribution $\mathcal{G}$
(as planning policy updates) will change $P_{\mathcal{G}}$, and thus the
optimal policy $\pi^{*}$ shifts. Thus, the control policy $\pi^{l}$ needs to
be updated for the new goal distribution. This can be achieved by training the
control policy with any RL algorithm by sampling the goal $g$ with the new
goal distribution [38].

###  IV-C Overall Pipeline

We summarize the training and deployment pipeline of our approach in this
section. First, users express their goals without delving into intricate maps,
as exemplified by
$\square_{[0,60]}(\Diamond_{[0,30]}A\land\Diamond_{[0,30]}B\land\Diamond_{[0,30]}C)$
in Fig. 1. Next, the $AvoidObstacles$ specification is derived from map images
with the method discussed in Sec. IV-A3. For policy training, we initially
train $\pi^{l}$ using goals sampled from $\pi^{h}_{\phi}$ while keeping
$\pi^{h}_{\phi}$ fixed. Then, we train the planning policy $\pi^{h}_{\phi}$
using (4) with the control policy $\pi^{l}$ fixed. This back-and-forth is
important as simultaneous updates can introduce instability due to the
evolving objectives in (4) and (12). We iterate through this process until
both policies converge. During deployment, the planning policy
$\pi^{h}_{\phi}$ generates a path $\tau$ from the initial position $x_{0}$ and
an arbitrary map image $I$. The control policy $\pi^{l}$ follows $\tau$ to
achieve the task $\phi$.

Figure 3: In the Sequence task, the Turtlebot3 must visit the three blue
regions sequentially. The Cover task requires the robot to traverse all three
blue regions in any sequence. For the Branch task, the robot chooses between
visiting either the two blue regions or the two green regions. The Loop task
asks the robot to cyclically traverse the three blue regions. Lastly, the
Signal task requires the robot to loop among three blue regions and exit the
loop after visiting the lower blue region twice, and finally visiting the
yellow region. The yellow tape in the center is used for calibrating the
coordinates.

##  V Experiments

###  V-A Setup

####  V-A1 Tasks

The tasks evaluated are depicted in Fig. 3. The STL specifications of the
Sequence task is $\bigwedge_{i=1}^{N}\Diamond_{[t_{i},t_{i+1}]}\phi_{i}$,
where $\phi_{i}$ is a predicate that corresponds to reaching each blue region.
The robot must visit the region specified by
$\phi_{1},\phi_{2},\cdots,\phi_{N}$ in sequence. The Cover task is specified
by $\bigwedge_{i=1}^{N}\Diamond_{[0,T]}\phi_{i}$. Here the robot is asked to
cover all the specified regions in any given order. The Branch task is
specified by
$\bigvee_{i=1}^{\frac{N}{2}}(\Diamond_{[0,T]}\phi_{i}\land\Diamond_{[0,T]}\phi_{\frac{N}{2}+i})$,
the robot must either visit the branch of blue regions ($\phi_{1}$,
$\phi_{3}$) or the branch of green regions ($\phi_{2}$, $\phi_{4}$). The Loop
task
$\square_{[0,T-\lfloor\frac{T}{M}\rfloor]}(\bigwedge_{i=1}^{N}\Diamond_{[0,\lfloor\frac{T}{M}\rfloor]}\phi_{i})$
asks the robot to cyclically traverse the blue regions specified by $\phi_{i}$
for $M$ times. Lastly, the Signal task is specified by
$\square_{[0,T-\lfloor\frac{T}{M}\rfloor]}(\bigwedge_{i=1}^{N}\Diamond_{[0,\lfloor\frac{T}{M}\rfloor]}\phi_{i})\
\mathcal{U}_{[0,1]}\psi_{1}\land\Diamond_{[0,T]}\psi_{2}$. The robot should
loop among the three blue regions ($\phi_{i}$) until it visits the lower blue
region twice ($\psi_{1}$) and finally visits the yellow region ($\psi_{2}$).
All these specifications are appended with $\land\square_{[0,T]}SDF(\cdot,M)$
extracted from map $M$.

####  V-A2 Observation, Action, and Dynamics

The Doggo (shown in Fig. 1) features a 74-dimensional observation space: it
has joint states (44D), IMU data (12D), position (2D), and a 16-beam LiDAR for
obstacle detection. Its action space is 12-dimensional, comprising 8 hip and 4
ankle control commands. On the other hand, TurtleBot3 (shown in Fig. 3)
utilizes 4-dimensional IMU data (including x-axis position, y-axis position,
and the sin and cos of rotation) and a 36-dimensional down-sampled LiDAR for
measuring obstacle distances. Its action commands are dedicated to controlling
the robot’s linear and angular velocity.

###  V-B Policy Performance and Alignment

####  V-B1 Metrics

We evaluate the planning and control policies using Success Rate (SR) and Time
to Reach (TtR) metrics. SR quantifies the proportion of successful plans. A
plan is considered successful if the path $\tau$ satisfies the condition
($\phi(\tau)>0$) without any collisions when followed by the control policy.
TtR calculates the duration (in seconds) needed to achieve the final
destination. Each learned policy is evaluated on 1000 episodes with different
map layouts. There are 10 obstacles for TurtleBot3 with $2.42\times 2.42$
meters maps and 15 obstacles for Doggo with $3\times 3$ meters maps.

TABLE I: Alignment Experiments. The Success Rate (SR) and Time to Reach (TtR) of the control policy are evaluated on 1000 episodes with randomly sampled map layouts. The performance of aligned policies is reported in the column “w.”, while the unaligned policies is reported in the column “w.o.”. The aligned policies perform significantly better than the unaligned policies. All the aligned policies have SR above 96%. Robot | Doggo | Turtlebot3
---|---|---
Metrics |  SR $\uparrow$ |  TtR $\downarrow$ |  SR $\uparrow$ |  TtR $\downarrow$
Align | w.o. | w. | w.o. | w. | w.o. | w. | w.o. | w.
Seq | 87.1% |  97.7% | 64.6 | 46.3 | 79.1% | 98.2% | 105.6 | 92.7
Cover | 83.9% |  96.4% | 73.7 | 57.0 | 73.1% | 96.1% | 92.5 | 81.9
Branch | 87.2% |  98.2% | 56.4 | 44.6 | 91.1% | 97.2% | 81.4 | 73.8
Loop | 83.1% |  98.5% | 70.6 | 51.3 | 85.1% | 98.2% | 114.3 | 94.5
Signal | 79.9% |  97.1% | 72.1 | 57.3 | 79.6% | 97.8% | 118.1 | 106.5

####  V-B2 Performance and Alignment

Experiments in Table I show the performance of our final trained policies and
the effect of policy alignment. The unaligned planning policy $\pi^{h}_{\phi}$
is trained with STL objective $\phi$ without control feedback, and the
unaligned control policy is trained on random goals. The aligned policies are
trained with our approach. All the aligned policies have SR above 96%, which
is significantly higher than the unaligned policies. The aligned policies also
have lower TtR than the unaligned policies. The results demonstrate that our
approach can work effectively on high-dimensional control tasks with various
specifications and maps with high SR and low TtR. Moreover, the results also
show that the alignment between planning and control policies is essential for
good performance, justifying our algorithm design.

####  V-B3 Policy Performance In A Real TurtleBot3

TABLE II: Deploy trained policy to real TurtleBot3. The first row shows the specs. The second and third rows are identical to TABLE I. The results are computed over 6 sampled episodes each.  Spec | Seq | Signal
---|---|---
Metrics |  SR$\uparrow$ |  TtR $\downarrow$ |  SR $\uparrow$ |  TtR $\downarrow$
Align | w.o. | w. | w.o. | w. | w.o. | w. | w.o. | w.
Stat. | 3/6 |  5/6 | 142.7 | 108.3 | 1/6 |  6/6 | 144.7 | 109.1

We evaluated our policies with and without alignment on a real Turtlebot3
robot for the Sequence and Signal tasks. The results in Table II show that our
policies can work effectively in the real world with high SR. Compared with
the unaligned policies, the aligned policies have higher SR and lower TtR.

###  V-C Sample Efficiency and Comparison with Reward Shaping

We focused on the number of samples required for our policies to reach
convergence. Each sample represents a transition, moving from one observation
$o_{t}$ to another $o_{t+1}$ upon taking an action $a_{t}$ and acquiring the
corresponding reward $r^{l}_{t}$. The high-level reward $r^{h}$ is determined
after we collect a full trajectory $\tau$.

To evaluate the efficacy of our methodology, we compare it with the Reward
Shaping (RS) strategy [2] illustrated in (5). Additionally, we benchmarked
against the Reward Machine (RM) [5] strategy that shapes rewards with the
progression of a pre-defined planned path. For example, given a planned path
$A,B,C$, if the planning policy did not reach $A$, the reward is shaped to
encourage the planning policy to reach $A$; if the planning policy reached $A$
but did not reach $B$, the reward is shaped to encourage the planning policy
to reach $B$, and so on. Notice that the RM strategy requires an additional
solver to generate the planned path, which is a stronger requirement than
ours. Our experimentation spanned two distinct settings:

####  V-C1 Two-Stage Training

Figure 4: Align pretrained policies on Cover and Loop tasks. The x-axis is the
number of samples, and the y-axis is the normalized score. The upper black
dashed line is the score of the aligned policies (“w.” columns in Table I),
and the lower black dashed line is the score of the unaligned policies (“w.o.”
columns in Table I). The values are computed over 5 different random seeds.
The results show that our approach can converge with fewer samples than RS and
RM.

In this setup, we first train the planning policy $\pi^{h}_{\phi}$ with STL
objective $\phi$ without control feedback, while the control policy is trained
on random goals. Later, in our approach, $\pi^{h}_{\phi}$ was finetuned using
(4). In contrast, RS trains the $\pi^{h}_{\phi}$ with (5), while RM trains
$\pi^{h}_{\phi}$ with rewards crafted from a path saftisfying $\phi$.
Following this, for all methods, $\pi^{l}$ is trained on goals sampled from
$\pi^{h}_{\phi}$. Unlike RS and RM, our method decouples the STL objective
from the reward and converges faster (Fig. 4). The trade-off between sample
efficiency and reward function complexity is also evident in RS and RM, with
RM outperforming RS with simplified reward functions.

TABLE III: Converging samples when training from scratch. The data shows the number of samples required for each algorithm to converge to the scores near Table I ($\pm 5\%$). The data is scaled by the coefficient $c$ next to the robot name. A dash “-” means that it does not converge in $100\times c$ samples. The results are computed over 5 random seeds.  Robot |  Doggo ($\times 10^{7}$) |  Turtlebot3 ($\times 10^{6}$)
---|---|---
Algo | Ours | RS | RM | Ours | RS | RM
Cover | 5.1 $\pm$ 1.2 | - |  27.4 $\pm$ 7.1 | 7.9 $\pm$ 1.9 |  69.2 $\pm$ 17.1 |  30.9 $\pm$ 5.4
Loop | 5.7 $\pm$ 0.9 | - | - | 9.1 $\pm$ 2.8 | - |  48.7 $\pm$ 4.9

####  V-C2 End-to-End Training

Alternatively, we conduct evaluations by training from scratch where both
$\pi^{h}_{\phi}$ and $\pi^{l}$ were initialized with random weights. In Table
III, we observe that our approach consistently requires fewer samples to
converge when compared to the other two approaches. The results also show RM
performs better than RS, which is consistent with the results in Fig. 4.

###  V-D More Scalable STL Planning

One alternative approach for our problem is solving the STL specifications
with an STL solver (such as STLPY [12]) and tracking the solved path with a
goal-conditioned control policy. However, this approach has its limitations.
First, manual obstacle specification is limited to linear or quadratic
constraints. Second, paths produced by STLPY, while sound and complete w.r.t
specifications, can still lead to collisions or stalls when tracking with an
unaligned control policy. In the Loop task with 100 timesteps, STLPY paths
succeed 14 out of 20 times for Doggo and 13 out of 20 times for TurtleBot3
with the optimal robustness parameter [12]. The planning policy’s SR is 98.5%
for Doggo and 98.2% for TurtleBot3 on Loop task.

Figure 5:  Planning time against time horizon and obstacle count. The spec for
the left is
$\square_{[0,T-\lfloor\frac{T}{3}\rfloor]}(\bigwedge_{i=1}^{3}\Diamond_{[0,\lfloor\frac{T}{3}\rfloor]}\phi_{i})$,
where $T$ is the time horizon. The spec for the right is
$\square_{[0,27]}(\bigwedge_{i=1}^{3}\Diamond_{[0,13]}\phi_{i})\land\bigwedge_{i=1}^{n}Obs_{i}$,
where $\bigwedge_{i}Obs_{i}$ represents the rectangle obstacle predicates with
linear constraints, and $n$ is the number of obstacles. Results from 20 runs
per data point show STLPY’s solving time grows exponentially with horizon and
obstacles, but the neural planning policy remains consistently fast ($<0.5$
secs).

Third, as noted by [12], the STL solver has scalability issues shown in Fig.
5. The planning time for the STL solver increases exponentially with the time
horizon and the number of obstacles.

##  VI Conclusions

In this paper, we present a new method combining differentiable specifications
with constrained RL, which reduces training sample complexity and aligns
planning and control policies more efficiently than current reward-shaping
techniques. Our method performs well in long-horizon planning in cluttered
environments, extracts complex obstacle details from images, and uses them
efficiently with our neural planning policy. It also demonstrates advantages
in high-dimensional dynamics.

## References

  * [1] J. Fu and U. Topcu, “Probably approximately correct mdp learning and control with temporal logic constraints,” _arXiv preprint arXiv:1404.7073_ , 2014\.
  * [2] X. Li, C.-I. Vasile, and C. Belta, “Reinforcement learning with temporal logic rewards,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2017, pp. 3834–3839.
  * [3] M. Hasanbeig, Y. Kantaros, A. Abate, D. Kroening, G. J. Pappas, and I. Lee, “Reinforcement learning for temporal logic control synthesis with probabilistic satisfaction guarantees,” in _2019 IEEE 58th Conference on Decision and Control (CDC)_. IEEE, 2019, pp. 5338–5343.
  * [4] K. Jothimurugan, R. Alur, and O. Bastani, “A composable specification language for reinforcement learning tasks,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [5] R. T. Icarte, T. Q. Klassen, R. Valenzano, and S. A. McIlraith, “Reward machines: Exploiting reward function structure in reinforcement learning,” _Journal of Artificial Intelligence Research_ , vol. 73, pp. 173–208, 2022\.
  * [6] M. Hasanbeig, D. Kroening, and A. Abate, “LCRL: Certified policy synthesis via logically-constrained reinforcement learning,” in _International Conference on Quantitative Evaluation of SysTems_. Springer, 2022.
  * [7] Y. Jiang, S. Bharadwaj, B. Wu, R. Shah, U. Topcu, and P. Stone, “Temporal-logic-based reward shaping for continuing learning tasks,” _arXiv preprint arXiv:2007.01498_ , 2020.
  * [8] A. K. Bozkurt, Y. Wang, M. M. Zavlanos, and M. Pajic, “Control synthesis from linear temporal logic specifications using model-free reinforcement learning,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 10 349–10 355.
  * [9] Z. Xu, I. Gavran, Y. Ahmad, R. Majumdar, D. Neider, U. Topcu, and B. Wu, “Joint inference of reward machines and policies for reinforcement learning,” _Proceedings of the International Conference on Automated Planning and Scheduling_ , vol. 30, no. 1, pp. 590–598, Jun. 2020.
  * [10] H. Zhang and Z. Kan, “Temporal logic guided meta q-learning of multiple tasks,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 3, pp. 8194–8201, 2022.
  * [11] G. Dulac-Arnold, D. Mankowitz, and T. Hester, “Challenges of real-world reinforcement learning,” _arXiv preprint arXiv:1904.12901_ , 2019.
  * [12] V. Kurtz and H. Lin, “Mixed-integer programming for signal temporal logic with fewer binary variables,” _IEEE Control Systems Letters_ , 2022.
  * [13] A. Pnueli, “The temporal logic of programs,” in _18th Annual Symposium on Foundations of Computer Science (sfcs 1977)_ , 1977, pp. 46–57.
  * [14] O. Maler and D. Nickovic, “Monitoring temporal properties of continuous signals,” in _Formal Techniques, Modelling and Analysis of Timed and Fault-Tolerant Systems_. Springer, 2004, pp. 152–166.
  * [15] G. E. Fainekos, H. Kress-Gazit, and G. J. Pappas, “Temporal logic motion planning for mobile robots,” in _Proceedings of the 2005 IEEE International Conference on Robotics and Automation_. IEEE, 2005, pp. 2020–2025.
  * [16] M. Kloetzer and C. Belta, “A fully automated framework for control of linear systems from temporal logic specifications,” _IEEE Transactions on Automatic Control_ , vol. 53, no. 1, pp. 287–297, 2008.
  * [17] H. Kress-Gazit, G. E. Fainekos, and G. J. Pappas, “Temporal-logic-based reactive mission and motion planning,” _IEEE Transactions on Robotics_ , vol. 25, no. 6, pp. 1370–1381, 2009.
  * [18] T. Wongpiromsarn, U. Topcu, and R. M. Murray, “Receding horizon temporal logic planning,” _IEEE Transactions on Automatic Control_ , vol. 57, no. 11, pp. 2817–2830, 2012.
  * [19] C. Yang, M. L. Littman, and M. Carbin, “On the (in)tractability of reinforcement learning for LTL objectives,” in _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022_ , 2022, pp. 3650–3658.
  * [20] J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in _International conference on machine learning_. PMLR, 2015, pp. 1889–1897.
  * [21] J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” in _International conference on machine learning_. PMLR, 2017, pp. 22–31.
  * [22] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” _arXiv preprint arXiv:1707.06347_ , 2017\.
  * [23] K. Leung, N. Aréchiga, and M. Pavone, “Backpropagation through signal temporal logic specifications: Infusing logical structure into gradient-based methods,” _Int. Journal of Robotics Research_ , 2022.
  * [24] K. Leung, N. Aréchiga, and M. Pavone, “Back-propagation through signal temporal logic specifications: Infusing logical structure into gradient-based methods,” in _International Workshop on the Algorithmic Foundations of Robotics_. Springer, 2020, pp. 432–449.
  * [25] N. Ratliff, M. Zucker, J. A. Bagnell, and S. Srinivasa, “Chomp: Gradient optimization techniques for efficient motion planning,” in _2009 IEEE International Conference on Robotics and Automation_. IEEE, 2009, pp. 489–494.
  * [26] M. Campana, F. Lamiraux, and J.-P. Laumond, “A gradient-based path optimization method for motion planning,” _Advanced Robotics_ , vol. 30, no. 17-18, pp. 1126–1144, 2016.
  * [27] C. Dawson and C. Fan, “Robust counterexample-guided optimization for planning from differentiable temporal logic,” _arXiv preprint arXiv:2203.02038_ , 2022\.
  * [28] A. H. Qureshi, A. Simeonov, M. J. Bency, and M. C. Yip, “Motion planning networks,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 2118–2124.
  * [29] R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” _Machine learning_ , vol. 8, pp. 229–256, 1992.
  * [30] C. Wu, A. Rajeswaran, Y. Duan, V. Kumar, A. M. Bayen, S. Kakade, I. Mordatch, and P. Abbeel, “Variance reduction for policy gradient with action-dependent factorized baselines,” _arXiv preprint arXiv:1803.07246_ , 2018.
  * [31] Y. Gilpin, V. Kurtz, and H. Lin, “A smooth robustness measure of signal temporal logic for symbolic control,” _IEEE Control Systems Letters_ , vol. 5, no. 1, pp. 241–246, 2021.
  * [32] A. Stooke, J. Achiam, and P. Abbeel, “Responsive safety in reinforcement learning by pid lagrangian methods,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 9133–9143.
  * [33] C. Zhang, D. Han, Y. Qiao, J. U. Kim, S.-H. Bae, S. Lee, and C. S. Hong, “Faster segment anything: Towards lightweight sam for mobile applications,” _arXiv preprint arXiv:2306.14289_ , 2023.
  * [34] S. Mehta and M. Rastegari, “Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer,” _arXiv preprint arXiv:2110.02178_ , 2021\.
  * [35] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [36] R. Malladi, J. A. Sethian, and B. C. Vemuri, “Shape modeling with front propagation: A level set approach,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 17, no. 2, pp. 158–175, 1995.
  * [37] J. L. Bentley, “Multidimensional binary search trees used for associative searching,” _Commun. ACM_ , vol. 18, no. 9, p. 509–517, sep 1975. [Online]. Available: https://doi.org/10.1145/361002.361007
  * [38] T. Schaul, D. Horgan, K. Gregor, and D. Silver, “Universal value function approximators,” in _International conference on machine learning_. PMLR, 2015, pp. 1312–1320.