# Quantum Transport of Two Dimensional Dirac Electrons Across Different Tilts

Rasha Al-Marzoog1 Ali Rezaei1 Zahra Noorinejad2 Mohsen Amini1 Ebrahim
Ghanbari-Adivi1 S. A. Jafari3 1 Department of Physics, Faculty of Physics,
University of Isfahan, Isfahan 81746-73441, Iran  2 Department of Physics,
Shahreza Branch, Islamic Azad University, Shahreza, Iran  3 II. Physikalisches
Institute, RWTH Aachen University, 52074 Aachen, Germany

###### Abstract

To understand the role of tilting of the Dirac cone, we consider the quantum
transport in a heterojunction of two-dimensional Dirac materials where
electrons tunnel between upright Dirac cone and tilted Dirac cone side. We
study the role of tilt in characterizing the transmission as well as the fate
of the pseudospin of the Dirac electrons uppon a sudden change in the tilting.

##  1 Introduction

The observation of Dirac-like fermions in graphene [1], with its remarkable
electronic properties, significantly broadened the horizons of condensed
matter physics and resulted in the inception of Dirac-fermion systems. These
materials reveal captivating electronic properties, most notably characterized
by the existence of linear conic energy dispersions in the proximity of Dirac
points within their momentum space [2]. Due to the linear nature of the energy
dispersion, the low-energy charge carriers within these materials emulate
massless Dirac fermions, leading to the emergence of relativistic behavior.
For instance, Klein tunneling, a fundamental quantum mechanical phenomenon
rooted in the principles of relativistic physics, is a direct evidence of this
unique behavior [3].

Expanding upon the widely studied case of graphene, Dirac cones can, in
general, possess tilted dispersions along a preferred direction of wave
vector. This unique feature introduces additional complexity and richness to
the electronic properties of two-dimensional materials. In materials with
tilted Dirac cones, the linear energy-momentum relation remains intact, but
the tilt in the cone not only results in anisotropic behaviors, opening up
exciting possibilities for tailoring their specific properties [4, 5, 6, 7],
but also naturally leads to an emergent spacetime metric [8]. Based on a
parameter known as the tilt parameter ($\zeta$), the tilted Dirac cone can be
classified into four primary types. These types include the untilted type
$(\zeta=0)$, type-I or subcritical $(0<\zeta<1)$, type-II or overcritical
$(\zeta>1)$, and type-III or critical $(\zeta=1)$ [9, 10, 11].

The tilt observed in the energy dispersion cones has been proven to have a
substantial influence on various properties of these fermions [11, 12, 13].
For instance, the impact of tilted velocity and anisotropic Dirac cones on
phenomena like asymmetric and oblique Klein tunneling and valley-dependent
electron retroreflection has been examined in $8$-Pmmn borophene [14, 15]. In
this context, variations in the tilt magnitude lead to non-universal behavior
in the anomalous Hall conductivity [16]. Specifically, small (subcritical)
tilts can induce an asymmetric Pauli blockade, resulting in finite free-
electron Hall responses [17] and non-zero photocurrents [18]. Conversely,
large (overcritical) tilts cause the Fermi surface to deviate from a point-
like structure, resulting in the emergence of a gap in the Landau-level
spectrum and the absence of chiral zero modes when the magnetic field
direction falls outside the cone [11]. Recent discussions have also
highlighted the considerable variation in optical conductivity among different
tilt types in two-dimensional materials with tilted Dirac cones [27, 28].
Furthermore, due to the anisotropy associated with the tilting of the bands,
these materials also provide a rich solid-state space-time platform, marked by
the corresponding non-Minkowski metric [19, 20, 21, 22]. Hence, it becomes
both intriguing and advantageous to inquire into how the characteristic
tilting of the Dirac cones affects the transmission of Dirac fermions across
interfaces that connect regions with varying types of tilted Dirac cones. This
question becomes particularly pertinent as active endeavors are underway to
manipulate and precisely tune the tilt in Dirac materials [23, 24, 25, 26].

In this study, we undertake a theoretical investigation into the transmission
characteristics of Dirac fermions across interfaces linking regions with
differing types of energy dispersion tilt. The rest of the paper is organized
as follows. In section 2.1, we first introduce the Hamiltonian and the tilted
Dirac cone eigenstates. Then, based on the continuity equation, we derive the
appropriate boundary conditions between two regions with different tilts and
calculate the spin rotation at the interface. In section 2.2, we obtain the
probability of electron transmission and reflection from a non-tilted region
to a tilted region for normal and oblique incidence. In section 2.3, we
calculate the probability of transmission and reflection from an intermediate
ribbon with arbitrary tilt and demonstrate that for half of the width region,
the transmission probability is equal to unity, and both reflection and
transmission exhibit periodic behavior with respect to the ribbon thickness.

##  2 Theoretical formalism

###  2.1 Model Hamiltonian

We initiate our theoretical framework by examining the low-energy transmission
properties of tilted Dirac fermions, which can be described by the following
Hamiltonian,

$\hat{\mathcal{H}}=v_{x}p_{x}\hat{\sigma}_{x}+v_{y}p_{y}\hat{\sigma}_{y}+v_{t}p_{x}\hat{\mathbf{I}}_{2\times
2}.$ (1)

In this Hamiltonian, $p_{x}$ and $p_{y}$ represent the electron momentum
components in the $x$ and $y$ directions, respectively. $\hat{\sigma}_{x}$ and
$\hat{\sigma}_{y}$ are the Pauli matrix operators, and $\hat{\mathbf{I}}$ is
the $2\times 2$ unit matrix. The model parameters $v_{x}$ and $v_{y}$ indicate
the anisotropic velocities, while $v_{t}$ represents the tilt velocity that
represents _nematicity_ of the Dirac electrons mingling space and time. The
introduction of the tilt parameter $\zeta=v_{t}/v_{x}$ allows us to consider
the band tilting along the $x$-direction. As previously discussed, Dirac
materials can generally be classified into four distinct phases: the untilted
phase $(\zeta=0)$, type-I or sub-critically tilted phase $(0<\zeta<1)$, type-
II or over-critically tilted phase $(\zeta>1)$, and type-III phase
$(\zeta=1)$. Diagonalizing the Hamiltonian provided in Eq. (1), the
eigenvalues and eigenvectors for both the conduction and valence bands can be
straightforwardly expressed as:

$E_{\lambda,\mathbf{k}}=v_{t}k_{x}+\lambda
v_{x}\sqrt{k_{x}^{2}+\gamma_{1}^{2}k_{y}^{2}},$ (2)

and

$\psi_{\lambda,\mathbf{k}}(\mathbf{r})=\frac{1}{\sqrt{2}}\left[\begin{array}[]{c}1\\\
\lambda\frac{(k_{x}+i\gamma_{1}k_{y})}{\sqrt{k_{x}^{2}+\gamma_{1}^{2}k_{y}^{2}}}\end{array}\right]e^{i\mathbf{k}\cdot\mathbf{r}},$
(3)

which will be used in the following. In this expression,
$\gamma_{1}=\frac{v_{y}}{v_{x}}$ represents the mere spatial anisotropy,
$\mathbf{k}=(k_{x},k_{y})$ denotes the wave-vector measured from the Dirac
point, $\mathbf{r}=(x,y)$ represents the position vector, and $\lambda=\pm 1$
indicates the conduction (+1) and valence (-1) bands, respectively.

###  2.2 Single interface heterojunction

We can begin our study by considering a heterojunction formed by two different
Dirac materials, each with distinct types of tilt in their energy bands, as
illustrated in Fig. 1. As depicted in the figure, the Dirac cone on the left
side ($x<0$) remains untilted ($\zeta_{A}=0$), while it exhibits tilt
($\zeta_{B}\neq 0$), on the right side ($x>0$). Consequently, the tilt
velocity in Eq.(1) becomes a function of $x$ and can be expressed as

$v_{t}(x)=v_{t}\Theta(x),$ (4)

where $\Theta$ denotes the Heaviside unit step function. It is important to
highlight that in our investigation, we specifically address the scenario with
Dirac cones tilted in the $k_{x}$-direction, which is perpendicular to the
interface. This configuration is distinct from cases where the tilt is
parallel to the interface, as discussed in a previous study [29].

Figure 1: Tilted Dirac cones in two regions.

We are interested in understanding the transmission properties across the
interface between regions $A$ and $B$, as illustrated in Fig. 1. Before
delving into the calculations of transmission and reflection coefficients, let
us establish the boundary conditions by utilizing the continuity of current
density. In elementary quantum mechanics, the probability density $\rho$ and
current density $\mathbf{J}$ are defined in terms of the wavefunction $\psi$
as $\rho=\psi^{\dagger}\psi$ and $\mathbf{J}=\psi^{\dagger}\mathbf{p}\psi$.
These quantities satisfy the continuity equation:

$\mathbf{\nabla}\cdot\mathbf{J}=-\frac{\partial\rho}{\partial t}.$ (5)

Hence, the perpendicular component of the current density should satisfy the
condition $J_{Ax}=J_{Bx}$, which implies

$\psi_{A}^{\dagger}(v_{Ax}\sigma_{x}+v_{At}\mathbf{\hat{I}})\psi_{A}=\psi_{B}^{\dagger}(v_{Bx}\sigma_{x}+v_{Bt}\mathbf{\hat{I}})\psi_{B}.$
(6)

The above equation for the Weyl semimetals of type II is also used in Ref.
[31] and Ref. [32]. It can be shown that by using the unitary operator

$U=\frac{1}{\sqrt{2}}\left(\begin{array}[]{cc}1&1\\\
1&-1\end{array}\right)=\frac{1}{\sqrt{2}}(\sigma_{x}+\sigma_{z}),$ (7)

Eq. (6) can be written as follows:

$\psi_{A}^{\prime\dagger}(v_{Ax}\sigma_{z}+v_{At}\mathbf{\hat{I}})\psi^{\prime}_{A}=\psi_{B}^{\prime\dagger}(v_{Bx}\sigma_{z}+v_{Bt}\mathbf{\hat{I}})\psi^{\prime}_{B},$
(8)

where $\psi_{A(B)}^{\prime}=U\psi_{A(B)}$ and
$\sigma_{z}=U^{\dagger}\sigma_{x}U.$ Also, Eq. (8) can be expressed in the
following matrix form:

$v_{Ax}\psi^{\prime\dagger}_{A}\left(\begin{array}[]{cc}1+\zeta_{A}&0\\\
0&1-\zeta_{A}\end{array}\right)\psi^{\prime}_{A}=v_{Bx}\psi^{\prime\dagger}_{B}\left(\begin{array}[]{cc}1+\zeta_{B}&0\\\
0&1-\zeta_{B}\end{array}\right)\psi^{\prime}_{B},$ (9)

The boundary condition that satisfies the above equation is expressed as
follows:

$\sqrt{v_{Ax}}\left(\begin{array}[]{cc}\sqrt{1+\zeta_{A}}&0\\\
0&\sqrt{1-\zeta_{A}}\end{array}\right)\psi^{\prime}_{A}=\sqrt{v_{Bx}}\left(\begin{array}[]{cc}\sqrt{1+\zeta_{B}}&0\\\
0&\sqrt{1-\zeta_{B}}\end{array}\right)\psi^{\prime}_{B},$ (10)

The above result is also presented in Ref. [31]. Finally, in the previous
bases, we arrive at the following conclusion:

$M_{A}\psi_{A}=M_{B}\psi_{B},$ (11)

where

$M_{A(B)}=\sqrt{v_{A(B),x}}U^{\dagger}\left(\begin{array}[]{cc}\sqrt{1+\zeta_{A(B)}}&0\\\
0&\sqrt{1-\zeta_{A(B)}}\end{array}\right)U,$ (12)

and the coefficient matrix $M_{A(B)}$ for each region can be defined as

$M_{A(B)}=\begin{bmatrix}a_{A(B)}&b_{A(B)}\\\ b_{A(B)}&a_{A(B)}\end{bmatrix},$
(13)

where

$a_{A(B)}=\frac{\sqrt{1+\zeta_{A(B)}}+\sqrt{1-\zeta_{A(B)}}}{2},$ (14a)
$b_{A(B)}=\frac{\sqrt{1+\zeta_{A(B)}}-\sqrt{1-\zeta_{A(B)}}}{2}.$ (14b)

It is evident that when the tilt velocities in both regions are the same,
i.e., $v_{Ax}=v_{Bx}$, it results in $M_{A}=M_{B}$, and therefore, the
boundary condition of Eq. (11) simplifies to $\psi_{A}=\psi_{B}$. This
condition remains valid even if $v_{Ax}=v_{Bx}=0$, which indicates no tilt in
the $k_{x}$-direction (although tilt may still exist in the $k_{y}$-direction,
as in Ref. [29]). However, in our current work, due to the presence of tilt in
the direction perpendicular to the interface between regions $A$ and $B$, we
will make use of the boundary condition given in Eq. (11), as we will discuss
further.

Prior to concluding this section, we address the spin or pseudo-spin reversal
of electrons at the interface of two regions. For the sake of simplicity in
calculations, we revert to the bases of Eq. (8). With some calculations, it
can be demonstrated that Eq. (9) can be straightforwardly transformed into the
following equation:

$\psi^{\prime}_{2}=(a^{\prime}\mathbf{\hat{I}}+b^{\prime}\sigma_{z})\psi^{\prime}_{1}$
(15)

Where $a^{\prime}$ and $b^{\prime}$ are given by the following expressions:

$a^{\prime}=\frac{a_{B}a_{A}-b_{B}b_{A}}{a_{B}^{2}-b_{B}^{2}}$

and

$b^{\prime}=\frac{a_{B}b_{A}-b_{B}a_{A}}{a_{B}^{2}-b_{B}^{2}}$

If we assume that the ”spin” direction $\psi^{\prime}_{1}$ in terms of
spherical coordinates is given by

$\psi^{\prime}_{1}=\begin{pmatrix}\cos(\frac{\theta}{2})\\\
\sin(\frac{\theta}{2})e^{i\theta_{i\phi}}\end{pmatrix},$ (16)

then using Eq. (15), we have

$\psi^{\prime}_{2}=\begin{pmatrix}(a^{\prime}+b^{\prime})\cos(\frac{\theta}{2})\\\
(a^{\prime}-b^{\prime})\sin(\frac{\theta}{2})e^{i\theta_{i\phi}}\end{pmatrix},$
(17)

If we examine the spin direction $\psi^{\prime}_{2}$, we arrive at the
following conclusion:

$\psi^{\prime}_{2}=N\begin{pmatrix}\cos(\frac{\theta^{\prime}}{2})\\\
\sin(\frac{\theta^{\prime}}{2})e^{i\theta_{i\phi}}\end{pmatrix},$ (18)

where N is the norm of $\psi^{\prime}_{2}$, and
$\tan(\frac{\theta^{\prime}}{2})=(\tan(\frac{\theta}{2}))(\frac{a+b}{a-b}).$
In other words, since $a^{\prime}$ and $b^{\prime}$ are real numbers, during
the passage through the boundary, the polar angle of the spin relative to the
axis along which the tilt occurs changes, while the azimuthal angle
perpendicular to the tilt direction remains unchanged.

The above calculations demonstrate that at the interface of the two regions,
the pseudo-spin direction remains in the x-y plane, but its orientation with
respect to the x-axis in the x-y plane changes.

####  2.2.1 Normal incidence analysis

Let’s consider the case where an electron approaches from the left side of
this heterojunction. To simplify our analysis, we’ll begin with the assumption
of a normally incident electron, which implies $k_{y}=0$. We will also
consider that the Dirac cone in region $B$ is in a sub-critically tilted phase
$(0<\zeta_{B}<1)$. In this case, the wave functions in regions $A$ and $B$ can
be expressed as:

$\psi(x)=\begin{cases}e^{ik_{x}x}\begin{pmatrix}1\\\
1\end{pmatrix}+re^{-ik_{x}x}\begin{pmatrix}1\\\ -1\end{pmatrix},&x<0,\\\
t\begin{pmatrix}1\\\ 1\end{pmatrix}e^{ik_{x}x},&x>0.\end{cases}$ (19)

Here, $r$ represents the reflection amplitude and $t$ is the transmission
amplitude and the two components of the wavefunction correspond to the two
regions, namely $A$ and $B$. By applying the boundary conditions (11) at the
interface $x=0$ and considering $\zeta_{A}=0$, we can derive the following
equations:

$1+r=(\sqrt{1+\zeta_{B}})t,$ (20a) $1-r=(\sqrt{1+\zeta_{B}})t.$ (20b)

which results in the following soultions

$r=0,$ (21) $t=\frac{1}{\sqrt{1+\zeta_{B}}}.$ (22)

We can also calculate the transmission coefficient $T$, which is defined in
terms of the incident and transmitted probability current densities
$J_{\text{in}}$ and $J_{\text{tr}}$ as:

$T=\frac{J_{\text{tr}}}{J_{\text{in}}}=\frac{J_{Bx}}{J_{Ax}}.$ (23)

As discussed earlier, it is clear that the transmitted probability current
density
$J_{\text{tr}}=J_{Bx}=\psi_{B}^{\dagger}(v_{x}\sigma_{x}+v_{t})\psi_{B}=\frac{v_{x}+v_{t}}{1+\zeta_{B}}$,
and the incident probability current density
$J_{\text{in}}=J_{Ax}=\psi_{A}^{\dagger}(v_{x}\sigma_{x})\psi_{A}=v_{x}$.
Therefore, if the electron arrives with normal incidence, the probability of
it passing through is 1, meaning $T=1$. This is often referred to as the Klein
tunneling effect [30], which remains valid even in the presence of tilt in the
energy band dispersion where $t\neq 1$.

Now, it becomes intriguing to explore the scenario when $\zeta_{B}\geq 1$. In
this situation, it would be impossible to write the wave function, as given in
Eq. (19), for region $B$. This impossibility arises from the fact that, in
region $B$, one cannot find a particle with a group velocity moving from right
to left. Furthermore, if a particle crosses the boundary, it won’t have the
possibility to escape. This is reminiscent of the event horizon of a black
hole.

####  2.2.2 Oblique incidence analysis

Figure 2: Transmission from region 1 to region 2. (a) The boundary of the
first and the second region, (s represents the pseudo-spin direction.) (b) The
intersections of Fermi surface and Dirac cones in two regions. The cones for
the 2nd region is tilted.

Let’s now explore the case of oblique incidence at the interface between
regions $A$ and $B$. We consider an incoming electron with a given wavevector
$\mathbf{k}_{\text{i}}$, which reflects with wavevector
$\mathbf{k}_{\text{r}}$ in region $A$, and we denote the transmitted electron
with wavevector $\mathbf{k}_{\text{t}}$ in region $B$. Referring to the angles
illustrated in Fig. 2, we can establish the wavevectors as
$\mathbf{k}_{\alpha}=(k_{x\alpha},k_{y\alpha})=|\mathbf{k}_{\alpha}|(\cos(\theta_{\alpha}),\sin(\theta_{\alpha}))$
for $\alpha=\text{i, r, t}$. Given the conservation of energy $E$ and momentum
projection $k_{y}$ (i.e., parallel to the interface), we can immediately
conclude $k_{r}=k_{i}$, $\theta_{r}=\theta_{i}$, $k_{t}y=k_{i}y$,

$\theta_{t}=\cot^{-1}{\left(\frac{\zeta_{B}\pm\sqrt{\zeta_{B}^{2}-\zeta_{B}^{2}\cos^{2}{\theta_{i}}+\cos^{2}{\theta_{i}}}}{(\zeta_{B}^{2}-1)\sin(\theta_{i})},\right)}$
(24)

and $k_{xt}$ reads as

${k}_{xt}=\frac{\zeta_{B}E\pm\sqrt{\zeta_{B}^{2}E^{2}-\zeta_{B}^{2}k^{2}_{ix}+k^{2}_{ix}}}{\zeta_{B}^{2}-1}.$
(25)

The associated wave functions for each case can be expressed as:

$\psi^{i(t)}(\textbf{r})=\begin{pmatrix}1\\\
e^{i\theta_{i(t)}}\end{pmatrix}e^{i\mathbf{k}_{\mathbf{i(t)}}\cdot\mathbf{r}},$
(26a) $\psi^{r}(\textbf{r})=\begin{pmatrix}1\\\
-e^{-i\theta_{r}}\end{pmatrix}e^{i\textbf{k}_{\textbf{r}}\cdot\textbf{r}},$
(26b)

These equations allows us to find the wave functions in both regions as

$\psi_{A}=\psi^{i}+r\psi^{r}\quad\text{for}\quad x<0,and$ (27)
$\psi_{B}=\psi^{t}\quad\text{for}\quad x>0.$ (28)

Inserting this wavefunction into Eq. (11), we obtain the following equation:

$\left[\begin{array}[]{c}1+r\\\
e^{i\theta_{i}}-re^{-i\theta_{i}}\end{array}\right]=M_{B}\left[\begin{array}[]{c}t\\\
te^{i\theta_{t}}\end{array}\right],$ (29)

where $M_{B}$ is given by Eq. (13). We can simplify this equation into the
following relations for transmission and reflection amplitudes:

$1+r=a_{B}t+b_{B}te^{i\theta_{t}},$ (30a)
$e^{i\theta_{i}}-re^{-i\theta_{i}}=b_{B}t+a_{B}e^{i\theta_{t}},$ (30b)

in which $a_{B}$ and $b_{B}$ are defined in Eq. (14). After some
straightforward algebra, the solutions for $r$ and $t$ can be expressed as:

$t=\frac{e^{i\theta_{i}}+e^{-i\theta_{i}}}{a_{B}(e^{-i\theta_{i}}+e^{i\theta_{t}})+b_{B}(1+e^{i\theta_{t}-\theta_{i}})},$
(31a) $r=t(a_{B}+b_{B}e^{i\theta_{t}})-1.$ (31b)

Figure 3: The probability of transmission and reflection for various incident
angles (region 1 to 2). The particle enters the region with tilted Dirac cones
from the region without tilt.

The next step is to determine the transmission coefficient, $T$. This involves
calculating $J_{\text{t}}$ and $J_{\text{in}}$. One can obtain $J_{\text{t}}$
as follows:

$J_{\text{tr}}=v_{x}\psi_{B}^{\dagger}(\sigma_{x}+\zeta_{B}\hat{\mathbf{I}})\psi_{B}=2|t|v_{x}(\cos(\theta_{t})+\zeta_{B}).$
(32)

For $J_{\text{in}}$, first, calculate the total current density in the
$x$-direction, $J_{x\text{A}}$, which can be expressed as:

$J_{x\text{A}}=v_{x}\psi_{A}^{\dagger}\sigma_{x}\psi_{A}=J_{\text{in}}+J_{\text{r}},$
(33)

where $J_{\text{in}}$ and $J_{\text{r}}$ are contributions from the incident
and reflected wavefunctions, respectively:

$J_{\text{in}}=2|1+r|v_{x}\cos{(\theta_{i}-\delta_{1})},$ (34)

and

$J_{\text{r}}=2|1+r|r^{*}v_{x}\cos{(\theta_{i}+\delta_{2})}.$ (35)

Here, $\delta_{1}$ and $\delta_{2}$ are the phase factors associated with the
$1+r$ and $(1+r)r^{*}$ expressions, respectively. Thus, we can calculate the
transmission probability as:

$T=\frac{J_{\text{t}}}{J_{\text{in}}}=\frac{|t|^{2}[\cos{(\theta_{t})+\alpha}]}{2|1+r|\cos{(\theta_{i}-\delta_{1})}}.$
(36)

Similarly, the reflection probability can be determined as:

$R=\frac{J_{\text{r}}}{J_{\text{in}}}=\frac{|r|\cos{(\theta_{i}+\delta_{2})}}{\cos{(\theta_{i}-\delta_{1})}}.$
(37)

In Fig. 3, we have visually represented the behavior of the transmission
probability ($T$) and reflection probability ($R$) as functions of the
incident angle $\theta_{i}$. As illustrated, the phenomenon of Klein tunneling
remains prominent for incident angles less than $45^{\circ}$, with $T$
reaching nearly 1. However, as $\theta_{i}$ exceeds $45^{\circ}$, the
probability of reflection becomes notably significant. It’s worth noting that
in this figure, the tilt parameter is assumed to be $\zeta_{B}=0.4$.

Figure 4: Transmission from region 2 to region 1. (a) The boundary of the
first and the second region, (s represents the pseudo-spin direction.) (b) The
intersections of Fermi surface and Dirac cones in two regions. The cones for
the 1st region is tilted., (c) The probability of transmission and reflection
for various incident angles (region 2 to 1). The particle enters the region
without tilted Dirac cones from the region with them.

Before closing this subsection it is interesting to ask what will happen if we
reverse the incident direction from right to left on the hetrojunction. To
explore the scenario of transmission from the tilted side to the untilted one
(reversing the incident direction), we need to consider the corresponding
angles and wavevectors, which are depicted in Fig. 4. By applying the
conservation laws discussed earlier, we can determine $\theta_{r}$ and
$\theta_{t}$ using the following relations:

$\theta_{r}=\cot^{-1}\left(\frac{\zeta_{B}+\sqrt{\zeta_{B}^{2}-\zeta_{B}^{2}\cos^{2}{\theta_{t}}+\cos^{2}{\theta_{t}}}}{(\zeta_{B}^{2}-1)\sin(\theta_{t})}\right),$
(38a)
$\theta_{i}=\cot^{-1}\left(\frac{\zeta_{B}-\sqrt{\zeta_{B}^{2}-\zeta_{B}^{2}\cos^{2}{\theta_{t}}+\cos^{2}{\theta_{t}}}}{(\zeta_{B}^{2}-1)\sin(\theta_{t})}\right).$
(38b)

The same analysis can be extended to determine the transmission and reflection
coefficients for this case. In Fig. 4, we present the graphical
representations of $T(\theta_{i})$ and $R(\theta_{i})$ when $\zeta_{B}=0.4$.
The overall behavior is qualitatively similar to the previous case, with one
significant observation. In this scenario, there exists a critical angle,
$\theta_{i}^{c}=\cot^{-1}\left(\frac{2\zeta_{B}}{\zeta_{B}^{2}-1}\right)$. If
$\theta_{i}>\theta_{i}^{c}$, it is impossible to transmit a particle from
right to left.

###  2.3 Double interface heterojunction

Figure 5: Incident, reflected and transmitted angles for two boundaries in the
presence of the tilted ribbon.

Now, let’s explore a heterojunction with double interfaces, as depicted in
Fig. 5. This setup comprises three regions, denoted as 1, 2, and 3. Regions 1
and 3 contain untilted Dirac cones, while the central region, region 2,
features a tilted Dirac cone, represented by $\zeta$. For the structure
depicted in Fig. 5, we can write the corresponding wave functions for each
region as follows:

$\psi(x)=\begin{cases}\left[\begin{array}[]{c}1\\\
e^{i\theta_{i}}\end{array}\right]e^{i\mathbf{k}_{i}\cdot\mathbf{r}}+r\left[\begin{array}[]{c}1\\\
e^{i\theta_{r}}\end{array}\right]e^{i\mathbf{k}_{r}\cdot\mathbf{r}}&\text{for
}x<0,\\\ A\left[\begin{array}[]{c}1\\\
e^{i\theta_{A}}\end{array}\right]e^{i\mathbf{k}_{A}\cdot\mathbf{r}}+B\left[\begin{array}[]{c}1\\\
e^{i\theta_{B}}\end{array}\right]e^{i\mathbf{k}_{B}\cdot\mathbf{r}}&\text{for
}0<x<l,\\\ t\left[\begin{array}[]{c}1\\\
e^{i\theta_{t}}\end{array}\right]e^{i\mathbf{k}_{t}\cdot\mathbf{r}}&\text{for
}x>l,\end{cases}$ (39)

where $\theta_{A}$, $\theta_{B}$, $k_{A,x}$, $k_{B,x}$, and $k_{t,x}$ can be
obtained using the same conservation laws as in the previous section. This
results in the following equations:

$\theta_{A}=\cot^{-1}{\left(\frac{-\zeta-\sqrt{\zeta^{2}-\zeta^{2}\cos^{2}{\theta_{t}}+\cos^{2}{\theta_{t}}}}{(1-\zeta^{2})\sin(\theta_{t})}\right)},$
(40)
$\theta_{B}=\cot^{-1}{\left(\frac{-\zeta+\sqrt{\zeta^{2}-\zeta^{2}\cos^{2}{\theta_{t}}+\cos^{2}{\theta_{t}}}}{(1-\zeta^{2})\sin(\theta_{t})}\right)},$
(41)
$k_{A,x}=\frac{-\zeta+\sqrt{\zeta^{2}-\zeta^{2}\cos^{2}{\theta_{t}}+\cos^{2}{\theta_{t}}}}{1-\zeta^{2}},$
(42)
$k_{B,x}=\frac{-\zeta-\sqrt{\zeta^{2}-\zeta^{2}\cos^{2}{\theta_{t}}+\cos^{2}{\theta_{t}}}}{1-\zeta^{2}},$
(43)

and

$k_{t,x}=\frac{E}{v_{x}}\cos\theta_{t}.$ (44)

To obtain the coefficients $r$, $A$, $B$, and $t$, you need to apply the
boundary condition (Eq. (11)) on both interfaces, $x=0$ and $x=l$. This
results in the following two equations:

$\psi_{1}(0)=M\psi_{2}(0),$ (45) $M\psi_{2}(l)=\psi_{3}(l).$ (46)

After performing the necessary calculations, you can obtain the transmission
and reflection coefficients, $T$ and $R$. While we won’t present the
complicated expressions here, you can visualize the behavior of $T$ in Fig. 6.

Figure 6: The probabilities of transmission and reflection for (a) $\zeta=0.4$
(left to right) and (b) $\zeta=0.4$ (right to left).

Fig. 6 illustrates the probability of transmission from left to right, while
Fig. 6 shows the probability of transmission from right to left. As observed,
their behaviors are opposite to each other. In other words, in regions where
the probability of transmission from left to right is unity, the probability
of transmission from right to left is less than unity. Another important point
is that the transmission probability is a periodic function of the thickness
of the intermediate ribbon, and the period of oscillation is dependent on
parameter $\zeta$.

##  3 Summary

In this study, we derived boundary conditions governing transmission and
reflection between two regions characterized by different tilts. Utilizing the
continuity equation, we established these boundary conditions, demonstrating
their consistency with existing literature. The spin rotation at the interface
was quantified, and leveraging the derived boundary conditions, we
investigated the probabilities of electron transmission and reflection during
the transition from a non-tilted region to a tilted one. Moreover, we
determined the probability of massless electron transmission through a tilted
region, revealing a periodic dependence on the width of an intermediate
region, with this periodicity being independent of the incident angle.
Notably, we observed that, for sufficiently large region widths, the
transmission probability approaches unity. Lastly, we highlighted the symmetry
of transmission probabilities between left-to-right and right-to-left
directions.

## Acknowledgment

The authors express their gratitude for the valuable discussions with M.
Soltani. M.A. acknowledges the support received through the Abdus Salam (ICTP)
short visit program.

## References

## References

  * [1] K. S. Novoselov, A. K. Geim, S. V. Morozov, D. Jiang, Y. Zhang, S. V. Dubonos, I. V. Grigorieva, and A. A. Firsov, Science 306, 666 (2004).
  * [2] A. H. Castro Neto, F. Guinea, N. M. R. Peres, K. S. Novoselov, and A. K. Geim, Rev. Mod. Phys. 81, 109 (2009).
  * [3] M. I. Katsnelson, K. S. Novoselov, and A. K. Geim, Nat. Phys. 2, 620 (2006).
  * [4] C.-C. Liu, W. Feng, and Y. Yao, Phys. Rev. Lett. 107, 076802 (2011).
  * [5] X. Qian, J. Liu, L. Fu, and J. Li, Science 346, 1344 (2014).
  * [6] H.Y. Lu, A.S.Cuamba, S.Y. Lin, L. Hao, R. Wang, H. Li, Y.Y. Zhao, and C.S. Ting, Phys. Rev. B 94, 195423 (2016).
  * [7] S. Li, Y. Liu, Z.-M. Yu, Y. Jiao, S. Guan, X.-L. Sheng, Y. Yao, and S.A. Yang, Phys. Rev. B 100, 205102 (2019).
  * [8] G. E. Volovik, JETP Letters, 114, 236 (2021)
  * [9] G.E. Volovik and K. Zhang, J. Low Temp. Phys. 189, 276 (2017).
  * [10] G.E. Volovik, Phys. Usp. 61, 89 (2018).
  * [11] A. A. Soluyanov, D. Gresch, Z. Wang, Q. Wu, M. Troyer, X. Dai, and B. A. Bernevig, Nature (London) 527, 495 (2015).
  * [12] M. Trescher et al., Phys. Rev. B 91, 115135 (2015).
  * [13] J. F. Steiner, A. V. Andreev, and D. A. Pesin, Phys. Rev. Lett. 119, 036601 (2017).
  * [14] Sh.-H. Zhang, W. Yang, Oblique Klein tunneling in $8$-Pmmn borophene p-n junctions, Phys. Rev. B 97 (2018).
  * [15] X. Zhou, Valley-dependent electron retroreflection and anomalous Klein tunneling in an $8$-Pmmn borophene- based n-p-n junction, Phys. Rev. B 100 (2019) 195139.
  * [16] A. A. Zyuzin and R. P. Tiwari, JETP Lett. 103, 717 (2016).
  * [17] J. F. Steiner, A. V. Andreev, and D. A. Pesin, Phys. Rev. Lett. 119, 036601 (2017).
  * [18] Q. Ma et al., Nat. Phys. 13, 842 (2017).
  * [19] Z. Jalali-Mola and S. A. Jafari, Polarization tensor for tilted Dirac fermion materials: Covariance in deformed Minkowski spacetime, Phys. Rev. B 100, 075113 (2019).
  * [20] T. Farajollahpour and S. A. Jafari, Synthetic non-Abelian gauge fields and gravitomagnetic effects in tilted Dirac cone systems, Phys. Rev. Res. 2, 023410 (2020).
  * [21] Z. Jalali-Mola and S. A. Jafari, Undamped transverse electric mode in undoped two-dimensional tilted Dirac cone materials, Phys. Rev. B 102, 245148 (2020).
  * [22] Z. Jalali-Mola and S. A. Jafari, Tilt-induced many-body corrections to optical conductivity of tilted Dirac cone materials, Phys. Rev. B 104, 085152 (2021)
  * [23] M. O. Goerbig, J.-N. Fuchs, G. Montambaux, and F. Piechon, Phys. Rev. B 78, 045415 (2008).
  * [24] Y. Lu et al., NPJ Comput. Mater. 2, 16011 (2016); C.-H. Li,
  * [25] Y.-J. Long, L.-X. Zhao, L. Shan, Z.-A. Ren, J.-Z. Zhao, H.-M. Weng, X. Dai, Z. Fang, C. Ren, and G.-F. Chen, Phys. Rev. B 95, 125417 (2017).
  * [26] Y. Yekta, H. Hadipour, and S. A. Jafari, Tuning the tilt of the Dirac cone by atomic manipulations in $8$-Pmmn borophene, Commun. Phys. 6, 46 (2023).
  * [27] A. Wild, E. Mariani, and M. E. Portnoi, Phys. Rev. B 105, 205306 (2022).
  * [28] C.-Y. Tan, J.-T. Hou, C.-X. Yan, H. Guo, and H.-R. Chang, Phys. Rev. B 106, 165404 (2022).
  * [29] E. Pattrawutthiwong, W. Choopan, W. Liewrian, Physics Letters A, 393,127154 (2021).
  * [30] N. Dombey and A. Calogeracos, Phys. Rep., 315, 41 (1999).
  * [31] K. Hashimoto and Y. Matsuo, Phys. Rev. B 102, 195128 (2020).
  * [32] E. Witten, La Rivista del Nuovo Cimento, 39 (2016).


$(\Lambda^{(\mathcal{E}-1)w+2}_{w})^{(1)}\geq\lambda,~{}\Lambda^{\mathcal{E}w+1}_{w}\leq\lambda+wH$.
∎

######  Lemma 10 (Upper bound of $\Delta^{p}_{k}(s,a)$).

For any $(s,a)\in\mathcal{S}\times\mathcal{A}$ and given $\delta\in(0,1)$, the
following holds with probability $1-\delta$.

$\Delta_{p,k}^{1}(s,a)\leq
B^{k}_{p,w}+(\Lambda^{k}_{w}(s,a))^{1/2}\cdot|\mathcal{S}|\cdot\sqrt{\frac{H^{2}}{2}\log{\left(\frac{H}{\delta\lambda}\right)}}+\lambda\Lambda^{k}_{w}(s,a)$

###### Proof of lemma 10.

For any $(s,a)\in\mathcal{S}\times\mathcal{A}$, $\Delta^{p}_{k}(s,a)$ is
represented as follows.

$\displaystyle\Delta^{p}_{k}(s,a)$
$\displaystyle=||P^{(k+1)}(\cdot|s,a)-\widehat{P}^{(k+1)}(\cdot|s,a)||_{1}$
$\displaystyle=||o^{p}_{(k+1)}(\cdot,s,a)-\widehat{o}^{p}_{(k+1)}(\cdot,s,a)||_{1}$
$\displaystyle=\sum_{s^{\prime}\in\mathcal{S}}\left|\frac{\sum_{t=k-w+1}^{k}n_{t}(s^{\prime},s,a)}{\lambda+\sum_{t=k-w+1}^{k}n_{t}(s,a)}-o^{p}_{(k+1)}(s^{\prime},s,a)\right|$
$\displaystyle=\Lambda^{k}_{w}(s,a)\sum_{s^{\prime}\in\mathcal{S}}\left|\sum_{t=k-w+1}^{k}n_{t}(s^{\prime},s,a)-\left(\lambda+\sum_{t=k-w+1}^{k}n_{t}(s,a)\right)o^{p}_{(k+1)}(s^{\prime},s,a)\right|$
$\displaystyle\leq\Lambda^{k}_{w}(s,a)\sum_{s^{\prime}\in\mathcal{S}}\left(\left|\sum_{t=k-w+1}^{k}\left(n_{t}(s^{\prime},s,a)-n_{t}(s,a)o^{p}_{(k+1)}(s^{\prime},s,a)\right)\right|+\left|\lambda
o^{p}_{(k+1)}(s^{\prime},s,a)\right|\right)$
$\displaystyle\leq\Lambda^{k}_{w}(s,a)\sum_{s^{\prime}\in\mathcal{S}}\left|\sum_{t=k-w+1}^{k}\left(n_{t}(s^{\prime},s,a)-n_{t}(s,a)o^{p}_{(k+1)}(s^{\prime},s,a)\right)\right|+\lambda\Lambda^{k}_{w}(s,a)$
(D.76)

Recall that $n_{t}(s^{\prime},s,a),~{}n_{t}(s,a)$ is defined as

$\displaystyle n_{t}(s^{\prime},s,a)$
$\displaystyle=\sum_{h=0}^{H-1}\mathbbm{1}\left[(s^{\prime},s,a)=(s^{t}_{h+1},s^{t}_{h},a^{t}_{h})\right]$
$\displaystyle=\sum_{h=0}^{H-1}\mathbbm{1}\left[(s,a)=(s^{t}_{h},a^{t}_{h})\right]\cdot\mathbbm{1}\left[s^{\prime}=s^{t}_{h+1}\right]$
(D.77) $\displaystyle n_{t}(s,a)$
$\displaystyle=\sum_{h=0}^{H-1}\mathbbm{1}\left[(s,a)=(s^{t}_{h},a^{t}_{h})\right]$
(D.78)

where $\mathbbm{1}[\cdot]$ is an indicator function. Plugging the equations
(D.77),(D.78) into the equations (D.76) gives the following,

$\displaystyle\Lambda^{k}_{w}(s,a)\sum_{s^{\prime}\in\mathcal{S}}\left|\sum_{t=k-w+1}^{k}\left(n_{t}(s^{\prime},s,a)-n_{t}(s,a)o^{p}_{(k+1)}(s^{\prime},s,a)\right)\right|$
$\displaystyle=\Lambda^{k}_{w}(s,a)\sum_{s^{\prime}\in\mathcal{S}}\left|\sum_{t=k-w+1}^{k}\left(\sum_{h=0}^{H-1}\mathbbm{1}\left[(s,a)=(s^{t}_{h},a^{t}_{h})\right]\cdot\mathbbm{1}\left[s^{\prime}=s^{t}_{h+1}\right]\right.\right.$
$\displaystyle\left.\left.-\sum_{h=0}^{H-1}\mathbbm{1}\left[(s,a)=(s^{t}_{h},a^{t}_{h})\right]\cdot
o^{p}_{(k+1)}(s^{\prime},s,a)\right)\right|$
$\displaystyle=\Lambda^{k}_{w}(s,a)\sum_{s^{\prime}\in\mathcal{S}}\left|\sum_{t=k-w+1}^{k}\left(\sum_{h=0}^{H-1}\mathbbm{1}\left[(s,a)=(s^{t}_{h},a^{t}_{h})\right]\left(\mathbbm{1}\left[s^{\prime}=s^{t}_{h+1}\right]-o^{p}_{(k+1)}(s^{\prime},s,a)\right)\right)\right|$
$\displaystyle\leq\underbrace{\Lambda^{k}_{w}(s,a)\sum_{s^{\prime}\in\mathcal{S}}\bigg{|}\sum_{t=k-w+1}^{k}\left(\sum_{h=0}^{H-1}\mathbbm{1}\left[(s,a)=(s^{t}_{h},a^{t}_{h})\right]\left(\mathbbm{1}\left[s^{\prime}=s^{t}_{h+1}\right]-o^{p}_{t}(s^{\prime},s,a)\right)\right)\bigg{|}}_{\leavevmode\hbox
to11.76pt{\vbox to11.76pt{\pgfpicture\makeatletter\hbox{\hskip
5.88237pt\lower-5.88237pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{5.68237pt}{0.0pt}\pgfsys@curveto{5.68237pt}{3.13832pt}{3.13832pt}{5.68237pt}{0.0pt}{5.68237pt}\pgfsys@curveto{-3.13832pt}{5.68237pt}{-5.68237pt}{3.13832pt}{-5.68237pt}{0.0pt}\pgfsys@curveto{-5.68237pt}{-3.13832pt}{-3.13832pt}{-5.68237pt}{0.0pt}{-5.68237pt}\pgfsys@curveto{3.13832pt}{-5.68237pt}{5.68237pt}{-3.13832pt}{5.68237pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-4.47221pt}{-2.25555pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{2.1}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}$
$\displaystyle+\underbrace{\Lambda^{k}_{w}(s,a)\sum_{s^{\prime}\in\mathcal{S}}\bigg{|}\sum_{t=k-w+1}^{k}\left(\sum_{h=0}^{H-1}\mathbbm{1}\left[(s,a)=(s^{t}_{h},a^{t}_{h})\right]\left(o^{p}_{t}(s^{\prime},s,a)-o^{p}_{(k+1)}(s^{\prime},s,a)\right)\right)\bigg{|}}_{\leavevmode\hbox
to11.76pt{\vbox to11.76pt{\pgfpicture\makeatletter\hbox{\hskip
5.88237pt\lower-5.88237pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{5.68237pt}{0.0pt}\pgfsys@curveto{5.68237pt}{3.13832pt}{3.13832pt}{5.68237pt}{0.0pt}{5.68237pt}\pgfsys@curveto{-3.13832pt}{5.68237pt}{-5.68237pt}{3.13832pt}{-5.68237pt}{0.0pt}\pgfsys@curveto{-5.68237pt}{-3.13832pt}{-3.13832pt}{-5.68237pt}{0.0pt}{-5.68237pt}\pgfsys@curveto{3.13832pt}{-5.68237pt}{5.68237pt}{-3.13832pt}{5.68237pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-4.47221pt}{-2.25555pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{2.2}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}$

The term 2.1 can be upperbounded by utilizing the Lemma (34), (43) of the
paper [31]. We first denote for any $t\in[K],~{}s^{\prime}\in\mathcal{S}$, the
random variable
$\eta^{t}(s^{\prime})\coloneqq\sum_{h=0}^{H-1}\left(\mathbbm{1}\left[s^{\prime}=s^{t}_{h+1}\right]-o^{p}_{t}(s^{\prime},s^{t}_{h},a^{t}_{h})\right)$.
Now, for given $s^{\prime}\in\mathcal{S}$ the sequence
$\\{\eta^{\tau}(s^{\prime})\\}_{\tau=1}^{\infty}$ is zero-mean and $H/2$-sub
Gaussian random variable. From the lemma (43) of the paper [31], set
$Y=\lambda\mathbb{I},~{}X_{t}=\sum_{h=0}^{H-1}\mathbbm{1}\left[(s,a)=(s^{t}_{h},a^{t}_{h})\right]$.Then,
for given $\delta\in(0,1)$, the following holds with probability $1-\delta$
for any $(s,a)\in\mathcal{S}\times\mathcal{A}$,

$\displaystyle\left|(\Lambda^{k}_{w}(s,a))^{1/2}\sum_{t=k-w+1}^{k}\left(\sum_{h=0}^{H-1}\mathbbm{1}\left[(s,a)=(s^{t}_{h},a^{t}_{h})\right]\cdot\sum_{h=0}^{H-1}\mathbbm{1}[s^{\prime}=s^{t}_{h+1}]-o_{t}^{p}(s^{\prime},s,a)\right)\right|$
$\displaystyle\leq\sqrt{\frac{H^{2}}{2}\log{\left(\frac{(\Lambda^{k}_{w}(s,a))^{-1/2}\cdot\lambda^{-1/2}}{\delta/H}\right)}}$
$\displaystyle=\sqrt{\frac{H^{2}}{2}\log{\left(\frac{H}{\delta}\cdot\frac{1}{(\Lambda^{k}_{w}(s,a))^{1/2}\cdot\lambda^{1/2}}\right)}}$
$\displaystyle\leq\sqrt{\frac{H^{2}}{2}\log{\left(\frac{H}{\delta}\cdot\frac{1}{\lambda}\right)}}$
(D.79)

Then the following holds with probability $1-\delta$,

$\displaystyle=(\Lambda^{k}_{w}(s,a))^{1/2}\sum_{s^{\prime}\in\mathcal{S}}\left|(\Lambda^{k}_{w}(s,a))^{1/2}\sum_{t=k-w+1}^{k}\left(\sum_{h=0}^{H-1}\mathbbm{1}\left[(s,a)=(s^{t}_{h},a^{t}_{h})\right]\cdot\sum_{h=0}^{H-1}\mathbbm{1}[s^{\prime}=s^{t}_{h+1}]-o_{t}^{p}(s^{\prime},s,a)\right)\right|$
$\displaystyle\leq(\Lambda^{k}_{w}(s,a))^{1/2}\cdot|\mathcal{S}|\cdot\sqrt{\frac{H^{2}}{2}\log{\left(\frac{H}{\delta\lambda}\right)}}$

The term 2.2 can be bounded as follows,

2.2
$\displaystyle\leq\Lambda^{k}_{w}(s,a)\sum_{s^{\prime}\in\mathcal{S}}\sum_{t=k-w+1}^{k}\sum_{h=0}^{H-1}\mathbbm{1}\left[(s,a)=(s^{t}_{h},a^{t}_{h})\right]\bigg{|}o^{p}_{t}(s^{\prime},s,a)-o^{p}_{(k+1)}(s^{\prime},s,a)\bigg{|}$
$\displaystyle=\Lambda^{k}_{w}(s,a)\sum_{t=k-w+1}^{k}\sum_{h=0}^{H-1}\mathbbm{1}\left[(s,a)=(s^{t}_{h},a^{t}_{h})\right]\sum_{s^{\prime}\in\mathcal{S}}\bigg{|}o^{p}_{t}(s^{\prime},s,a)-o^{p}_{(k+1)}(s^{\prime},s,a)\bigg{|}$
$\displaystyle=\Lambda^{k}_{w}(s,a)\sum_{t=k-w+1}^{k}\sum_{h=0}^{H-1}\mathbbm{1}\left[(s,a)=(s^{t}_{h},a^{t}_{h})\right]\bigg{|}\bigg{|}o^{p}_{t}(\cdot,s,a)-o^{p}_{(k+1)}(\cdot,s,a)\bigg{|}\bigg{|}_{1}$
$\displaystyle\leq\max_{t\in[k-w+1,k]}\left(\bigg{|}\bigg{|}o^{p}_{t}(\cdot,s,a)-o^{p}_{(k+1)}(\cdot,s,a)\bigg{|}\bigg{|}_{1}\right)\cdot\left(\Lambda^{k}_{w}(s,a)\sum_{t=k-w+1}^{k}\sum_{h=0}^{H-1}\mathbbm{1}\left[(s,a)=(s^{t}_{h},a^{t}_{h})\right]\right)$
$\displaystyle\leq\max_{t\in[k-w+1,k]}\left(\bigg{|}\bigg{|}o^{p}_{t}(\cdot,s,a)-o^{p}_{(k+1)}(\cdot,s,a)\bigg{|}\bigg{|}_{1}\right)\cdot
1$ $\displaystyle\leq B^{k}_{p,w}(\Delta_{\pi})$ (D.80)

Then, combining the equation (D.76), (D.79), (D.80), the
$\Delta_{p,k}^{1}(s,a)$ can be represented as follows,

$\displaystyle\Delta_{p,k}^{1}(s,a)\leq
B^{k}_{p,w}(\Delta_{\pi})+(\Lambda^{k}_{w}(s,a))^{1/2}\cdot|\mathcal{S}|\cdot\sqrt{\frac{H^{2}}{2}\log{\left(\frac{H}{\delta\lambda}\right)}}+\lambda\Lambda^{k}_{w}(s,a)$

∎

######  Lemma 11 (Upper bound of $\bar{\Delta}^{p}_{K}$).

For given $\delta\in(0,1)$, the following holds with probability $1-\delta$.

$\bar{\Delta}^{p}_{K}\leq\left(|\mathcal{S}|\sqrt{\frac{H^{2}}{2}\log{\left(\frac{H}{\delta\lambda}\right)}}+\lambda\right)(K-1)\sqrt{\frac{H}{w}}\sqrt{\log{\left(\frac{\lambda+wH}{\lambda}\right)}}+wHB_{p}(\Delta_{\pi})$

###### Proof of lemma 11.

The total empirical forecasting transition probability model error
$\bar{\Delta}^{p}_{K}$ can be represented as follows,

$\displaystyle\bar{\Delta}^{p}_{K}=$
$\displaystyle\sum_{k=1}^{K-1}\sum_{h=0}^{H-1}\bar{\Delta}^{p}_{k,h}$
$\displaystyle=$
$\displaystyle\sum_{k=1}^{K-1}\sum_{h=0}^{H-1}\Delta^{p}_{k}(s^{k+1}_{h},a^{k+1}_{h})$
$\displaystyle\leq$
$\displaystyle\sum_{k=1}^{K-1}\sum_{h=0}^{H-1}\left((\Lambda^{k}_{w}(s^{k+1}_{h},a^{k+1}_{h}))^{1/2}|\mathcal{S}|\sqrt{\frac{H^{2}}{2}\log{\left(\frac{H}{\delta\lambda}\right)}}\right)$
$\displaystyle+\sum_{k=1}^{K-1}\sum_{h=0}^{H-1}\left(\max_{t\in[k-w+1,k]}\bigg{|}\bigg{|}o^{p}_{t}(\cdot,s^{k+1}_{h},a^{k+1}_{h})-o^{p}_{(k+1)}(\cdot,s^{k+1}_{h},a^{k+1}_{h})\bigg{|}\bigg{|}_{1}\right)$
$\displaystyle+\sum_{k=1}^{K-1}\sum_{h=0}^{H-1}\left(\lambda\Lambda^{k}_{w}(s^{k+1}_{h},a^{k+1}_{h})\right)$
$\displaystyle\leq$
$\displaystyle\left(|\mathcal{S}|\sqrt{\frac{H^{2}}{2}\log{\left(\frac{H}{\delta\lambda}\right)}}+\lambda\right)\sum_{k=1}^{K-1}\sum_{h=0}^{H-1}\left((\Lambda^{k}_{w}(s^{k+1}_{h},a^{k+1}_{h}))^{1/2}\right)$
$\displaystyle+\sum_{k=1}^{K-1}\sum_{h=0}^{H-1}\left(\max_{t\in[k-w+1,k]}\bigg{|}\bigg{|}o^{p}_{t}(\cdot,s^{k+1}_{h},a^{k+1}_{h})-o^{p}_{(k+1)}(\cdot,s^{k+1}_{h},a^{k+1}_{h})\bigg{|}\bigg{|}_{1}\right)$
$\displaystyle\leq$
$\displaystyle\left(|\mathcal{S}|\sqrt{\frac{H^{2}}{2}\log{\left(\frac{H}{\delta\lambda}\right)}}+\lambda\right)(K-1)\sqrt{\frac{H}{w}}\sqrt{\log{\left(\frac{\lambda+wH}{\lambda}\right)}}+wHB_{p}(\Delta_{\pi})$

∎

Prelimnary for proof of theorem 3.

The W-LSE involves solving a following joint optimization problem over
$\phi^{r}_{f}\in\mathbb{R}^{|\mathcal{S}||\mathcal{A}|},\phi^{p}_{f}\in\mathbb{R}^{|\mathcal{S}|^{2}|\mathcal{A}|},q\in\mathbb{R}^{N}$
to obtain a minimum upper bound of the dynamic regret:

$\displaystyle\min_{\phi^{\diamond}_{f},q}\mathcal{L}\left(\phi^{\diamond}_{f},q~{};~{}\square_{1:N}\right)\text{
where
}\mathcal{L}\left(\phi^{\diamond}_{f},q~{};~{}\square_{1:N}\right)=\sum_{t=1}^{N}q_{t}\left(\widehat{\square}_{\phi^{\diamond}_{f}}^{k+1}-\square_{t}\right)^{2}+\text{disc}(q)+\frac{1}{wH}\cdot\lambda||\phi^{\diamond}_{f}||_{2}$
(D.81)

where $\diamond=r\text{ or }p$. If $\diamond=r$, then $\square=R(s,a)$ and if
$\diamond=p$, then $\square=P(s^{\prime},s,a)$.
${\square}_{\phi^{\diamond}_{f}}$ means that $\square$ is parameterized by
$\phi^{\diamond}_{f}$, and $\square_{1:N}$ are observed data of $\square$, and
the
disc$(q)\coloneqq\sup_{f\in\mathcal{F}}\left(\mathbb{E}[f(\widehat{\square}^{k+1}~{}|~{}\square_{1:N}]-\sum_{t=1}^{N}q_{t}\mathbb{E}[\widehat{\square}^{t}|\square_{1:t-1}]\right)$
measures the non-stationarity of the environment. disc$(q)$ could be measured
and upperbounded by observed data. For example, if $\diamond=r$ and
$\square=R$, then $\phi^{r}_{f}$ parameterizes the future reward function
$\widehat{R}_{\phi^{r}_{f}}^{k+1}$, $N$ is the total number of visits of
$(s,a)$ up to episode $k$, $R_{1:N}(s,a)$ is the set of reward values
$\\{R_{1}(s,a),R_{1}(s,a),..,R_{N}(s,a)\\}$ that the agent has received when
visiting $(s,a)$. We demonstrate the modified upper bound of
$\mathfrak{R}_{\mathcal{I}}$ when utilizing W-LSE. Before, we define
forecasting reward model error as
$\Delta_{r,k}^{1}(s,a)=\left|\left(R^{(k+1)}-\widetilde{R}^{(k+1)}\right)(s,a)\right|$,
forecasting transition probability model error as
$\Delta_{p,k}^{1}(s,a)=\left|\left|\left(P^{(k+1)}-\widehat{P}^{(k+1)}\right)(\cdot~{}|~{}s,a)\right|\right|_{1}$
where $\widetilde{R},\widehat{P}^{(k+1)}$ are predicted reward,transition
probability from meta-function $\mathcal{I}_{g\circ f}$ (Appendix D.2).

###### Proof of Theorem 4 .

We bring the theorem 7 of the paper [22] to offer the upper bound of the
$l_{2}$-norm of the reward gap between $R^{(k+1)}(s,a)$ and the
$\widetilde{R}^{(k+1)}(s,a)$ as follows. Inspired from the theorem 7 of the
paper [22], let us denote
$X_{k,h}=(s^{k}_{h},a^{k}_{h})\in\mathcal{S}\times\mathcal{A},~{}Y_{k,h}=R^{(k)}(s^{k}_{h},a^{k}_{h})\in\mathbb{R}$
and assume the environment provides agent an noisy reward
$\widehat{Y}_{k,h}=Y_{k,h}+\eta$ with zero-mean gaussian
$\eta\sim\mathcal{N}(0,b)$. Define the kernel
$\Psi(x)=\varphi(x)\in\mathbb{R}^{|\mathcal{S}||\mathcal{A}|}$ where
$\varphi(x)$ is the one-hot vector that we have defined on section A.1. Now,
let function $r(x)=c^{\top}\varphi(x)$ where the vector
$c\in\mathbb{R}^{|\mathcal{S}||\mathcal{A}|}$ is same as the estimated future
reward vector
$\widetilde{R}^{(k+1)}\in\mathbb{R}^{|\mathcal{S}||\mathcal{A}|}$ and $r(x)$
is same as the estimated future reward when $x=(s,a)$, namely
$\widetilde{R}^{(k+1)}(s,a)$. Assume, Then, for given data until episode $k$,
$\mathcal{D}_{data}=\\{(X_{1,0},\widehat{Y}_{1,0}),(X_{1,1},\widehat{Y}_{1,1}),..,(X_{k,H-1},\widehat{Y}_{k,H-1})\\}$,
we denote
$\mathcal{D}^{(s,a)}_{data}\coloneqq\\{(X_{k,h},\widehat{Y}_{k,h})~{}|~{}X_{k,h}=(s,a)\text{
where }(X_{k,h},\widehat{Y}_{k,h})\in\mathcal{D}_{data}\\}$. We relabel
$\mathcal{D}^{(s,a)}_{data}$ as
$\\{\left((s,a),\widehat{Y}_{1}\right),\left((s,a),\widehat{Y}_{2}\right),...,\left((s,a),\widehat{Y}_{N}\right)\\}$
where $N(s,a)=\sum_{t=1}^{k}n_{t}(s,a)$ is the total number of visitation of
$(s,a)$ until $k$ episodes (Definition (D.78)). We deonte $N$ as $N(s,a)$, and
$\sum_{t=1}^{N}q_{t}=1$. Then for any $(s,a)\in\mathcal{S}\times\mathcal{A}$,
the following holds with probability at least $1-\delta$ for all functions
$r\in\\{x\rightarrow c^{\top}\Psi(x):||c||_{2}\leq\Lambda\\}$,

$\displaystyle\mathbb{E}[\left(r(s,a)-\widehat{Y}_{N+1}\right)^{2}|\mathcal{D}^{(s,a)}_{data}]$
$\displaystyle\leq\sum_{t=1}^{N}q_{t}\left(r(s,a)-\widehat{Y}_{t}\right)^{2}+\text{disc}(q)+\mathcal{O}\left((\text{log}^{2}N)\Lambda
r||q||_{2}\right)$
$\displaystyle\leq\sum_{t=1}^{N}q_{t}\left(r(s,a)-\widehat{Y}_{t}\right)^{2}+\text{disc}(q)+\frac{1}{wH}\cdot\lambda||\bar{r}||_{2}$
(D.82)

where we let the function $\mathcal{O}\left((\text{log}^{2}N)\Lambda
r||q||_{2}\right)$ can be represented as
$\frac{1}{wH}\cdot\lambda||\bar{r}||_{2}$ where $\lambda$ is a constant. Take
the expectation over $\eta$ on both inequailty.

$\displaystyle\mathbb{E}_{\eta}\left[\mathbb{E}\left[\left(r(s,a)-\widehat{Y}_{N+1}\right)^{2}|\mathcal{D}^{(s,a)}_{data}\right]\right]$
$\displaystyle\leq\mathbb{E}_{\eta}\left[\sum_{t=1}^{N}q_{t}\left(r(s,a)-\widehat{Y}_{t}\right)^{2}+\text{disc}(q)+\frac{1}{wH}\cdot\lambda||\bar{r}||_{2}\right]$
$\displaystyle\mathbb{E}\left[\left(r(s,a)-\widehat{Y}_{N+1}\right)^{2}|\mathcal{D}^{(s,a)}_{data}\right]$
$\displaystyle\leq\sum_{t=1}^{N}\mathbb{E}_{\eta}\left[q_{t}\left(r(s,a)-\widehat{Y}_{t}\right)^{2}\right]+\text{disc}(q)+\frac{1}{wH}\cdot\lambda||\bar{r}||_{2}$

The LHS of equation (D.82) can be expressed as follows.

$\displaystyle\mathbb{E}\left[(r(s,a)-\widehat{Y}_{N+1}-\eta)^{2}\right]$
$\displaystyle=\mathbb{E}_{\eta}\left[(r(s,a)-Y_{N+1})^{2}\right]+\mathbb{E}_{\eta}\left[\eta^{2}\right]$
$\displaystyle=(r(s,a)-Y_{N+1})^{2}+\mathbb{E}\left[\eta^{2}\right]$ (D.83)

Also the
term$\sum_{t=1}^{N}\mathbb{E}_{\eta}\left[q_{t}\left(r(s,a)-\widehat{Y}_{t}\right)^{2}\right]$
of the RHS of equation (D.82) is represented as follows.

$\displaystyle\sum_{t=1}^{N}\mathbb{E}_{\eta}[q_{t}\left(r(s,a)-\widehat{Y}_{t}\right)^{2}]$
$\displaystyle=\sum_{t=1}^{N}\mathbb{E}_{\eta}[q_{t}\left(\left(r(s,a)-Y_{t}\right)^{2}+\eta^{2}\right)]$
$\displaystyle=\sum_{t=1}^{N}\mathbb{E}_{\eta}\left[q_{t}\left(\left(r(s,a)-Y_{t}\right)^{2}\right)\right]+\sum_{t=1}^{N}\mathbb{E}_{\eta}\left[q_{t}\eta^{2}\right]$
$\displaystyle=\sum_{t=1}^{N}q_{t}\left(\left(r(s,a)-Y_{t}\right)^{2}\right)+\mathbb{E}_{\eta}\left[\eta^{2}\right]$

Then the term $\mathbb{E}_{\eta}[\eta^{2}]$ removed by both side, then the
following holds,

$\displaystyle(r(s,a)-Y_{N+1})^{2}\leq\sum_{t=1}^{N}q_{t}\left(\left(r(s,a)-Y_{t}\right)^{2}\right)+\text{disc}(q)+\frac{1}{wH}\cdot\lambda||\bar{r}||_{2}$
(D.85)

Recall the definition of $r(s,a)=\widetilde{R}^{(k+1)}(s,a)$,
$Y_{t}=R_{t}(s,a)$. Since $t$ matches with one of $(k,h)\in[K]\times[H]$
pairs, we can rewrite
$\sum_{t=1}^{N}q_{t}\left(r(s,a)-\widehat{Y}_{t}\right)^{2}=\sum_{k^{\prime}=1}^{k-1}\sum_{h=0}^{H-1}q_{(k^{\prime},h)}\left(r(s,a)-Y_{(k,h)}\right)^{2}=\sum_{k^{\prime}=1}^{k-1}\sum_{h=0}^{H-1}q_{(k^{\prime},h)}\left(\widetilde{R}^{(k+1)}(s,a)-R^{k^{\prime}}_{h}(s,a)\right)^{2}=\sum_{k^{\prime}=1}^{k-1}\sum_{h=0}^{H-1}q_{(k^{\prime},h)}\left(\widetilde{R}^{(k+1)}(s,a)-R^{k^{\prime}}(s,a)\right)^{2}$
where if $(s,a)$ is not visited at step $h$ of episode $k$, then corresponding
$q_{(k^{\prime},h)}$ is zero. Then, the following holds,

$\displaystyle\Delta^{1}_{r,k}(s,a)\leq$
$\displaystyle\sqrt{\min_{q,\bar{r}}\left(\sum_{k^{\prime}=1}^{k-1}\sum_{h=0}^{H-1}q_{(k^{\prime},h)}\left(\widetilde{R}^{(k+1)}(s,a)-R^{k^{\prime}}(s,a)\right)^{2}+\text{disc}(q)+\frac{1}{wH}\cdot\lambda||\bar{r}||_{2}\right)}$
$\displaystyle\leq$
$\displaystyle\sqrt{\min_{q,\bar{r}}\left(\left(\max_{1\leq k^{\prime}\leq
k}\left(\widetilde{R}^{(k+1)}(s,a)-R^{k^{\prime}}(s,a)\right)\right)^{2}\left(\sum_{k^{\prime}=1}^{k-1}\sum_{h=0}^{H-1}q_{(k^{\prime},h)}\right)+\text{disc}(q)+\frac{1}{wH}\cdot\lambda||\bar{r}||_{2}\right)}$

Similar process for $\Delta^{1}_{p,k}$ provides the following inequality. For
any $(s^{\prime},s,a)\in\mathcal{S}\times\mathcal{S}\times\mathcal{A},$

$\displaystyle\left|P^{(k+1)}(s^{\prime}~{}|~{}s,a)-\widehat{P}^{(k+1)}(s^{\prime}~{}|~{}s,a)\right|$
$\displaystyle\leq\sqrt{\min_{q,\bar{p}}\left(\sum_{k^{\prime}=1}^{k-1}\sum_{h=0}^{H-1}q_{(k^{\prime},h)}\left(\widehat{P}^{(k+1)}(s^{\prime}|s,a)-P^{k^{\prime}}(s^{\prime}|s,a)\right)^{2}+\text{disc}(q)+\frac{1}{wH}\cdot\lambda||\bar{p}||_{2}\right)}$

holds. Then, recall the definition of $\Delta^{1}_{p,k}(s,a)$.

$\displaystyle\Delta^{1}_{p,k}(s,a)$
$\displaystyle\leq\sum_{s^{\prime}\in\mathcal{S}}\sqrt{\min_{q,\bar{p}}\left(\sum_{k^{\prime}=1}^{k-1}\sum_{h=0}^{H-1}q_{(k^{\prime},h)}\left(\widehat{P}^{(k+1)}(s^{\prime}|s,a)-P^{k^{\prime}}(s^{\prime}|s,a)\right)^{2}+\text{disc}(q)+\frac{1}{wH}\cdot\lambda||\bar{p}||_{2}\right)}$
$\displaystyle\leq|\mathcal{S}|\sqrt{\min_{q,\bar{p}}\left(\sum_{k^{\prime}=1}^{k-1}\sum_{h=0}^{H-1}q_{(k^{\prime},h)}\left|\left|\widehat{P}^{(k+1)}(\cdot|s,a)-P^{k^{\prime}}(\cdot|s,a)\right|\right|_{\infty}^{2}+\text{disc}(q)+\frac{1}{wH}\cdot\lambda||\bar{p}||_{2}\right)}$

Now, recall the Corollary 5,6 and definition of $\mathfrak{R}_{\mathcal{I}}$.
Fix $(s,a)$, then $\mathfrak{R}_{\mathcal{I}}(s,a)$

$\displaystyle\mathfrak{R}_{\mathcal{I}}=$
$\displaystyle\frac{1}{1-\gamma}\sum_{k=1}^{K-1}\bar{\iota}_{\infty}^{k+1}+\sum_{k=1}^{K-1}\sum_{h=0}^{H-1}-\iota^{k+1}_{h}+C_{p}\sqrt{K-1}$
$\displaystyle\leq$
$\displaystyle\frac{1}{1-\gamma}\sum_{k=1}^{K-1}\left(\Delta^{1}_{r,k}(s,a)+\Delta^{1}_{p,k}(s,a)\frac{\gamma\hat{r}_{max}}{1-\gamma}-2\Gamma^{k}_{w}(s,a)\right)$
$\displaystyle+\sum_{k=1}^{K-1}\sum_{h=0}^{H-1}\left(\Delta^{1}_{r,k}(s,a)+\Delta^{p}_{k}(s,a)\frac{\gamma\hat{r}_{max}}{1-\gamma}+2\Gamma^{k}_{w}(s,a)\right)$
$\displaystyle+C_{p}\sqrt{K-1}$ $\displaystyle\leq$
$\displaystyle\frac{1}{1-\gamma}\sum_{k=1}^{K-1}\left(\Delta^{1}_{r,k}(s,a)+\Delta^{1}_{p,k}(s,a)\frac{\gamma}{1-\gamma}\left(\tilde{r}_{\text{max}}+\max(2\Gamma^{k}_{w}(s,a))\right)-2\Gamma^{k}_{w}(s,a)\right)$
$\displaystyle+H\sum_{k=1}^{K-1}\left(\Delta^{1}_{r,k}(s,a)+\Delta^{p}_{k}(s,a)\frac{\gamma}{1-\gamma}\left(\tilde{r}_{\text{max}}+\max(2\Gamma^{k}_{w}(s,a))\right)+2\Gamma^{k}_{w}(s,a)\right)$
$\displaystyle+C_{p}\sqrt{K-1}$ $\displaystyle\leq$
$\displaystyle\sum_{k=1}^{K-1}\bigg{(}\underbrace{\left(\frac{1}{1-\gamma}+H\right)\Delta^{1}_{r,k}(s,a)+\frac{\gamma\tilde{r}_{\text{max}}}{1-\gamma}\left(\frac{1}{1-\gamma}+H\right)\Delta^{1}_{p,k}(s,a)}_{\leavevmode\hbox
to7.49pt{\vbox to7.49pt{\pgfpicture\makeatletter\hbox{\hskip
3.74467pt\lower-3.74467pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{3.54468pt}{0.0pt}\pgfsys@curveto{3.54468pt}{1.95769pt}{1.95769pt}{3.54468pt}{0.0pt}{3.54468pt}\pgfsys@curveto{-1.95769pt}{3.54468pt}{-3.54468pt}{1.95769pt}{-3.54468pt}{0.0pt}\pgfsys@curveto{-3.54468pt}{-1.95769pt}{-1.95769pt}{-3.54468pt}{0.0pt}{-3.54468pt}\pgfsys@curveto{1.95769pt}{-3.54468pt}{3.54468pt}{-1.95769pt}{3.54468pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-1.75pt}{-2.25555pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{1}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}+$
$\displaystyle+\frac{\gamma}{1-\gamma}\left(\frac{1}{1-\gamma}+H\right)\max(2\Gamma^{k}_{w}(s,a))\Delta^{1}_{p,k}(s,a)\bigg{)}$
$\displaystyle+\sum_{k=1}^{K-1}2\left(-\frac{1}{1-\gamma}+H\right)\Gamma^{k}_{w}(s,a)$
$\displaystyle+C_{p}\sqrt{K-1}$

Now we let the term 1 as $2(\frac{1}{1-\gamma}+H)\Gamma^{k}_{w}(s,a)$, that is
if we redefine the exploration bonus term as follow,

$\displaystyle\Gamma^{k}_{w}(s,a)=\frac{1}{2}\Delta^{1}_{r,k}(s,a)+\frac{\gamma\tilde{r}_{\text{max}}}{2(1-\gamma)}\Delta^{1}_{p,k}(s,a)$

Also, note that
$\Delta^{p}_{k}(s,a)=\sum_{s^{\prime}\in\mathcal{S}}\left|\left(\widehat{P}^{(k+1)}-P^{(k+1)}\right)(s^{\prime}|s,a)\right|\leq\left|\mathcal{S}\right|$.
Then the following holds,

$\displaystyle\mathfrak{R}_{\mathcal{I}}$
$\displaystyle\leq\sum_{k=1}^{K-1}\left(4H\Gamma^{k}_{w}(s,a)+\frac{2\gamma}{1-\gamma}\left(\frac{1}{1-\gamma}+H\right)\max(\Gamma^{k}_{w}(s,a))\left|\mathcal{S}\right|\right)$
$\displaystyle\leq\sum_{k=1}^{K-1}\left(4H+\frac{2\gamma}{1-\gamma}\left(\frac{1}{1-\gamma}+H\right)\right)\max(\Gamma^{k}_{w}(s,a))$
$\displaystyle=\left(4H+\frac{2\gamma\left|\mathcal{S}\right|}{1-\gamma}\left(\frac{1}{1-\gamma}+H\right)\right)\sum_{k=1}^{K-1}\max(\Gamma^{k}_{w}(s,a))$
$\displaystyle\leq\left(4H+\frac{2\gamma\left|\mathcal{S}\right|}{1-\gamma}\left(\frac{1}{1-\gamma}+H\right)\right)\sum_{k=1}^{K-1}\left(\frac{1}{2}\max(\Delta^{1}_{r,k}(s,a))+\frac{\gamma\tilde{r}_{\text{max}}}{2(1-\gamma)}\max(\Delta^{1}_{p,k}(s,a))\right)$
$\displaystyle=\left(4H+\frac{2\gamma\left|\mathcal{S}\right|}{1-\gamma}\left(\frac{1}{1-\gamma}+H\right)\right)\sum_{k=1}^{K-1}\left(\frac{1}{2}\Delta^{1}_{r,k}(s,a)+\frac{\gamma\tilde{r}_{\text{max}}}{2(1-\gamma)}\Delta^{1}_{p,k}(s,a)\right)$
$\displaystyle=\left(4H+\frac{2\gamma\left|\mathcal{S}\right|}{1-\gamma}\left(\frac{1}{1-\gamma}+H\right)\right)\left(\frac{1}{2}\sum_{k=1}^{K-1}\Delta^{1}_{r,k}(s,a)+\frac{\gamma\tilde{r}_{\text{max}}}{2(1-\gamma)}\sum_{k=1}^{K-1}\Delta^{1}_{p,k}(s,a)\right)$

Note that above upper bound for $\mathfrak{R}_{\mathcal{I}}$ holds for
following given upper bound condition of
$\Delta^{1}_{r,k}(s,a),\Delta^{1}_{p,k}(s,a)$ ,

$\displaystyle\Delta^{1}_{r,k}(s,a)$
$\displaystyle\leq\sqrt{\min_{q,\bar{r}}\left(\sum_{k^{\prime}=1}^{k-1}\sum_{h=0}^{H-1}q_{(k^{\prime},h)}\left(\widetilde{R}^{(k+1)}(s,a)-R^{k^{\prime}}(s,a)\right)^{2}+\text{disc}(q)+\frac{1}{wH}\cdot\lambda||\bar{r}||_{2}\right)}$
$\displaystyle\Delta^{1}_{p,k}(s,a)$
$\displaystyle\leq\sum_{s^{\prime}\in\mathcal{S}}\sqrt{\min_{q,\bar{p}}\left(\sum_{k^{\prime}=1}^{k-1}\sum_{h=0}^{H-1}q_{(k^{\prime},h)}\left(\widehat{P}^{(k+1)}(s^{\prime}|s,a)-P^{k^{\prime}}(s^{\prime}|s,a)\right)^{2}+\text{disc}(q)+\frac{1}{wH}\cdot\lambda||\bar{p}||_{2}\right)}$

∎

###### Proof of Remark 1.

The proof starts from the equation (D.85). Let us denote $r^{sw},q^{sw}$ as
follows.

$\displaystyle q^{sw}_{t}$ $\displaystyle=\begin{cases}\frac{1}{wH}&\text{if
}t\in(k-w,k]\\\ 0&\text{otherwise}\end{cases},$ $\displaystyle r^{sw}$
$\displaystyle=\operatorname*{arg\,min}_{\bar{r}}\left(\lambda||\bar{r}||^{2}+\sum_{t=1}^{N}\left(r(s,a)-\widehat{Y}_{t}\right)^{2}\right)$
(D.86)

where $r_{sw}$ is the same reward estimation with equation D.7. Then the
minimum over $\bar{r},q$ of the equation (D.82) yields the following
inequality,

$\displaystyle\min_{\bar{r},q}\left(\sum_{t=1}^{N}q_{t}\left(r(s,a)-\widehat{Y}_{t}\right)^{2}+\text{disc}(q)+\frac{1}{wH}\cdot\lambda||\bar{r}||_{2}\right)$
(D.87)
$\displaystyle\leq\min_{\bar{r}}\left(\sum_{t=1}^{N}q^{sw}_{t}\left(r(s,a)-\widehat{Y}_{t}\right)^{2}+\text{disc}(q^{sw})+\frac{1}{Hw}\cdot\lambda||\bar{r}||_{2}\right)$
$\displaystyle\leq\frac{1}{Hw}\underbrace{\min_{\bar{r}}\left(\sum_{t=1}^{N}(Hw)\cdot
q^{sw}_{t}\left(r(s,a)-\widehat{Y}_{t}\right)^{2}+\lambda||\bar{r}||_{2}\right)}_{\leavevmode\hbox
to7.49pt{\vbox to7.49pt{\pgfpicture\makeatletter\hbox{\hskip
3.74467pt\lower-3.74467pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{
{{}}\hbox{\hbox{{\pgfsys@beginscope\pgfsys@invoke{
}{{}{{{}}}{{}}{}{}{}{}{}{}{}{}{}{{}\pgfsys@moveto{3.54468pt}{0.0pt}\pgfsys@curveto{3.54468pt}{1.95769pt}{1.95769pt}{3.54468pt}{0.0pt}{3.54468pt}\pgfsys@curveto{-1.95769pt}{3.54468pt}{-3.54468pt}{1.95769pt}{-3.54468pt}{0.0pt}\pgfsys@curveto{-3.54468pt}{-1.95769pt}{-1.95769pt}{-3.54468pt}{0.0pt}{-3.54468pt}\pgfsys@curveto{1.95769pt}{-3.54468pt}{3.54468pt}{-1.95769pt}{3.54468pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} }{{{{}}\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{1.0}{0.0}{0.0}{1.0}{-1.75pt}{-2.25555pt}\pgfsys@invoke{
}\hbox{{\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }\hbox{{1}}
}}\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}} }
\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{{{}}}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}+\text{disc}(q_{sw})$
(D.88)

The term 1 is the optimization problem of equation (D.86) where its minimizer
is $r^{sw}$. The inspection through equation (D.87) $\sim$ (D.88) provides
that the optimal $q^{*},\bar{r}^{*}$, the minimizer of equation (D.87), can
provide smaller value than $q^{sw},r^{sw}$. Since the RHS of the equation
(D.85) is same as the equation (D.87), $q^{*},\bar{r}^{*}$ provide a tighter
upper bound of the LHS term of equation (D.82) than $q^{sw},r^{sw}$.
Therefore, the equation (D.83) provides that the optimal $q^{*},\bar{r}^{*}$
gives a tighter upper bound of the term $\Delta^{1}_{r,k}$ using than
$q^{sw},\bar{r}^{sw}$. Repeat similar analysis for the upper bound of the
$\Delta^{1}_{p,k}$. Then, by Corollary 5,6, the tighter upper bound of
$\Delta^{1}_{r,k}(s,a),\Delta^{1}_{p,k}(s,a)$ provides the smaller upper bound
of $-\iota^{K}_{H},\bar{\iota}^{K}_{\infty}$ and leads to tighter upper bound
of $\mathfrak{R}_{\mathcal{I}}$.

∎

##  Appendix E Experimental design and results

###  E.1 Environment setting details

Reward function design.

All three environments share the same reward function structure and have an
identical goal. The reward function $R$ consists of three parts
$\mathcal{R}=\mathcal{R}_{h}+\mathcal{R}_{f}-\mathcal{R}_{c}$ where
$\mathcal{R}_{h}$ is the healthy reward,
$\mathcal{R}_{f}=k_{f}(x_{t+1}-x_{t})/\Delta t,k_{f}>0$ is the forward reward,
and $R_{c}$ is the control cost. Agents have a goal to run faster in the $+x$
direction, so the faster they run, the higher the forward reward
$\mathcal{R}_{f}$ they receive. We modified the environment to make the
agent’s desired directions change as the episode goes by. To be specific, we
designed the forward reward $R_{f}$ to change as episodes progress as
$\mathcal{R}^{k}_{f}=o_{k}\cdot k_{f}(x_{t+1}-x_{t})/\Delta t$ where $o_{k}$
is the sine function $a\text{sin}(wbk)$ where $a,b>0$ are constant. Positive
$o_{k}$ causes the agent to desire the forward $+x$ direction as an optimal
policy, and negative $o_{k}$ causes it to desire the backward $-x$ direction.
We generated different speeds of non-stationary by changing the frequency
variable $w$ through [1,2,3,4,5].

Non-stationary variable $o_{k}$ generator.

  1. 1.

Sine function : The non-stationary variable $o_{k}$ is designed as
$o_{k}=\sin{\left(2\pi wk/37\right)}$ where $w$ is the integer speed of the
environment change and $k$ is the episode. We changed $w$ among [1,2,3,4,5].
We divided $2\pi wk$ by 37, the prime number, to make sure that the
environment has various non-stationary modes and to avoid certain non-
stationary variables appearing frequently.

  2. 2.

Real data : we brought the stroke price data to model the non-stationary real
data. We use real world data that the function that generates the non-
staionary variable is not known.

Figure 4: Nonstationary variable from real dataA,B

Non-stationary variable $o_{k}$ generator (ablation study).  We provide for
$o_{k}$ that we used in our expierments, $B(G)$ satisfies the property of the
time-elapsing variation budget that $B(G)$ increases as $G$ increases as
follows. For the ablation study, we generate $o_{k}=\sin{(2\pi\cdot G\cdot
k}/37)$ where $G=[38,76,114,152,190]$. We estimated $B(G)$ as
$\sum_{k=1}^{150}|o_{k+1}-o_{k}|$

| $\Delta_{t}=1/G=38$ | $\Delta_{t}=2/G=76$ | $\Delta_{t}=3/G=114$ | $\Delta_{t}=4/G=152$ | $\Delta_{t}=5/G=190$
---|---|---|---|---|---
$B(G)$ | 15.98 | 31.85 | 47.49 | 62.79 | 77.64

###  E.2 Hyperparameters and implementation details

Training Details.

For the ARIMA model that serves as a forecatser $f$, we used auto$\\_$arima
function of pmdarima python package to find the optimal $p,q,d$. To compare
the result between ProST-G and MBPO, we both trained the MBPO and ProST-G with
initial learning rate $lr=3e-4$ with the decaying parameter $0.999$. For
ProST-G, We added uniform noise $\eta\sim\text{Unif}([-b,b])$ in observed non-
stationary variable $o^{k}$ to generate noisy non-stationary variable
$\hat{o}_{k}=o_{k}+\eta$ with different noise bound $b\in[0.01,0.03,0.05])$.

To compare the result between ProST-G with ProOLS,ONPG,FTML, we trained these
three baselines with eight different initial learning rate
$lr=\\{1e-3,3e-3,5e-3,7e-3,1e-2,3e-2,5e-2,7e-2\\}$.

Hyper parameters.

Letter | hyper parameters | Swimmer-v2 | Half cheetah-v2 | Hopper-v2
---|---|---|---|---
$K$ | episodes | 100 | 150 | 150
$H$ | environment steps per episodes | 100
$G$ | policy updates per epochs | 50
$\widehat{H}$ | model rollout length |  1 $\rightarrow$ 15 over episodes 20 $\rightarrow$150
$N$ | iteration of policy update and policy evaluation | 1
$M$ |  model rollout batch size ($D_{syn}$) | 1e5
$\tau$ | entropy regularization parameter | 0.2
$\gamma$ | reward discounting factor | 0.99

Note that $\widehat{H}$ increases linearly within a certain range as episode
goes by. We denote $h_{min}\rightarrow h_{max}$ over episodes
$k_{min}\rightarrow k_{max}$ as
$\widehat{H}(k)=\min(\max(h_{min}+(k-k_{min})/(k_{max}-k_{min})\cdot(h_{max}-h_{min}),h_{min}),h_{max})$.

###  E.3 Full results

####  E.3.1 Non-stationarity : sin wave

(1) Swimmer-v2

Figure 5: (a) $\sim$(e) the average rewards of ProST-G, and the three
baselines : ProOLS, ONPG, FTML for 5 different speeds ($x$-axis is episode).
The shaded area of ProST-G is 95 % confidence area among 3 different noise
bounds (0.01, 0.03, 0.05), and the shaded ares of three baselines are the 95 %
confidence area among 8 different learning rates (0.001, 0.003, 0.005, 0.007,
0.01, 0.03, 0.05, 0.07). Figure 6: (a) $\sim$(e) the average reweards of
ProST-G and MBPO. The shaded area of ProST-G is 95 % confidence area among 3
different noise bound (0.01, 0.03, 0.05).

(2) Halfcheetah-v2

Figure 7: (a) $\sim$(e) the average rewards of ProST-G, and the three
baselines : ProOLS, ONPG, FTML for 5 different speeds ($x$-axis is episode).
The shaded area of ProST-G is 95 % confidence area among 3 different noise
bounds (0.01, 0.03, 0.05), and the shaded ares of three baselines are the 95 %
confidence area among 8 different learning rates (0.001, 0.003, 0.005, 0.007,
0.01, 0.03, 0.05, 0.07). Figure 8: (a) $\sim$(e) the average reweards of
ProST-G and MBPO ($x$-axis is episode). The shaded area of ProST-G is 95 %
confidence area among 3 different noise bound (0.01, 0.03, 0.05).

(3) Hopper-v2

Figure 9: (a) $\sim$(e) the average rewards of ProST-G, and the three
baselines : ProOLS, ONPG, FTML for 5 different speeds ($x$-axis is episode).
The shaded area of ProST-G is 95 % confidence area among 3 different noise
bound (0.01, 0.03, 0.05), and the shaded ares of three baselines are the 95 %
confidence area among 8 different learning rates (0.001, 0.003, 0.005, 0.007,
0.01, 0.03, 0.05, 0.07). Figure 10: (a) $\sim$(e) the average reweards of
ProST-G and MBPO ($x$-axis is episode). The shaded area of ProST-G is 95 %
confidence area among 3 different noise bound (0.01, 0.03, 0.05).

####  E.3.2 Non-stationarity : real data

The shaded area of ProST-G is 95 % confidence area among 3 different noise
bounds (0.01, 0.03, 0.05), and the shaded ares of three baselines are the 95 %
confidence area among 8 different learning rates (0.001, 0.003, 0.005, 0.007,
0.01, 0.03, 0.05, 0.07).

(1) Swimmer-v2

Figure 11: (a) average reward with ProST-G and MBPO on real data A,B ($x$-axis
is episode). (b) average reward with ProST-G and three baselines on realdata
A. (c) average reward with ProST-G and three baselines on realdata B.

(2) Halfcheetah-v2

Figure 12: (a) average reward with ProST-G and MBPO on real data A,B ($x$-axis
is episode). (b) average reward with ProST-G and three baselines on realdata
A. (c) average reward with ProST-G and three baselines on realdata B.

(3) Hopper-v2

Figure 13: (a) average reward with ProST-G and MBPO on real data A,B ($x$-axis
is episode). (b) average reward with ProST-G and three baselines on realdata
A. (c) average reward with ProST-G and three baselines on realdata B.

##  Appendix F Meta algorithm design

###  F.1 Meta-algorithm $\mathcal{I}_{g\circ f}[A]$

1 Set : $k_{f}=1$

2 Init : policy $\pi^{0}$, forecaster $f_{\phi^{0}_{f}}$, model estimator
$g_{\phi^{0}_{g}}$ , two dataset $\mathcal{D}_{env},\mathcal{D}_{syn}$

3for  _episode $k$_ do

4 Execute the agent with $\pi^{k}$ in a environment $\mathcal{M}_{k}$ and add
a trajectory to $\mathcal{D}_{env}$.

/* Meta function $\mathcal{I}_{g\circ f}$ */

/* (1) Observe and forecast : */

5 Observe a noisy non-stationary variable $\hat{o}_{k}$

6 Update $f_{\phi_{f}},g_{\phi_{g}}$ using $\mathcal{D}_{env}$ and
$\hat{o}_{k-(w-1):k}$.

7 Use $f_{\phi^{k}_{f}},g_{\phi^{k}_{g}}$ to predict the future
$\widehat{\mathcal{P}}^{k+1},\widehat{\mathcal{R}}^{k+1}$ and construct future
MDP $\widehat{\mathcal{M}}_{k+1}$

/* Baseline $A$ */

/* (2) Optimize : */

8 Roll out synthetic trajectories in the $\widehat{\mathcal{M}}_{k+1}$ and add
them to $\mathcal{D}_{syn}$

9

10 Use $\mathcal{D}_{syn}$to evaluate and update $\pi^{k}$ to
$\widehat{\pi}^{k+1}$

11 end for

Algorithm 1 Meta-algorithm $\mathcal{I}_{g\circ f}[A]$

###  F.2 ProST-T algorithm

1 Set : $k_{f}=1$

2 Init : policy $\pi^{k}$ , forecaster $f_{\phi^{k}_{f}}$, tabular reward
model $g^{R}_{k}$, tabular transition probability model $g^{P}_{k}$,
forecasted state-action value $\widehat{Q}^{\boldsymbol{\cdot},k+1}$, empty
dataset $\mathcal{D}_{env},\mathcal{D}_{syn}$

3 Explore $w$ episodes and add $(\tau^{-k},\hat{o}_{-k})$ to $D_{env}$ where
$k\in[w]$ before starts

4 for  _episodes $k=1,..,K$_ do

5 Rollout a trajectory $\tau_{k}$ using $\pi^{k}$and
$\mathcal{D}_{env}=\mathcal{D}_{env}\cup\\{\tau_{k}\\}$

6 Observe a noisy non-stationary variable $\hat{o}_{k}$

/* Meta function $\mathcal{I}_{g\circ f}$ : (1) update $f,g$ */

7 Update $f_{\phi_{f}}$ :
$\phi^{k}_{f}\leftarrow\operatorname*{arg\,min}_{\phi}\mathcal{L}_{f}(\hat{o}_{k-(w-1):k};\phi)$

8 Update $g^{P}_{k}(s^{\prime},s,a,o)$

9 Update $g^{R}_{k}(s,a,o)$

/* Meta function $\mathcal{I}_{g\circ f}$ : (2) predict
$\widehat{\mathcal{P}}^{k+1},\widehat{\mathcal{R}}^{k+1}$ */

10 Forecast $1$ episode ahead non-stationary variable :
$\hat{o}_{k+1}=f_{\phi^{k}_{f}}(\hat{o}_{k-(w-1):k})$

11 Forecast transition probability function:
$\widehat{g}^{P}_{k+1}=g^{P}_{k}(\cdot,\hat{o}_{k+1})$

12 Forecast reward function:
$\widehat{g}^{R}_{k+1}=g^{R}_{k}(\cdot,\hat{o}_{k+1})$

13 Reset $\mathcal{D}_{syn}$ to empty.

/* Baseline $A$ : NPG with entropy regularization */

14 Set $\widehat{\pi}^{(0)}\leftarrow\pi^{k}$

15 for  _$g=0,..,G-1$_ do

16 Evaluate $Q_{\tau}^{\widehat{\pi}^{(g)}}$ using the rollouts from the
future model $\widehat{g}^{P}_{k+1},\widehat{g}^{R}_{k+1}$

17 Update $\widehat{\pi}$ : $\widehat{\pi}^{(g+1)}\leftarrow
1/Z^{(t)}\cdot\left(\widehat{\pi}^{(g)}\right)^{1-\frac{\eta\tau}{1-\gamma}}\exp{\left((\eta\widehat{Q}_{\tau}^{\widehat{\pi}^{(g)}})/(1-\gamma)\right)}$

18 where
$Z^{(t)}=\sum_{a\in\mathcal{A}}\left(\widehat{\pi}^{(g)}\right)^{1-\frac{\eta\tau}{1-\gamma}}\exp{\left((\eta\widehat{Q}_{\tau}^{\widehat{\pi}^{(g)}})/(1-\gamma)\right)}$

19 end for

20 Set $\pi^{k+1}\leftarrow\widehat{\pi}^{(G)}$

21 end for

I’ve removed the
commands after lines where they were not needed, and your LaTeX code should
now compile without the "There’s no line here to end" error. Please test it in
your LaTeX editor to ensure it works as expected.

###  F.3 ProST-G algorithm

(1) Forecaster $f$. We adopt the ARIMA model to forecast $\hat{o}_{k+1}$ from
the noisy observed $\widehat{o}_{k-(w-1):k}$. ARIMA model is one of the most
general class of models for forecasting a time series which can be made to be
stationary by taking a difference among the data. For given time series data
$X_{t}$, we define ARIMA$(p,d,q)$ as given by
$X_{t}-\alpha_{1}X_{t-1}-...-\alpha_{p}X_{t-p}=\epsilon_{t}+\theta_{1}\epsilon_{t-1}+...+\theta_{q}\epsilon_{t-q}$
where $\alpha_{i}$ are the parameters of the autoregressive part of the model,
the $\theta_{i}$ are the parameters of the moving average part and
$\epsilon_{t}$ are the error terms that take $d$ times difference between
$X_{t}$s, which we assume to be independent and follow a normal distribution
with zero mean.

(2) Model predictor $g$. We use a bootstrap ensemble of dynamic models
$\\{g^{1}_{\phi_{g}},g^{1}_{\phi_{g}},...,g^{M}_{\phi_{g}}\\}$. Each ensemble
model is a probabilistic neural network whose output is parameterized by the
mean vector $\mu$ and the diagonal vector of the standard deviation
$\text{Diag}(\Sigma)$ of a Gaussian distribution ,
$g^{i}_{\phi_{g}}(s_{h+1},r_{h}|s_{h},a_{h},\widehat{o}_{k+1})=\mathcal{N}(\mu^{i}_{\phi_{g}}(s_{h},a_{h}),\Sigma^{i}_{\phi_{g}}(s_{h},a_{h}))$.
To efficiently handle uncertainty due to the non-stationary environment, we
designed each neural network to be a probabilistic model to capture the
aleatoric uncertainty, the noise of the output, and learn multiple models as
bootstrap ensemble to handle the epistemic uncertainty, the uncertainty in the
model parameters. Then we predict $s_{h+1},r_{h}$ from a model uniformly
chosen from its ensemble randomly that admits different transitions along a
single model rollout to be sampled from different dynamics modes.

(3) Baseline algorithm $A$. We adopt soft-actor critic (SAC) as our policy
optimization algorithm. SAC alternates the policy evaluation step and the
policy optimization step. For a given policy $\widehat{\pi}$, it estimates the
forecasted $\widehat{Q}^{\widehat{\pi},k+1}$ value using Bellman backup
operator and optimizes the policy that minimizes the expected KL-divergence
between $\pi$ and the exponential of the difference
$\widehat{Q}^{\widehat{\pi},k+1}-\widehat{V}^{\widehat{\pi},k+1}$ :
$\mathbb{E}_{s\sim
D_{syn}}[D_{KL}(\widehat{\pi}||\exp{(\widehat{Q}^{\widehat{\pi},k+1}-\widehat{V}^{\widehat{\pi},k+1})}]$

1 Set : $k_{f}=1$

2 Init : policy $\pi^{k}$, forecaster $f_{\phi^{k}_{f}}$, model estimator
$g_{\phi^{k}_{g}}$ , two dataset $\mathcal{D}_{env},\mathcal{D}_{syn}$

3

4Explore $w$ episodes and add $(\tau^{-k},\hat{o}_{-k})$ to $D_{env}$ where
$k\in[w]$ before starts

5 for  _episodes $k=1,..,K$_ do

6 Execute the agent with $\pi^{k}$ in a environment $\mathcal{M}_{k}$ and add
a trajectory to $\mathcal{D}_{env}$.

/* Meta function $\mathcal{I}_{g\circ f}$ : (1) update $f,g$ */

7 Observe a noisy non-stationary variable $\hat{o}_{k}$

8 Optimize $f_{\phi^{k}_{f}}$ on $\hat{o}_{k-(w-1):k}$

9 Optimize $g_{\phi^{k}_{g}}$ on $\mathcal{D}_{env}$

/* Meta function $\mathcal{I}_{g\circ f}$ : (2) predict $f,g$ */

10 Forecast $\hat{o}_{k+1}=f_{\phi^{k}_{f}}(\hat{o}_{k-(w-1):k})$

11 Forecast model : $\widehat{g}_{k+1}=g_{\phi^{k}_{g}}(\cdot,\hat{o}_{k+1})$

12 Reset $\mathcal{D}_{syn}$ to empty.

/* Baseline $A$ : SAC */

13 Set $\widehat{\pi}^{k+1}\leftarrow\pi^{k}$

14 for  _epochs $n=1,...,N$_ do

15 for  _model rollouts $m=1,..,M$_ do

16 Sample $\hat{s}^{m}_{0}$ uniformly from $D_{env}$.

17 Perform a $\widehat{H}$-step model rollout using
$\hat{a}^{m}_{h}=\widehat{\pi}^{k+1}(\hat{s}^{m}_{h})$,
$\hat{s}^{m}_{h+1}=\widehat{g}_{k+1}(\hat{s}^{m}_{h},\hat{a}^{m}_{h})$ and add
a rollout to $\mathcal{D}_{syn}$ .

18

19 end for

20

21 for  _updates $g=1,..,G$_ do

22 Evaluate and update forecasted policy $\widehat{\pi}^{k+1}$ on
$\mathcal{D}_{syn}$

23 end for

24

25 end for

26 Set $\pi_{k+1}\leftarrow\widehat{\pi}^{k+1}$

27

28 end for

Algorithm 2 Forecasted-Model Based Policy Optimization ((ProST-G)

##  Appendix G Experiment Platforms and Licenses

###  G.1 Platforms

All experiments are done on 12 Intel Xeon CPU E5-2690 v4 and 2 Tesla V100
GPUs.

###  G.2 Licenses

We have used the following libraries/ repos for our python codes:

  * •

Pytorch (BSD 3-Clause "New" or "Revised" License).

  * •

OpenAI Gym (MIT License).

  * •

Numpy (BSD 3-Clause "New" or "Revised" License).

  * •

Official codes distributed from the paper [7]: to compare the four baselines.

  * •

Official codes distributed from the paper [24]: to build PMT-G.