# Maximum principle for the weak solutions of the Cauchy problem for the
fourth-order hyperbolic equations

Kateryna Buryachenko
Humboldt-Universität zu Berlin, Berlin, Germany
Vasyl’ Stus Donetsk National University, Vinnytsia, Ukraine

###### Abstract

We investigate the maximum principle for the weak solutions to the Cauchy
problem for the hyperbolic fourth-order linear equations with constant complex
coefficients in the plane bounded domain.

MSC (2010):

Keywords: the Cauchy problem, the maximum principle, hyperbolic fourth-order
PDEs, weak solutions, L-traces.

## Introduction

We concern here the problem of proving the analog of maximum principle for the
weak solutions of the Cauchy problem for the fourth-order linear hyperbolic
equations with the complex constant coefficients and homogeneous non-
degenerate symbol in some plane bounded domain $\Omega\in{R}^{2}$:

$L(\partial_{x})u=a_{0}\frac{\partial^{4}u}{\partial
x_{1}^{4}}+a_{1}\frac{\partial^{4}u}{\partial x_{1}^{3}\partial
x_{2}}+a_{2}\frac{\partial^{4}u}{\partial x_{1}^{2}\partial
x_{2}^{2}}+a_{3}\frac{\partial^{4}u}{\partial x_{1}\partial
x_{2}^{3}}+a_{4}\frac{\partial^{4}u}{\partial x_{2}^{4}}=f(x).$ (0.1)

Here coefficients $a_{j}\in C,\,j=0,\,1,...,\,4$ are constant, $f(x)\in
L^{2}(\Omega),$ $\partial_{x}=\left(\frac{\partial}{\partial
x_{1}},\,\frac{\partial}{\partial x_{2}}\right).$ We assume, that Eq.(0.1) is
hyperbolic, that means that all roots of characteristics equation

$L(1,\,\lambda)=a_{0}\lambda^{4}+a_{1}\lambda^{3}+a_{2}\lambda^{2}+a_{3}\lambda+a_{4}=0$

are prime, real and are not equal to $\pm i$, that means that the symbol of
Eq.(0.1) is non-degenerate or that the Eq. (0.1) is a principal-type equation.
The equations for which the roots of the corresponding characteristic equation
are multiple and can take the values $\pm i$ are called the equation with
degenerate symbol (see [6]).

The main novelty of the paper is to prove the analog of maximum principle for
the weak solutions of the Cauchy problem for the fourth-order hyperbolic
equations. Most of these equations serve as mathematical models of many
physical processes and attract the interest of researchers. The most famous of
them are elasticity beam equations (Timoshenko beam equations with and without
internal damping) [9], short laser pulse equation [11], equations which
describe the structures are subjected to moving loads, equation of Euler-
Bernoulli beam resting on two-parameter Pasternak foundation and subjected to
a moving load or mass [22] and others.

Due to evident practice application, these models need more exact tools for
studying, and as consequence, to attract fundamental knowledge. As usual, most
of these models are studied by the analytical-numerical methods (Galerkin’s
methods).

On the other hand, the maximum principle is an efficient tool for fundamental
knowledge of PDEs: the range of problems, which maximum principle allows to
study, belongs to a class of quite actual problems of well-posedness of so-
called general boundary-value problems for higher-order differential equations
originating from the works by L. Hormander and M.Vishik who used the theory of
extensions to prove the existence of well-posed boundary-value problems for
linear differential equations of arbitrary order with constant complex
coefficients in a bounded domain with smooth boundary. This theory got its
present-day development in the works by G. Grubb [12], L.Hormander [13], and
A. Posilicano [20]. Later, the problem of well-posedness of boundary-value
problems for various types of second order differential equations and systems
was studied by V. Burskii and A. Zhedanov [3], which developed a method of
traces associated with a differential operator and applied this method to
establish the Poncelet, Abel and Goursat problems, and also by I.Kmit [14]. In
the previous works of author (see [7]—[8]) there have been developed
qualitative methods of studying the Cauchy problems and nonstandard in the
case of hyperbolic equations the Dirichlet and Neumann problems for the linear
fourth-order equations (moreover, for an equation of any even order
$2m,\,m\geq 2,$ ) with the help of operator methods (L-traces, theory of
extension, moment problem, method of duality equation-domain and others).

As concern the maximum principle, at the present time there are not any
results for the fourth order equations even in linear case. As it was
mentioned above, the maximum principle even for the simple case of one
dimensional wave equation [1], and for the second-order telegraph equation
[16] are quite different from those for elliptic and parabolic cases.

Following R.Ortega, A.Robles-Perez [19], we introduce the definition of the
maximum principle for the hyperbolic equations, which will be used later.

Definition 1. [19]. Let $\mathcal{L}=\mathcal{L}u$ be linear differential
operator, acting on functions $u:\,D\to R$, in some domain $D$. These
functions will belong to the certain family $\mathcal{B},$ which includes
boundary conditions or others requirements. It is said that $\mathcal{L}$
satisfies the maximum principle, if

$\mathcal{L}u\geq 0,\,\,u\in\mathcal{B},$

implies $u\geq 0$ in $D$.

In further works of these authors (see [16], [17],[18]) there was studied the
maximum principle for weak bounded twice periodical solutions from the space
$L^{\infty}$ of the telegraph equation with parameter $\lambda$ in lower term,
one-, two-, and -tree dimensional spaces, and which includes the cases of
variables coefficients. The precise condition for $\lambda$ under which the
maximum principle still valid was font. There was also introduced a method of
upper and lower solutions associated with the nonlinear equation, which allows
to obtain the analogous results (uniqueness, existence and regularity
theorems) for the telegraph equations with external nonlinear forcing,
applying maximum principle. There was considered also the case when the
external forcing belongs to a certain space of measures.

The maximum principle for general quasilinear hyperbolic systems with
dissipation was proved by [15]. There were given two estimates for the
solution to the general quasilinear hyperbolic system (see and introduced the
concept of dissipation (strong dissipation and weak dissipation), then state
some maximum principles of quasilinear hyperbolic systems with dissipation.
Using the maximum principle there were reproved the existence and uniqueness
theorems of the global smooth solution to the Cauchy problem for considered
quasilinear hyperbolic system.

So, the problem to prove the maximum principle for the weak solutions stills
more complicated and at that time becomes more interesting in the case of
fourth-order hyperbolic equations, especially, in the case of non-classical
boundary value problems with weak-regularity data. There are no results on
maximum principle even for model case of linear 2-dimensional fourth-order
hyperbolic equations with the constant coefficients and without lower terms.
Moreover, we can not use the term of usual traces in the cases of initial data
of weak regularity, and we come to the notions of $L-$traces, the traces,
which associated with differential operator (see the Section 1). Let us remind
(see, for example, [2]), that $L-$ traces exist for the weak solutions from
space $L^{2}$ even in the situations when classical notions of traces does not
work for such solutions.

##  1 Statement of the problem and auxiliary definitions

Let us start to establish the maximum principle for the weak solutions to the
Cauchy problem for the Eq.(0.1) in some admissible planar domain. Naturally,
that in the hyperbolic case the characteristics of the equation play a crucial
role.

Let $C_{j},\,j=1,\,2,\,3,\,4$ be characteristics. We call the angle of
characteristics slop the solution to the equation
$-\tan\varphi_{j}=\lambda_{j}$, and angle between $j-$ and $k-$
characteristics: $\varphi_{k}-\varphi_{j}\neq\pi l,\,l\in Z,$ where
$\lambda_{j}\neq\pm i$ are real and prime roots of the characteristics
equation, $j,\,k=1,\,2,\,3,\,4$.

Let also $\Gamma_{0}:=\\{x_{1}\in[a,\,b],\,x_{2}=0\\},$ and define the domain
$\Omega$ as a domain which is restricted by the characteristics
$C_{j},\,j=1,\,2,\,3,\,4$ and $\Gamma_{0}.$ Consider also the following Cauchy
problem for the Eq.(0.1) on $\Gamma_{0}:$

$u|_{\Gamma_{0}}=\varphi(x),\,u^{\prime}_{\nu}|_{\Gamma_{0}}=\psi(x),\,u^{\prime\prime}_{\nu\nu}|_{\Gamma_{0}}=\sigma(x),\,u^{\prime\prime\prime}_{\nu\nu\nu}|_{\Gamma_{0}}=\chi(x),$
(1.1)

where $\varphi,\,\psi,\,\sigma$ and $\chi$ are given weak regular functions on
$\Gamma_{0},$ in general case $\varphi,\,\psi,\,\sigma,\,\chi\in
L^{2}(\Gamma_{0}),\,\nu-$ is outer normal of $\Gamma_{0}.$

Definition 2. [21]. We call a domain
$D:=\\{(x_{1},\,x_{2}):\,x_{1}\in(-\infty,\,+\infty),\,x_{2}>0\\}$ in the
half-plane $x_{2}>0$ an admissible domain if it has the property that for each
point $C\in D$ the corresponding characteristics domain $\Omega$ is also in
$D$. More generally, $D$ is admissible, if it is the finite or countable union
of characteristics 5-angles (in the case of fourth-order equations with
constant coefficients, that is existence of 4 different and real
characteristics lines).

Establishment of the maximum principle in this situation allows us to obtain a
local properties of the solution to Cauchy problem (0.1)—(1.1) on the
arbitrary interior point $C\in D.$

We will consider the weak solution to the problem (0.1)—(1.1) from the $D(L)$,
domain of definition of maximal operator, associated with the differential
operation $\mathcal{L}$ in Eq.(0.1). Following [2], [6], we remind the
corresponding definitions.

In the bounded domain $\Omega$ we consider the linear differential operation
$\mathcal{L}$ of the order $m,\,m\geq 2,$ and formally adjoint
$\mathcal{L}^{+}$:

$\mathcal{L}(D_{x})=\sum\limits_{|\alpha|\leq
m}a_{\alpha}D^{\alpha},\,\mathcal{L}^{+}(D_{x})=\sum\limits_{|\alpha|\leq
m}D^{\alpha}(a_{\alpha}),$ (1.2)

where
$\alpha=(\alpha_{1},\,\alpha_{2},...\alpha_{n}),\,|\alpha|=\alpha_{1}+\alpha_{2}+...+\alpha_{n}$
is multi-index. Note, that for Eq.(0.1) $n=2,\,m=4.$

Definition 3. Minimum operator. [2]. Let us consider the differential
operation (1.2) on functions from the space $C_{0}^{\infty}(\Omega).$ The
minimum operator $L_{0}$ is called the extension of the operation from
$C_{0}^{\infty}(\Omega)$ to the set
$D(L_{0}):=\overline{C_{0}^{\infty}(\Omega)}.$ The closure is realized in the
norm of the graph of operator $L$:
$||u||_{L}^{2}:=||u||_{L_{2}(\Omega)}^{2}+||Lu||^{2}_{L_{2}(\Omega)}.$

Definition 4. Maximum operator. [2]. The maximum operator $L$ is define as the
restriction of the differential operation $\mathcal{L}(D_{x})$ to the set
$D(L):=\\{u\in L^{2}(\Omega):\,Lu\in L^{2}(\Omega)\\}.$

Definition 5. [2]. The operator $\tilde{L}$ is define as the extension of the
minimum operator $L_{0},$ to the set
$D(\tilde{L}):=\overline{C^{\infty}(\bar{\Omega})}.$

Definition 6. Regular operator. [2]. The maximum operator is called regular if
$D(L)=D(\tilde{L}).$

It is easy to see, that in the case of the fourth-order differential operation
(0.1), maximal operator is regular and $D(L)=D(\tilde{L})=H^{4}(\Omega),$
$D(L_{0})=\stackrel{{\scriptstyle 0}}{{H^{4}}}(\Omega).$

The definition of a weak solution to the problem (0.1)—(1.1) from the space
$D(L)$ is closely connected with the notion of $L-$ traces, that is traces,
which are associated with the differential operator $L$.

Definition 7. L-traces. [6]. Assume, that for a function $u\in D(\tilde{L})$
there exist linear continuous functionals $L_{(p)}u$ over the space
$H^{m-p-1/2}(\partial\Omega),\,p=0,1,2...,m-1$, such that the following
equality is satisfied:

$(Lu,v)_{L^{2}(\Omega)}-(u,L^{+}v)_{L^{2}(\Omega)}=\sum\limits_{j=0}^{m-1}(L_{(m-1-j)}u,\,\partial^{(j)}_{\nu}v).$
(1.3)

The functionals $L_{(p)}u$ are called the $L_{(p)}-$ traces of the function
$u\in D(\tilde{L}).$ Here $(\cdot,\,\cdot)_{L^{2}(\Omega)}$ is a scalar
product in Hilbert space $L^{2}(\Omega)$.

Finally, we are going to the definition of the weak solution to the problem
(0.1)—(1.1):

Definition 8. We will call the function $u\in D(L)$ a weak solution to the
Cauchy problem (1)-(2), if it satisfies to the following integral identity

$(f,\,v)_{L^{2}(\Omega)}-(u,\,L^{+}v)_{L^{2}(\Omega)}=\sum\limits_{j=0}^{3}(L_{(3-j)}u,\,\partial^{(j)}_{\nu}v),$
(1.4)

for any functions $v\in C_{0}^{\infty}(\Omega).$ The functionals $L_{(p)}u$ is
called the $L_{(p)}-$ traces of the function $u$, $p=0,\,1,\,2,\,3,$ and
completely determined by the initial functions
$\varphi,\,\psi,\,\sigma,\,\chi$ by the following way [6]:

$L_{(0)}u=-L(x)u|_{\partial\Omega}=-L(\nu)\varphi;$
$L_{(1)}u=L(\nu)\psi+\alpha_{1}\varphi_{\tau}^{\prime}+\alpha_{2}\varphi;$
$L_{(2)}u=-L(\nu)\sigma+\beta_{1}\psi_{\tau}^{\prime}+\beta_{2}\psi+\beta_{3}\varphi_{\tau\tau}^{\prime\prime}+\beta_{4}\varphi_{\tau}^{\prime}+\beta_{5}\varphi;$
(1.5)
$L_{(3)}u=L(\nu)\chi+\delta_{1}\varphi_{\tau\tau\tau}^{\prime\prime\prime}+\delta_{2}\sigma+\delta_{3}\psi_{\tau\tau}^{\prime\prime}+\delta_{4}\psi_{\tau}^{\prime}+\delta_{5}\psi+\delta_{6}\varphi_{\tau\tau}^{\prime\prime}+\delta_{7}\varphi_{\tau}^{\prime}+\delta_{8}\varphi.$

Here $\alpha_{i},\,i=1,\,2,\,\beta_{j},\,j=1,\,2,...,\,5,$ and
$\delta_{k},\,k=1,\,...,\,9$ are smooth functions, completely determined by
the coefficients of the Eq.(0.1).

Remark 1. We can use a general form of the operators $\gamma_{j}$ in the left-
hand side of the identity (1.4) instead of operators of differentiation
$\partial^{(j)}_{\nu}v$. Indeed, we define $\gamma_{j}=p_{j}\gamma,$ where
$\gamma:\,u\in
H^{m}(\Omega)\to(u|_{\partial\Omega},\,...,\,u_{\nu}^{(m-1)}|_{\partial\Omega})\in
H^{(m)}=H^{m-1/2}(\partial\Omega)\times
H^{m-3/2}(\partial\Omega)\times...\times H^{1/2}(\partial\Omega),$ and
$p_{j}:\,H^{(m)}\to H^{m-j-1/2}(\partial\Omega)-$ projection.

As it has been mentioned above, some examples show (see [2]) that in the
general case do not exist ordinary traces of solutions $u\in D(L)$ in the
sense of distributions even for the simplest hyperbolic equations. Indeed, for
the wave equation $Lu=\frac{\partial^{2}u}{\partial x_{1}\partial x_{2}}=0$ in
the unit disk $K:\,|x|=1$, the solution $u(x)=(1-x_{1}^{2})^{-\frac{5}{2}}$
belongs to $L^{2}(K)$, but $<u|_{\partial K},1>_{\partial K}=\infty$ it means
that $\lim_{r\to 1-0}\int\limits_{|x|=r}u(x)ds_{x}=\infty$, such a way the
trace $u|_{\partial K}$ does not exist even as a distribution. However, for
every solution $u\in L^{2}(K)$ the $L_{(0)}-$trace
$L_{(0)}u:=-L(x)u(x)|_{|x|=1}=-x_{1}x_{2}u(x)|_{|x|=1}\in L^{2}(\partial K).$
Likewise, $L_{(1)}-$ trace $L_{(1)}u$ exists for every $u\in L^{2}(K)$:

$L_{(1)}u=\left(L(x)u^{\prime}_{\nu}+L^{\prime}_{\tau}u^{\prime}_{\tau}+\frac{1}{2}L^{\prime\prime}_{\tau\tau}u\right)|_{\partial
K}\in H^{-\frac{3}{2}}(\partial K).$

where $\tau$ is the angular coordinate and $u^{\prime}_{\tau}$ is the
tangential derivative, and $L(x)=x_{1}x_{2}-$ symbol of the operator
$L=\frac{\partial^{2}}{\partial x_{1}\partial x_{2}}$.

##  2 Maximum principle for the weak solutions of Cauchy problem

We prove here the first simple case of the maximum principle for the weak
solution of the Cauchy problem (0.1)—(1.1) in the admissible plane domain
$\Omega,$ restricted by the different and non-congruent characteristics
$C_{j},\,j=1,\,2,...,\,4$ and initial line $\Gamma_{0}$.

Theorem 1. Maximum principle. Let $u\in D(L)$ and satisfy the following
inequalities:

$Lu=f\leq 0,\,\,\,x\in D,$ (2.1)

and

$L_{(0)}u\mid_{\Gamma_{0}}\geq 0,\,L_{(1)}u|_{\Gamma_{0}}\geq
0,\,L_{(2)}u|_{\Gamma_{0}}\geq 0,\,L_{(3)}u|_{\Gamma_{0}}\geq 0,$ (2.2)

then $u\leq 0$ in $D.$ Proof.

Due to homogeneity of the symbol,
$L(\xi)=a_{0}\xi_{1}^{4}+a_{1}\xi_{1}^{3}\xi_{2}+a_{2}\xi_{1}^{2}\xi_{2}^{2}+a_{3}\xi_{1}\xi_{2}^{3}+a_{4}\xi_{2}^{4}=<\xi,\,a^{1}><\xi,\,a^{2}><\xi,\,a^{3}>\times\\\
<\xi,\,a^{4}>,\,\xi=(\xi_{1},\,\xi_{2})\in R^{2},$ we can rewrite equation
(0.1) in the following form:

$<\nabla,\,a^{1}><\nabla,\,a^{2}><\nabla,\,a^{3}><\nabla,\,a^{4}>u=f(x).$
(2.3)

The vectors $a^{j}=(a_{1}^{j},\,a_{2}^{j}),\,j=1,\,2,\,3,\,4$ are determined
by the coefficients $a_{i},\,i=0,\,1,\,2,\,3,\,4,$ and
$<a,\,b>=a_{1}\bar{b}_{1}+a_{2}\bar{b}_{2}$ is a scalar product in $C^{2}$. It
is easy to see that vector $a^{j}$ is a tangent vector of $j-$th
characteristic, slope $\varphi_{j}$ of which is determined by
$-\tan\varphi_{j}=\lambda_{j},\,j=1,\,2,\,3,\,4.$ In what follows, we also
consider the vectors
$\tilde{a}^{j}=(-\bar{a}_{2}^{j},\,\bar{a}_{1}^{j}),\,j=1,\,2,\,3,\,4.$ It is
obvious that $<\tilde{a}^{j},\,a^{j}>=0,$ so $\tilde{a}^{j}$ is a normal
vector of $j-$th characteristic.

Use the definitions 7 and 8 for the case $m=4,$ that is fourth-order operator
in Eq.(0.1), and domain $\Omega,$ which is restricted by the characteristics
$C_{j},\,j=1,\,2,\,3,\,4$ and $\Gamma_{0}:$

$\int\limits_{\Omega}\\{Lu\cdot\bar{v}-u\cdot\overline{L^{+}v}\\}dx=\sum\limits_{k=0}^{3}\int\limits_{\partial\Omega}L_{(3-k)}u\cdot\partial^{(k)}_{\nu}v\,ds=$
$=\sum\limits_{k=0}^{3}\int\limits_{C_{1}}L_{(3-k)}u\cdot\partial^{(k)}_{\nu}v\,ds+\sum\limits_{k=0}^{3}\int\limits_{C_{2}}L_{(3-k)}u\cdot\partial^{(k)}_{\nu}v\,ds+\sum\limits_{k=0}^{3}\int\limits_{C_{3}}L_{(3-k)}u\cdot\partial^{(k)}_{\nu}v\,ds+\sum\limits_{k=0}^{3}\int\limits_{C_{4}}L_{(3-k)}u\cdot\partial^{(k)}_{\nu}v\,ds+$
$+\sum\limits_{k=0}^{3}\int\limits_{\Gamma_{0}}L_{(3-k)}u\cdot\partial^{(k)}_{\nu}v\,ds.$
(2.4)

Taking into account the representation (2.3), we arrive to

$\int\limits_{\Omega}Lu\cdot\bar{v}\,dx=\int\limits_{\Omega}<\nabla,\,a^{1}><\nabla,\,a^{2}><\nabla,\,a^{3}><\nabla,\,a^{4}>u\cdot\bar{v}\,dx=$
$\int\limits_{\partial\Omega}<\nu,\,a^{1}>\cdot<\nabla,\,a^{2}><\nabla,\,a^{3}><\nabla,\,a^{4}>u\cdot\bar{v}\,ds-$
$\int\limits_{\Omega}<\nabla,\,a^{2}><\nabla,\,a^{3}><\nabla,\,a^{4}>u\cdot\overline{<\nabla,\,a^{1}>v}\,dx,$

and further:

$\int\limits_{\Omega}Lu\cdot\bar{v}\,dx=\int\limits_{\partial\Omega}<\nu,\,a^{1}>\cdot<\nabla,\,a^{2}><\nabla,\,a^{3}><\nabla,\,a^{4}>u\cdot\bar{v}\,ds-$
$\int\limits_{\partial\Omega}<\nu,\,a^{2}>\cdot<\nabla,\,a^{3}><\nabla,\,a^{4}>u\cdot\overline{<\nabla,\,a^{1}>v}\,ds+$
$+\int\limits_{\partial\Omega}<\nu,\,a^{3}>\cdot<\nabla,\,a^{4}>u\cdot\overline{<\nabla,\,a^{2}><\nabla,\,a^{1}>v}\,ds-$
$-\int\limits_{\partial\Omega}<\nu,\,a^{4}>\cdot
u\cdot\overline{<\nabla,\,a^{3}><\nabla,\,a^{2}><\nabla,\,a^{1}>v}\,ds+$
$+\int\limits_{\Omega}u\cdot\overline{<\nabla,\,a^{4}><\nabla,\,a^{3}><\nabla,\,a^{2}><\nabla,\,a^{1}>v}\,dx.$

Since
$<\nabla,\,a^{4}><\nabla,\,a^{3}><\nabla,\,a^{2}><\nabla,\,a^{1}>v=L^{+}v,$
and determining

$\tilde{L}_{(0)}u:=<\nu,\,a^{4}>u,\,\,\tilde{L}_{(1)}u:=<\nu,\,a^{3}>\cdot<\nabla,\,a^{4}>u,$
$\tilde{L}_{(2)}u:=<\nu,\,a^{2}>\cdot<\nabla,\,a^{3}><\nabla,\,a^{4}>u,$
$\tilde{L}_{(3)}u=L_{(3)}u=<\nu,\,a^{1}>\cdot<\nabla,\,a^{2}><\nabla,\,a^{3}><\nabla,\,a^{4}>u$

that are analogues of $L-$traces from the formula (2.4). Such a way we have

$\int\limits_{\Omega}\\{Lu\cdot\bar{v}-u\cdot\overline{L^{+}v}\\}\,dx=\int\limits_{\partial\Omega}L_{(3)}u\cdot\bar{v}\,ds-\int\limits_{\partial\Omega}\tilde{L}_{(2)}u\cdot\overline{<\nabla,\,a^{1}>v}\,ds+$
$+\int\limits_{\partial\Omega}\tilde{L}_{(1)}u\cdot\overline{<\nabla,\,a^{2}><\nabla,\,a^{1}>v}\,ds-\int\limits_{\partial\Omega}\tilde{L}_{(0)}u\cdot\overline{<\nabla,\,a^{3}><\nabla,\,a^{2}><\nabla,\,a^{1}>v}\,ds.$
(2.5)

Difference between formulas (2.4) and (2.5) is that natural $L_{(3-k)}$ traces
in (2.4) are multiplied by $k-$ derivative by outer normal $\nu$ of truncated
function $v:\,\partial^{(k)}_{\nu}v,$ on the other hand, in (2.5) we
determined by $\tilde{L}_{(3-k)}$ some expressions which multiplied by
differential operators $L_{k}^{+}v$ of order $k$ and which can serve as
analogous of natural $L_{(3-k)}$ traces, $k=0,\,1,\,2,\,3.$ So, in the (2.5)

$L^{+}_{1}v:=<\nabla,\,a^{1}>v,\,L^{+}_{2}v:=<\nabla,\,a^{2}><\nabla,\,a^{1}>v,$
$L_{0}^{+}v=v,\,L^{+}_{3}v:=<\nabla,\,a^{3}><\nabla,\,a^{2}><\nabla,\,a^{1}>v.$

Let $v\in KerL^{+}$ in (2.5) and calculate $L-$ traces on
$\partial\Omega=C_{1}\cup C_{2}\cup C_{3}\cup C_{4}\cup\Gamma_{0}$. For
instance, for $L_{(3)}u$ we obtain:
$L_{(3)}u=<\nu,\,a^{1}><\nabla,\,a^{2}><\nabla,\,a^{3}><\nabla,\,a^{4}>u,$ and
use that
$<\nabla,\,a^{j}>u=<\nu,\,a^{j}>u^{\prime}_{\nu}+<\tau,\,a^{j}>u^{\prime}_{\tau},\,j=1,\,2,\,3,\,4,$
where $\nu-$ normal vector, $\tau-$ tangent vector. Due to presence the
product $<\nu,\,a^{1}>,$ $L_{(3)}u=0$ on characteristic $C_{1},$ normal vector
$\tilde{a}^{1}$ of which is orthogonal to the vector $a^{1}$. On the other
parts of $\partial\Omega$ there will vanish the terms containing $<\nu,a^{j}>$
on $C_{j}$. After that

$\int\limits_{\partial\Omega}<\nu,\,a^{1}><\nabla,\,a^{2}><\nabla,\,a^{3}><\nabla,\,a^{4}>u=\int\limits_{\Gamma_{0}}L_{(3)}u\,ds+$
$<\tilde{a}^{2},\,a^{1}><a^{2},\,a^{2}><\tilde{a}^{2},\,a^{3}><\tilde{a}^{2},\,a^{4}>\int\limits_{C_{2}}u_{\nu\nu\tau}\,ds+$
$<\tilde{a}^{3},\,a^{1}><\tilde{a}^{3},\,a^{2}><a^{3},\,a^{3}><\tilde{a}^{3},\,a^{4}>\int\limits_{C_{3}}u_{\nu\nu\tau}\,ds+$
$<\tilde{a}^{4},\,a^{1}><\tilde{a}^{4},\,a^{2}><\tilde{a}^{4},\,a^{3}><a^{4},\,a^{4}>\int\limits_{C_{4}}u_{\nu\nu\tau}\,ds+$
$\left\\{<\tilde{a}^{2},\,a^{1}><a^{2},\,a^{2}><\tilde{a}^{2},\,a^{3}><a^{2},\,a^{4}>+<\tilde{a}^{2},\,a^{1}><a^{2},\,a^{2}><a^{2},\,a^{3}><\tilde{a}^{2},\,a^{4}>\right\\}\int\limits_{C_{2}}u_{\tau\tau\nu}\,ds+$
$\left\\{<\tilde{a}^{3},\,a^{1}><\tilde{a}^{3},\,a^{2}><a^{3},\,a^{3}><a^{3},\,a^{4}>+<\tilde{a}^{3},\,a^{1}><a^{3},\,a^{2}><a^{3},\,a^{3}><\tilde{a}^{3},\,a^{4}>\right\\}\int\limits_{C_{3}}u_{\tau\tau\nu}\,ds+$
$\left\\{<\tilde{a}^{4},\,a^{1}><\tilde{a}^{4},\,a^{2}><a^{4},\,a^{3}><a^{4},\,a^{4}>+<\tilde{a}^{4},\,a^{1}><a^{4},\,a^{2}><\tilde{a}^{4},\,a^{3}><a^{4},\,a^{4}>\right\\}\int\limits_{C_{4}}u_{\tau\tau\nu}\,ds+$
$<\tilde{a}^{2},\,a^{1}><a^{2},\,a^{2}><a^{2},\,a^{3}><a^{2},\,a^{4}>\int\limits_{C_{2}}u_{\tau\tau\tau}\,ds+$
$<\tilde{a}^{3},\,a^{1}><a^{3},\,a^{2}><a^{3},\,a^{3}><a^{3},\,a^{4}>\int\limits_{C_{3}}u_{\tau\tau\tau}\,ds+$
$<\tilde{a}^{4},\,a^{1}><a^{4},\,a^{2}><a^{4},\,a^{3}><a^{4},\,a^{4}>\int\limits_{C_{4}}u_{\tau\tau\tau}\,ds+\alpha_{4,1}\int\limits_{C_{2}}u_{\nu\nu}\,ds+\alpha_{4,2}\int\limits_{C_{3}}u_{\nu\nu}\,ds+\alpha_{4,3}\int\limits_{C_{4}}u_{\nu\nu}\,ds+$
$\alpha_{5,1}\int\limits_{C_{2}}u_{\nu\tau}\,ds+\alpha_{5,2}\int\limits_{C_{3}}u_{\nu\tau}\,ds+\alpha_{5,3}\int\limits_{C_{4}}u_{\nu\tau}\,ds+\alpha_{6,1}\int\limits_{C_{2}}u_{\tau\tau}\,ds+\alpha_{6,2}\int\limits_{C_{3}}u_{\tau\tau}\,ds+\alpha_{6,3}\int\limits_{C_{4}}u_{\tau\tau}\,ds+$
$\alpha_{7,1}\int\limits_{C_{2}}u_{\nu}\,ds+\alpha_{7,2}\int\limits_{C_{3}}u_{\nu}\,ds+\alpha_{7,3}\int\limits_{C_{4}}u_{\nu}\,ds+\alpha_{8,1}\int\limits_{C_{2}}u_{\tau}\,ds+\alpha_{8,2}\int\limits_{C_{3}}u_{\tau}\,ds+\alpha_{8,3}\int\limits_{C_{4}}u_{\tau}\,ds.$

Here correspondent coefficients $\alpha_{i,j}$ were numerated as follows:
first index $i$ indicates the derivative of $u$:
$1)\,u_{\nu\nu\tau},\,2)\,u_{\nu\tau\tau},\,3)\,u_{\tau\tau\tau},\,4)\,u_{\nu\nu},\,5)\,u_{\nu\tau},\,6)\,u_{\tau\tau},\,7)\,u_{\nu}\,8)\,u_{\tau},$
the second index $j$ indicates $j+1-$th characteristic, $j=1,\,2,\,3.$ So, now
the formula (2.4) has the form:

$\int\limits_{\Omega}Lu\,dx=\int\limits_{\Gamma_{0}}L_{(3)}u\,ds+\alpha_{1,1}\int\limits_{C_{2}}u_{\nu\nu\tau}\,ds+\alpha_{1,2}\int\limits_{C_{3}}u_{\nu\nu\tau}\,ds+\alpha_{1,3}\int\limits_{C_{4}}u_{\nu\nu\tau}\,ds+$
$\alpha_{2,1}\int\limits_{C_{2}}u_{\tau\tau\nu}\,ds+\alpha_{2,2}\int\limits_{C_{3}}u_{\tau\tau\nu}\,ds+\alpha_{2,3}\int\limits_{C_{4}}u_{\tau\tau\nu}\,ds+$
$\alpha_{3,1}\int\limits_{C_{2}}u_{\tau\tau\tau}\,ds+\alpha_{3,2}\int\limits_{C_{3}}u_{\tau\tau\tau}\,ds+\alpha_{3,3}\int\limits_{C_{4}}u_{\tau\tau\tau}\,ds+$
$\alpha_{4,1}\int\limits_{C_{2}}u_{\nu\nu}\,ds+\alpha_{4,2}\int\limits_{C_{3}}u_{\nu\nu}\,ds+\alpha_{4,3}\int\limits_{C_{4}}u_{\nu\nu}\,ds+$
$\alpha_{5,1}\int\limits_{C_{2}}u_{\nu\tau}\,ds+\alpha_{5,2}\int\limits_{C_{3}}u_{\nu\tau}\,ds+\alpha_{5,3}\int\limits_{C_{4}}u_{\nu\tau}\,ds+\alpha_{6,1}\int\limits_{C_{2}}u_{\tau\tau}\,ds+\alpha_{6,2}\int\limits_{C_{3}}u_{\tau\tau}\,ds+\alpha_{6,3}\int\limits_{C_{4}}u_{\tau\tau}\,ds+$
$\alpha_{7,1}\int\limits_{C_{2}}u_{\nu}\,ds+\alpha_{7,2}\int\limits_{C_{3}}u_{\nu}\,ds+\alpha_{7,3}\int\limits_{C_{4}}u_{\nu}\,ds+\alpha_{8,1}\int\limits_{C_{2}}u_{\tau}\,ds+\alpha_{8,2}\int\limits_{C_{3}}u_{\tau}\,ds+\alpha_{8,3}\int\limits_{C_{4}}u_{\tau}\,ds.$

Coefficients $\alpha_{i,j}$ are constant and depend on only from coefficients
$a_{0},\,a_{1},\,a_{2},\,a_{3},\,a_{4}.$ By analogous way we calculate others
$L-$ traces, $L_{(0)}u,\,L_{(1)}u$ and $L_{(2)}u.$

To obtain the statement of the Theorem 1, we choose some arbitrary point $C\in
D$ in admissible plane domain $D,$ draw through this point two arbitrary
characteristics, $C_{1}$ and $C_{2}$. Another two characteristics ($C_{3}$ and
$C_{4}$) we draw through the ends $a$ and $b$ of initial line $\Gamma_{0}$. We
determine the points $O_{1}$ and $O_{2}$ as intersections of $C_{1},\,C_{3}$
and $C_{2},\,C_{4}$ correspondingly: $O_{1}=C_{1}\cap C_{3},\,O_{2}=C_{2}\cap
C_{4}.$ Such a way, domain $\Omega$ is a pentagon $aO_{1}CO_{2}b.$ The value
of the function $u$ at the point $C\in D,\,u(C)$ we estimate from the last
equality, integrating by the characteristics $C_{1}$ and $C_{2}$ and using
conditions (1.1), (1.5)—(2.2). Since, the chosen point $C\in D$ is arbitrary,
we arrive at $u\leq 0$ in $D.$

Remark 2. In the case of classical solution of the Cauchy problem for the
second order hyperbolic equations of the general form with the constant
coefficients the statement of the Theorem 1 coincides with the result of [21].
In this case conditions (2.2) have usual form without using the notion of
$L-$traces (see [21]):

$u|_{\Gamma_{0}}\leq 0,\,\,u^{\prime}_{\nu}|_{\Gamma_{0}}\leq 0.$

## Acknowledgments

The author thanks to Prof. Iryna Kmit for the drawing the author’s attention
to the problem of the maximum principle for high-order hyperbolic PDEs and
systems, that allowed to apply developed earlier methods of investigations of
the fourth-order PDEs. This work is partially supported by the Volkswagen
Foundation (the project A131968 ”From Modeling and Analysis to
Approximation”).

## References

  * [1] S. Agmon, L. Nirenberg and M. Protter, Comm. on Pure and Applied Math, 1, (1953).
  * [2] V.P. Burskii V.P. Methods for Studing Boundary Value Problems for General Differential Equations (Naukova dumka, Kiev, 2002).
  * [3] V.P. Burskii, A.S. Zhedanov, Comm. on Pure Appl. Analysis, 12, 1587-1633 (2013).
  * [4] V.P. Burskii, International Journal of Modern Physics A, 37, 20-21 (2022).
  * [5] V.P. Burskii, K.O. Buryachenko, Mathematical Notes, 77, 461-470 (2005).
  * [6] V.P. Burskii, K.O. Buryachenko, Journal of Mathematical Sciences, 190, 539-566 (2013).
  * [7] K.O. Buryachenko, Nonlinear boundary value problems, 1, 44-49 (2000).
  * [8] K.O. Buryachenko, Differential Equations, 51, 477-486 (2015).
  * [9] A. Capsoni, G.M. Vigano, and Kh. Bani-Hani, International Journal of Mechanical Sciences, 73, 27-39 (2013).
  * [10] S. Clain, SIAM J. Num.Anal. (2013).
  * [11] G. Fichera, Rend. Math. Acc. Lincei. (1997).
  * [12] G. Grubb. Distributions and Operators (Graduate Texts in Mathematics), 252 (Springer, New York, 2009).
  * [13] L. Hörmander. The Analysis of Linear Partial Differential Operators I-IV (Springer, Berlin, 1983).
  * [14] I. Kmit, L. Recke, Journal of Dynamics and Differential Equations, 1, (2022).
  * [15] Kong De-Xing, Nonlinear Analysis, Theory, Methods and Applications, 32, 871-880 (1998). p.
  * [16] J. Mawhin, R. Ortega, and A.M. Robles-Perez, Journal of Differential Equations, 208, 42-63 (2005).
  * [17] J. Mawhin, R. Ortega, and A.M. Robles-Perez, Journal Math.Anal.Appl., 251, 695-709 (2000).
  * [18] J. Mawhin J, R. Ortega, and A.M. Robles-Perez, C.R.Acad.Sci.Paris,Ser.I, 334, 1089-1094 (2002).
  * [19] R. Ortega, A.M. Robles-Perez, Journal of Math. Analysis and Applications, 221, 625-651 (1998).
  * [20] A. Posilicano, Operat. Matrices, 2, (2008).
  * [21] M. Protter, H. Weinberger, Maximum Principle in Differential Equations (Springer-Verlag, Berlin, 1984).
  * [22] R. Uzzal, Shock and vibration, (2012).


# Improving Intrinsic Exploration by Creating Stationary Objectives

Roger Creus Castanyer
Mila Québec AI Institute
Université de Montréal &Joshua Romoff
Ubisoft LaForge
<EMAIL_ADDRESS>&Glen Berseth
Mila Québec AI Institute
Université de Montréal
&
{roger.creus-castanyer<EMAIL_ADDRESS>
###### Abstract

Exploration bonuses in reinforcement learning guide long-horizon exploration
by defining custom intrinsic objectives. Several exploration objectives like
count-based bonuses, pseudo-counts, and state-entropy maximization are non-
stationary and hence are difficult to optimize for the agent. While this issue
is generally known, it is usually omitted and solutions remain under-explored.
The key contribution of our work lies in transforming the original non-
stationary rewards into stationary rewards through an augmented state
representation. For this purpose, we introduce the Stationary Objectives For
Exploration (SOFE) framework. SOFE requires identifying sufficient statistics
for different exploration bonuses and finding an efficient encoding of these
statistics to use as input to a deep network. SOFE is based on proposing state
augmentations that expand the state space but hold the promise of simplifying
the optimization of the agent’s objective. We show that SOFE improves the
performance of several exploration objectives, including count-based bonuses,
pseudo-counts, and state-entropy maximization. Moreover, SOFE outperforms
prior methods that attempt to stabilize the optimization of intrinsic
objectives. We demonstrate the efficacy of SOFE in hard-exploration problems,
including sparse-reward tasks, pixel-based observations, 3D navigation, and
procedurally generated environments.

##  1 Introduction

Intrinsic objectives have been widely used to improve exploration in
reinforcement learning (RL), especially in sparse-reward and no-reward
settings. In the case of Markov Decision Processes (MDPs) with a finite and
small set of states, count-based exploration methods perform near-optimally
when paired with tabular RL algorithms (Strehl & Littman, 2008; Kolter & Ng,
2009). Count-based methods keep track of the agent’s frequency of state visits
to derive an exploration bonus that can be used to encourage structured
exploration. While much work has studied how to extend these methods to larger
state spaces and continuous environments (Bellemare et al., 2016; Lobel et
al., 2023; Tang et al., 2017), count-based methods introduce unstable learning
dynamics that have not been thoroughly studied and can make it impossible for
the agent to discover optimal policies. Specifically, any reward distribution
that depends on the counts (i.e. the state-visitation frequencies) is non-
stationary because the dynamics for the counts change as the agents generate
new experiences, and the agent does not have access to the information needed
to estimate these dynamics. In an MDP, the convergence of policies and value
functions relies on the transition dynamics and the reward distribution being
stationary (Sutton & Barto, 2018). The non-stationarity of count-based rewards
induces a partially observable MDP (POMDP), as the dynamics of the reward
distribution are unobserved by the agent. In a POMDP, there are no guarantees
for an optimal Markovian (i.e. time-homogeneous) policy to exist (Alegre et
al., 2021; Cheung et al., 2020; Lecarpentier & Rachelson, 2019). In general,
optimal policies in POMDPs will require non-Markovian reasoning to adapt to
the dynamics of the non-stationary rewards (Seyedsalehi et al., 2023). Despite
this issue, count-based methods are usually paired with RL algorithms that are
designed to converge to Markovian policies and hence might attain suboptimal
performance. Previous research has either overlooked or attempted to address
the non-stationarity issue in intrinsic rewards (Singh et al., 2010). Some
efforts to tackle this problem involve completely separating the exploration
and exploitation policies (Schäfer et al., 2021; Whitney et al., 2021).
However, these approaches add an additional layer of complexity to the RL loop
and can introduce unstable learning dynamics. In this work, we introduce a
framework to define stationary objectives for exploration (SOFE). SOFE
provides an intuitive algorithmic modification to eliminate the non-
stationarity of the intrinsic rewards, making the learning objective stable
and stationary. With minimal complexity, SOFE enables both tractable and end-
to-end training of a single policy on the combination of intrinsic and
extrinsic rewards.

SOFE is described in Section 4 and consists of augmenting the original states
of the POMDP by including the state-visitation frequencies or a representative
embedding. SOFE proposes a state augmentation that effectively formulates the
intrinsic reward distribution as a deterministic function of the state, at the
cost of forcing the agent to operate over a larger set of states. We
hypothesize that RL agents with parametrized policies are better at
generalizing across bigger sets of states than at optimizing non-stationary
rewards. We evaluate the empirical performance of SOFE in different
exploration modalities and show that SOFE enables learning better exploration
policies. We present SOFE as a method to solve the non-stationarity of count-
based rewards. However, we show that SOFE provides orthogonal gains to other
exploration objectives, including pseudo-counts and state-entropy
maximization. Furthermore, our experiments in Section 5 show that SOFE is
agnostic to the RL algorithm and robust in many challenging environment
specifications, including large 3D navigation maps, procedurally generated
environments, sparse reward tasks, pixel-based observations, and continuous
action spaces. Videos of the trained agents and summarized findings can be
found on our supplementary webpage111https://sites.google.com/view/sofe-
webpage/home.

##  2 Related Work

Exploration in RL Exploration is a central challenge in RL. Classical
exploration strategies explore in an aleatoric fashion. $\epsilon$-greedy
(Sutton & Barto, 2018) samples random actions during training for the sake of
exploration. Adding random structured noise in the action space (Lillicrap et
al., 2015; Fujimoto et al., 2018) can enable exploration in continuous spaces.
Maximum entropy RL provides a framework to find optimal policies that are as
diverse as possible, and hence better explore the space of solutions (Haarnoja
et al., 2018; Levine et al., 2020). For hard-exploration tasks, structured
exploration has been studied through the lens of hierarchical RL (Gehring et
al., 2021; Eysenbach et al., 2018). State-entropy maximization has been
proposed to explore efficiently, in an attempt to learn policies that induce a
uniform distribution over the state-visitation distribution (Seo et al., 2021;
Lee et al., 2019; Zisselman et al., 2023). In MDPs with sparse reward
distributions, exploration bonuses (i.e. intrinsic rewards) provide proxy
objectives to the agents that can induce state-covering behaviors, hence
allowing agents to find the sparse rewards. Count-based methods (Auer, 2002)
derive an exploration bonus from state-visitation frequencies. Importantly,
the inverse counts of a given state measure its novelty and hence provide a
suitable objective to train exploratory agents. This property makes count-
based exploration an appealing technique to drive structured exploration.
However, count-based methods do not scale well to high-dimensional state
spaces (Bellemare et al., 2016). Pseudo-counts provide a framework to
generalize count-based methods to high-dimensional and partially observed
environments (Tang et al., 2017; Lobel et al., 2023; Bellemare et al., 2016).

In modern deep RL applications, many popular methods enable exploration by
defining exploration bonuses in high-dimensional state spaces (Laskin et al.,
2021), and among them are curiosity-based (Pathak et al., 2017; Burda et al.,
2018), data-based (Yarats et al., 2021) and skill-based (Eysenbach et al.,
2018; Lee et al., 2019). Recently, elliptical bonuses have achieved great
results in contextual MDPs with high-dimensional states (Henaff et al., 2022).
These methods aim to estimate novelty in the absence of the true state-
visitation frequencies. Henaff et al. (2022) showed that elliptical bonuses
provide the natural generalization of count-based methods to high-dimensional
observations. In this work, we show that SOFE improves the performance of
count-based methods in small MDPs and pseudo-counts in environments with high-
dimensional observations (e.g. images), further improving the performance of
the state-of-the-art exploration algorithm E3B in contextual MDPs.
Additionally, our results show that SOFE provides orthogonal gains to
exploration objectives of different natures like state-entropy maximization.

Non-stationary objectives A constantly changing (i.e. non-stationary) MDP
induces a partially observed MDP (POMDP) if the dynamics of the MDP are
unobserved by the agent. In Multi-Agent RL, both the transition and reward
functions are non-stationary because these are a function of other learning
agents that evolve over time (Zhang et al., 2021a; Papoudakis et al., 2019).
In contextual MDPs, the transition and reward functions can change every
episode and hence require significantly better generalization capabilities,
which might not emerge naturally during training (Cobbe et al., 2020; Henaff
et al., 2022; Wang et al., 2022). For MDPs with non-stationary rewards, meta-
learning and continual learning study adaptive algorithms that can adapt to
moving objectives (Beck et al., 2023). Learning separate value functions for
non-stationary rewards has also been proposed (Burda et al., 2018). Schäfer et
al. (2021) proposed DeRL, which entirely decouples the training process of an
exploratory policy from the exploitation policy. While DeRL mitigates the
effect of the non-stationary intrinsic rewards in the exploitation policy, the
exploration policy still faces a hard optimization problem. Importantly, there
might not exist an optimal Markovian policy for a POMDP (Seyedsalehi et al.,
2023). Hence, RL algorithms can only achieve suboptimal performance in these
settings.

Many exploration bonuses are non-stationary by definition. In particular,
count-based methods are non-stationary since the state-visitation frequencies
change during training (Singh et al., 2010; Şimşek & Barto, 2006). We note
that this issue is also present in many of the popular deep exploration
methods that use an auxiliary model to compute the intrinsic rewards like ICM
(Pathak et al., 2017), RND (Burda et al., 2018), E3B (Henaff et al., 2022),
density models (Bellemare et al., 2016; Tang et al., 2017) and many others
(Lobel et al., 2023; Raileanu & Rocktäschel, 2020; Flet-Berliac et al., 2021;
Zhang et al., 2021b). In these cases, the non-stationarity is caused by the
weights of the auxiliary models also changing during training. In this work,
we argue that non-stationarity should not be implicit when an exploration
bonus is defined. For this reason, we introduce SOFE, which proposes an
intuitive modification to intrinsic objectives that eliminates their non-
stationarity and facilitates the optimization process.

##  3 Preliminaries

Reinforcement Learning (RL) uses MDPs to model the interactions between a
learning agent and an environment. An MDP is defined as a tuple
$\mathcal{M}=\left(\mathcal{S},\mathcal{A},\mathcal{R},\mathcal{T},\gamma\right)$
where $\mathcal{S}$ is the state space, $\mathcal{A}$ is the action-space,
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the
extrinsic reward function,
$\mathcal{T}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$ is
a transition function and $\gamma$ is the discount factor. The objective of
the agent is to learn a policy that maximizes the expected discounted sum of
rewards across all possible trajectories induced by the policy. If the MDP is
non-stationary, then there exists some unobserved environment state that
determines the dynamics of the MDP, hence inducing a partially observed MDP
(POMDP), which is also a tuple
$\mathcal{M^{\prime}}=\left(\mathcal{S},\mathcal{O},\mathcal{A},\mathcal{R},\mathcal{T},\gamma\right)$
where $\mathcal{O}$ is the observation space and the true states
$s\in\mathcal{S}$ are unobserved. In a POMDP, the transition and reward
functions might not be Markovian with respect to the observations, and
therefore, the policy training methods may not converge to an optimal policy.
To illustrate this, consider an MDP where the reward distribution is different
at odd and even time steps. If the states of the MDP are not augmented with an
odd/even component, the rewards appear to be non-stationary to an agent with a
Markovian policy. In this case, a Markovian policy will not be optimal over
all policies. The optimal policy will have to switch at odd/even time steps.
In this work, we extend the previous argument to intrinsic exploration
objectives in RL. In the following sections, we uncover the implicit non-
stationarity of several exploration objectives and propose a novel method to
resolve it.

###  3.1 Exploration Bonuses and Intrinsic Rewards

In hard-exploration problems, exploration is more successful if directed,
controlled, and efficient. Exploration bonuses provide a framework to decouple
the original task from the exploration one and define exploration as a
separate RL problem. In this framework, the extrinsic rewards provided by the
environment are aggregated with the intrinsic rewards (i.e. exploration
bonuses) to build an augmented learning target. By directing the agent’s
behavior towards custom exploration bonuses, this formulation induces
exploratory behaviors that are state-covering and are well-suited for long-
horizon problems.

Central to SOFE is the introduction of the parameters $\phi_{t}$ in the
formulation of exploration bonuses $\mathcal{B}(s_{t},a_{t}|\phi_{t})$, which
enables reasoning about the dynamics of the intrinsic reward distributions.
The parameters of the intrinsic reward distribution $\phi_{t}$ determine how
novelty is estimated and exploration is guided, and if they change over time
then $\mathcal{B}$ is non-stationary. In the following, we unify count-based
methods, pseudo-counts, and state-entropy maximization under the same
formulation, which includes $\phi_{t}$. In the next section, we present SOFE
as a solution to their non-stationarity.

####  3.1.1 Count-Based Methods

Count-based methods keep track of the agent’s frequencies of state visits to
derive an exploration bonus. Formally, the counts keep track of the visited
states until time $t$, and so $\mathcal{N}_{t}(s)$ is equal to the number of
times the state $s$ has been visited by the agent until time $t$. Two popular
intrinsic reward distributions derived from counts that exist in prior work
are:

$\mathcal{R}(s_{t},a_{t},s_{t+1}|\phi_{t})=\mathcal{B}(s_{t},a_{t},s_{t+1}|\phi_{t})=\frac{\beta}{\sqrt{\mathcal{N}_{t}(s_{t+1}|\phi_{t})}}$
(1)

where $\beta$ weights the importance of the count-based bonus, and:

$\mathcal{R}(s_{t},a_{t},s_{t+1}|\phi_{t})=\mathcal{B}(s_{t},a_{t},s_{t+1}|\phi_{t})=\big{\\{}\begin{array}[]{lr}1,&\text{if
}\mathcal{N}_{t}(s_{t+1}|\phi_{t})=0\\\ 0,&\text{else }\end{array}$ (2)

Note that the state-visitation frequencies $\mathcal{N}_{t}$ are the
sufficient statistics for $\phi_{t}$ and hence for the count-based rewards in
Equations 1 and 2. That is, the state-visitation frequencies are the only
dynamically changing component that induces non-stationarity in count-based
rewards.

Equation 1 (Strehl & Littman, 2008) produces a dense learning signal since
$\mathcal{B}(s_{t},a_{t},s_{t+1}|\phi_{t})\neq 0$ unless
$\mathcal{N}_{t}(s_{t+1})=\infty$ which is unrealistic in practice. Equation 2
(Henaff et al., 2023) defines a sparse distribution where the agent is only
rewarded the first time it sees each state, similar to the objective of the
travelling salesman problem. Throughout the paper, we refer to Equations 1 and
2 as $\sqrt{}$-reward and salesman reward.

####  3.1.2 Pseudo-Counts

To enable count-based exploration in high-dimensional spaces, the notion of
visitation counts has been generalized to that of pseudo-counts (Bellemare et
al., 2016). Prior work has estimated pseudo-counts through density models
(Bellemare et al., 2016), neural networks (Ostrovski et al., 2017), successor
representations (Machado et al., 2020), and samples from the Rademacher
distribution (Lobel et al., 2023). Recently, Henaff et al. (2022) proposed
elliptical bonuses (E3B) as a natural generalization of count-based methods.
An appealing property of E3B is that it models the complete set of state-
visitation frequencies over the state space, and not only for the most recent
state222Previous pseudo-count methods allowed the agent to query a density
model with a single state and obtain its pseudo-count. However, E3B maintains
a model of the state-visitation frequencies over the complete state space. The
latter is key for SOFE to obtain sufficient statistics of the E3B reward in
Equation 3. Concretely, the E3B algorithm produces a bonus:

$\begin{split}\mathcal{B}(s_{t},a_{t},s_{t+1}|\phi_{t})&=\psi_{t}(s_{t+1})^{T}C^{-1}_{t}\psi_{t}(s_{t+1})\\\
C_{t}&=\sum_{t=0}^{T}\psi_{t}(s_{t})\psi_{t}(s_{t})^{T}\end{split}$ (3)

where $\psi_{t}$ is an auxiliary model that produces low-dimensional
embeddings from high-dimensional observations. Since the ellipsoid is updated
after each transition, the exploration bonus is non-stationary. The matrix
$C_{t}$ defines an ellipsoid in the embedding space, which encodes the
distribution of observed embeddings in a given trajectory. Since $C_{t}$ is
the only moving component of Equation 3, it is a sufficient statistic to
characterize the non-stationarity of the reward distribution. Note that in an
MDP with finite state space, where $\psi$ is the one-hot encoding of the
states, the exploration bonus in Equation 3 becomes a count-based bonus
similar to Equation 1. Concretely, $C_{t-1}^{-1}$ becomes a diagonal matrix
with the inverse state-visitation frequencies for each state in the elements
of the diagonal (Henaff et al., 2022).

####  3.1.3 State-Entropy Maximization

State-entropy maximization is a widely used exploration objective that
consists of training policies to induce a uniform distribution over the state-
marginal visitation distribution (Lee et al., 2019). A canonical formulation
of this problem is presented in Berseth et al. (2019). Maximizing the state-
entropy objective corresponds to training policies to maximize the following
reward distribution (Berseth et al., 2019):

$\mathcal{R}(s_{t},a_{t},s_{t+1}|\phi_{t})=\mathcal{B}(s_{t},a_{t},s_{t+1}|\phi_{t})=-\log
p_{\phi_{t}}(s_{t+1})$ (4)

The parameters $\theta$ define the policy and $\phi$ are the parameters of a
generative model which estimates the state-marginal distribution
$d^{\pi_{\theta}}(s_{t})$. Note that the sufficient statistics of the
generative distribution are also sufficient statistics for the intrinsic
reward in Equation 4. Throughout the paper, we refer to this algorithm as
surprise maximization (S-Max) and use the Gaussian distribution to model
trajectories of states with $p_{\phi_{t}}$. Hence the sufficient statistics of
$p_{\phi_{t}}$ for the reward reward in Equation 4 are
$\phi_{t}=(\mu_{t}\cup\sigma_{t}^{2})$. We present the details of S-Max in
Section A.8.

##  4 Stationary Objectives For Exploration

In the following, we present a training framework for Stationary Objectives
for Exploration (SOFE). Any exploration bonus
$\mathcal{B}(s_{t},a_{t},s_{t+1}|\phi_{t})$ derived from dynamically changing
parameters will define a non-stationary reward function. Without any
modification, exploration bonuses define a POMDP:
$\mathcal{M}=\left(\mathcal{S},\mathcal{O},\mathcal{A},\mathcal{B},\mathcal{T},\gamma\right)$.
For simplicity, we have fully replaced the task-reward $\mathcal{R}$ with the
exploration bonus $\mathcal{B}$, and we consider that the only unobserved
components in the POMDP are the parameters of the reward distribution333This
assumption holds true if the agent has access to sufficient statistics of the
transition dynamics (e.g. grid environments), and makes SOFE transform a POMDP
into a fully observed MDP. Even when there are unobserved components of the
true states apart from the parameters of the intrinsic reward distributions,
we empirically show that SOFE mitigates the non-stationary optimization,
yielding performance gains.. Hence, we argue that the unobserved states $s\in
S$ satisfy $s_{t}=o_{t}\>\cup\>\phi_{t}$. Note that the transition function of
the POMDP is generally only Markovian if defined over the state space and not
over the observation space:
$\mathcal{T}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$.

Figure 1: SOFE enables agents to observe the sufficient statistics of the
intrinsic rewards and use them for decision-making.

The sufficient statistics for exploration bonuses are always available during
training as they are explicitly computed to produce the intrinsic rewards.
However, current RL methods do not allow the agents to observe them. Hence,
any method that aims to solve $\mathcal{M}$ faces optimizing a non-stationary
objective, which is difficult to optimize, as it can require non-Markovian
properties like memory, continual learning, and adaptation, and may only find
suboptimal policies. In this work, we argue that non-stationarity should not
be implicit in the formulation of an exploration objective. For this reason,
we propose SOFE, which augments the state space by defining an augmented MDP
$\hat{\mathcal{M}}=\left(\hat{\mathcal{S}},\mathcal{A},\mathcal{B},\mathcal{T},\gamma\right)$
where $\hat{\mathcal{S}}=\left\\{\mathcal{O}\>\cup\>\phi\right\\}$, with
$\mathcal{O}$ being the observations from $\mathcal{M}$. Note that we get rid
of the observation space $\mathcal{O}$ in the definition of
$\hat{\mathcal{M}}$ because by augmenting the original observations from
$\mathcal{M}$ with the sufficient statistics for $\mathcal{B}$ we effectively
define a fully observed MDP. This simple modification allows instantiating the
same exploration problem in a stationary and Markovian setting. That is the
optimal policies in $\hat{\mathcal{M}}$ are also optimal in $\mathcal{M}$.
This is true since the transition and reward functions are identical in
$\mathcal{M}$ and $\hat{\mathcal{M}}$. We note that the update rule for the
parameters $\phi_{t}$ must be Markovian, meaning that these can be updated
after every step without requiring information other than $s_{t}$ and
$s_{t+1}$. For example, counts only increment by one for the state that was
most recently visited: $\mathcal{N}_{t+1}(s)=\mathcal{N}_{t}(s)\>\forall
s\in\\{S-s_{j}\\}$, where $s_{j}=s_{t+1}$ and
$\mathcal{N}_{t+1}(s_{j})=\mathcal{N}_{t}(s_{j})+1$. The latter also applies
to E3B and S-Max, since the ellipsoid $C_{t}$ and parameters of the generative
model are updated incrementally with every new transition (see Equation 3 and
Section A.8). Given the sufficient statistics, the intrinsic reward
distributions in Equations 1,2, 3, 4 become fully Markovian, and hence are
invariant across time.

##  5 Experiments

SOFE is designed to improve the performance of exploration tasks. To evaluate
its efficacy, we study three questions: (1) How much does SOFE facilitate the
optimization of non-stationary exploration bonuses? (2) Does this increased
stationarity improve exploration for downstream tasks? (3) How well does SOFE
scale to image-based state inputs where approximations are needed to estimate
state-visitation frequencies?

Figure 2: We use 3 mazes and a large 3D map to evaluate both goal-reaching and
purely exploratory behaviors. Maze 1: a fully connected, hard-exploration
maze; Maze 2: a maze with open spaces and a goal; Maze 3: same as Maze 1 but
with 3 doors which an intelligent agent should use for more efficient
exploration; 3D map: a large map with continuous state and action spaces.

To answer each of these research questions, we run the experiments as follows.
(1) We use three different mazes without goals to investigate how SOFE
compares to vanilla count-based methods and S-Max in reward-free exploration.
Concretely, we evaluate whether SOFE allows for better optimization of purely
exploratory behaviors. We also use a large 3D environment with continuous
state and action spaces which introduces complex challenges as it requires
more than purely navigation skills.

Secondly (2), we use a 2D maze with a goal and sparse extrinsic reward
distribution. This is a hard-exploration task where the extrinsic reward is
only non-zero if the agent reaches the goal, which requires a sequence of 75
coordinated actions. We evaluate whether SOFE enables better optimization of
the joint objective of intrinsic and task rewards. Furthermore, we use the
DeepSea sparse-reward hard-exploration task from the DeepMind suite (Osband et
al., 2019) and show that SOFE achieves better performance than DeRL (Schäfer
et al., 2021) which attempts to stabilize intrinsic rewards by training
decoupled exploration and exploitation policies.

Thirdly (3), we apply SOFE on the E3B (Henaff et al., 2022) algorithm as
argued in Section 4 to demonstrate the effectiveness of the approach with an
imperfect representation of the state-visitation frequencies. We use the
MiniHack-MultiRoom-N6-v0 task, originally used for E3B in Henaff et al.
(2023), and the Procgen-Maze task (Cobbe et al., 2020). In both environments,
the task is to navigate to the goal location in a procedurally generated map
and the extrinsic reward is only non-zero if the agent reaches the goal. Both
environments return pixel observations. Minihack additionally returns natural
language observations. However, the Procgen-Maze task is more challenging
because each episode uses unique visual assets, requiring an additional level
of generalization, while in Minihack, different episodes only vary in the map
layout. Additionally, we include the Habitat environment (Szot et al., 2021)
to evaluate purely exploratory behaviors and show the results in Section A.2.

We provide the details of the network architectures, algorithm
hyperparameters, and environment specifications in Section A.4. Furthermore,
we provide an in-depth analysis of the behaviors learned by SOFE in Section
A.3, which uncovers valuable insights on how SOFE learns to drive exploration
more efficiently.

###  5.1 Reward-Free Exploration

In this section, we focus on the first research question and consider the
reward-free setting to evaluate purely exploratory behaviors. We use the 3
mazes in Figure 2 and measure map coverage, which correlates with exploration
in navigation environments. In Figure 3, we show how SOFE enables agents to
explore the mazes better than vanilla count-based methods. Even though we fix
the count-based rewards described in Equations 1 and 2, SOFE generally enables
RL agents to better optimize them, achieving higher state coverage. Section
A.10 contains the results across all algorithms and exploration modalities.

Figure 3: Episodic state-visitation for A2C agents during training. The first
row represents SOFE, which uses both the count-based rewards and state
augmentation (+ C. + Aug.), and the second row represents training with the
count-based rewards only (+ C.). Although optimizing for the same reward
distribution, our method achieves better exploration performance.

We also run experiments in a large 3D environment from the Godot RL repository
(Beeching et al., 2021a), to evaluate SOFE’s ability to scale to continuous
state and action spaces. This environment contains challenging dynamics that
require exploratory agents to master a variety of skills, from avoiding lava
and water to using jump pads efficiently (Alonso et al., 2020; Beeching et
al., 2021b). Figure 4 shows that SOFE also scales to these more complex
settings, enabling SAC (Haarnoja et al., 2018) agents to achieve higher map
coverage across different exploration modalities.

Figure 4: Map coverage achieved by SAC agents in a complex 3D map. Blue curves
represent agents that use count-based rewards (+ C.); Red curves represent
SOFE, which uses both count-based rewards and the state augmentations from
SOFE (+ C. + Aug.). Even though we use the same learning objective, SOFE
facilitates its optimization and achieves better exploration. Shaded areas
represent one standard deviation. Results are averaged from 6 seeds.

Additionally, we show that SOFE stabilizes the state-entropy maximization
objective. Figure 5 shows the episodic map coverage achieved in Maze 2 by the
vanilla S-Max algorithm compared to the augmented S-Max with SOFE. These
results provide further evidence that SOFE is a general framework that tackles
the non-stationarity of exploration objectives and provides orthogonal gains
across objectives of different natures.

Figure 5:  Episodic state coverage achieved by S-Max (blue) and SOFE S-Max
(red) in Maze 2. When augmented with SOFE, agents better optimize for the
state-entropy maximization objective. Shaded areas represent one standard
deviation. Results are averaged from 6 seeds.

###  5.2 Exploration for Sparse Rewards

In the previous section, we showed that SOFE enables RL agents to better
explore the state space. In this section, we evaluate whether SOFE can achieve
better performance on hard-exploration tasks.

We evaluate count-based methods and SOFE in Maze 2 in Figure 2. For each of
the RL algorithms, we compare training with the sparse extrinsic reward only
and training with the extrinsic and intrinsic rewards with and without SOFE.
Figure 6 shows that SOFE significantly improves the performance of RL agents
in this hard-exploration task. Our results confirm that extrinsic rewards are
not enough to solve such hard-exploration tasks and show that SOFE is
significantly more effective than vanilla count-based methods, achieving the
highest returns across multiple RL algorithms. PPO (Schulman et al., 2017),
PPO+LSTM (Cobbe et al., 2020), and A2C (Mnih et al., 2016) achieve near-
optimal goal-reaching performance only when using SOFE. Importantly, policies
equipped with LSTMs have enough capacity to model the non-stationary intrinsic
rewards (Ni et al., 2021) and could learn to count implicitly (Suzgun et al.,
2018). However, our results show that SOFE further improves the performance of
recurrent policies when optimizing for non-stationary intrinsic rewards.

Figure 6: Interquantile mean (IQM) of episode extrinsic rewards for episodic
(left) and global (right) exploration across multiple algorithms. Green bars
represent training with the sparse reward; blue bars represent additionally
using count-based rewards (+ C.); and red bars represent additionally using
count-based rewards and SOFE (+ C. + Aug.). We compute confidence intervals
using stratified bootstrapping with 6 seeds. SOFE generally achieves
significantly higher extrinsic rewards across RL algorithms and exploration
modalities. Without exploration bonuses, agents fail to obtain a non-zero
extrinsic reward during training.

Additionally, we compare SOFE to DeRL (Schäfer et al., 2021) in the DeepSea
environment and show the results in Table 1. DeRL entirely decouples the
training process of an exploratory policy from the exploitation policy to
stabilize the optimization of the exploration objective. SOFE is degrees of
magnitude less complex than DeRL as it only requires training an additional
feature extractor. Still, SOFE achieves better results in the harder
variations of the DeepSea environment.

Algorithm | DeepSea 10 | DeepSea 14 | DeepSea 20 | DeepSea 24 | DeepSea 30
---|---|---|---|---|---
DeRL-A2C | $0.98\pm 0.10$ | $0.65\pm 0.23$ | $0.42\pm 0.16$ | $0.07\pm 0.10$ | $0.09\pm 0.08$
DeRL-PPO | $0.61\pm 0.20$ | $0.92\pm 0.18$ | $-0.01\pm 0.01$ | $0.63\pm 0.27$ | $-0.01\pm 0.01$
DeRL-DQN | $\textbf{0.98}\pm\textbf{0.09}$ | $\textbf{0.95}\pm\textbf{0.17}$ | $0.40\pm 0.08$ | $0.53\pm 0.27$ | $0.10\pm 0.10$
SOFE-A2C | $0.94\pm 0.19$ | $0.45\pm 0.31$ | $0.11\pm 0.25$ | $0.08\pm 0.14$ | $0.04\pm 0.09$
SOFE-PPO | $0.77\pm 0.29$ | $0.67\pm 0.33$ | $0.13\pm 0.09$ | $0.07\pm 0.15$ | $0.09\pm 0.23$
SOFE-DQN | $0.97\pm 0.29$ | $0.78\pm 0.21$ | $\textbf{0.70}\pm\textbf{0.28}$ | $\textbf{0.65}\pm\textbf{0.26}$ | $\textbf{0.42}\pm\textbf{0.33}$
Table 1: Average performance of DeRL and SOFE with one standard deviation over
100,000 evaluation episodes in the DeepSea environment. SOFE achieves better
performance in the harder exploration variations of the DeepSea environment
while consisting of a less complex method. Additional details on the
hyperparameters used for SOFE and DeRL are presented in Section A.4.

###  5.3 Exploration in High-dimensional Environments

In this section, we evaluate SOFE and E3B (Henaff et al., 2022), the state-of-
the-art exploration algorithm for high-dimensional contextual MDPs. E3B
tackles the challenging problem of estimating the true state-visitation
frequencies from pixel observations. As argued in Section 5.3, the ellipsoid
is the only moving component of the E3B objective. Hence, we evaluate whether
including either the diagonal or the full ellipsoid in the state enables
better exploration. We optimize the E3B objective with IMPALA (Espeholt et
al., 2018) as proposed in Henaff et al. (2022). Section A.4 contains the
details of the policy architecture.

Figure 7 shows that SOFE also improves the performance of pseudo-count-based
methods, providing empirical evidence that reducing the non-stationarity of a
reward distribution enables better optimization even in high-dimensional
environments. In Section A.2, we include experiments with E3B and SOFE in the
reward-free setting using the Habitat simulator. These show that SOFE improves
the sample efficiency of E3B.

Figure 7: Interquantile mean (IQM) of the episode extrinsic rewards in
Minihack (left) and Procgen (right). The vanilla E3B algorithm can solve
MiniHack as shown in Henaff et al. (2022). However, we find that E3B can only
consistently reach the goals in Procgen-Maze when using SOFE. As shown in
Henaff et al. (2022), ICM (Pathak et al., 2017) and RND (Burda et al., 2018)
fail to provide a good exploration signal in procedurally generated
environments. Results are averaged from 6 seeds.

##  6 Conclusion

We identify that exploration bonuses can be non-stationary by definition,
which can complicate their optimization, resulting in suboptimal policies. To
address this issue, we have introduced a novel framework that creates
stationary objectives for exploration (SOFE). SOFE is based on capturing
sufficient statistics of the intrinsic reward distribution and augmenting the
MDP’s state representation with these statistics. This augmentation transforms
the non-stationary rewards into stationary rewards, simplifying the
optimization of the agent’s objective. We have identified sufficient
statistics of count-based methods, the state-entropy maximization objective,
and E3B. Our experiments provide compelling evidence of the efficacy of SOFE
across various environments, tasks, and reinforcement learning algorithms,
even improving the performance of the state-of-the-art exploration algorithm
in procedurally generated environments. Using augmented representations, SOFE
significantly improves exploration behaviors, particularly in challenging
tasks with sparse rewards and across multiple exploration modalities.
Additionally, SOFE extends to large continuous state and action spaces,
showcasing its versatility.

## References

  * Alegre et al. (2021) Lucas N Alegre, Ana LC Bazzan, and Bruno C da Silva.  Quantifying the impact of non-stationarity in reinforcement learning-based traffic signal control.  _PeerJ Computer Science_ , 7:e575, 2021.
  * Alonso et al. (2020) Eloi Alonso, Maxim Peter, David Goumard, and Joshua Romoff.  Deep reinforcement learning for navigation in aaa video games.  _arXiv preprint arXiv:2011.04764_ , 2020.
  * Auer (2002) Peter Auer.  Using confidence bounds for exploitation-exploration trade-offs.  _Journal of Machine Learning Research_ , 3(Nov):397–422, 2002.
  * Bamford (2021) Christopher Bamford.  Griddly: A platform for ai research in games.  _Software Impacts_ , 8:100066, 2021.
  * Beck et al. (2023) Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa Zintgraf, Chelsea Finn, and Shimon Whiteson.  A survey of meta-reinforcement learning.  _arXiv preprint arXiv:2301.08028_ , 2023.
  * Beeching et al. (2021a) Edward Beeching, Jilles Dibangoye, Olivier Simonin, and Christian Wolf.  Godot reinforcement learning agents.  _arXiv preprint arXiv:2112.03636._ , 2021a.
  * Beeching et al. (2021b) Edward Beeching, Maxim Peter, Philippe Marcotte, Jilles Debangoye, Olivier Simonin, Joshua Romoff, and Christian Wolf.  Graph augmented deep reinforcement learning in the gamerland3d environment.  _arXiv preprint arXiv:2112.11731_ , 2021b.
  * Bellemare et al. (2016) Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos.  Unifying count-based exploration and intrinsic motivation.  _Advances in neural information processing systems_ , 29, 2016.
  * Berseth et al. (2019) Glen Berseth, Daniel Geng, Coline Devin, Nicholas Rhinehart, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine.  Smirl: Surprise minimizing reinforcement learning in unstable environments.  _arXiv preprint arXiv:1912.05510_ , 2019.
  * Brockman et al. (2016) Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym.  _arXiv preprint arXiv:1606.01540_ , 2016.
  * Burda et al. (2018) Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov.  Exploration by random network distillation.  _arXiv preprint arXiv:1810.12894_ , 2018.
  * Cheung et al. (2020) Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu.  Reinforcement learning for non-stationary markov decision processes: The blessing of (more) optimism.  In _International Conference on Machine Learning_ , pp. 1843–1854. PMLR, 2020.
  * Cobbe et al. (2020) Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman.  Leveraging procedural generation to benchmark reinforcement learning.  In _International conference on machine learning_ , pp. 2048–2056. PMLR, 2020.
  * Espeholt et al. (2018) Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al.  Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.  In _International conference on machine learning_ , pp. 1407–1416. PMLR, 2018.
  * Eysenbach et al. (2018) Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine.  Diversity is all you need: Learning skills without a reward function.  _arXiv preprint arXiv:1802.06070_ , 2018.
  * Flet-Berliac et al. (2021) Yannis Flet-Berliac, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist.  Adversarially guided actor-critic.  _arXiv preprint arXiv:2102.04376_ , 2021.
  * Fujimoto et al. (2018) Scott Fujimoto, Herke Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  In _International conference on machine learning_ , pp. 1587–1596. PMLR, 2018.
  * Gehring et al. (2021) Jonas Gehring, Gabriel Synnaeve, Andreas Krause, and Nicolas Usunier.  Hierarchical skills for efficient exploration.(10 2021), 2021.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International conference on machine learning_ , pp. 1861–1870. PMLR, 2018.
  * Henaff et al. (2022) Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rocktäschel.  Exploration via elliptical episodic bonuses.  _Advances in Neural Information Processing Systems_ , 35:37631–37646, 2022.
  * Henaff et al. (2023) Mikael Henaff, Minqi Jiang, and Roberta Raileanu.  A study of global and episodic bonuses for exploration in contextual mdps.  _arXiv preprint arXiv:2306.03236_ , 2023.
  * Kolter & Ng (2009) J Zico Kolter and Andrew Y Ng.  Near-bayesian exploration in polynomial time.  In _Proceedings of the 26th annual international conference on machine learning_ , pp. 513–520, 2009.
  * Laskin et al. (2021) Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel.  Urlb: Unsupervised reinforcement learning benchmark.  _arXiv preprint arXiv:2110.15191_ , 2021.
  * Lecarpentier & Rachelson (2019) Erwan Lecarpentier and Emmanuel Rachelson.  Non-stationary markov decision processes, a worst-case approach using model-based reinforcement learning.  _Advances in neural information processing systems_ , 32, 2019.
  * Lee et al. (2019) Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov.  Efficient exploration via state marginal matching.  _arXiv preprint arXiv:1906.05274_ , 2019.
  * Levine et al. (2020) Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Lillicrap et al. (2015) Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_ , 2015.
  * Lobel et al. (2023) Sam Lobel, Akhil Bagaria, and George Konidaris.  Flipping coins to estimate pseudocounts for exploration in reinforcement learning.  _arXiv preprint arXiv:2306.03186_ , 2023.
  * Machado et al. (2020) Marlos C Machado, Marc G Bellemare, and Michael Bowling.  Count-based exploration with the successor representation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 5125–5133, 2020.
  * Mnih et al. (2016) Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In _International conference on machine learning_ , pp. 1928–1937. PMLR, 2016.
  * Ni et al. (2021) Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov.  Recurrent model-free rl can be a strong baseline for many pomdps.  _arXiv preprint arXiv:2110.05038_ , 2021.
  * Osband et al. (2019) Ian Osband, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari, Satinder Singh, et al.  Behaviour suite for reinforcement learning.  _arXiv preprint arXiv:1908.03568_ , 2019.
  * Ostrovski et al. (2017) Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos.  Count-based exploration with neural density models.  In _International conference on machine learning_ , pp. 2721–2730. PMLR, 2017.
  * Papoudakis et al. (2019) Georgios Papoudakis, Filippos Christianos, Arrasy Rahman, and Stefano V Albrecht.  Dealing with non-stationarity in multi-agent deep reinforcement learning.  _arXiv preprint arXiv:1906.04737_ , 2019.
  * Pathak et al. (2017) Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell.  Curiosity-driven exploration by self-supervised prediction.  In _International conference on machine learning_ , pp. 2778–2787. PMLR, 2017.
  * Raffin et al. (2021) Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann.  Stable-baselines3: Reliable reinforcement learning implementations.  _The Journal of Machine Learning Research_ , 22(1):12348–12355, 2021.
  * Raileanu & Rocktäschel (2020) Roberta Raileanu and Tim Rocktäschel.  Ride: Rewarding impact-driven exploration for procedurally-generated environments.  _arXiv preprint arXiv:2002.12292_ , 2020.
  * Ramakrishnan et al. (2021) Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra.  Habitat-matterport 3d dataset (HM3d): 1000 large-scale 3d environments for embodied AI.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2021.  URL https://arxiv.org/abs/2109.08238.
  * Samvelyan et al. (2021) Mikayel Samvelyan, Robert Kirk, Vitaly Kurin, Jack Parker-Holder, Minqi Jiang, Eric Hambro, Fabio Petroni, Heinrich Küttler, Edward Grefenstette, and Tim Rocktäschel.  Minihack the planet: A sandbox for open-ended reinforcement learning research.  _arXiv preprint arXiv:2109.13202_ , 2021.
  * Savva et al. (2019) Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruv Batra.  Habitat: A Platform for Embodied AI Research.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019.
  * Schäfer et al. (2021) Lukas Schäfer, Filippos Christianos, Josiah P Hanna, and Stefano V Albrecht.  Decoupled reinforcement learning to stabilise intrinsically-motivated exploration.  _arXiv preprint arXiv:2107.08966_ , 2021.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Seo et al. (2021) Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee.  State entropy maximization with random encoders for efficient exploration.  In _International Conference on Machine Learning_ , pp. 9443–9454. PMLR, 2021.
  * Seyedsalehi et al. (2023) Erfan Seyedsalehi, Nima Akbarzadeh, Amit Sinha, and Aditya Mahajan.  Approximate information state based convergence analysis of recurrent q-learning.  _arXiv preprint arXiv:2306.05991_ , 2023.
  * Şimşek & Barto (2006) Özgür Şimşek and Andrew G Barto.  An intrinsic reward mechanism for efficient exploration.  In _Proceedings of the 23rd international conference on Machine learning_ , pp. 833–840, 2006.
  * Singh et al. (2010) Satinder Singh, Richard L Lewis, Andrew G Barto, and Jonathan Sorg.  Intrinsically motivated reinforcement learning: An evolutionary perspective.  _IEEE Transactions on Autonomous Mental Development_ , 2(2):70–82, 2010.
  * Strehl & Littman (2008) Alexander L Strehl and Michael L Littman.  An analysis of model-based interval estimation for markov decision processes.  _Journal of Computer and System Sciences_ , 74(8):1309–1331, 2008.
  * Sutton & Barto (2018) Richard S Sutton and Andrew G Barto.  _Reinforcement learning: An introduction_.  MIT press, 2018.
  * Suzgun et al. (2018) Mirac Suzgun, Yonatan Belinkov, and Stuart M Shieber.  On evaluating the generalization of lstm models in formal languages.  _arXiv preprint arXiv:1811.01001_ , 2018.
  * Szot et al. (2021) Andrew Szot, Alex Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, Noah Maestre, Mustafa Mukadam, Devendra Chaplot, Oleksandr Maksymets, Aaron Gokaslan, Vladimir Vondrus, Sameer Dharur, Franziska Meier, Wojciech Galuba, Angel Chang, Zsolt Kira, Vladlen Koltun, Jitendra Malik, Manolis Savva, and Dhruv Batra.  Habitat 2.0: Training home assistants to rearrange their habitat.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2021.
  * Tang et al. (2017) Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel.  # exploration: A study of count-based exploration for deep reinforcement learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. (2022) Kaixin Wang, Kuangqi Zhou, Bingyi Kang, Jiashi Feng, and YAN Shuicheng.  Revisiting intrinsic reward for exploration in procedurally generated environments.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Whitney et al. (2021) William F Whitney, Michael Bloesch, Jost Tobias Springenberg, Abbas Abdolmaleki, Kyunghyun Cho, and Martin Riedmiller.  Decoupled exploration and exploitation policies for sample-efficient reinforcement learning.  _arXiv preprint arXiv:2101.09458_ , 2021.
  * Yarats et al. (2021) Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto.  Reinforcement learning with prototypical representations.  In _International Conference on Machine Learning_ , pp. 11920–11931. PMLR, 2021.
  * Zhang et al. (2021a) Kaiqing Zhang, Zhuoran Yang, and Tamer Başar.  Multi-agent reinforcement learning: A selective overview of theories and algorithms.  _Handbook of reinforcement learning and control_ , pp. 321–384, 2021a.
  * Zhang et al. (2021b) Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian.  Noveld: A simple yet effective exploration criterion.  _Advances in Neural Information Processing Systems_ , 34:25217–25230, 2021b.
  * Zisselman et al. (2023) Ev Zisselman, Itai Lavie, Daniel Soudry, and Aviv Tamar.  Explore to generalize in zero-shot rl.  _arXiv preprint arXiv:2306.03072_ , 2023.

##  Appendix A Appendix

###  A.1 Stationarity of the count-based rewards

In this Section, we show that the state-visitation frequencies must be
observed by the agent in order to make the count-based objective fully
Markovian and stationary.

If the expectation of the count-based bonus is constant, then the bonus is
stationary. Let $\mathcal{N}_{t}$ be the table of state-visitation frequencies
at time step $t$, and let $\mathcal{N}_{t}(s_{t})$ be the count of visits to
state $s_{t}$ at time $t$. The count-based reward is given by
$\mathcal{B}(s_{t})=\frac{1}{\mathcal{N}(s_{t})}$. Without loss of generality,
we have defined $\mathcal{B}$ as a function of a single state as the count-
based bonus $\mathcal{B}(s_{t},a_{t},s_{t+1})=\frac{1}{\mathcal{N}(s_{t+1})}$
only depends on the next state. The proof holds for any monotonically
decreasing function of the counts.

The expectation of $\mathcal{B}(s_{t})$ is given by:

$\displaystyle E[\mathcal{B}(s_{t})]$
$\displaystyle=\sum_{s}P(s_{t}=s)\cdot\frac{1}{\mathcal{N}_{t}(s_{t})}$
$\displaystyle=\frac{1}{\mathcal{N}_{t}(s_{t})}\cdot
P(s_{t}=s_{t})+\sum_{s\neq s_{t}}P(s_{t}=s)\cdot\frac{1}{\mathcal{N}_{t}(s)}$

Importantly, the counts provide information about the state visitation
distribution of the policy. Assuming that the policy visitation distribution
$P(s_{t}=s)$ is proportional to the state visit counts ($P(s_{t}=s)\propto
N_{t}(s)$), we can write:

$\displaystyle P(s_{t}=s)$
$\displaystyle=\frac{\mathcal{N}_{t}(s)}{\sum_{s^{\prime}}\mathcal{N}_{t}(s^{\prime})}$
$\displaystyle E[\mathcal{B}(s_{t})]$
$\displaystyle=\frac{1}{\mathcal{N}_{t}(s_{t})}\cdot\frac{\mathcal{N}_{t}(s_{t})}{\sum_{s^{\prime}}\mathcal{N}_{t}(s^{\prime})}+\sum_{s\neq
s_{t}}\frac{\mathcal{N}_{t}(s)}{\sum_{s^{\prime}}\mathcal{N}_{t}(s^{\prime})}\cdot\frac{1}{\mathcal{N}_{t}(s)}$

Simplifying the expression, we get:

$\displaystyle E[\mathcal{B}(s_{t})]$
$\displaystyle=\frac{1}{\sum_{s^{\prime}}\mathcal{N}_{t}(s^{\prime})}+\sum_{s\neq
s_{t}}\frac{1}{\sum_{s^{\prime}}\mathcal{N}_{t}(s^{\prime})}$

The terms in the summation are constant across time because the summation is
over all states except $s_{t}$, and the denominator
$\sum_{s^{\prime}}\mathcal{N}_{t}(s^{\prime})$ is based on the count of visits
to each state, which the agent can observe with SOFE.

Additionally, the update rule of the counts is also Markovian, as for each
transition $(s_{t},s_{t+1})$ we have:
$\mathcal{N}_{t+1}(s)=\mathcal{N}_{t}(s)\>\forall s\in\\{S-s_{j}\\}$, where
$s_{j}=s_{t+1}$ and $\mathcal{N}_{t+1}(s_{j})=\mathcal{N}_{t}(s_{j})+1$.

###  A.2 Reward-Free exploration with SOFE and E3B

Figure 8: Map coverage on a held-out set of 100 3D scenes of the HM3D dataset.
The E3B agents trained using SOFE explore the new scenes better.

As in Section 5.1, we evaluate if SOFE can enable better optimization of the
non-stationary exploration bonus, in this case for E3B. We consider the
reward-free setting for purely exploratory behaviors. For this reason, we use
the Habitat simulator (Savva et al., 2019; Szot et al., 2021) and the HM3D
dataset (Ramakrishnan et al., 2021), which contains 1,000 different scenes of
photorealistic apartments for 3D navigation. We train E3B and our proposed
augmented versions for 10M environment steps and measure their map coverage in
a set of 100 held-out scenes. We optimize the E3B exploration bonus with PPO
(Schulman et al., 2017) which requires 31 hours in a machine with a single
GPU. We show the results in Figure 8. In Figure 9 we show the learning curves
corresponding to the results presented in Section 5.3.

Figure 9: E3B (blue) outperforms the other deep exploration baselines in the
hard-exploration and partially-observable tasks of MiniHack and Procgen-Maze.
With SOFE (Red and Green), we further increase the performance of E3B.

###  A.3 Analysis of the behaviours learned by SOFE

By using SOFE on count-based methods, RL agents extract features from the
state-visitation frequencies and use them for decision-making. To better
understand how the agents use the augmented information, we artificially
create an object $\mathcal{N}_{0}$ with
$\mathcal{N}_{0}(s_{i})>0\>\forall_{i}\in\\{\mathcal{S}-s_{j}\\}$ and
$\mathcal{N}_{0}(s_{j})=0$. Intuitively, we communicate to the agent that all
states in the state space but $s_{j}$ have already been visited through the
state-visitation frequencies. We evaluate PPO agents pre-trained on reward-
free episodic exploration and show the results in Figure 10. Results show that
augmented agents efficiently direct their exploration towards the unvisited
states, self-identifying these as goals. This reveals how the agents leverage
the augmented information for more efficient exploration.

Figure 10: Analysis of how an RL agent uses SOFE for better exploration.
Yellow boxes show the agent’s starting position. Red boxes show the goals’
positions, which are the only positions for which we set
$\mathcal{N}_{0}(s_{j})=0$, and green traces show the agent’s trajectory.
Augmented agents pre-trained on episodic exploration efficiently direct their
exploration toward the unvisited states.

###  A.4 Training Details

####  A.4.1 Network Architecture

We use Stable-Baselines3 (Raffin et al., 2021) to run our experiments in the
mazes, Godot maps, and DeepSea. For DQN, PPO, A2C, and SAC we use the same CNN
to extract features from the observation. The CNN contains 3 convolutional
layers with kernel size of $(3\times 3)$, stride of 2, padding of 1, and 64
channels. The convolutional layers are followed by a fully connected layer
that produces observation embeddings of dimension 512. For the augmented
agents, we use an additional CNN with the same configuration to extract
features from $\phi_{t}$. The augmented agents concatenate the representations
from the observation and the parameters $\phi_{t}$ and feed these to the
policy for decision-making, while the vanilla methods (e.g. counts, S-Max)
only extract features from the observations. We show the CNN architecture in
Figure A.4.1.

For Minihack and Procgen, we use the official E3B codebase, which contains
baselines for ICM and RND, and uses IMPALA to optimize the exploration
bonuses. We use the same policy architecture as in Henaff et al. (2022), which
contains an LSTM. We ran the experiments in Minihack and Procgen for 100M
steps. For the augmented agents, we design a CNN that contains 5 convolutional
layers with a kernel size of $(3\times 3)$ and stride and padding of 1, batch
normalization layers after every convolutional layer, max-pooling layers with
a kernel size of $(2\times 2)$ and stride of 1, followed by a fully-connected
layer that produces embeddings of dimension 1024. This architecture allows to
extract features from the 512x512 ellipsoids, which are later passed together
with the observation features to the policy for decision-making. We show the
CNN architecture in Figure LABEL:fig:e3b_cnn_arch.

###  A.5 Algorithm Hyperparameters

####  A.5.1 DQN

Hyperparameter | Value
---|---
Num. Envs | 16
Learning Rate | 0.0001
Buffer Size | 1000000
Learning Starts | 50000
Batch Size | 32
Tau | 1.0
Gamma | 0.99
Train Frequency | 4
Gradient Steps | 4
Target Update Interval | 10000
Exploration Fraction | 0.1
Exploration Initial Epsilon | 1.0
Exploration Final Epsilon | 0.05
Max Grad Norm | 10
Table 2: Hyperparameters for the DQN Implementation

####  A.5.2 PPO

Hyperparameter | Value
---|---
Num. Envs | 16
Learning Rate | 0.0003
N Steps | 2048
Batch Size | 64
N Epochs | 10
Gamma | 0.99
GAE Lambda | 0.95
Clip Range | 0.2
Normalize Advantage | True
Entropy Coefficient | 0.0
Value Function Coefficient | 0.5
Max Grad Norm | 0.5
Table 3: Hyperparameters for the PPO Implementation

###  A.6 A2C

Hyperparameter | Value
---|---
Num. Envs | 16
Learning Rate | 0.0007
N Steps | 5
Gamma | 0.99
GAE Lambda | 1.0
Entropy Coefficient | 0.0
Value Function Coefficient | 0.5
Max Grad Norm | 0.5
RMS Prop Epsilon | $1\times 10^{-5}$
Use RMS Prop | True
Normalize Advantage | False
Table 4: Hyperparameters for the A2C Implementation

###  A.7 Environment Details

####  A.7.1 Mazes

We designed the mazes in Figure 2 with Griddly (Bamford, 2021). The 3 mazes
are of size 32x32. The agents observe entity maps: matrices of size
$(map\\_size,map\\_size)$ of entity id’s (e.g. 0 for the floor, 1 for the
wall, and 2 for the agent). The action space is discrete with the four-
movement actions (i.e. up, right, down, left). In the following, we now show
an example of the observation space of a 5x5 variation of the mazes we use
throughout the paper following the OpenAI Gym interface (Brockman et al.,
2016).

⬇

obs_shape = env.observation_space.shape

# obs_shape == (3,5,5)

obs, reward, done, info = env.step( ... )

# obs = [

[ # avatar in these locations

[0,0,0,0,0],

[0,1,0,0,0],

[0,0,0,0,0],

[0,0,0,0,0],

[0,0,0,0,0]

],

[ # wall in these locations

[1,1,1,1,1],

[1,0,0,0,1],

[1,0,0,0,1],

[1,0,0,0,1],

[1,1,1,1,1]

],

[ # goal in these locations

[0,0,0,0,0],

[0,0,0,0,0],

[0,0,0,0,0],

[0,0,0,1,0],

[0,0,0,0,0]

]

]

####  A.7.2 DeepSea

Figure 11: The DeepSea environment.

The DeepSea environment is taken from (Osband et al., 2019) and has been used
to evaluate the performance of intrinsic exploration objectives (Schäfer et
al., 2021). DeepSea represents a hard-exploration task in a $N\times N$ grid
where the agent starts in the top left and has to reach a goal in the bottom
right location. At each timestep, the agent moves one row down and can choose
whether to descend to the left or right. The agent observes the 2D one-hot
encoding of the grid and receives a small negative reward of $\frac{-0.01}{N}$
for moving right and 0 rewards for moving left. Additionally, the agent
receives a reward of +1 for reaching the goal and the episode ends after $N$
timesteps. Hence, it is very hard for an agent trained on extrinsic rewards
only to solve the credit assignment problem and realize that going right is
the optimal action. The episodes last exactly $N$ steps, and the complexity of
the task can be incremented by increasing $N$.

####  A.7.3 Godot Map

We use the Godot game engine to design the 3D world used in Section 5.1, which
we open-source together with the code. We show a global view of the map in
Figure 12. The map is of size 120x120 and has continuous state and action
spaces. To apply count-based methods we discretize the map in bins of size 5,
resulting in a 24x24 object $\mathcal{N}_{t}$. The observations fed to the
agent are the result of shooting several raycasts from the agent’s
perspective. The observations also contain global features like the agent’s
current velocity, rotation, and position. The action space contains 3
continuous dimensions that control the velocity, rotation, and jumping
actions.

Figure 12: Global view of the 3D Godot map.

####  A.7.4 Minihack Multiroom

We use the Multiroom-N6 task from the Minihack suite (Samvelyan et al., 2021)
to evaluate the performance of E3B and our proposed augmentation, as
originally used in Henaff et al. (2022). The environment provides pixel and
natural language observations and generates procedurally generated maps at
each episode. The rewards are only non-zero when the agent finds the goal
location in a maze that contains 6 rooms. We use the same policy architecture
described in Section C.1.1 in Henaff et al. (2022).

Figure 13: An observation from MiniHack.

####  A.7.5 Procgen Maze

We use the Procgen-Maze task from the Procgen benchmark (Cobbe et al., 2020)
to evaluate the performance of E3B and our proposed augmentation. We use the
memory distribution of mazes. The mazes are procedurally generated at each
episode, have different sizes, and use different visual assets. Procgen-Maze
provides pixel observations and the rewards are only non-zero if the agent
finds the goal location.

Figure 14: Procedurally generated mazes from the Procgen-Maze environment.

###  A.8 State-entropy Maximization

In this section, we provide the pseudo-code for the surprise-maximization
algorithm presented in Section 3.1.3. Note that the update rule for the
sufficient statistics of the generative model $p_{\phi_{t}}$ is Markovian as
it is updated at each timestep with the new information from the next state.
As mentioned in the paper, we use a Gaussian distribution to model
$p_{\phi_{t}}$, and hence when using SOFE, we pass its mean and standard
deviation to the RL agents.

Algorithm 1 Surprise Maximization

1:while not converged do

2: $\beta\leftarrow\\{\\}$ $\triangleright$ Initialize replay buffer

3: for $episode=0,\dots,\text{M}$ do

4: $s_{0}\leftarrow p(s_{0});\tau_{0}\leftarrow\\{s_{0}\\}$

5: for $t=0,\dots,\text{T}$ do

6: $a_{t}\sim\pi_{\theta}(a_{t}|s_{t})$ $\triangleright$ Get action

7: $s_{t+1}\sim T(s_{t+1}|s_{t},a_{t})$ $\triangleright$ Step dynamics

8: $r_{t}\leftarrow-\log p_{\phi_{t}}(s_{t+1})$ $\triangleright$ Compute
intrinsic reward

9: $\tau_{t+1}\leftarrow\tau_{t}\cup\\{s_{t+1}\\}$ $\triangleright$ Update
trajectory

10: $\phi_{t+1}\leftarrow\mathcal{U}(\tau_{t+1})$ $\triangleright$ Update mean
and variance

11: $\beta\leftarrow\beta\cup\\{(s_{t},a_{t},r_{t},s_{t+1})\\}$

12: end for

13: end for

14: $\theta\leftarrow RL(\theta,\beta)$ $\triangleright$ Update policy

15:end while

###  A.9 Exploration for Sparse Rewards

In this section, we show the complete set of results for Section 5.2. The
results include confidence intervals and learning curves for DQN, A2C, PPO,
and PPO-LSTM for the task of reaching the goal in Maze 2 in Figure 2. We also
include the partially-observable setting, where the agent does not observe the
full maze but 5x5 agent-centred observation.

###  A.10 Reward-free Exploration

In this section, we show the complete set of results for Section 5.1. The
results include learning curves for DQN, A2C, PPO and PPO-LSTM measuring the
map coverage achieved by these algorithms in the 3 mazes in Figure 2.

####  A.10.1 DQN

####  A.10.2 A2C

####  A.10.3 PPO

####  A.10.4 PPO-LSTM