# Epitaxial growth and characterization of (001) [NiFe/M]20 (M = Cu, CuPt and
Pt) superlattices

Movaffaq Kateb<EMAIL_ADDRESS>Science Institute, University of
Iceland, Dunhaga 3, IS-107 Reykjavik, Iceland  Condensed Matter and Materials
Theory Division, Department of Physics, Chalmers University of Technology,
SE-412 96 Gothenburg, Sweden  Jon Tomas Gudmundsson<EMAIL_ADDRESS>Science
Institute, University of Iceland, Dunhaga 3, IS-107 Reykjavik, Iceland  Space
and Plasma Physics, School of Electrical Engineering and Computer Science, KTH
Royal Institute of Technology, SE-100 44, Stockholm, Sweden  Snorri Ingvarsson
<EMAIL_ADDRESS>Science Institute, University of Iceland, Dunhaga 3, IS-107
Reykjavik, Iceland

###### Abstract

We present optimization of [(15 Å) Ni80Fe20/(5 Å) M]20 single crystal
multilayers on (001) MgO, with M being Cu, Cu50Pt50 and Pt. These
superlattices were characterized by high resolution X-ray reflectivity (XRR)
and diffraction (XRD) as well as polar mapping of important crystal planes. It
is shown that cube on cube epitaxial relationship can be obtained when
depositing at substrate temperature of 100 ∘C regardless of the lattice
mismatch (5% and 14% for Cu and Pt, respectively). At lower substrate
temperatures poly-crystalline multilayers were obtained while at higher
substrate temperatures {111} planes appear at $\sim$10∘ off normal to the film
plane. It is also shown that as the epitaxial strain increases, the easy
magnetization axis rotates towards the direction that previously was assumed
to be harder, i.e. from [110] to [100], and eventually further increase in the
strain makes the magnetic hysteresis loops isotropic in the film plane. Higher
epitaxial strain is also accompanied with increased coercivity values. Thus,
the effect of epitaxial strain on the magnetocrystalline anisotropy is much
larger than what was observed previously in similar, but polycrystalline
samples with uniaxial anisotropy (Kateb et al. 2021).

NiFe; Superlattice; Magnetic Anisotropy; Microstructure; Substrate Temperature

###### pacs:

75.30.Gw, 75.50.Bb, 73.50.Jt, 81.15.Cd

††preprint: Surf. Interfaces

##  I Introduction

Since the discovery of the giant magneto-resistance (GMR) effect by Fert
(Baibich  _et al._ , 1988) and Grünberg (Binasch  _et al._ , 1989) in the late
1980s, magnetic multilayers have been widely studied. In many cases they
present unique features that cannot be achieved within the bulk state namely
inter-layer exchange coupling (Parkin  _et al._ , 1990), magnetic damping, due
to the interface (Mizukami  _et al._ , 2001; Ingvarsson  _et al._ , 2002)
rather than alloying (Ingvarsson  _et al._ , 2004), and perpendicular magnetic
anisotropy (Johnson  _et al._ , 1996).

The GMR discovery, without a doubt, was an outcome of the advances in
preparation methods such as molecular beam epitaxy (MBE), that enabled
deposition of multilayer films with nanoscale thicknesses (Fert, 2008). Thus,
a great deal of effort has been devoted to enhancing the preparation methods
over the years using both simulations (Zhou and Wadley, 1998, 2004; Zhou  _et
al._ , 2004; Ene  _et al._ , 2005) and experiments (cf. Ref. (Kateb  _et al._
, 2021a) and references therein). Permalloy (Py) multilayers with non-magnetic
(NM) Pt (Corrêa  _et al._ , 2016; Kateb  _et al._ , 2021a; Ley Domínguez  _et
al._ , 117) or Cu (Rook  _et al._ , 1991; van Loyen  _et al._ , 2000; Hecker
_et al._ , 2002; Ene  _et al._ , 2005; Heitmann  _et al._ , 2000) as spacers
have been studied extensively in recent years. Various deposition methods have
been utilized for preparing magnetic multilayers such as MBE (Rook  _et al._ ,
1991), pulsed laser deposition (PLD) (Shen  _et al._ , 2004), ion beam
deposition (Ueno and Tanoue, 1995; Ene  _et al._ , 2005), dc magnetron
sputtering (dcMS) (Parkin  _et al._ , 1990; Corrêa  _et al._ , 2016; Hecker
_et al._ , 2002; van Loyen  _et al._ , 2000), and more recently, high power
impulse magnetron sputtering (HiPIMS) (Kateb  _et al._ , 2021a).

Permalloy (Py) is a unique material with regards to studying magnetic
anisotropy, which has been shown to strongly depend on the preparation method
(Kateb  _et al._ , 2019a). For instance, uniaxial anisotropy can be induced in
polycrystalline Py by several means (Kateb and Ingvarsson, 2021). However, it
has been thought that the cubic symmetry of single crystal Py encourages
magneto-crystalline anisotropy, while uniaxial anisotropy cannot be achieved.
We have recently shown that using HiPIMS deposition one can decrease the Ni3Fe
(L12) order, but maintain the single crystal form, to achieve uniaxial
anisotropy. We attributed this to the high instantaneous deposition rate
during the HiPIMS pulse (Kateb  _et al._ , 2021b), which limits ordering
compared to dcMS that present cubic (biaxial) anisotropy. Regarding Py
multilayers there has been a lot of focus on magneto-dynamic properties
recently while the effects of interface strain on magnetic anisotropy has not
received much attention. Rook  _et al._ (1991) prepared polycrystalline Py/Cu
multilayers by MBE and reported a weak anisotropy in them, i.e. hysteresis
loops along both the hard and easy axes with complete saturation at higher
fields. They compared the coercivity values ($H_{\rm c}$) and saturation
fields of their samples to $H_{\rm c}$ and anisotropy field ($H_{\rm k}$) of
sputter deposited multilayers showing uniaxial anisotropy and concluded that
the latter gives more than twice harder properties. They also reported an
increase in $H_{\rm c}$ with Py thickness and attributed this to the interface
strain that relaxes with increased thickness. Corrêa  _et al._ (2016) prepared
nanocrystalline Py/Pt multilayers on rigid and flexible substrates and in both
cases obtained weak anisotropy but two orders of magnitude larger $H_{\rm c}$.
Unfortunately, they did not mention any change in magnetic anisotropy upon
straining the flexible substrate.

Recently we showed that utilizing increased power to the dcMS process, and in
particular, by using HiPIMS deposition that the interface sharpness in
polycrystalline [Py/Pt]20 multilayers can be improved, due to increased
ionization of the sputtered species (Kateb  _et al._ , 2021a). Briefly, in
dcMS deposition the film forming material is composed mostly of neutral atoms
(Gudmundsson, 2020), while in HiPIMS deposition a significant fraction of the
film forming material consists of ions (Gudmundsson  _et al._ , 2012; Lundin
_et al._ , 2020). In fact we have shown that higher ionization of the film-
forming material leads to smoother film surfaces and sharper interfaces using
molecular dynamics simulations Kateb  _et al._ (2019b, 2020). We also showed
that by changing the non-magnetic spacer material one can increase interface
strain that is accompanied with higher $H_{\rm c}$, $H_{\rm k}$ and limited
deterioration of uniaxial anisotropy Kateb  _et al._ (2021a).

Another aspect of preparation is that deposition chambers for multilayers
mostly benefit from oblique deposition geometry, which encourage uniaxial
anisotropy in Py. The origin of uniaxial anisotropy induced by oblique
deposition has been proposed to be self-shadowing, but this has not been
systematically verified. We demonstrated uniaxial anisotropy, even in
atomically smooth films with normal texture, which indicates lack of self-
shadowing (Kateb and Ingvarsson, 2017; Kateb  _et al._ , 2018a). We also
showed that oblique deposition is more decisive in definition of anisotropy
direction than application of an in-situ magnetic field for inducing uniaxial
magnetic anisotropy. Also for polycrystalline Py films oblique deposition by
HiPIMS presents a lower coercivity and anisotropy field than when dcMS
deposition is applied (Kateb  _et al._ , 2018b; Hajihoseini  _et al._ , 2019;
Kateb, 2019). While none of the above mentioned results verify self-shadowing
they are consistent with our interpretation of the order i.e. oblique
deposition induces more disorder than in-situ magnetic field and HiPIMS
produce more disorder than dcMS. Note that the level of order in polycrystals
cannot be easily observed by X-ray diffraction. In this regard we proposed a
method for mapping the resistivity tensor that is very sensitive to level of
order in Py (Ingvarsson  _et al._ , 2017; Kateb and Ingvarsson, 2021). We
reported much higher coercivity and deterioration of uniaxial anisotropy in
(111) Py/Pt multilayers obtained by HiPIMS deposition of the Py layers (Kateb
_et al._ , 2021a). We attributed the latter effect to the interface sharpness
and higher epitaxial strain when HiPIMS is utilized for Py deposition.

Here, we study the properties of Py superlattices deposited by dcMS with Pt,
Cu and CuPt as non-magnetic spacers. Pt and Cu were chosen as spacer because
they have lattice parameters of 3.9 and 3.5 Å, respectively, and therefore
provide varying strain to the Py film which has lattice constant of 3.54 Å. In
this regard, calibration of the substrate temperature during deposition with
respect to the desired thickness is of prime importance Loloee and Crimp
(2002). It is worth mentioning that dcMS deposition is expected to give more
ordered single crystal (001) Py layers in which crystalline anisotropy is
dominant (Kateb  _et al._ , 2019a). This enables understanding to what extent
interface strain will affect magnetocrystalline anisotropy of Py which we will
show is much larger than the changes in uniaxial anisotropy in our latest
study (Kateb  _et al._ , 2021a). Section II discusses the deposition method
and process parameters for the fabrication of the superlattices and the
characterization methods applied. In Section III the effects of substrate
temperature on the properties of the Py/Cu system are studied followed by
exploring the influence of varying the lattice parameter of the non-magnetic
layer on the structural and magnetic properties of the superlattice. The
findings are summarized in Section IV.

##  II Experimental apparatus and methods

The substrates were one side polished single crystal (001) MgO (Crystal GmbH)
with surface roughness $<$5 Å and of dimensions 10 mm$\times$10 mm$\times$0.5
mm. The MgO substrates were used as received without any cleaning but were
baked for an hour at 600 ∘C in vacuum for dehydration, cooled down for about
an hour, and then maintained at the desired temperature $\pm$0.4 ∘C during the
deposition. The superlattices were deposited in a custom built UHV magnetron
sputter chamber with a base pressure below $5\times 10^{-7}$ Pa. The chamber
is designed to support 5 magnetron assemblies and targets, which are all
located 22 cm away from substrate holder with a 35∘ angle with respect to
substrate normal. The shutters were controlled by a LabVIEW program (National
Instruments). The deposition was made with argon of 99.999 % purity as the
working gas using a Ni80Fe20 at.% and Cu targets both of 75 mm diameter and a
Pt target of 50 mm in diameter.

The Py depositions were performed at 150 W dc power (MDX 500 power supply from
Advanced Energy) at argon working gas pressure of 0.25 Pa which gives
deposition rate of 1.5 Å/s. Both pure Cu and Pt buffer layers were deposited
at dc power of 20 W. For the deposition of CuPt alloy we calibrated Cu50Pt50
at. % at dc power at 10 and 13 W for Cu and Pt, respectively. This selection
of powers provide a similar deposition rate of 0.45 Å/s in all cases. In order
to ensure that the film thickness is as uniform as possible, we rotate the
sample at $\sim$12.8 rpm. These deposition processes were repeated to
fabricate superlattices consisting of 20 repetitions of 15 Å Py and 5 Å Pt, Cu
or Cu50Pt50 at. % (CuPt).

X-ray diffraction measurements (XRD) were carried out using a X’pert PRO
PANalitical diffractometer (Cu Kα1 and Kα2 lines, wavelength 0.15406 and
0.15444 nm, respectively) mounted with a hybrid monochromator/mirror on the
incident side and a 0.27∘ collimator on the diffracted side. We would like to
remark that, Kα2 separation at $2\theta=55^{\circ}$ is only 0.2∘ and much less
at the smaller angles i.e. where our multilayer peaks are located. This is an
order of magnitude smaller than the full width half maximum (FWHM) of our
multiplayer and satellite peaks. A line focus was used with a beam width of
approximately 1 mm. The film thickness, mass density, and surface roughness,
was determined by low-angle X-ray reflectivity (XRR) measurements with an
angular resolution of 0.005∘, obtained by fitting the XRR data using the
commercial X’pert reflectivity program, that is based on the Parrat formalism
(Parratt, 1954) for reflectivity.

The magnetic hysteresis was recorded using a home-made high sensitivity
magneto-optic Kerr effect (MOKE) looper. We use a linearly polarized He-Ne
laser of wavelength 632.8 nm as a light source, with Glan-Thompson polarizers
to further polarize and to analyze the light after Kerr rotation upon
reflection off the sample surface. The Glan-Thompson polarizers linearly
polarize the light with a high extinction ratio. They are cross polarized near
extintion, i.e. their polarization states are near perpendicular and any
change in polarization caused by the the Kerr rotation at a sample’s surface
is detected as a change in power of light passing through the analyzer. The
coercivity was read directly from the easy axis loops. The anisotropy field is
obtained by extrapolating the linear low field trace along the hard axis
direction to the saturation magnetization level, a method commonly used when
dealing with effective easy axis anisotropy.

##  III Results and discussion

###  III.1 Effect of substrate temperature on structural and magnetic
properties

Figure 1 shows the XRR results from Py/Cu superlattices deposited at different
substrate temperatures. The $\Lambda$ and $\delta$ indicated in the figure are
inversely proportional to the superlattice period and the total thickness,
respectively. It can be clearly seen that the fringes decay faster for a Py/Cu
superlattice deposited at substrate temperature of 21 ∘C and 200 ∘C than when
deposited at 100 ∘C. This indicates lower surface roughness obtained in the
Py/Cu superlattice deposited at 100 ∘C. When deposited at room temperature,
the large lattice mismatch between MgO and Py/Cu does not allow depositing a
high quality superlattice. For substrate temperature of 200 ∘C, however, it is
difficult to grow a continuous Cu layer with such a low thickness (5 Å). This
is due to the dewetting phenomenon which causes the minimum Cu thickness that
is required to maintain its continuity to be 12 Å. Earlier, it has been shown
that for substrate temperature up to 100 ∘C Py/(1 Å) Cu showed a limited
intermixing upon annealing (van Loyen  _et al._ , 2000). The optimum substrate
temperature for deposition obtained here is very close to 156 ∘C which has
earlier been reported for the deposition of (001) Fe/MgO (Moubah  _et al._ ,
2016) and (001) Fe84Cu16/MgO (Warnatz  _et al._ , 2020) superlattices. We
would like to remark that in our previous study we deposited 5 nm Ta
underlayer to reduce the substrate surface roughness (Kateb  _et al._ ,
2021a). However, Ta on MgO is non-trivial due to the large lattice mismatch
(22%). Besides, Ta underlyer encourages polycrystalline $\langle 111\rangle$
texture normal to substrate surface that does not serve our purpose here.

Figure 1: Comparison of the XRR pattern from [Py/Cu]20 superlattices deposited
on (001) MgO at different substrate temperatures. The $\Lambda$ and $\delta$
are inversely proportional to the Py/Cu period and total thickness,
respectively.

Figure 2 shows the result of symmetric ($\theta-2\theta$) XRD scan normal to
the film for Py/Cu superlattices deposited at different substrate
temperatures. It can be seen that no Cu and Py peak were detected in the
superlattice deposited at room temperature. Thus, epitaxial growth of Py and
Cu were suppressed by the low substrate temperature. Furthermore, we studied
room temperature deposited Py/Cu using grazing incidence XRD which indicated a
polycrystalline structure (not shown here). For substrate temperature of 100 –
200 ∘C there are clear (002) Py/Cu peaks indicating an epitaxial relationship
in the (001) Py $\|$ (001) Cu $\|$ (001) MgO stack. However, there is no sign
of satellite peaks due to the $\Lambda$ (Py/Cu) period. We explain this
further when comparing the Py/Cu, Py/Pt, and Py/CuPt superlattices in Section
III.2.

Figure 2: Comparison of the XRD pattern from [Py/Cu]20 superlattices deposited
on (001) MgO at different substrate temperatures. The intense peak belongs to
(002) planes of MgO and and the other peak is due to (002) planes of Py/Cu
multilayer.

Figure 3 shows the pole figures from the {200} and {111} planes for Py/Cu
superlattices deposited at different substrate temperatures. For the Py/Cu
superlattice deposited at 21 ∘C, there is only a peak in the middle of the
{111} pole figure that indicates a weak $\langle 111\rangle$ contribution
normal to the film plane. For a superlattice deposited with substrate
temperature of 100 ∘C the {200} pole figure indicates an intense spot at
$\psi=0$ that is corresponding to (002) Py/Cu planes parallel to the
substrate. There is also a weaker four-fold spot at $\psi=90^{\circ}$ and
$\phi=0,90,180$ and $270^{\circ}$ from the {200} planes parallel to the
substrate edges. In the {111} pole figure only four-fold points appear at
$\psi=54.7^{\circ}$ and with 45∘ shifts in $\phi$ with respect to substrate
edges. These are the characteristics of the so-called cube on cube epitaxy
achieved at 100 ∘C. For deposition with substrate temperature of 200 ∘C,
however, there is a weak {111} ring at $\psi=7.5^{\circ}$. Note that these
{111} planes were not detected by normal XRD because the (111) Py/Cu peak
appears at $2\theta\simeq 42^{\circ}$ which is masked by the strong (002) MgO
peak normal to the film plane.

Figure 3: Comparison of the {200} and {111} pole figures from [Py/Cu]20
superlattices deposited on (001) MgO at substrate temperatures of 21, 100, and
200∘C. The background is removed for better illustration.

Figure 4 compares the MOKE response of Py/Cu superlattices deposited at
different substrate temperatures. For a superlattice deposited at room
temperature, uniaxial anisotropy along the [100] direction is evident. This is
expected since the oblique deposition in a co-deposition chamber tends to
induce uniaxial anisotropy in Py films (Kateb and Ingvarsson, 2017; Kateb  _et
al._ , 2018b, a, 2019a). However, the oblique deposition cannot overcome
magnetocrystalline anisotropy due to symmetry in an ordered single crystal Py
(Kateb  _et al._ , 2019a). Thus, the low substrate temperature must be
accounted for the limiting order in the Py layer and presence of uniaxial
anisotropy.

Figure 4: MOKE response of different [Py/Cu]20 superlattices deposited on
(001) MgO at substrate temperatures (a) 21, (b) 100 and (c) 200 ∘C. Each
legend indicates probing orientation in the substrate plane.

For deposition at higher substrate temperatures, however, biaxial anisotropy
was obtained with the easy axes along the [110] directions in plane. It is
worth mentioning that the bulk crystal symmetry gives the easy axis along the
[111] direction which is forced into the film plane along the [110] direction
due to shape anisotropy (Kateb  _et al._ , 2019a). In the Py/Cu superlattice
grown at 100 ∘C (Figure 4 (b)), $\langle 1\bar{1}0\rangle$ is clearly an easy
direction, with a very low $H_{\rm c}$ of 0.7 Oe and double-hysteresis loops
along the $\langle 110\rangle$ direction that saturates at 1.2 Oe. For the
Py/Cu superlattice deposited at 200∘C (Figure 4 (c)), it seems the double-
hysteresis loops overlap and the other easy axis gives a step that in total
present increased coercivity. With increasing substrate temperature not only
do the coercivities vary but also the shapes of the hysteresis curves are
different. When the substrate temperature during deposition is 21∘C the
magnetization, shown in Figure 4 (a), is much like we obtain for
polycrystalline single layer films. When the substrate temperature is higher,
as shown in Figures 4 (b) and (c), however, the anisotropy has rotated by 45
degrees and the hysteresis loops have changed. The intermediate steps in the
hysteresis curves are caused by antiferromagnetic alignment of the magnetic
layers, that minimizes the exchange and dipolar magnetic interactions. In some
cases this results in perfectly zero magnetic remanence, while in other cases
the cancellation is not perfect. The non-magnetic Cu spacer layer is only 5 Å
in our case, just at the onset of the first antiferromagnetic exchange
coupling peak observed by Parkin (1992). Double hysteresis curves have been
observed in the Py/Cu system (Hecker  _et al._ , 2002). Note that Ni is
miscible in Cu and during annealing a mixing of Ni and Cu is possible. Such
intermixing causes a decrease of magnetic homogenity and a reduction in the
GMR (van Loyen  _et al._ , 2000; Hecker  _et al._ , 2002).

###  III.2 Effect of strain on structural and magnetic properties

In order to explore the influence of strain on the magnetic properties we
deposited NM layers of Pt and Cu50Pt50 at. % alloy in addition to Cu discussed
in Section III.1. Pt has lattice constant of 3.9 Å which is larger than of Py,
which has lattice constant of 3.54 Å. Therefore, by going from Cu to Cu50Pt50
and then to Pt the strain is gradually increased. Figure 5 shows XRR results
from different superlattices deposited on (001) MgO at 100 ∘C. Note that the
$\Lambda$ peak is suppressed in the Py/Cu superlattice. One may think this
arises from a diffused Py/Cu interface that leads to smooth density variation
at the interface. This is not the case here and the $\Lambda$ peaks intensity
decreases due the similar density of Py and Cu. The latter has been shown to
reduce the resolution of the XRR measurement in Si/SiO2 by a few orders of
magnitude Tiilikainen  _et al._ (2007).

Figure 5: XRR measurements from the various superlattices, [Py/Cu]20,
[Py/Pt]20, and [Py/CuPt]20, deposited on (001) MgO at substrate temperature of
100 ∘C.

The layers thickness and their mass density as well as surface and interface
roughness obtained by fitting XRR results for deposition at substrate
temperature of 100∘C are summarized in table 1. The period $\Lambda$ is in all
cases about 19 Å with $t_{\rm Py}\sim 16$ Å and $t_{\rm NM}\sim 3$ Å. The film
mass density of the Py layers is the highest (8.74 g/cm3) in the Py/Cu stack
but is lowest (7.45 g/cm3) in the Py/CuPt stack.

Table 1: The Py and NM layer thicknesses ($t$), roughness (Ra) and density ($\rho$) extracted by fitting the XRR results of different superlattices deposited on (001) MgO at 100∘C substrate temperature. Sample |  $t$ (Å) | Ra (Å) |  $\rho$ (g/cm3)
---|---|---|---
| Py | NM | $\Lambda$ | Py | NM | Py | NM
Py/Cu | 15.8 | 3.46 | 19.3 | 7.62 | 6.25 | 8.74 | 9.8
Py/Pt | 15.9 | 2.97 | 18.9 | 5.92 | 3.24 | 8.55 | 27.2
Py/CuPt | 15.9 | 3.43 | 19.3 | 2.25 | 4.94 | 7.45 | 26.8

Figure 6 shows the XRD results from Py/NM superlattices deposited on (001) MgO
at substrate temperature of 100 ∘C. The most intense peak, indicated by the
vertical dashed line, belong to the (002) planes of the MgO substrate. Rather
than exhibiting separate peaks for Py and NM, a single main (002) Py/NM peak
is evident from all the superlattices and indicated by $0$ in the figure. This
peak is closer to the Py side due to the higher thickness of Py layers
compared to the NM layers. The other peaks, indicated by $\pm$ are satellite
peaks. The asymmetric intensity of the satellite peaks is associated with the
strain in direction normal to the substrate. It is also clear that the main
(002) Py/CuPt peak is located between the Py/Cu and Py/Pt peaks. Due to the
lack of any other peaks, we conclude that these superlattices are single
crystalline with their $\langle 001\rangle$ orientation normal to the
substrate surface.

Figure 6: Comparison of the XRD results from the various superlattices,
[Py/Cu]20, [Py/Pt]20 and [Py/CuPt]20, deposited at substrate temperature of
100 ∘C. All the peaks are due to the (002) plane and the vertical dashed line
indicate the (002) peak position for the bulk state.

The satellite peaks suggest the period $\Lambda$ to be of 18.8 and 19.2 Å for
Py/Pt and Py/CuPt, respectively, which is slightly off compared to the values
given in table 1.

Figure 7 shows the {002} and {111} pole figures from different superlattices
deposited on (001) MgO at substrate temperature of 100 ∘C. Since the Py/Pt and
Py/CuPt superlattices exhibit multiple (002) peaks the pole figure were
obtained for the main peak (indicated by $0$ in figure 6). It can be seen that
all the pole figures are very similar. All these pole figures indicate a cube
on cube epitaxial relationship.

Figure 7: Comparison of the {002} and {111} pole figures from the various
superlattices, [Py/Cu]20, [Py/Pt]20 and [Py/CuPt]20, deposited on (001) MgO at
substrate temperature of 100 ∘C. The background is removed for better
illustration.

Figure 8 depicts the MOKE response from Py/Pt and Py/CuPt superlattices
prepared at 100 ∘C. For the Py/Pt superlattice we did not detect any clear
easy direction in the film plane, the film appeared almost isotropic in the
film plane with $H_{\rm c}$ of 60 – 75 Oe. The hysteresis in the [100] and
[110] directions are displayed in figure 8(a). Aside from $H_{\rm c}$ of 3 Oe,
the Py/CuPt superlattice presents biaxial anisotropy similar to Py/Cu, cf.
figure 4(b) and (c). However, a Py/Cu superlattice exhibits an easy axis along
the [110] directions, while an easy axis appears along the [100] orientations
for a Py/CuPt superlattice. Note that the [100] directions are harder than
both the [110] and [111] directions. However, forcing easy axes along the
[100] direction in the single crystal Py on (001) MgO has been reported
previously (Michelini  _et al._ , 2002).

Figure 8: MOKE response from (a) [Py/Pt]20 and (b) [Py/CuPt]20 superlattices.
The legend indicate probing direction in the substrate plane.

For the polycrystalline but highly (111) textured Py/M multilayers we observed
limited change in coercivity and opening in hard axis with interface strain
due to the choice of M (Kateb  _et al._ , 2021a). Here, for Py/Pt the
coercivity increases an order of magnitude and cubic anisotropy is almost
destroyed.

##  IV Summary

In summary it is shown that Py superlattices can be successfully deposited on
(001) MgO within a narrow substrate temperature window around 100 ∘C. For
small lattice mismatch of 5% superlattice the easy axes detected along the
[110] directions is similar to the single crystal Py. It is also shown that
the moderate lattice mismatch (7%) rotates the easy axes towards the [100]
orientation and the coercivity increases. The higher lattice mismatch of 14%
present nearly isotropic behaviour and a very high coercivity, simultaneously.
Thus, the results indicate that the changes in magnetocrystalline anisotropy
due to epitaxial strain are much larger than the changes we observed earlier
in the case of uniaxial anisotropy.

###### Acknowledgements.

The authors would like to acknowledge helpful comments and experimental help
from Dr. Fridrik Magnus and Einar B. Thorsteinsson. This work was partially
supported by the Icelandic Research Fund Grant Nos. 228951, 196141, 130029 and
120002023.

## References

  * Baibich  _et al._ (1988) M. N. Baibich, J. M. Broto, A. Fert, F. N. Van Dau, F. Petroff, P. Etienne, G. Creuzet, A. Friederich, and J. Chazelas, Phys. Rev. Lett. 61, 2472 (1988).
  * Binasch  _et al._ (1989) G. Binasch, P. Grünberg, F. Saurenbach, and W. Zinn, Physical Review B 39, 4828 (1989).
  * Parkin  _et al._ (1990) S. S. P. Parkin, N. More, and K. P. Roche, Physical Review Letters 64, 2304 (1990).
  * Mizukami  _et al._ (2001) S. Mizukami, Y. Ando, and T. Miyazaki, Jpn. J. Appl. Phys. 40, 580 (2001).
  * Ingvarsson  _et al._ (2002) S. Ingvarsson, L. Ritchie, X. Liu, G. Xiao, J. Slonczewski, P. Trouilloud, and R. Koch, Physical Review B 66, 214416 (2002).
  * Ingvarsson  _et al._ (2004) S. Ingvarsson, G. Xiao, S. S. P. Parkin, and R. H. Koch, Applied Physics Letters 85, 4995 (2004).
  * Johnson  _et al._ (1996) M. T. Johnson, P. J. H. Bloemen, F. J. A. Den Broeder, and J. J. de Vries, Reports on Progress in Physics 59, 1409 (1996).
  * Fert (2008) A. Fert, Rev. Mod. Phys. 80, 1517 (2008).
  * Zhou and Wadley (1998) X. Zhou and H. Wadley, Journal of Applied Physics 84, 2301 (1998).
  * Zhou and Wadley (2004) X. Zhou and H. Wadley, Philosophical Magazine 84, 193 (2004).
  * Zhou  _et al._ (2004) X. Zhou, R. Johnson, and H. Wadley, Physical Review B 69, 144113 (2004).
  * Ene  _et al._ (2005) C. B. Ene, G. Schmitz, R. Kirchheim, and A. Hütten, Acta Materialia 53, 3383 (2005).
  * Kateb  _et al._ (2021a) M. Kateb, J. T. Gudmundsson, and S. Ingvarsson, Journal of Magnetism and Magnetic Materials 538, 168288 (2021a).
  * Corrêa  _et al._ (2016) M. A. Corrêa, R. Dutra, T. L. Marcondes, T. J. A. Moric, F. Bohn, and R. L. Sommer, Materials Science and Engineering B 211, 115 (2016).
  * Ley Domínguez  _et al._ (117) D. Ley Domínguez, R. J. Sáenz-Hernández, A. F. Arzate, A. I. Arteaga Duran, C. E. Ornelas Gutiérrez, O. Solís Canto, M. E. Botello-Zubiate, F. J. Rivera-Gómez, A. Azevedo, G. L. da Silva, S. M. Rezende, and J. A. Matutes-Aquino, Journal of Applied Physics 17, 17D910 (117).
  * Rook  _et al._ (1991) K. Rook, A. M. Zeltser, J. O. Artman, D. E. Laughlin, M. H. Kryder, and R. M. Chrenko, Journal of Applied Physics 69, 5670 (1991).
  * van Loyen  _et al._ (2000) L. van Loyen, D. Elefant, D. Tietjen, C. M. Schneider, M. Hecker, and J. Thomas, Journal of Applied Physics 87, 4852 (2000).
  * Hecker  _et al._ (2002) M. Hecker, D. Tietjen, H. Wendrock, C. M.Schneider, N.Cramer, L. Malkinski, R. E. Camley, and Z. Celinski, Journal of Magnetism and Magnetic Materials 247, 62 (2002).
  * Heitmann  _et al._ (2000) S. Heitmann, A. Hütten, T. Hempel, W. Schepper, G. Reiss, and C. Alof, Journal of Applied Physics 87, 4849 (2000).
  * Shen  _et al._ (2004) J. Shen, Z. Gai, and J. Kirschner, Surface Science Reports 52, 163 (2004).
  * Ueno and Tanoue (1995) M. Ueno and S. Tanoue, Journal of Vacuum Science and Technology A 13, 2194 (1995).
  * Kateb  _et al._ (2019a) M. Kateb, J. T. Gudmundsson, and S. Ingvarsson, AIP Advances 9, 035308 (2019a).
  * Kateb and Ingvarsson (2021) M. Kateb and S. Ingvarsson, Journal of Magnetism and Magnetic Materials 532, 167982 (2021).
  * Kateb  _et al._ (2021b) M. Kateb, J. T. Gudmundsson, P. Brault, A. Manolescu, and S. Ingvarsson, Surface and Coatings Technology 426, 127726 (2021b).
  * Gudmundsson (2020) J. T. Gudmundsson, Plasma Sources Science and Technology 29, 113001 (2020).
  * Gudmundsson  _et al._ (2012) J. T. Gudmundsson, N. Brenning, D. Lundin, and U. Helmersson, Journal of Vacuum Science and Technology A 30, 030801 (2012).
  * Lundin  _et al._ (2020) D. Lundin, T. Minea, and J. T. Gudmundsson, eds.,  _High Power Impulse Magnetron Sputtering: Fundamentals, Technologies, Challenges and Applications_ (Elsevier, Amsterdam, The Netherlands, 2020).
  * Kateb  _et al._ (2019b) M. Kateb, H. Hajihoseini, J. T. Gudmundsson, and S. Ingvarsson, Journal of Vacuum Science and Technology A 37, 031306 (2019b).
  * Kateb  _et al._ (2020) M. Kateb, J. T. Gudmundsson, and S. Ingvarsson, Journal of Vacuum Science and Technology A 38, 043006 (2020).
  * Kateb and Ingvarsson (2017) M. Kateb and S. Ingvarsson, in  _2017 IEEE Sensors Applications Symposium (SAS)_ (IEEE, Glassboro, New Jersey, 13–15 March 2017) pp. 1–5.
  * Kateb  _et al._ (2018a) M. Kateb, E. Jacobsen, and S. Ingvarsson, Journal of Physics D: Applied Physics 52, 075002 (2018a).
  * Kateb  _et al._ (2018b) M. Kateb, H. Hajihoseini, J. T. Gudmundsson, and S. Ingvarsson, J. Phys. D Appl. Phys. 51, 285005 (2018b).
  * Hajihoseini  _et al._ (2019) H. Hajihoseini, M. Kateb, S. Ingvarsson, and J. T. Gudmundsson, Beilstein Journal of Nanotechnology 10, 1914 (2019).
  * Kateb (2019) M. Kateb,  _Induced uniaxial magnetic anisotropy in polycrystalline, single crystal and superlattices of permalloy_ , Ph.D. thesis, University of Iceland (2019).
  * Ingvarsson  _et al._ (2017) S. Ingvarsson, M. Kateb, and C. Scheuner, in  _Bulletin of the American Physical Society_, Vol. 62 (APS, 2017) pp. APS March Meeting 2017, abstract E47.012.
  * Loloee and Crimp (2002) R. Loloee and M. A. Crimp, Journal of Applied Physics 92, 4541 (2002).
  * Parratt (1954) L. G. Parratt, Physical Review 95, 359 (1954).
  * Moubah  _et al._ (2016) R. Moubah, F. Magnus, T. Warnatz, G. K. Pálsson, V. Kapaklis, V. Ukleev, A. Devishvili, J. Palisaitis, P. Persson, and B. Hjörvarsson, Physical Review Applied 5, 044011 (2016).
  * Warnatz  _et al._ (2020) T. Warnatz, B. E. Skovdal, F. Magnus, H. Stopfel, D. Primetzhofer, A. Stein, R. Brucas, and B. Hjörvarsson, Journal of Magnetism and Magnetic Materials 496, 165864 (2020).
  * Parkin (1992) S. S. P. Parkin, Appl. Phys. Lett. 60, 512 (1992).
  * Tiilikainen  _et al._ (2007) J. Tiilikainen, J.-M. Tilli, V. Bosund, M. Mattila, T. Hakkarainen, J. Sormunen, and H. Lipsanen, Journal of Physics D: Applied Physics 40, 7497 (2007).
  * Michelini  _et al._ (2002) F. Michelini, L. Ressier, J. Degauque, P. Baulés, A. R. Fert, J. P. Peyrade, and J. F. Bobo, Journal of Applied Physics 92, 7337 (2002).


# Discrete State-Action Abstraction
via the Successor Representation

Amnon Attali
Department of Computer Science
University of Illinois Urbana-Champaign
Champaign, IL 61820
<EMAIL_ADDRESS>
Pedro Cisneros-Velarde
Department of Computer Science
University of Illinois Urbana-Champaign
Champaign, IL 61820
<EMAIL_ADDRESS>
Marco Morales
Department of Computer Science
University of Illinois Urbana-Champaign
Champaign, IL 61820
<EMAIL_ADDRESS>
Nancy M. Amato
Department of Computer Science
University of Illinois Urbana-Champaign
Champaign, IL 61820
<EMAIL_ADDRESS>

###### Abstract

When reinforcement learning is applied with sparse rewards, agents must spend
a prohibitively long time exploring the unknown environment without any
learning signal. Abstraction is one approach that provides the agent with an
intrinsic reward for transitioning in a latent space. Prior work focuses on
dense continuous latent spaces, or requires the user to manually provide the
representation. Our approach is the first for automatically learning a
discrete abstraction of the underlying environment. Moreover, our method works
on arbitrary input spaces, using an end-to-end trainable regularized successor
representation model. For transitions between abstract states, we train a set
of temporally extended actions in the form of options, i.e., an action
abstraction. Our proposed algorithm, Discrete State-Action Abstraction (DSAA),
iteratively swaps between training these options and using them to efficiently
explore more of the environment to improve the state abstraction. As a result,
our model is not only useful for transfer learning but also in the online
learning setting. We empirically show that our agent is able to explore the
environment and solve provided tasks more efficiently than baseline
reinforcement learning algorithms. Our code is publicly available at
https://github.com/amnonattali/dsaa.

##  1 Introduction

Reinforcement learning (RL) provides a general framework for solving search
problems through the formalism of a Markov Decision Process (MDP); yet with
that generality, it sacrifices some basic properties one might expect of a
search algorithm. In particular, one should not explore a state multiple
times, which is why basic search methods, such as Dijkstra’s algorithm, keep
track of the explored frontier in memory. Such basic search methods perform
well when state and action spaces are small and discrete, but don’t
necessarily translate to more complex environments, such as large or
continuous ones.

Humans have several intuitive ways to efficiently explore in complex
scenarios. One is by _abstracting states_ , thereby exploring a simpler, more
structured model of the environment. For example, consider searching for an
exit in a large room using only touch: we would never blindly roam the center,
but rather follow the walls. By abstracting states based on the property “can
contain exit”, we greatly reduce the set of states we have to explore in the
first place. Another, closely related way we explore efficiently is by
_abstracting actions_. Ingrained skills, or temporally extended actions,
impose a prior on the types of action sequences that help solve problems. For
example, when exploring we don’t move forward and then immediately backwards.
Randomly choosing the next direction to move is rarely a good option, and so
skills ensure that we explore new environments efficiently.

In this work we are concerned with incorporating the intuitive concepts of
state abstraction and action abstraction – the latter also known as _skill
selection_ – into reinforcement learning. By learning a discrete state
abstraction, we take advantage of a simple and natural definition for action
abstraction, namely, abstract actions are policies which help the agent
navigate between pairs of abstract states [Abel et al., 2020]. We show that
such discrete state-action abstraction allows an agent to navigate efficiently
between and within regions of the original MDP.

We motivate our interest in discrete abstractions in two ways. Firstly, many
of the decisions an agent must make in the world are discrete and depend on
discrete objects with properties that are not continuously comparable. For
example, a room has a discrete number of doors, and the length of the optimal
path out of the room does not change continuously as a function of the number
of doors. While we can still model such decision problems using continuous
representations, it has been shown that policies trained in such continuous
spaces struggle precisely at points of discontinuity [Tang and Hauser, 2019].

The second motivation is that classical algorithms for planning in discrete
spaces are better understood and provide stronger guarantees; in fact, we
often deal with continuous spaces by discretizing them [Kavraki et al., 1996].
Moreover, a discrete abstraction represents an explicit reduction in the size
of the state space, in which both the depth and branching factor of future
search can be easily controlled. Thus, it is simpler and more efficient to
reuse our discrete abstraction to navigate the environment.

###  1.1 Contributions

Our main contribution is a novel method to learn a discrete abstraction by
partitioning an arbitrary state space from a dataset of transitions which
explore that same space. In particular, we cluster states with a similar
_successor representation_ (SR) as being part of the same abstract state.
Unlike prior works on the SR (e.g., Machado et al. [2018]), our approach is
end-to-end trainable and uses a comparatively weaker max-entropy
regularization. Our neural network model resembles a discrete variational
autoencoder, in which an encoder computes the abstraction and a decoder
computes the SR. In addition, our use of the SR is more general, relying on
options rather than uniform random exploration.

To demonstrate the effectiveness of our method, we propose a new algorithm
called Discrete State-Action Abstraction (DSAA), which creates a discrete
state-action abstraction pair, using _options_ for modeling the action
abstraction. Prior work on the SR often requires a single, prohibitively long,
offline training phase, which produces a large set of options. In contrast,
our method iteratively improves the learned abstraction, making it useful in
both the online, and transfer learning settings.

Experimentally, we validate our approach in simulation on simple discrete and
continuous control environments that are nevertheless difficult for off-the-
shelf RL algorithms because they have sparse reward, high diameter (distance
to reward), and bottleneck states. We also show our method is robust to noise
in the input state. Finally, we show that using the learned abstraction and
corresponding options improves exploration efficiency, consequently speeding
up the solving of environment tasks.

##  2 Background

###  2.1 Reinforcement Learning

We consider the model-free reinforcement learning (RL) problem with an
underlying infinite-horizon Markov Decision Process (MDP):
$\mathcal{M}=(\mathcal{X},\mathcal{A},p,r,\gamma,x_{0})$, with state space
$\mathcal{X}$, action space $\mathcal{A}$, $p(x^{\prime}\mid x,a)$ being the
unknown environment dynamics giving the probability of transitioning to state
$x^{\prime}$ having taken action $a$ at state $x$, reward function
$r:\mathcal{X}\times\mathcal{A}\to[0,1]$, discount factor $0<\gamma<1$, and
initial state $x_{0}$.

Let $x_{t}\in\mathcal{X},a_{t}\in\mathcal{A}$ be the agent’s state and action
respectively at time $t$, and $\pi(a_{t}\mid x_{t})$ be the agent’s policy,
determining the probability distribution of agent actions at state $x_{t}$.
Given a policy $\pi$, the expected return (i.e., discounted sum of rewards)
the agent would obtain if action $a$ is taken at initial state $x$, is
described by the _Q-value function_ :
$Q^{\pi}(x,a)=\mathbb{E}_{p,\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r(x_{t},a_{t})\bigg{|}a_{0}=a,x_{0}=x\right]$,
where $a_{t}\sim\pi(\cdot|x_{t})$ and $x_{t+1}\sim p(\cdot|x_{t},a_{t})$. The
agent’s goal is to compute a policy $\pi$ which maximizes the expected return
from the starting state $x_{0}$, i.e.,
$\pi\in\arg\max_{\bar{\pi}}\mathbb{E}_{a\sim\bar{\pi}(\cdot\mid
x)}[Q^{\bar{\pi}}(x_{0},a)]$.

Since the environment dynamics $p$ is unknown, a common approach in RL is to
iteratively improve an estimate of the Q-value function, while simultaneously
exploring the environment using the induced policy. However, we highlight that
when the reward function is sparse, such methods suffer from a long and
uninformed (unrewarding) random exploration phase.

###  2.2 Options Framework

Sutton et al. [1999] presented the options framework to extend the classic
formalism of an MDP to a semi-MDP, in which we can replace primitive single-
step actions with temporally extended policies in the form of _options_. An
option in an MDP is a 3-tuple $o=(\mathcal{I}_{o},\pi_{o},\mathcal{T}_{o})$,
where $\mathcal{I}_{o},\mathcal{T}_{o}\subseteq\mathcal{X}$, and $\pi_{o}$ is
a policy that initiates in some state $x_{0}\in\mathcal{I}_{o}$ and terminates
in any state in the set $\mathcal{T}_{o}$. Intuitively, an option describes a
local subproblem of navigating the agent between regions of the state space.
As an example, each primitive action $a\in\mathcal{A}$ can be represented by
an option which initiates everywhere, terminates immediately, and its policy
is to always take action $a$: $o_{a}=(\mathcal{X},\pi_{a}=a,\mathcal{X})$.

A common perspective is to think of options as funnels that take the agent
from anywhere in the state space to some goal. For example, one can create
options so that the termination set of one lies in the initiation set of
another, thus allowing for option chaining [Bagaria and Konidaris, 2019]. In
general, the quality of the options depends on the partition of the state
space, such that each option takes the agent to a useful sub-goal. Our work
can be viewed as a method for designing such a partition based on the
successor representation.

###  2.3 Successor Representation

The successor representation (SR) of a state, also known as the discounted
state occupancy, models the visitation density of a given policy $\pi$
starting from each state. In a discrete state space $\mathcal{X}$ we can
simply keep count of these visitation frequencies in a matrix
$\Psi_{\pi}\in\mathbb{R}^{|\mathcal{X}|\times|\mathcal{X}|}$ whose $(u,v)$-th
entry is

$\Psi_{\pi}(u,v)=\mathbb{E}_{p,\pi}\left[\sum_{t=1}^{\infty}\gamma^{t}\delta_{x_{t},v}\bigg{|}x_{0}=u\right],$

where $\delta_{\cdot,\cdot}$ is the Kronecker delta (with value 1 if the
arguments are equal, and 0 otherwise). Thus, the SR of state $v$ is the $v$-th
row of the matrix $\Psi_{\pi}$.

Much like the Q-value function of an MDP, the SR satisfies a Bellman-like
equation which relates the SR of a state to its neighbors by

$\Psi_{\pi}(u,v)=\delta_{u,v}+\gamma\mathbb{E}_{x\sim
p(\cdot|u,\pi(u))}\Psi_{\pi}(x,v).$

When the environment dynamics $p$ are unknown and we instead have a dataset
$\mathcal{D}$ of transitions collected using a policy $\pi$, this equation can
be used to learn an estimate of the SR,
$\psi:\mathcal{X}\to\mathbb{R}^{|\mathcal{X}|}$ which minimizes the so-called
temporal difference (TD) error $\mathbb{E}_{x,x^{\prime}\sim
D}\left[\psi(x)-(\mathbbm{1}(x)+\gamma\psi(x^{\prime}))\right]^{2}$ where
$\mathbbm{1}(x)$ is a one-hot encoding of the discrete state $x$ (the $x$-th
standard unit vector).

The SR has a natural extension, the Deep Successor Representation (DSR), in
which we replace this count based density with a discounted sum of features
[Kulkarni et al., 2016]. An important advantage of our work is that it uses
the SR of a discrete encoding of the state space, and so we do not need to
resort to the more complex DSR.

We highlight that the SR depends heavily on the given policy $\pi$ used to
explore the environment. Past work (e.g., [Machado et al., 2018]) has focused
on the setting where $\pi$ is a uniform random policy (i.e., a random walk),
and so the properties and/or usefulness of the SR with more general policies
remains an open problem. This work represents a largely experimental foray
into that problem.

Under a uniform random policy $\pi$ in a discrete state space, $\Psi_{\pi}$ is
closely related to the graph Laplacian, namely their eigenvectors are parallel
[Machado et al., 2018]. When considering random walks on graphs, the
eigenvectors of the Laplacian provide many useful properties of the graph. For
example, Cheeger’s Inequality tells us that the second eigenvalue gives a
bound on the cut with minimum connectivity in the graph. Such a cut represents
a bottleneck region, or narrow passage, which is known as a major challenge
for planning algorithms and has been used as a method for identifying subgoals
[Şimşek et al., 2005, Ichter et al., 2020]).

In this work we focus on the SR itself rather than its eigenvectors. We
provide some additional observations about the SR that make it conducive to
planning, also illustrated in Figure 1:

  * •

SR as a proxy for distance: In an $N$-dimensional state space, the SR of a
state is an $N$-dimensional vector embedding, which we can compare using the
Euclidean distance to other SRs.

  * •

Each option has a predictable effect on the SR: Thinking of options as
funnels, we can see that augmenting the action space with an option has the
effect of making states which belong to the same initiation set of the option
have a more similar SR relative to states outside this initiation set. In
other words, the borders of option initiation sets create sharp differences in
SR.

Figure 1: We provide intuition regarding the successor representation in a two
room environment with black walls where each room has $8\times 8$ cells with a
single open cell connecting them. We compute the euclidean distance
$||\psi(x)-\psi(\bar{x})||$ between the SR of every cell $x$ and the SR of a
fixed reference state $\bar{x}\in\\{[4,4],[4,9],[4,14]\\}$ (middle of left
room, center hallway, middle of right room). The top row shows distances
computed for a uniform random policy, and the bottom row for a uniform random
policy augmented with a wall-hugging option. This option follows walls
counterclockwise until there are no walls adjacent. The SR partitions the
environment based on agent dynamics, and options have a large effect on the
SR, namely by altering relative distances between certain states.

##  3 Related Work

Abstraction falls under the broader category of Intrinsic Motivation (IM)
methods. IM methods provide an intrinsic reward to the agent to guide and
structure the exploration process, which makes exploration more efficient in
the absence of environment rewards (i.e., when rewards are sparse). Oudeyer
and Kaplan [2009] describe three broad types of IM: knowledge, competence, and
abstraction.

Knowledge-based IM methods reward the agent for improving its knowledge of the
environment. Strehl and Littman [2008] and Bellemare et al. [2016] reward the
agent for visiting states that have not been visited before, extending a known
principle from simple search. Curiosity [Sekar et al., 2020] rewards the agent
for visiting states in which a trained world model fails, or has high
uncertainty, thus encouraging the policy to sample in those regions and
improve the world model.

Competence-based IM methods reward the agent for accomplishing sub-tasks in
the environment. Such methods train a Universal Value Function Approximation
[Schaul et al., 2015] which models the value of state-action pairs conditioned
on the goal (reward). By providing a curriculum of tasks to the agent, such
methods can iteratively increase the agent’s competence, until it can achieve
the extrinsic environment reward. Hindsight Experience Replay [Andrychowicz et
al., 2017] treats every trajectory as a successful sample of reaching the last
state in that trajectory. Nasiriany et al. [2019] train goal conditioned
policies by identifying goals in a latent space trained from a variational
autoencoder (VAE).

Abstraction-based IM methods typically reward the agent for exploring a latent
space. The focus is therefore on learning a latent space that captures
environment dynamics, in contrast to _dense representations_ , such as those
learned by a VAE, which encode information irrelevant to planning. For
example, Feudal Networks [Vezhnevets et al., 2017] trains a manager worker
system where the manager maps observations to a latent space and produces a
direction in that space. Then, the worker is rewarded for achieving this
desired direction. Another example is Eigenoption Discovery [Machado et al.,
2018, 2021], which learns a latent space based on a modified VAE whose decoder
produces future observations, thereby capturing environment dynamics. The
eigenvectors of the DSR in this latent space are then used as individual
reward functions for a set of options. Here the DSR is trained offline, using
a uniform random policy for exploration.

A common theme in abstraction methods [Erraqabi et al., 2021, Li et al., 2021]
is that contrastive losses, which enforce MDP dynamics on the latent space by
bringing the representations of nearby states closer together, encourage the
representation to capture “slow features”. These are features that change
slowly as the agent transitions in the environment. Therefore, capturing such
features yields efficient environment exploration. Jonschkowski and Brock
[2015] add additional losses on top of the contrastive loss, motivated by real
world priors such as Netwon’s Laws.

We note that all the above methods learn continuous abstractions. Like us,
Giannakopoulos et al. [2021] learn a discrete abstraction, but they use
observation reconstruction (i.e., an autoencoder) as a powerful regularizer
that encodes information irrelevant to planning.

Ramesh et al. [2019] provide a method which resembles ours in that they
cluster states based on the SR in order to define options that go to cluster
centers. Unlike theirs, our method learns the abstraction, and therefore the
intrinsic reward, in a fully differentiable manner, i.e., without using
external clustering algorithms. In addition, their method restricts the input
state space to be discrete, whereas ours learns this clustering from arbitrary
input spaces.

Finally, we note that ultimately our form of abstraction can be thought of as
extending the classical idea of _state space discretization_. Classical
robotics algorithms, such as Probabilistic Roadmaps [Kavraki et al., 1996],
iteratively build a discrete model of the continuous search space, where
discrete nodes are sampled and connected via local planners. Our options are
like these local planners, except they are learned rather than specified ahead
of time. Our abstract states are much like these discrete sampled nodes, but
defined instead through a more general state-encoding process.

##  4 Method: The DSAA Algorithm

Figure 2: Our algorithm DSAA alternates between exploration and abstraction
update. During exploration (top), we simultaneously train options to navigate
between abstract states (line 16 in Algorithm 1), and use those options to
explore the environment efficiently (line 19). The abstraction update phase
(bottom) uses transition data that was gathered during exploration (line 14)
for updating the abstraction (line 23) using the induced successor
representation.

We start with a high level description of our proposed algorithm Discrete
State-Action Abstraction (DSAA), followed by a deeper explanation of the
motivation and implementation decisions behind its components. Our algorithm
is described in Algorithm 1 and a pictorial representation in Figure 2.

DSAA iteratively swaps between two phases: an _exploration phase_ , in which
it explores the environment using a trained set of options $\mathcal{O}$, and
an _abstraction update phase_ , where it uses the exploration data to improve
an abstraction $\phi$ based on the successor representation $\psi$ induced by
the same data. Consequently, during the exploration phase, rather than
choosing among primitive actions, we choose among the available options
uniformly at random.

The options are defined as follows: each option is a trained policy which
navigates from primitive states in one abstract state to another. As such, one
can think of our method as performing a (lazy) random walk on an _abstract
graph_ whose nodes correspond to abstract states, and edges correspond to
options which transition between them. This abstract graph updates as we
update the abstraction and identify new transitions between pairs of abstract
states.

More specifically, for a fixed number of abstract states $N$, our algorithm
produces:

  1. 1.

The abstraction $\phi:\mathcal{X}\to[N]$, which maps primitive states to
discrete abstract states. Let
$\phi^{-1}(s)=\\{x\in\mathcal{X}\mid\phi(x)=s\\}$.

  2. 2.

Option policies $o_{s,s^{\prime}}\in\mathcal{O}$, where
$o_{s,s^{\prime}}=(\phi^{-1}(s),\pi_{s,s^{\prime}},\mathcal{X}\setminus\phi^{-1}(s))$
for some $(s,s^{\prime})\in[N]\times[N]$. $\pi_{s,s^{\prime}}$ initiates in
the preimage of abstract state $s$ and terminates upon leaving it.
$\pi_{s,s^{\prime}}$ is rewarded for transitioning into $s^{\prime}$ as
follows: $r_{s,s^{\prime}}(x,a)=\delta_{\phi(x^{\prime}),s^{\prime}}$, where
$x^{\prime}\sim p(\cdot|x,a)$ given the current state and action $x,a$. We
note that this is a slight departure from our previous discussion of options,
since we only reward the policy for specific transitions into the terminating
set.

  3. 3.

An abstract MDP represented by the abstract graph $G=(V,E)$, with node set
$V=[N]$ and edge set $E=\\{(s,s^{\prime})\in[N]\times[N]\mid
o_{s,s^{\prime}}\in\mathcal{O}\\}$. For any $v\in V$, the neighbors of $v$ are
$N_{G}(v)=\\{u\in V\mid(v,u)\in E\\}$.

  4. 4.

The successor representation model $\psi:[N]\to\mathbb{R}^{N}$ computes the SR
of each abstract state.

Following initialization (lines 1-5 in Algorithm 1), during the exploration
phase, at time step $i$, we take $s_{i}=\phi(x_{i})$, randomly sample $s_{g}$
from the neighborhood of $s_{i}$ in $G$ (lines 8, 18), and then select the
current action according to the option policy
$a_{i}\sim\pi_{s_{i},s_{g}}(x_{i})$ (line 10). The resulting transition is
added to the dataset $\mathcal{D}$ (line 14) and used for continuing the
training $\pi_{s_{i},s_{g}}$ using a standard out of the box reinforcement
learning algorithm (we use Soft Q-learning, line 16). The conditional on line
17 ensures that we continue using the same option until its termination when
we transition to a different abstract state. Note, however, that the training
of $\pi_{s_{i},s_{g}}$ will only have a non-zero reward if
$x_{i+1}\in\phi^{-1}(s_{g})$.

Then, in the abstraction update phase, the dataset of transitions is used for
updating the abstraction using Eq. (1) + Eq. (2) (lines 22-23), discussed in
more detail in Subsections 4.1 and 4.2 below. In practice we perform this
update using stochastic gradient descent (SGD) on batches sampled from
$\mathcal{D}$.

Algorithm 1 shows the _unsupervised_ setting, where we explore the environment
without an environment provided reward. In this setting the algorithm runs for
a fixed number of iterations (this would be the stopping criterion in line
24). In the _online_ setting (e.g., the experiments in section 5.2) we
additionally reward the trained option policies corresponding to self loop
edges with the environment reward. In other words we add the environment
reward to the option policy if $s_{g}=s_{i}$: replace line 13 with
$r_{i}=\delta_{s_{i+1},s_{g}}+\delta_{s_{i},s_{g}}\times(\text{environment\\_reward})$.
To solve the full environment task we then train a model-based tabular
Q-learning agent, whose MDP is described by the abstract graph $G$. The state
space for this agent is the discrete set of abstract states $[N]$, and the
action space is defined by $N_{G}(s)$ when the agent is in state $s\in[N]$,
corresponding to the options available at that abstract state. This abstract
agent’s policy is trained to maximize the environment reward by selecting
which option to follow at each abstract state (replace line 18 with this
agent). Thus, in the online setting we terminate the algorithm (line 24) upon
convergence of the abstract agent to a high return policy. Note that the
reward the abstract agent receives upon selecting an option is the sum of
environment rewards along the temporally extended sequence of primitive
actions (e.g., as is standard in Sutton et al. [1999]).

Algorithm 1 Discrete State-Action Abstraction (DSAA)

1:Input: $N$, E_iters, MDP
$\mathcal{M}=(\mathcal{X},\mathcal{A},p,r,\gamma,x_{0})$

2:Initialize $\phi,\psi,\pi,s_{0}=\phi(x_{0})$

3:repeat

4: Initialize abstract graph as $G=(V=[N],E=I_{n})$ $\triangleright$ Each node
has a self loop

5: Initialize replay buffer as $\mathcal{D}=\emptyset$

6: # EXPLORATION PHASE:

7: $\triangleright$ The option (abstract goal) is chosen from the neighborhood
of the current abstract state

8: $s_{g}=$ random.choice($N_{G}(s_{0})$)

9: for $i=0$ to E_iters do $\triangleright$ In this loop we explore the
environment using options

10: $a_{i}\sim\pi_{s_{i},s_{g}}(x_{i})$ $\triangleright$ Select action from
corresponding option

11: $x_{i+1}\sim p(\cdot|x,a_{i})$ $\triangleright$ Step in the environment

12: $s_{i+1}=\phi(x_{i+1})$ $\triangleright$ Compute next abstract state

13: $r_{i}=\delta_{s_{i+1},s_{g}}$ $\triangleright$ Compute option reward: $1$
if we transitioned to $s_{g}$

14: Add $(x_{i},a_{i},x_{i+1},r_{i})$ to $\mathcal{D}$ $\triangleright$ Save
transition

15: $E=E\cup(s_{i},s_{i+1})$ $\triangleright$ Add edge to graph if it didn’t
exist

16: Update option $\pi_{s_{i},s_{g}}$ with batch from $\mathcal{D}$
$\triangleright$ RL update (Soft Q-Learning)

17: if $s_{i}\neq s_{i+1}$ then $\triangleright$ Start new option if abstract
state changes

18: $s_{g}=$ random.choice($N_{G}(s_{i+1})$)

19: end if

20: end for

21: # ABSTRACTION UPDATE PHASE:

22:
$\mathcal{L}_{A}(\phi,\psi;\mathcal{D})=\mathcal{L}_{H}(\phi;\mathcal{D})+\mathcal{L}_{SR}(\psi;\phi,\mathcal{D})$
$\triangleright$ Compute loss as Eq. (1) + Eq. (2)

23: Update $(\phi,\psi)\in\arg\min\mathcal{L}_{A}$

24:until stopping criterion (see Section 4)

###  4.1 Abstraction $\phi$

We implement our discrete abstraction as a feed-forward neural network encoder
$\phi:\mathcal{X}\to\Delta(N)$ which maps observations $x\in\mathcal{X}$ to
points on the $N$-simplex (i.e., a distribution over the $N$ abstract states).
Our model resembles the encoder of a Discrete Variational Autoencoder (D-VAE),
where we employ the Gumbel-Softmax [Jang et al., 2017] to sample from the
output distribution. Thus, we regularize the distribution, as in a VAE, to be
similar to a uniform distribution prior $\mathcal{U}(N)$ over $N$ discrete
states, by minimizing the KL-Divergence over a dataset of collected samples
$\mathcal{D}$,

$\mathcal{L}_{H}(\phi;\mathcal{D})=\mathbb{E}_{x\sim\mathcal{D}}KL(\phi(\cdot\mid
x)\mid\mid\mathcal{U}(N))=\mathbb{E}_{x\sim\mathcal{D}}[-H(\phi(\cdot\mid
x))],$ (1)

which is equivalent to maximizing the entropy of the discrete distribution
over $\mathcal{D}$.

Intuitively, we motivate the use of such regularization as follows: a uniform
prior encourages a partition of the environment into similar sized parts.
Segmenting the state space into abstract states, or regions, turns our
original large MDP into a set of smaller sub-problems restricted to the pre-
image of each abstract state. From a divide and conquer perspective, we would
prefer these problems to be approximately the same size. Otherwise, we may end
up with very large abstract states, in which computing the seemingly local
options between pairs of abstract states is just as hard as solving the
original task.

To minimize (1), the Gumbel-Softmax allows us to sample from the distribution
while still backpropogating through it. We qualitatively verify in Appendix A
that this is necessary, and that treating $\phi$ as a classifier using simple
softmax activation is insufficient for learning a useful abstraction.

###  4.2 Successor Representation $\psi$

We use the successor representation as a _decoder_ to the encoder $\phi$. We
recall that the SR is a policy conditioned representation of the state space
in which the difference in SR between states indicates divergent policy
behavior at those states. Therefore, our approach is to find a clustering of
the state space such that states within the same cluster have similar SR in a
provided dataset – similar to what has been done in Ramesh et al. [2019]. We
train the successor representation function $\psi$ using the TD error:

$\mathcal{L}_{SR}(\psi;\phi,\mathcal{D})=\mathbb{E}_{x,x^{\prime}\sim
D}\left[\psi(\phi(x))-(\phi(x)+\gamma\psi(\phi(x^{\prime})))^{-}\right]^{2}$
(2)

where $(\cdot)^{-}$ indicates a fixed target, treated as a constant during
back-propagation.

We implement the SR as a feed-forward neural network
$\psi:\Delta(N)\to\mathbb{R}^{N}$, a relaxation of
$\psi:[N]\to\mathbb{R}^{N}$. Back-propagating through $\psi$ into the many-to-
one $\phi$ ensures that when $\phi$ maps two input states to the same abstract
state, those inputs should have similar SR under the induced abstraction. We
note that zero is a fixed point of Eq. (2), meaning $\phi\equiv 0$ and
$\psi\equiv 0$ minimizes the loss. As a consequence, prior works tend to
compute their abstraction in other ways, e.g., by not back-propagating through
$\psi$ [Machado et al., 2018] or resorting to dense regularization such as
observation reconstruction [Kulkarni et al., 2016]. Our work shows that with a
discrete abstraction, and with Eq. (1) as regularization, we can avoid the
degenerate case $\phi\equiv 0$ and $\psi\equiv 0$.

###  4.3 Abstract Graph

The set of edges $E$ of $G$ is expressed as an adjacency matrix which is
initialized to the identity, meaning each vertex has a single self-loop edge.
This self-loop edge lets us perform a lazy random walk over the abstract
graph, allowing some probability of exploring within the same abstract state.
In the online learning case we provide the self loop option policy with the
environment reward. We observe that our algorithm introduces an exploration-
exploitation trade-off: at each step the agent either _explores_ within its
current abstract state, or _exploits_ a trained option to reach a new abstract
state.

During the exploration phase, much like Roderick et al. [2018], if we
transition between two abstract states such that $(s,s^{\prime})\not\in E$, we
add this directed edge to $E$. The abstract graph is reset each time the
abstraction is updated. Once an edge $(s,s^{\prime})$ is added to $E$, we
begin updating the option policy $\pi_{s,s^{\prime}}$.

(a) 4 abstract states

(b) 8 abstract states

(c) 16 abstract states with noisy input

(d) Samples by percent explored

(e) State occupancy for random exploration versus exploration with options. We
also report the entropy $H$ of the normalized occupancy.

Figure 3: The FourRooms environment experiments [Sutton et al., 1999]. In the
top row we show examples of learned abstractions for different numbers of
fixed abstract states. In each subfigure, each abstract state is represented
by a different number-color pair. Note that each abstract state corresponds to
an intuitive cluster in the SR space; for example, no abstract state extends
to more than one room. In 3(c) we show abstraction robustness to noise. In the
bottom left we compare exploration efficiency between our method and uniform
random exploration. In the bottom right we show that if episode length is
restricted to 100 steps, a uniform random policy (left) spends much time near
the start, and struggles to reach the entire environment, whereas our learned
abstraction (right) helps the agent explore diverse states efficiently.

##  5 Experiments

###  5.1 FourRooms: unsupervised exploration in a discrete space

We first test our method in a discrete environment where we report
unsupervised learning results, meaning exploration efficiency in the absence
of environment reward.

In FourRooms the agent’s space is discrete and composed of four rooms
connected by small hallways. This classic problem demonstrates the difficulty
of RL agents in navigating through narrow passages. We visually show that our
learned abstraction intuitively partitions the environment based on bottleneck
states (Figures 3(a) and 3(b) show resulting abstractions for $N=4$ and $8$
respectively). Moreover, DSAA exhibits better exploration sample efficiency
than a baseline, reaching a larger set of states in the environment in a more
diverse manner (Figures 3(d) and 3(e)). We emphasize that uniform random
exploration is the behavior of the optimal uninformed agent in the absence of
intrinsic motivation.

We also show our method’s robustness to noise: despite augmenting the input
state with uniformly sampled noise each time the agent steps in the
environment, our method is still capable of learning a qualitatively useful
abstraction, in which, for example, each abstract state is a connected region
(Figure 3(c)). Noise is a distracting feature that makes every state seem
different to an RL agent, weakening the credit assignment necessary to
propagate values from regions of high to low reward. One desirable property of
any abstraction method is to filter such features out.

###  5.2 Arm2D: solving tasks in a continuous space

(a) Arm2D environment

(b) Abstract state visualization

(c) Example abstract graph with 8 abstract states

(d) Sample arm positions in different abstract states

(e) Online RL comparison results

Figure 4: Arm2D environment experiments. In (a) we visualize the 5D continuous
state space composed of 3 joint angles and the 2D object center position. In
each task the agent receives a sparse reward only when it moves the object
below a certain height in the vertical y-direction. In (d) we visualize arm
positions at boundaries between abstract states; and in (b) we show a 2D slice
of the 5D state, colored by abstract state. In (c) we visualize an example
abstract graph for 8 abstract states. Finally in (e) we show return curves for
DSAA and a Soft Q-Learning baseline on both the "hard" and "easy" tasks shown
in (a), averaged over 10 random seeds. The standard deviations, which we only
visualize for the "easy" task, demonstrate the consistency of our method.
Although the options can be used for transfer learning, we show in (e) that
our method improves over a baseline agent even in the online setting.

In these second set of experiments we show the capability of DSAA to solve
tasks online in a continuous control environment.

In Arm2D we design a continuous control task for a three joint manipulator
(robot arm) on a 2D-plane. The arm must move an object (ball) down below a
certain height, despite the arm starting below the ball, at which point only
it obtains a non-zero reward: the lower the height the ball must reach, the
more difficult the task, since the sparse reward is more distant; see Figure
4(a). The state space is 5-dimensional, including the three joint angles and
the 2D object position. For a more detailed description of the Arm2D
environment, we refer to Subsection B.1 of the appendix.

One aspect which makes these tasks difficult for an RL agent is the one
directional nature of certain actions: if during exploration an action pushes
the object up, the task becomes significantly harder for the rest of the
episode. In Figure 4(b) we visually show a projection of our learned
abstraction onto the first two arm joints, where each abstract state is
colored differently and the other three dimensions are held fixed. In Figure
4(d) we visualize examples of arm positions at option boundaries. Finally in
Figure 4(e) we show a comparison between DSAA and a baseline Soft Q-Learning
agent at solving environment tasks online. DSAA solves tasks in the
environment with fewer samples from the environment and more consistently than
the baseline. Moreover, these results are in the online setting, which shows
our learned options are not just reusable but help solving tasks as they are
learned.

In Subsection B.2 of the appendix we provide specific implementation details
for $\phi$, $\psi$, and $\pi_{s,s^{\prime}}$ in the context of these
experiments.

##  6 Conclusions and Future work

We presented a new method to perform a discrete abstraction on the state and
action spaces of a reinforcement learning problem using a dataset of
transitions in the environment. Our model is based on the successor
representation, grouping states together if the agent’s behavior after
visiting those states is similar. We showed that despite past concerns that
gradients from the successor representation would drive any abstraction
towards a trivial fixed point, with simple max entropy regularization we can
learn a useful abstraction in the discrete setting and in an end-to-end
manner. Finally, we showed that a DSAA agent equipped with our action
abstraction, provided as a set of options, navigates the environment more
efficiently than a baseline, making our abstraction useful for solving
downstream tasks.

A limitation of our work is that while our method may help an agent explore
the environment efficiently, that exploration is still heavily biased towards
the region that has already been explored. Performing a random walk in
abstract space limits the rate at which the abstraction extends to cover more
of the original state space across iterations. Given this limitation, one
direction for improvement is to train a higher level policy to bias the walk
over the abstract graph towards states which have been identified as the
frontier (e.g., see Sekar et al. [2020]).

Finally, as an interesting line of future work, it would be relevant to
provide theoretical guarantees on the convergence of clustering by successor
representation, as well as other properties related to our method.

## References

  * Abel et al. [2020] David Abel, Nate Umbanhowar, Khimya Khetarpal, Dilip Arumugam, Doina Precup, and Michael Littman.  Value preserving state-action abstractions.  In _International Conference on Artificial Intelligence and Statistics_ , pages 1639–1650. PMLR, 2020.
  * Tang and Hauser [2019] Gao Tang and Kris Hauser.  Discontinuity-sensitive optimal control learning by mixture of experts.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 7892–7898. IEEE, 2019.
  * Kavraki et al. [1996] Lydia E Kavraki, Petr Svestka, J-C Latombe, and Mark H Overmars.  Probabilistic roadmaps for path planning in high-dimensional configuration spaces.  _IEEE Transactions on Robotics and Automation_ , 12(4):566–580, 1996.
  * Machado et al. [2018] Marlos C Machado, Clemens Rosenbaum, Xiaoxiao Guo, Miao Liu, Gerald Tesauro, and Murray Campbell.  Eigenoption discovery through the deep successor representation.  In _6th International Conference on Learning Representations_ , 2018\.
  * Sutton et al. [1999] Richard S Sutton, Doina Precup, and Satinder Singh.  Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.  _Artificial intelligence_ , 112(1-2):181–211, 1999.
  * Bagaria and Konidaris [2019] Akhil Bagaria and George Konidaris.  Option discovery using deep skill chaining.  In _7th International Conference on Learning Representations_ , 2019\.
  * Kulkarni et al. [2016] Tejas D Kulkarni, Ardavan Saeedi, Simanta Gautam, and Samuel J Gershman.  Deep successor reinforcement learning.  _arXiv preprint arXiv:1606.02396_ , 2016.
  * Şimşek et al. [2005] Özgür Şimşek, Alicia P Wolfe, and Andrew G Barto.  Identifying useful subgoals in reinforcement learning by local graph partitioning.  In _International Conference on Machine Learning_ , pages 816–823, 2005.
  * Ichter et al. [2020] Brian Ichter, Edward Schmerling, Tsang-Wei Edward Lee, and Aleksandra Faust.  Learned critical probabilistic roadmaps for robotic motion planning.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 9535–9541. IEEE, 2020.
  * Oudeyer and Kaplan [2009] Pierre-Yves Oudeyer and Frederic Kaplan.  What is intrinsic motivation? a typology of computational approaches.  _Frontiers in neurorobotics_ , 1:6, 2009.
  * Strehl and Littman [2008] Alexander L Strehl and Michael L Littman.  An analysis of model-based interval estimation for markov decision processes.  _Journal of Computer and System Sciences_ , 74(8):1309–1331, 2008.
  * Bellemare et al. [2016] Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos.  Unifying count-based exploration and intrinsic motivation.  _Advances in Neural Information Processing Systems_ , 29, 2016.
  * Sekar et al. [2020] Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak.  Planning to explore via self-supervised world models.  In _International Conference on Machine Learning_ , pages 8583–8592. PMLR, 2020.
  * Schaul et al. [2015] Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver.  Universal value function approximators.  In _International Conference on Machine Learning_ , pages 1312–1320. PMLR, 2015.
  * Andrychowicz et al. [2017] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba.  Hindsight experience replay.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Nasiriany et al. [2019] Soroush Nasiriany, Vitchyr Pong, Steven Lin, and Sergey Levine.  Planning with goal-conditioned policies.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Vezhnevets et al. [2017] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu.  Feudal networks for hierarchical reinforcement learning.  In _International Conference on Machine Learning_ , pages 3540–3549. PMLR, 2017.
  * Machado et al. [2021] Marlos C Machado, Andre Barreto, and Doina Precup.  Temporal abstraction in reinforcement learning with the successor representation.  _arXiv preprint arXiv:2110.05740_ , 2021.
  * Erraqabi et al. [2021] Akram Erraqabi, Mingde Zhao, Marlos C Machado, Yoshua Bengio, Sainbayar Sukhbaatar, Ludovic Denoyer, and Alessandro Lazaric.  Exploration-driven representation learning in reinforcement learning.  In _ICML 2021 Workshop on Unsupervised Reinforcement Learning_ , 2021\.
  * Li et al. [2021] Siyuan Li, Lulu Zheng, Jianhao Wang, and Chongjie Zhang.  Learning subgoal representations with slow dynamics.  In _9th International Conference on Learning Representations_ , 2021\.
  * Jonschkowski and Brock [2015] Rico Jonschkowski and Oliver Brock.  Learning state representations with robotic priors.  _Autonomous Robots_ , 39(3):407–428, 2015.
  * Giannakopoulos et al. [2021] Petros Giannakopoulos, Aggelos Pikrakis, and Yannis Cotronis.  Neural discrete abstraction of high-dimensional spaces: A case study in reinforcement learning.  In _2020 28th European Signal Processing Conference (EUSIPCO)_ , pages 1517–1521. IEEE, 2021.
  * Ramesh et al. [2019] Rahul Ramesh, Manan Tomar, and Balaraman Ravindran.  Successor options: An option discovery framework for reinforcement learning.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)_ , pages 3304–3310, 2019.
  * Jang et al. [2017] Eric Jang, Shixiang Gu, and Ben Poole.  Categorical reparameterization with gumbel-softmax.  In _5th International Conference on Learning Representations_ , 2017\.
  * Roderick et al. [2018] Melrose Roderick, Christopher Grimm, and Stefanie Tellex.  Deep abstract q-networks.  In _AAMAS ’18: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems_ , pages 131–138, 2018.

## Appendix

##  Appendix A Additional ablations for the FourRooms environment

(a) Softmax with $\beta=1$

(b) Softmax with $\beta=10$

(c) Softmax with $\beta=100$

Figure 5: In these experiments we learn an abstraction with $4$ abstract
states. As demonstrated in Figure 3(a), each abstract state should intuitively
correspond to one of the rooms. Replacing Gumbel-Softmax with Softmax leads to
poor abstractions: we observe that abstract states occupy more than one room,
including the bottleneck connections between rooms, or are even disjoint
across two rooms (e.g., abstract state $0$ in (b)), which did not occur with
Gumbel-Softmax. Moreover, we observe that the size of each abstract state
remains unbalanced even with large weight coefficient $\beta$ for the entropy
term in the loss.

In the set of ablations shown in Figure 5, we qualitatively explore the
effects of certain implementation decisions on the resulting abstraction in
the FourRooms environment. In particular we show the importance of stochastic
categorical reparameterization via Gumbel-Softmax as opposed to
straightforward classification via Softmax, applied to the output logits of
the abstraction model $\phi$. We also explore the effects of adding a weight
coefficient $\beta>0$ to the entropy term in the abstraction update loss as
$\mathcal{L}_{A}=\mathcal{L}_{H}+\beta\mathcal{L}_{SR}$.

##  Appendix B The Arm2D environment

###  B.1 Description

We provide more detail about the Arm2D environment of Section 5.2. The action
space consists of 6 discrete actions: each one of the 3 arm joints has two
actions which increment or decrement its angle by $\delta$ degrees. For our
specific simulations we set $\delta=1$ degree. Each joint is limited between
$-90$ and $90$ degrees. The state spaces consists of 5 features: the 3 joint
angles and the (x,y) coordinate of the ball. While the joint angles are
discretized (by $\delta$), the ball location remains continuous. Each episode
was limited to 5000 steps at which point we reset the environment by bringing
the arm back to joint angles $[0,0,0]$ (horizontal position of the arm), and
the ball to $(x,y)=(13,13)$. The fixed end of the robot arm is located at
position $(0,0)$. The environment reward is a sparse 0 or 1, where 1 is only
given to the agent if it moves the ball below the green line ("easy" task at
height $y=11$) or red line ("hard" task at height $y=9$) shown in Figure 4(a).

###  B.2 Model architectures

Our algorithm trains three models: $\phi,\psi,\pi_{s,s^{\prime}}$. We
structure all three as simple feed-forward neural networks. We report the
specific values used for the experiments in Subsection 5.2

  * •

$\phi:\mathcal{X}\to\Delta(N)$ has two hidden layers with 128 and 256 neurons,
and LeakyReLU activation functions. The input is the 5 dimensional primitive
state $x$, and the output has $N$ neurons, one for each abstract state, which
is passed through a Gumbel-Softmax activation with temperature parameter
$\tau$. For these experiments we used $N=8,\tau=0.5$.

  * •

$\psi:\Delta(N)\to\mathbb{R}^{N}$ has two hidden layers with 64 and 128
neurons and LeakyReLU activation functions. It takes as input the output from
$\phi$ and maps it to $N$ neurons output.

  * •

Finally, we group our option policies as a single model with shared parameters
except for the final output layer. More precisely, we train a neural network
with four hidden layers of 64, 128, 256, 512 neurons and LeakyReLU activation.
It takes as input $x$ concatenated with $s=\phi(x)$, which in our case of
$N=8$ abstract states with state vector of dimension $5$, results in $8+5=13$
input states. Then for each option $\pi_{s,s^{\prime}}$ we have a single
linear layer with no activation that maps from the 512 neuron embedding to the
discrete actions of our agent, which in our case of $N=8$ abstract states and
$|\mathcal{A}|=6$ discrete actions, results in $8\times 6=48$ outputs. We note
there are other ways we can model the option policies, for example by not
sharing parameters between each option policy.

###  B.3 Hyper-parameter details

We now describe the hyperparameter settings we used in the experiments shown
in Figure 4(e). We highlight that we did not conduct a large scale parameter
search for the optimal configuration of the hyperparameters. In this sense our
results demonstrate our algorithm without excessive tuning.

We roll out each episode for $5000$ steps and set the replay buffer size for
the option policy to $100,000$ transitions ($20$ episodes). This is the same
replay buffer which we use to train the abstraction, and consequently we set
the number of exploration steps (E_iters in Algorithm 1) to $100,000$ as well.
We use Adam optimizer with learning rate $0.001$ and batch size $512$ for
updating both $\pi_{s,s^{\prime}}$ and $\phi,\psi$ (which are updated in
tandem as an encoder-decoder pair). Both the update equations for Q-learning
and the successor representation have a discount factor $\gamma$, which we set
to $0.95$. We use a target Q model for computing the temporal difference
target, which updates with a delay behind the online model of 20 iterations.
We reward each option policy when it transitions to the correct next abstract
state with 200 reward points – this reward needs to be high enough to balance
the entropy based reward the soft policy receives on each transition, but is
otherwise arbitrary.

We implement our models using PyTorch, without any external learning libraries
or outside implementations of Soft Q-learning. We run on CPU using an Intel(R)
Core(TM) i9-10980XE CPU @ 3.00GHz. The code can be found as part of the
Supplementary Material.