# Two-dimensional weak-type topological insulators in inversion symmetric
crystals

Sunam Jeon  Department of Energy Science, Sungkyunkwan University, Suwon
16419, Korea  Youngkuk Kim<EMAIL_ADDRESS>Department of Physics,
Sungkyunkwan University, Suwon 16419, Korea

###### Abstract

The Su-Schrieffer-Heeger (SSH) chain is an one-dimensional lattice that
comprises two dimerized sublattices. Recently, Zhu, Prodan, and Ahn (ZPA)
proposed in [L. Zhu, E. Prodan, and K. H. Ahn, Phys. Rev. B 99, 041117 (2019)]
that one-dimensional flat bands can occur at topological domain walls of a
two-dimensional array of the SSH chains. Here, we newly suggest a two-
dimensional topological insulator that is protected by inversion and time-
reversal symmetries without spin-orbit coupling. It is shown that the two-
dimensional SSH chains realize the proposed topological insulator. Utilizing
the first Stiefel-Whitney numbers, a weak type of $\mathbb{Z}_{2}$ topological
indices are developed, which identify the proposed topological insulator,
dubbed a two-dimensional Stiefel-Whitney insulator (2DSWI). The ZPA model is
employed to study the topological phase diagrams and topological phase
transitions. It is found that the phase transition occurs via the formation of
the massless Dirac points that wind the entire Brillouin zone. We argue that
this unconventional topological phase transition is a characteristic feature
of the 2DSWI, manifested as the one-dimensional domain wall states. The new
insight from our work could help efforts to realize topological flat bands in
solid-state systems.

Introduction - Recently, there has been a surge of interest in the correlated
electrons in flat bands Wu  _et al._ (2007); Wong  _et al._ (2013); Biesenthal
_et al._ (2019); Balents  _et al._ (2020). In particular, twisted bilayer
graphene Lopes dos Santos  _et al._ (2007); Dean  _et al._ (2010); Suárez
Morell  _et al._ (2010); Bistritzer and MacDonald (2011); Wang  _et al._
(2013); Kim  _et al._ (2016), in which interlayer twist angle tunes the
bandwidth to zero, has spurred a great deal of community interest, exhibiting
strong correlations of electrons and nontrivial band topology Cao  _et al._
(2018a, b); Song  _et al._ (2019); Brihuega  _et al._ (2012). The advent of
magic angle twisted bilayer graphene has renewed the interest in the
realization of flat bands in diverse systems Heikkilä  _et al._ (2011); Bi
_et al._ (2019); Hwang  _et al._ (2021). Notable examples include the recent
study by Zhu, Prodan, and Ahn (ZPA) Zhu  _et al._ (2019). Utilizing the two-
dimensional (2D) array of the Su-Schrieffer-Heeger (SSH) chains Su  _et al._
(1979, 1980); Heeger  _et al._ (1988), ZPA have shown that one-dimensional
(1D) flat bands can arise, being stabilized via the formation of topological
domain walls (DWs). Encouragingly, the proposed 1D flat bands have been
experimentally confirmed in a mechanical metamaterial Qian  _et al._ (2020),
whereas their condensed-matter realization is yet to be discovered.

Regarding efforts to search for topological materials, these days have
witnessed remarkable developments, based on topological quantum chemistry and
symmetry indicators Fang  _et al._ (2012); Bradlyn  _et al._ (2017); Po  _et
al._ (2017); Khalaf  _et al._ (2018); Ono and Watanabe (2018); Tang  _et al._
(2019a, b). Due to these high-throughput approaches, mass topological
materials have been catalogued Tang  _et al._ (2019c); Zhang  _et al._ (2019);
Vergniory  _et al._ (2019, 2021), started with the seminal work by Fu and Kane
Fu and Kane (2007). The Fu-Kane formula expresses the $\mathbb{Z}_{2}$
topological indices using parity eigenvalues, enabling the discovery of
archetypal topological materials, such as Bi1-xSbx Hasan and Moore (2011);
Hsieh  _et al._ (2008) and Bi2Se3 Xia  _et al._ (2009); Zhang  _et al._
(2009). Later, the strong $\mathbb{Z}_{2}$ index has been extended to the
$\mathbb{Z}_{4}$ index $\nu_{0}$, including a wider class of topological
phases with and without spin-orbit coupling (SOC) Benalcazar  _et al._ (2017);
Khalaf  _et al._ (2018); Langbehn  _et al._ (2017); Song  _et al._ (2017);
Schindler  _et al._ (2018); Song  _et al._ (2020). For example, without SOC,
$\nu_{0}$ = 2 characterizes a $\mathbb{Z}_{2}$ monopole nodal-line semimetal
in three dimensions Ahn  _et al._ (2018); Wang  _et al._ (2019), whereas it
characterizes higher-order topological insulators in two dimensions Park  _et
al._ (2019); Lee  _et al._ (2020). Interestingly, an insulators with
$\nu_{0}=0$ can contain nontrivial band topology, similar to inversion-
symmetric topological insulators Hughes  _et al._ (2011); Alexandradinata  _et
al._ (2014). However, this opportunity to encounter a novel topological phase
has remained widely unaddressed to date.

In this Letter we propose and characterize a novel topological class of
insulators with $\nu_{0}=0$ under inversion $\mathcal{P}$ and time-reversal
$\mathcal{T}$ symmetries without SOC, dubbed the 2D Stiefel-Whitney insulator
(2DSWI). Utilizing the first Stiefel-Whitney (SW) numbers, we develop two weak
$\mathbb{Z}_{2}$ indices ($\nu_{1}\nu_{2}$), which complement the strong
$\mathbb{Z}_{2}$ index $\nu_{0}=0$ and identify the 2DSWIs. Employing the ZPA
model, we show that the 2DSWI is realized in the 2D SSH chains under various
centrosymmetric deformations. The topological phase diagrams are found to
facilitate four distinct 2DSWI phases, indexed by
$(\nu_{0};\nu_{1}\nu_{2})=(0;00),(0;01),(0;10)$, and $(0;11)$, respectively.
In between distinct 2DSWIs, a topological semimetal with $\nu_{0}=1$ appears,
featuring a pair of gapless Dirac points (DPs). We show that the DPs mediate a
topological phase transition by winding the Brillouin zone (BZ), and their
trajectories manifest as 1D DW states between distinct 2DSWIs.

$\mathbb{Z}_{2}$ topological indices - We consider a 2D Bloch Hamiltonian
$\mathcal{H}({\boldsymbol{k}})$ that is invariant under $\mathcal{PT}$ and
$\mathcal{T}^{2}=1$. Since
$\left[\mathcal{PT},\mathcal{H}({\boldsymbol{k}})\right]=0$, 1D families of
real Hamiltonians on a closed loop $\mathcal{C}$ in the BZ,
$\mathcal{H}({\boldsymbol{k}})|_{{\boldsymbol{k}}\in\mathcal{C}}$, can be
characterized by the $\mathbb{Z}_{2}$-quantized first SW number
$\nu_{\mathcal{C}}=\frac{1}{\pi}P\oint_{\mathcal{C}}{\boldsymbol{A}}({\boldsymbol{k}})\cdot
d{\boldsymbol{k}}$ Ahn  _et al._ (2019), where $P$ is the path-ordering
operator, ${\boldsymbol{A}}({\boldsymbol{k}})=i\langle
u({\boldsymbol{k}})|{\boldsymbol{\nabla}}_{{\boldsymbol{k}}}|u({\boldsymbol{k}})\rangle$
is the Berry connection, and $u({\boldsymbol{k}})$ is the cell-periodic part
of the occupied Bloch function at ${\boldsymbol{k}}\in$ BZ. The number refers
to a topological invariant that characterizes the twist of real Bloch states
in momentum space Ahn  _et al._ (2019). The Zak phase Zak (1989) that is
$\mathbb{Z}_{2}$-quantized by the reality condition corresponds to the first
SW number. The SW number is defined for any given $k_{1}$ and $k_{2}$ lines as
$\nu_{1}(k_{1})=\frac{1}{\pi}P\int_{-\pi}^{\pi}A_{1}(k_{1},k_{2})dk_{2}$ and
$\nu_{2}(k_{2})=\frac{1}{\pi}P\int_{-\pi}^{\pi}A_{2}(k_{1},k_{2})dk_{1}$,
where $A_{i}$ ($i$ = 1, 2) is $k_{i}$-component of
${\boldsymbol{A}}({\boldsymbol{k}})$. In the presence of a direct band gap,
one-parameter families of real Hamiltonians on a $k_{i}$ line ($i=1,2$) are
adiabatically connected to those on $k^{\prime}_{i}$ line for any
$k^{\prime}_{i}\neq k^{\prime}_{i}$, resulting in the equivalent SW numbers
$\nu_{i}(k_{i})\equiv\nu_{i}(k^{\prime}_{i})$. Therefore, the $\mathbb{Z}_{2}$
indices $\nu_{1}$ and $\nu_{2}$ can unambiguously characterize a
$\mathcal{PT}$-symmetric insulating phase.

Figure 1: Diagrams depicting possible 2DSWIs indexed by ($\nu_{1}\nu_{2}$) (a)
2D bulk BZ and 1D edge BZ. $\Gamma_{i}$ are TRIMs. (b) TRIMs with
$\xi_{i}=+1(-1)$ are denoted by open (closed) circles at $\Gamma_{i}$. The
thick (red) line indicates possible edge states on the $(\nu_{1}\nu_{2})$
edge.

Inversion-symmetry indicators - The parity eigenvalues
$\xi_{n}(\Gamma_{a})=\pm 1$ of the occupied Bloch states at the four time-
reversal invariant momenta (TRIMs) $\Gamma_{a}$ [See Fig. 1(a)] provide a
symmetry indicator that diagnoses the 2DSWIs. The SW numbers $\nu_{1}$ and
$\nu_{2}$ on the $k_{1}=0$ ($\pi$) and $k_{2}$ = 0 ($\pi$) loops satisfy

$\displaystyle(-1)^{\nu_{1}}=\xi_{a}\xi_{d}~{}(\xi_{b}\xi_{c});~{}~{}(-1)^{\nu_{2}}=\xi_{a}\xi_{b}~{}(\xi_{d}\xi_{c}),$
(1)

where

$\displaystyle\xi_{a}=\prod_{n}\xi_{n}(\Gamma_{a}).$ (2)

($\nu_{1}\nu_{2}$) are well-defined if $\xi_{a}\xi_{d}=\xi_{b}\xi_{c}$ and
$\xi_{a}\xi_{b}=\xi_{d}\xi_{c}$, guaranteed by the strong $\mathbb{Z}_{2}$
index $\nu_{0}=0$ because

$\displaystyle(-1)^{\nu_{0}}=\xi_{a}\xi_{b}\xi_{c}\xi_{d}=1.$ (3)

Note that the strong index $\nu_{0}=0$ is also a necessary condition for an
insulating phase since the strong index $\nu_{0}=1$ dictates the presence of
the massless DPs in momentum space Kim  _et al._ (2015). Figure 1(b) shows
possible configurations of parity eigenvalues compatible with $\nu_{0}=0$. We
identify the 2DSWI as a $\mathcal{PT}$-symmetric topological insulator in
vanishing SOC, characterized by $\nu_{0}=0$ and $(\nu_{1}\nu_{2})\neq(00)$.

2DSWI as a stack of SSH chains - Similar to the weak topological insulators in
three dimensions Moore and Balents (2007); Fu and Kane (2007), which can be
viewed as a stack of 2D quantum spin Hall insulators, the 2DSWI with
$(\nu_{1}\nu_{2})\neq(00)$ can be viewed as a stack of 1D SSH chains along
${\boldsymbol{G}}_{\nu}=\nu_{1}{\boldsymbol{b}}_{1}+\nu_{2}{\boldsymbol{b}}_{2}$,
where ${\boldsymbol{b}}_{i}$ is the reciprocal lattice vector ($i=1,2$). In
stark contrast to the 3D weak topological insulators, however, by stacking the
SSH chains in two dimensions, every 1D line in the BZ has the first SW number.
This results in the boundary mode, for example, at any edge momentum
$\bar{k}_{i}$ when projected along the $k_{j}$-direction with $\nu_{i}=1$,
where $i$,$j$ = 1,2 and $i\neq j$. In general, the band topology of a 2DSWI
manifests as 1D topological states at the $(\nu_{1}\nu_{2})$ edge, as
illustrated in Fig. 1(b).

Figure 2:  Zhu-Prodan-Ahn model of the 2D SSH chains. (a) A primitive unit
cell of the 2D SSH chains, indicated by a grey parallelogram. The staggered
distortion, parameterized by ${\boldsymbol{d}}=(d_{x},d_{y})$, is applied to
the $A$ and $B$ sublattices. (b) Unit cell under a strain $e\neq 0$. Examples
of the 2D SSH chains: (c) Square lattice with $e=0$ and ${\boldsymbol{d}}=0$.
(d) Honeycomb lattice with $e=1/3$ and ${\boldsymbol{d}}=(1/3,0)$. The inter-
chain and intra-chain dimerizations are delineated by red and blue-colored
lines, respectively. A grey parallelogram indicates a primitive unit cell.

Zhu-Prodan-Ahn model - Using the ZPA model, we suggest a material realization
of the 2DSWI in 2D SSH chains. As shown in Fig. 2, a family of 2D SSH chains
are constructed by an alternating array of two SSH chains. The ZPA model
considers inter- and intra-chain dimerizations, parameterized by
${\boldsymbol{d}}$ and $e$. Here, ${\boldsymbol{d}}=(d_{x},d_{y})$ and $e$
describe an in-plane staggered distortion and a strain of the unit cell,
respectively [See Figs. 2(a) and 2(b)]. This family of 2D SSH chains includes
various 2D lattices, such as a square lattice with ${\boldsymbol{d}}=0$ and
$e=0$ [Figs. 2(c)] and a graphene-like honeycomb lattice with $e=1/3$,
${\boldsymbol{d}}=\left(1/3,0\right)$ [Fig. 2(d)].

A Hamiltonian, describing hopping of electrons in the 2D SSH chains, is given
by

$\displaystyle\mathcal{H}({\boldsymbol{k}})=h^{\ast}({\boldsymbol{k}})\sigma^{+}+h({\boldsymbol{k}})\sigma^{-},$
(4)

where $\sigma^{\pm}=\sigma_{x}\pm i\sigma_{y}$ are the Pauli matrices
associated with the sublattices and
$h({\boldsymbol{k}})=-t_{1}-t_{2}\mathrm{e}^{ik_{2}}-t_{3}\mathrm{e}^{-ik_{1}}-t_{4}\mathrm{e}^{-i(k_{1}+k_{2})}$.
The hopping parameters $t_{j}$ ($j=1,2,3,4$) are given by $t_{1}=1-e+2d_{x}$,
$t_{2}=1+e+2d_{y}$, $t_{3}=1-e-2d_{x}$, and $t_{4}=1+e-2d_{y}$, so that they
describe the deformations. Diagonalizing Eq. (4), one finds the following band
dispersion:

$\displaystyle E({\boldsymbol{k}})^{2}$
$\displaystyle=4(1+e^{2}+2d_{x}^{2}+2d_{y}^{2})+2\left[(1-e)^{2}-4d_{x}^{2}\right]\cos
k_{1}$ $\displaystyle+4\left[(1-e^{2})+4d_{x}d_{y}\right]\cos k_{2}$
$\displaystyle+4\left[(1-e^{2})-4d_{x}d_{y}\right]\cos(k_{1}+k_{2})$
$\displaystyle+2\left[(1+e)^{2}-4d_{y}^{2}\right]\cos(k_{1}+2k_{2}),$ (5)

where $k_{1}={\boldsymbol{k}}\cdot{\boldsymbol{a}}_{1}$ and
$k_{2}={\boldsymbol{k}}\cdot{\boldsymbol{a}}_{2}$ are dimensionless lattice
momenta. In good agreement with the ZPA results Zhu  _et al._ (2019), the
energy bands are calculated as fully gapped at
$(e,d_{x},d_{y})=\left(1/3,0,1/3\right)$ and
$(e,d_{x},d_{y})=\left(-1/3,1/3,0\right)$.

Figure 3: Topological phase diagrams with $\mathbb{Z}_{2}$ indices
($\nu_{0};\nu_{1}\nu_{2}$) in ${\boldsymbol{d}}$-space. (a) e = 0.25. (b) e =
-0.25. The colored (white) regions have the strong index $\nu_{0}=0$
($\nu_{0}=1$). (c) Trajectories of the two DPs ($D1$ and $D2$) in momentum
space along the path \small1⃝, \small2⃝, and \small3⃝ in (a). (d) Trajectories
along the path \small4⃝, \small5⃝, and \small6⃝ in (b).

Topological phase diagrams - Topological phase diagrams in
${\boldsymbol{d}}$-space are calculated using the inversion-symmetry
indicators ($\nu_{0};\nu_{1}$,$\nu_{2}$). Inversion operator, given by
$\mathcal{P}=\sigma_{x}$, readily evaluates the strong index $\nu_{0}$ as

$\displaystyle(-1)^{\nu_{0}}=\rm{sgn}\left[e\sin\left(\theta_{d}-\frac{\pi}{4}\right)\sin\left(\theta_{d}+\frac{\pi}{4}\right)\right].$
(6)

where $\theta_{d}=\tan^{-1}(d_{x}/d_{y})$. An insulating phase, facilitating
the 2DSWIs, is allowed when $\nu_{0}=0$ or, equivalently,
$e\sin\left(\theta_{d}-\frac{\pi}{4}\right)\sin\left(\theta_{d}+\frac{\pi}{4}\right)>0$.
A close inspection reveals that the bands are fully gapped when $\nu_{0}=0$.
Using the symmetry indicators, we obtain the weak indices ($\nu_{1}\nu_{2}$)
as

$\displaystyle(-1)^{\nu_{1}}=-\rm{sgn}\left[e\right]$ (7)

and

$\displaystyle(-1)^{\nu_{2}}=\rm{sgn}\left[\sin\left(\theta_{d}+\frac{\pi}{4}\right)\right].$
(8)

Figure 3 depicts the resultant phase diagrams. The phase boundaries always
occur at $|d_{y}|=|d_{x}|$, irrespective of $|{\boldsymbol{d}}|$ and $e$. The
sign of $e$ determines the metallic and insulating regions, depending on which
there exist two kinds of topological phase diagrams. For $e>0$, a metallic
(insulating) phase occurs in $|\theta_{d}|<\pi/4$ and $|\theta_{d}-\pi|<\pi/4$
($|\theta_{d}-\pi/2|<\pi/4$ and $|\theta_{d}+\pi/2|<\pi/4$) [Fig.3(a)]. For
$e<0$, the metallic and insulating phases switch the regions [Fig. 3(b)]. The
four insulating regions host distinct 2DSWIs with
($\nu_{0}$;$\nu_{1}$$\nu_{2}$) = (0;10), (0;11) [(0;01), and (0;00)] for $e>0$
($e<0$), respectively.

Topological phase transitions - The DPs that occur when $\nu_{0}=1$ are
stable, existing in the finite regions of ${\boldsymbol{d}}$-space
$e\sin\left(\theta_{d}-\frac{\pi}{4}\right)\sin\left(\theta_{d}+\frac{\pi}{4}\right)<0$.
The trajectories of the DPs as a function of the staggered dimerization angle
$\theta_{d}$ capture the topological phase transition of the 2DSWIs, driven by
the dimerization. Figures 3(c) and 3(d) show the trajectories of the DPs from
$\theta_{d}=-\pi/4$ to $\pi/4$ for $e=0.25$ and from $\theta_{d}=\pi/4$ to
$3\pi/4$ for $e=-0.25$, respectively. When $e>0$, the DPs traverse the BZ from
$\Gamma_{b}$ to $\Gamma_{c}$, winding the 2D BZ twice (once) along the
$k_{1}$\- ($k_{2}$-)direction [Fig. 3(c)]. On the other hand, when $e<0$, the
DPs travel from $\Gamma_{a}$ to $\Gamma_{d}$, winding the BZ once only along
the $k_{2}$-direction [Fig. 3(d)]. The BZ winding by DPs characterizes the
2DSWIs. The one-parameter families of real Hamiltonians on any $k_{i}$ line
($i$ = 1,2) defines a 1D topological insulator indexed by $\nu_{i}$. The weak
$\mathbb{Z}_{2}$ indices ($\nu_{1}\nu_{2}$) can change by changing all the
topological insulators at any $k_{i}\in[-\pi,\pi]$. This necessitates the
existence of the Dirac nodal line that winds the BZ during the phase
transition.

Figure 4: Atomic structure and electronic energy spectrum of the topological
DW in 2D SSH chains. (a) The atomic configuration of the DW geometry that maps
$\theta_{d}\in[0,2\pi]$ to $y$ under the cell-periodicity along $x$ with
${\boldsymbol{d}}=e=1/3$. The intensity represents the amplitude of the energy
eigenstates with $E=0$. $\theta_{d}$ is the staggered dimerization angle. (b)
Energy spectrum of the DW geometry. The intensity represents the localization
strength at the DW.

1D flat-band DW states - We argue that the ZPA 1D boundary modes are a
physical manifestation of the DPs that wind the BZ. ZPA have shown that the
DWs of the 2D SSH chain host the one-dimensional flat bands Zhu  _et al._
(2019), which we reproduce in Fig. 4. In view of the topological phase
transition, the DW geometry [Fig. 4(a)], in which the dimerization parameter
${\boldsymbol{d}}$ smoothly varies from $0$ to $2\pi$ along $y$, seamlessly
visits the four electronic phases of the phase diagram [Fig. 3(a)]. Therefore,
the topological phase transition should be captured in between the insulating
domains via the occurrence of two DPs that wind the edge BZ
$\overline{k}_{1}\in[-\pi,\pi]$ twice. Figure 4(b) shows the electronic energy
spectrum calculated from the DW structure. In good agreement with the previous
results Zhu  _et al._ (2019); Qian  _et al._ (2020), our calculations
reproduce the flat bands at $E=0$ in otherwise all gapped bulk states. We find
that there exist two mid-gap states per DW [See Fig. 4(b)], which cover the
whole $\overline{k}_{x}$ edge BZ twice. Associated with the topological phase
transition of $\nu_{1}$, these DW states can be considered as the projection
of the DP trajectories from $\theta_{d}=-\pi/4$ to $\pi/4$ along $k_{2}$.
Since the the DW states wind the $\overline{k}_{1}$ twice, the resulting
change of $\nu_{1}$ should be zero, which is in line with the adjacent 2DSWI
phases $(\nu_{0};\nu_{1}\nu_{2})=(0;11)$ and $0;10)$. This supports that the
1D DW states are a physical manifestation of the DPs that wind the BZ. We note
that the degeneracy of the zero modes are stabilized by the crystalline
symmetry of the DW geometry as discussed by ZPA Zhu  _et al._ (2019).

It is interesting to note that the trajectories of the DPs and, thus, the 1D
DW states can be viewed as a 1D projection of a Dirac nodal line that lives in
higher three dimensions. A 3D BZ can be constructed by adding additional
dimension from the polar coordinate $\theta_{d}\in[-\pi,\pi)$ to the 2D BZ.
Viewed from the three dimensions ($k_{1},k_{2},\theta_{d}$), one can consider
one-parameter families of the Hamiltonian
$\mathcal{H}(k_{1},k_{2},\theta_{d})$ in class AI Teo and Kane (2010), since
satisfy $[\mathcal{H}(k_{1},k_{2},\theta_{d}),\mathcal{PT}]=0$ and
$\left(\mathcal{PT}\right)^{-1}\theta_{d}\mathcal{PT}=\theta_{d}$. Therefore,
the strong index ($\nu_{0}=1$) is well-defined and dictate the presence of the
Dirac nodal line that threads the 2D BZ at $\theta_{d}=0$ ($\theta_{d}=\pi/2$)
for $e>0$ ($e<0$). This idea my help access higher-dimensional band topology
realized in a globally smooth spatially varying geometry of lower-dimensional
materials.

Conclusion - In summary, we have demonstrated that the combination of
inversion and time-reversal symmetries allows for the $\mathbb{Z}_{2}$
classification of topological insulators under vanishing spin-orbit
interactions. The proposed topological insulators are characterized by the
strong $\mathbb{Z}_{2}$ index $\nu_{0}=0$ and the weak $\mathbb{Z}_{2}$
indices $(\nu_{1}\nu_{2})\neq(00)$, indicated by parity eigenvalues at four
TRIMs. Diagnosing with the topological indices, we have shed light on the
centrosymmetric 2D SSH chains based on the ZPA model as a material realization
of the proposed topological insulators, dubbed the two-dimensional Stiefel-
Whitney insulators (2DSWIs). The 2DSWI features 1D topological boundary modes
as a physical manifestation of their characteristic topological phase
transitions. The proposed inversion-symmetry indicators can help identify the
2DSWIs in solid-state materials. Hopefully, our results simulate further
experimental and theoretical studies and lead to the discover a solid-state
platform that realizes topological flat bands.

Acknowledgement - This work was supported by National Research Foundation
(NRF) of Korea under the program of Basic Research Laboratory
(2020R1A4A3079707). Y.K. acknowledges the support from the NRF Grants
(2021R1A2C1013871). The computational resource was provided from the Korea
Institute of Science and Technology Information (KISTI) (KSC-2021-CRE-0116).

## References

  * Wu  _et al._ (2007) C. Wu, D. Bergman, L. Balents, and S. Das Sarma, Phys. Rev. Lett. 99, 070401 (2007).
  * Wong  _et al._ (2013) C. L. M. Wong, J. Liu, K. T. Law, and P. A. Lee, Phys. Rev. B 88, 060504 (2013).
  * Biesenthal  _et al._ (2019) T. Biesenthal, M. Kremer, M. Heinrich, and A. Szameit, Phys. Rev. Lett. 123, 183601 (2019).
  * Balents  _et al._ (2020) L. Balents, C. R. Dean, D. K. Efetov, and A. F. Young, Nature Physics 16, 725 (2020).
  * Lopes dos Santos  _et al._ (2007) J. M. B. Lopes dos Santos, N. M. R. Peres, and A. H. Castro Neto, Phys. Rev. Lett. 99, 256802 (2007).
  * Dean  _et al._ (2010) C. R. Dean, A. F. Young, I. Meric, C. Lee, L. Wang, S. Sorgenfrei, K. Watanabe, T. Taniguchi, P. Kim, K. L. Shepard, and J. Hone, Nature Nanotechnology 5, 722 (2010).
  * Suárez Morell  _et al._ (2010) E. Suárez Morell, J. D. Correa, P. Vargas, M. Pacheco, and Z. Barticevic, Phys. Rev. B 82, 121407 (2010).
  * Bistritzer and MacDonald (2011) R. Bistritzer and A. H. MacDonald, Proceedings of the National Academy of Sciences 108, 12233 (2011).
  * Wang  _et al._ (2013) L. Wang, I. Meric, P. Y. Huang, Q. Gao, Y. Gao, H. Tran, T. Taniguchi, K. Watanabe, L. M. Campos, D. A. Muller, J. Guo, P. Kim, J. Hone, K. L. Shepard, and C. R. Dean, Science 342, 614 (2013).
  * Kim  _et al._ (2016) K. Kim, M. Yankowitz, B. Fallahazad, S. Kang, H. C. P. Movva, S. Huang, S. Larentis, C. M. Corbet, T. Taniguchi, K. Watanabe, S. K. Banerjee, B. J. LeRoy, and E. Tutuc, Nano Letters 16, 1989 (2016).
  * Cao  _et al._ (2018a) Y. Cao, V. Fatemi, S. Fang, K. Watanabe, T. Taniguchi, E. Kaxiras, and P. Jarillo-Herrero, Nature 556, 43 (2018a).
  * Cao  _et al._ (2018b) Y. Cao, V. Fatemi, A. Demir, S. Fang, S. L. Tomarken, J. Y. Luo, J. D. Sanchez-Yamagishi, K. Watanabe, T. Taniguchi, E. Kaxiras, R. C. Ashoori, and P. Jarillo-Herrero, Nature 556, 80 (2018b).
  * Song  _et al._ (2019) Z. Song, Z. Wang, W. Shi, G. Li, C. Fang, and B. A. Bernevig, Phys. Rev. Lett. 123, 036401 (2019).
  * Brihuega  _et al._ (2012) I. Brihuega, P. Mallet, H. González-Herrero, G. Trambly de Laissardière, M. M. Ugeda, L. Magaud, J. M. Gómez-Rodríguez, F. Ynduráin, and J.-Y. Veuillen, Phys. Rev. Lett. 109, 196802 (2012).
  * Heikkilä  _et al._ (2011) T. T. Heikkilä, N. B. Kopnin, and G. E. Volovik, JETP Letters 94, 233 (2011).
  * Bi  _et al._ (2019) Z. Bi, N. F. Q. Yuan, and L. Fu, Phys. Rev. B 100, 035448 (2019).
  * Hwang  _et al._ (2021) Y. Hwang, J.-W. Rhim, and B.-J. Yang, “Flat bands with band crossings enforced by symmetry representation,” (2021), arXiv:2105.14919 [cond-mat.mes-hall] .
  * Zhu  _et al._ (2019) L. Zhu, E. Prodan, and K. H. Ahn, Phys. Rev. B 99, 041117 (2019).
  * Su  _et al._ (1979) W. P. Su, J. R. Schrieffer, and A. J. Heeger, Phys. Rev. Lett. 42, 1698 (1979).
  * Su  _et al._ (1980) W. P. Su, J. R. Schrieffer, and A. J. Heeger, Phys. Rev. B 22, 2099 (1980).
  * Heeger  _et al._ (1988) A. J. Heeger, S. Kivelson, J. R. Schrieffer, and W. P. Su, Rev. Mod. Phys. 60, 781 (1988).
  * Qian  _et al._ (2020) K. Qian, L. Zhu, K. H. Ahn, and C. Prodan, Phys. Rev. Lett. 125, 225501 (2020).
  * Fang  _et al._ (2012) C. Fang, M. J. Gilbert, and B. A. Bernevig, Phys. Rev. B 86, 115112 (2012).
  * Bradlyn  _et al._ (2017) B. Bradlyn, L. Elcoro, J. Cano, M. G. Vergniory, Z. Wang, C. Felser, M. I. Aroyo, and B. A. Bernevig, Nature 547, 298 (2017).
  * Po  _et al._ (2017) H. C. Po, A. Vishwanath, and H. Watanabe, Nature Communications 8 (2017), 10.1038/s41467-017-00724-z.
  * Khalaf  _et al._ (2018) E. Khalaf, H. C. Po, A. Vishwanath, and H. Watanabe, Phys. Rev. X 8, 031070 (2018).
  * Ono and Watanabe (2018) S. Ono and H. Watanabe, Phys. Rev. B 98, 115150 (2018).
  * Tang  _et al._ (2019a) F. Tang, H. C. Po, A. Vishwanath, and X. Wan, Science Advances 5, eaau8725 (2019a).
  * Tang  _et al._ (2019b) F. Tang, H. C. Po, A. Vishwanath, and X. Wan, Nature Physics 15, 470 (2019b).
  * Tang  _et al._ (2019c) F. Tang, H. C. Po, A. Vishwanath, and X. Wan, Nature 566, 486 (2019c).
  * Zhang  _et al._ (2019) T. Zhang, Y. Jiang, Z. Song, H. Huang, Y. He, Z. Fang, H. Weng, and C. Fang, Nature 566, 475 (2019).
  * Vergniory  _et al._ (2019) M. G. Vergniory, L. Elcoro, C. Felser, N. Regnault, B. A. Bernevig, and Z. Wang, Nature 566, 480 (2019).
  * Vergniory  _et al._ (2021) M. G. Vergniory, B. J. Wieder, L. Elcoro, S. S. P. Parkin, C. Felser, B. A. Bernevig, and N. Regnault, “All topological bands of all stoichiometric materials,” (2021), arXiv:2105.09954 .
  * Fu and Kane (2007) L. Fu and C. L. Kane, Phys. Rev. B 76, 045302 (2007).
  * Hasan and Moore (2011) M. Z. Hasan and J. E. Moore, Annual Review of Condensed Matter Physics 2, 55 (2011), https://doi.org/10.1146/annurev-conmatphys-062910-140432 .
  * Hsieh  _et al._ (2008) D. Hsieh, D. Qian, L. Wray, Y. Xia, Y. S. Hor, R. J. Cava, and M. Z. Hasan, Nature 452, 970 (2008).
  * Xia  _et al._ (2009) Y. Xia, D. Qian, D. Hsieh, L. Wray, A. Pal, H. Lin, A. Bansil, D. Grauer, Y. S. Hor, R. J. Cava, and M. Z. Hasan, Nature Physics 5, 398 (2009).
  * Zhang  _et al._ (2009) G. Zhang, H. Qin, J. Teng, J. Guo, Q. Guo, X. Dai, Z. Fang, and K. Wu, Applied Physics Letters 95, 053114 (2009).
  * Benalcazar  _et al._ (2017) W. A. Benalcazar, B. A. Bernevig, and T. L. Hughes, Science 357, 61 (2017).
  * Langbehn  _et al._ (2017) J. Langbehn, Y. Peng, L. Trifunovic, F. von Oppen, and P. W. Brouwer, Phys. Rev. Lett. 119, 246401 (2017).
  * Song  _et al._ (2017) Z. Song, Z. Fang, and C. Fang, Phys. Rev. Lett. 119, 246402 (2017).
  * Schindler  _et al._ (2018) F. Schindler, A. M. Cook, M. G. Vergniory, Z. Wang, S. S. P. Parkin, B. A. Bernevig, and T. Neupert, Science Advances 4, eaat0346 (2018).
  * Song  _et al._ (2020) Z.-D. Song, L. Elcoro, Y.-F. Xu, N. Regnault, and B. A. Bernevig, Phys. Rev. X 10, 031001 (2020).
  * Ahn  _et al._ (2018) J. Ahn, D. Kim, Y. Kim, and B.-J. Yang, Phys. Rev. Lett. 121, 106403 (2018).
  * Wang  _et al._ (2019) Z. Wang, B. J. Wieder, J. Li, B. Yan, and B. A. Bernevig, Phys. Rev. Lett. 123, 186401 (2019).
  * Park  _et al._ (2019) M. J. Park, Y. Kim, G. Y. Cho, and S. Lee, Phys. Rev. Lett. 123, 216803 (2019).
  * Lee  _et al._ (2020) E. Lee, R. Kim, J. Ahn, and B.-J. Yang, npj Quantum Materials 5 (2020), 10.1038/s41535-019-0206-8.
  * Hughes  _et al._ (2011) T. L. Hughes, E. Prodan, and B. A. Bernevig, Phys. Rev. B 83, 245132 (2011).
  * Alexandradinata  _et al._ (2014) A. Alexandradinata, X. Dai, and B. A. Bernevig, Phys. Rev. B 89, 155114 (2014).
  * Ahn  _et al._ (2019) J. Ahn, S. Park, D. Kim, Y. Kim, and B.-J. Yang, Chinese Physics B 28, 117101 (2019).
  * Zak (1989) J. Zak, Phys. Rev. Lett. 62, 2747 (1989).
  * Kim  _et al._ (2015) Y. Kim, B. J. Wieder, C. L. Kane, and A. M. Rappe, Phys. Rev. Lett. 115, 036806 (2015).
  * Moore and Balents (2007) J. E. Moore and L. Balents, Phys. Rev. B 75, 121306 (2007).
  * Teo and Kane (2010) J. C. Y. Teo and C. L. Kane, Phys. Rev. B 82, 115120 (2010).


# Understanding the World Through Action

Sergey Levine
Berkeley AI Research (BAIR)
UC Berkeley
<EMAIL_ADDRESS>

###### Abstract

The recent history of machine learning research has taught us that machine
learning methods can be most effective when they are provided with very large,
high-capacity models, and trained on very large and diverse datasets. This has
spurred the community to search for ways to remove any bottlenecks to scale.
Often the foremost among such bottlenecks is the need for human effort,
including the effort of curating and labeling datasets. As a result,
considerable attention in recent years has been devoted to utilizing unlabeled
data, which can be collected in vast quantities. However, some of the most
widely used methods for training on such unlabeled data themselves require
human-designed objective functions that must correlate in some meaningful way
to downstream tasks. I will argue that a general, principled, and powerful
framework for utilizing unlabeled data can be derived from reinforcement
learning, using general purpose unsupervised or self-supervised reinforcement
learning objectives in concert with offline reinforcement learning methods
that can leverage large datasets. I will discuss how such a procedure is more
closely aligned with potential downstream tasks, and how it could build on
existing techniques that have been developed in recent years.

> Keywords: Self-Supervised Learning, Embodied Learning

##  1 Introduction

Machine learning systems have mastered a breadth of challenging problems in
domains ranging from computer vision [1] to speech recognition [2] and natural
language processing [3], and yet the question of how to design learning-enable
systems that match the flexibility and generality of human reasoning remains
out of reach. This has prompted considerable discussion about what the
“missing ingredient” in modern machine learning might be, with a number of
hypotheses put forward as the big question that the field must resolve. Is the
missing ingredient causal reasoning [4], inductive bias [5], better algorithms
for self-supervised or unsupervised learning [6], or something else entirely?

This question is difficult one, and any answer must necessarily involve a
great deal of guesswork, but the lessons we can glean from recent progress in
artificial intelligence can provide us with several guiding principles.

The first lesson is in the “unreasonable” effectiveness of large, generic
models supplied with large amounts of training data. As eloquently articulated
by Sutton [7] in his essay on the “bitter lesson,” as well as a number of
other researchers in machine learning, a persistent theme in the recent
history of ML research has been that methods that effectively leverage large
amounts of computation and large amounts of data tend to often outperform
methods that rely on manually engineered priors and heuristics. While a full
discussion of the reasons for this trend are outside the scope of this
article, in short they could be summarized (or perhaps caricatured) as
follows: when we engineer biases or priors for our models, we are injecting
our own imperfect knowledge of how the world works, which biases the model
toward some solutions over others. When the model instead gleans this
knowledge from data, it will arrive at conclusions that are more accurate than
those that we engineered ourselves, and therefore will work better. Indeed, a
similar pattern has been observed in how people acquire proficiency. As
discussed by Dreyfus and Dreyfus [8], “rule-based” reasoning that follows
rules we can articulate clearly tends to only provide people with “novice-
level” performance at various skills, while “expert-level” performance is
associated with a mess of special cases, exceptions, and patterns that people
struggle to articulate clearly, and yet can leverage seamlessly in the moment
when the situation demands it. As Dreyfus points out, real human experts are
rarely able to articulate the rules they actually follow when exhibiting their
expertise, and so it should come as no surprise that in the same way that we
must acquire expertise from experience, so must our machines. And to do that,
they will need powerful, high-capacity models that impose comparatively few
biases and can handle the large amounts of experience that will be needed.

The second, more recent lesson is that manual labeling and supervision do not
scale nearly as well as unsupervised or self-supervised learning. We’ve
already seen that unsupervised pre-training has become standard in natural
language processing [9], and perhaps will soon become standard across other
fields. In a sense, this lesson is a corollary to the first one: if large
models and large datasets are the most effective, then anything that limits
how large the models and datasets can be will eventually end up as the
bottleneck. Human supervision can be one such bottleneck: if all data must be
labeled manually by a person, then less data will be available to the system
to learn from. However, here we reach a conundrum: current methods for
learning without human labels often violate the principles outlined in the
first lesson, requiring considerable human insight (which is often domain-
specific!) to engineer self-supervised learning objectives that allow large
models to acquire meaningful knowledge from unlabeled datasets. These include
relatively straightforward tasks such as language modeling [10], as well as
comparatively more esoteric tasks, such as predicting whether two transformed
images were produced by the same original image, or two different ones [11].
The latter is a widely used and successful approach in modern self-supervised
learning in computer vision. While such approaches can be effective up to a
point, it may well be that the next bottleneck we will face is in deciding how
to train large models without requiring manual labeling _or_ manual design of
self-supervised objectives, so as to acquire models that distill a deep and
meaningful understanding of the world and can perform downstream tasks with
robustness generalization, and even a degree of common sense.

I will argue that such methodology could be developed out of current
algorithms for learning-based control (reinforcement learning), though it will
require a number of substantial algorithmic innovations that will allow such
methods to go significantly beyond the kinds of problems they’ve been able to
tackle so far. Central to this idea is the notion that, in order to control
the environment in diverse and goal-directed ways, autonomous agents will
necessarily need to develop an understanding of their environment that is
causal and generalizable, and hence will address many of the shortcomings of
current supervised models. At the same time, this will require going beyond
the current paradigm in reinforcement learning in two important ways. First,
reinforcement learning algorithms require a task goal (i.e., a reward
function) to be specified by hand by the user, and then learn the behaviors
necessary to accomplish that task goal. This of course greatly limits their
ability to learn without human supervision. Second, reinforcement learning
algorithms in common use today are not inherently _data-driven_ , but rather
learn from online experience, and although such methods can be deployed
directly in real-world environments [12], online active data collection limits
their generalization in such settings [13], and many uses of reinforcement
learning instead take place in simulation, where there are few opportunities
to learn about how the _real_ world works.

##  2 Learning Through Action

Insofar as artificial intelligence systems are useful, it is because they
provide inferences that can be used to make decisions, which in turn affect
something in the world. Therefore, it is reasonable to conclude that a general
learning objective should be one that provides impetus to learn those things
that are most useful for affecting the world in meaningful ways. Making
decisions that create desired outcomes is the purview of reinforcement
learning and control. Therefore, we should consider how reinforcement learning
can provide the sort of automated and principled objectives for training high-
capacity models that can endow them with the ability to understand, reason,
and generalize.

However, this will require addressing the two limitations: reinforcement
learning requires manually-defined reward functions, and it requires an active
learning paradigm that is difficult to reconcile with the need to train on
large and diverse datasets. To address the issue with objectives, we can
develop algorithms that, instead of aiming to perform a single user-specified
task, rather aim to accomplish whatever outcomes they infer are possible in
the world. Potential objectives for such methods could include learning to
reach any feasible state, learning to maximize mutual information between
latent goals and outcomes, or learning through principled intrinsic motivation
objectives that lead to broad coverage of possible outcomes. To address the
issue with data, we must develop reinforcement learning algorithms that can
effectively utilize previously collected datasets. These are offline
reinforcement learning algorithms [13], and they can provide a path toward
training RL systems on broad and diverse datasets in much the same manner as
in supervised learning, followed by some amount of active online finetuning to
attain the best performance [14].

To provide a hypothetical example of a system that instantiates these ideas,
imagine a robot that performs a variety of manipulation tasks When given a
user-specified goal, the robot performs that goal. However, in its “spare
time,” the robot imagines potential outcomes that it can produce, and then
“practices” taking actions to produce them. Each such practice session deepens
its understanding of the causal structure in the world.

Of course, the notion of a real-world commercially deployed robotic system
that “plays” with its environment in this way might seem far-fetched (it is
also of course not a new idea [15]). This is precisely why offline RL is
important: since an offline algorithm would be comparatively indifferent to
the _source_ of the experience, the fraction of time that the robot spends
accomplishing user-specified objectives versus “playing” could be adjusted to
either extreme, and even a system that spends all of its time performing user-
specified tasks can still use all of its collected experience as _offline_
training data for learning to achieve any outcome. Such a system would still
“play” with its environment, but only virtually, in its “memories.”

While robotic systems might be the most obvious domain in which to instantiate
this design, it is not restricted to robotics, nor to systems that are
embodied in the world in an analogous way to people. Any system that has a
well-defined notion of actions can be trained in this way: recommender
systems, autonomous vehicles, systems for inventory management and logistics,
dialogue systems, and so forth. Online exploration may not be feasible in many
of these settings, but learning with unsupervised outcome-driven objectives
via offline RL is still possible. As mentioned previously, ML systems are
useful insofar as they enable making intelligent decisions. It therefore
stands to reason that any useful ML system is situated in a sequential process
where decision-making is possible, and therefore such a self-supervised
learning procedure should be applicable.

##  3 Unsupervised and Self-Supervised Reinforcement Learning

An unsupervised or self-supervised reinforcement learning method should
fulfill two criteria: it should learn behaviors that control the world in
meaningful ways, and it should provide some mechanism to learn to control it
in as _many_ ways as possible. This leaves considerable room for
interpretation, for example concerning the term “meaningful,” and a wide range
of potential definitions have been offered in the literature [16, 17, 18].
This problem should not be confused with the closely related problem of
exploration, which has also often been formulated as a problem of attaining
broad coverage [19], but which is not generally concerned with learning to
control the world in meaningful ways in the absence of a task objective. That
is, exploration methods provide an objective for _collecting_ data, rather
than _utilizing_ it.

Perhaps the most direct way to formulate a self-supervised RL objective is to
frame it as the problem of reaching a goal state [20]. The problem then
corresponds to training a goal-conditioned policy
$\pi(\mathbf{a}|\mathbf{s},\mathbf{g})$ with some choice of reward function
$r(\mathbf{s},\mathbf{g})$. Though this reward itself might constitutes a
manually designed objective, it’s possible to derive frameworks where the
reward function is a consequence of solving a well-defined inference problem,
such as the problem of predicting the action that is most likely to lead a
particular outcome [21, 22]. This problem formulation provides considerable
depth, with connections to density estimation [21], variational inference [23,
22], model-based reinforcement learning [24, 25, 26], and exploration [27].

What does a policy trained to reach all possible goals learn about the world?
As argued both in recent work [25, 26] and classic literature in RL [24],
solving such goal-conditioned RL problems corresponds to learning a kind of
dynamics model. Intuitively, being able to bring about any potential desired
outcome requires a deep understanding of how actions affect the environment
over a long horizon. Of course, one might then wonder – why not just learn a
dynamics model directly, of the form more commonly used in model-based RL?
Model learning is likely also an effective way to utilize diverse datasets
without a specific user-provided objective. However, there is good reason to
believe that goal-conditioned RL either solves a very similar problem [24, 25,
26] or, potentially, actually solves a problem that is more closely connected
to downstream tasks – unlike model-based RL, where the model objective is
largely disconnected from actually bringing about desired outcomes, the goal-
conditioned RL objective is connected to long-horizon outcomes very directly.
Therefore, insofar as the end goal of an ML system is to bring about desired
outcomes, we would expect that the objective of goal-conditioned RL would be
well-aligned.

However, current approaches suffer from a number of major limitations. Even
standard goal-conditioned RL methods can be difficult to use and unstable. But
even more importantly, goal-reaching does not span the full set of possible
tasks that could be specified in RL. Even if an agent learns to successfully
accomplish every outcome that is possible in a given environment, there may
not exist a single desired outcome that would maximize an arbitrary user-
specified reward function. It may still be that such a goal-conditioned policy
would have learned powerful and broadly applicable features, and could be
readily finetuned to the downstream task, but an interesting problem for
future work is to better understand whether more universal self-supervised
objectives could lift this limitation. A number of methods have been proposed
for unsupervised acquisition of _skills_ [28, 29, 30], so we might reasonably
ask whether more general and principled self-supervised reinforcement learning
objectives could be derived on this basis.

##  4 Offline Reinforcement Learning

As discussed previously, offline RL can make it possible to apply self-
supervised or unsupervised RL methods even in settings where online collection
is infeasible, and such methods can serve as one of the most powerful tools
for incorporating large and diverse datasets into self-supervised RL. This is
likely to be essential to make this a truly viable and general tool for large-
scale representation learning. However, offline RL presents a number of
challenges [13]. Foremost among these is that offline RL requires answering
_counterfactual_ questions: given data that shows one outcome, can we predict
what would have happened if we had taken a different action? This is of course
very challenging in general. Nonetheless, our understanding of offline RL has
progressed significantly over the past few years. Besides understanding how
distributional shift affects offline RL [31], the performance of offline RL
algorithms has advanced considerably, and new algorithms have been developed
that provide robustness guarantees [32], finetune online after offline
pretraining [14], and tackle a range of other problems in the offline RL
setting.

Advances in offline RL have the potential to significantly increase the
applicability of self-supervised RL methods. Using the tools of offline RL, it
is possible to construct self-supervised RL methods that do not require any
exploration on their own. Much like the “virtual play” mentioned in Section 2,
we can utilize offline RL in combination with goal-conditioned policies to
learn entirely from previously collected data [33, 34, 35]. However, major
challenges remain. Offline RL algorithms inherit many of the difficulties of
standard (deep) RL methods, including sensitivity to hyperparameters, but
these difficulties are further exacerbated by the fact that we cannot perform
multiple online trials to determine the best hyperparameters. In supervised
learning we can deal with such issues by using a validation set, but a
corresponding equivalent in offline RL is lacking. We need algorithms that are
more stable and reliable, as well as effective methods for evaluation, in
order to make such approaches truly broadly applicable.

##  5 Concluding Remarks

In this article, I discussed how self-supervised reinforcement learning
combined with offline RL could enable scalable representation learning. The
motivation behind this approach is that, insofar that learned models are
useful, it is because they allow us to make decisions that bring about the
desired outcome in the world. Therefore, self-supervised training with the
goal of bringing about _any_ possible outcome should provide such models with
the requisite understanding of how the world works. Self-supervised RL
objectives, such as those in goal-conditioned RL, have a close relationship
with model learning, and fulfilling such objectives is likely to require
policies to gain a functional and causal understanding of the environment in
which they are situated. However, for such techniques to be useful, it must be
possible to apply them at scale to real-world datasets. Offline RL can play
this role, because it enables using large, diverse previously collected
datasets. Putting these pieces together may lead to a new class of algorithms
that can understand the world through action, leading to methods that are
truly scalable and automated.

## References

  * Krizhevsky et al. [2012] A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  _Advances in neural information processing systems_ , 25:1097–1105, 2012.
  * Hinton et al. [2012] G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath, et al.  Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups.  _IEEE Signal processing magazine_ , 29(6):82–97, 2012.
  * Sutskever et al. [2014] I. Sutskever, O. Vinyals, and Q. Le.  Sequence to sequence learning with neural networks.  _arXiv preprint arXiv:1409.3215_ , 2014.
  * Schölkopf et al. [2021] B. Schölkopf, F. Locatello, S. Bauer, N. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio.  Toward causal representation learning.  _Proceedings of the IEEE_ , 109(5):612–634, 2021\.
  * Lake et al. [2017] B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman.  Building machines that learn and think like people.  _Behavioral and brain sciences_ , 40, 2017.
  * LeCun and Misra [2021] Y. LeCun and I. Misra.  Self-supervised learning: The dark matter of intelligence, 2021.  URL https://ai.facebook.com/blog/self-supervised-learning-the-dark-matter-of-intelligence/.
  * Sutton [2019] R. Sutton.  The bitter lesson, 2019.  URL http://www.incompleteideas.net/IncIdeas/BitterLesson.html.
  * Dreyfus and Dreyfus [1986] H. L. Dreyfus and S. E. Dreyfus.  From socrates to expert systems: The limits of calculative rationality.  In _Philosophy and technology II_ , pages 111–130. Springer, 1986\.
  * Devlin et al. [2018] J. Devlin, M. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Peters et al. [2018] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer.  Deep contextualized word representations.  _arXiv preprint arXiv:1802.05365_ , 2018.
  * Chen et al. [2020] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pages 1597–1607. PMLR, 2020.
  * Kalashnikov et al. [2018] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine.  Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation.  In _Conference on Robot Learning (CoRL)_ , 2018.
  * Levine et al. [2020] S. Levine, A. Kumar, G. Tucker, and J. Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  _arXiv preprint arXiv:2005.01643_ , 2020.
  * Nair et al. [2020] A. Nair, M. Dalal, A. Gupta, and S. Levine.  Accelerating online reinforcement learning with offline datasets.  _arXiv preprint arXiv:2006.09359_ , 2020.
  * Oudeyer et al. [2005] P.-Y. Oudeyer, F. Kaplan, V. V. Hafner, and A. Whyte.  The playground experiment: Task-independent development of a curious robot.  In _Proceedings of the AAAI Spring Symposium on Developmental Robotics_ , pages 42–47, 2005.
  * Lehman and Stanley [2011] J. Lehman and K. O. Stanley.  Abandoning objectives: Evolution through the search for novelty alone.  _Evolutionary computation_ , 19(2):189–223, 2011\.
  * Barto [2013] A. G. Barto.  Intrinsic motivation and reinforcement learning.  In _Intrinsically motivated learning in natural and artificial systems_ , pages 17–47. Springer, 2013.
  * Salge et al. [2014] C. Salge, C. Glackin, and D. Polani.  Empowerment–an introduction.  In _Guided Self-Organization: Inception_ , pages 67–114. Springer, 2014.
  * Bellemare et al. [2016] M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos.  Unifying count-based exploration and intrinsic motivation.  _arXiv preprint arXiv:1606.01868_ , 2016.
  * Kaelbling [1993] L. P. Kaelbling.  Learning to achieve goals.  In _IJCAI_ , pages 1094–1099, 1993.
  * Eysenbach et al. [2020] B. Eysenbach, R. Salakhutdinov, and S. Levine.  C-learning: Learning to achieve goals via recursive classification.  _arXiv preprint arXiv:2011.08909_ , 2020.
  * Rudner et al. [2021] T. G. Rudner, V. H. Pong, R. McAllister, Y. Gal, and S. Levine.  Outcome-driven reinforcement learning via variational inference.  _arXiv preprint arXiv:2104.10190_ , 2021.
  * Warde-Farley et al. [2018] D. Warde-Farley, T. Van de Wiele, T. Kulkarni, C. Ionescu, S. Hansen, and V. Mnih.  Unsupervised control through non-parametric discriminative rewards.  _arXiv preprint arXiv:1811.11359_ , 2018.
  * Dayan [1993] P. Dayan.  Improving generalization for temporal difference learning: The successor representation.  _Neural Computation_ , 5(4):613–624, 1993.
  * Pong et al. [2018] V. Pong, S. Gu, M. Dalal, and S. Levine.  Temporal difference models: Model-free deep rl for model-based control.  _arXiv preprint arXiv:1802.09081_ , 2018.
  * Janner et al. [2020] M. Janner, I. Mordatch, and S. Levine.  $\gamma$-models: Generative temporal difference learning for infinite-horizon prediction.  _arXiv preprint arXiv:2010.14496_ , 2020.
  * Pong et al. [2019] V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine.  Skew-fit: State-covering self-supervised reinforcement learning.  _arXiv preprint arXiv:1903.03698_ , 2019.
  * Gregor et al. [2016] K. Gregor, D. J. Rezende, and D. Wierstra.  Variational intrinsic control.  _arXiv preprint arXiv:1611.07507_ , 2016.
  * Eysenbach et al. [2018] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine.  Diversity is all you need: Learning skills without a reward function.  _arXiv preprint arXiv:1802.06070_ , 2018.
  * Sharma et al. [2019] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman.  Dynamics-aware unsupervised discovery of skills.  _arXiv preprint arXiv:1907.01657_ , 2019.
  * Kumar et al. [2019] A. Kumar, J. Fu, G. Tucker, and S. Levine.  Stabilizing off-policy q-learning via bootstrapping error reduction.  _arXiv preprint arXiv:1906.00949_ , 2019.
  * Kumar et al. [2020] A. Kumar, A. Zhou, G. Tucker, and S. Levine.  Conservative q-learning for offline reinforcement learning.  _arXiv preprint arXiv:2006.04779_ , 2020.
  * Tian et al. [2020] S. Tian, S. Nair, F. Ebert, S. Dasari, B. Eysenbach, C. Finn, and S. Levine.  Model-based visual planning with self-supervised functional distances.  _arXiv preprint arXiv:2012.15373_ , 2020.
  * Chebotar et al. [2021] Y. Chebotar, K. Hausman, Y. Lu, T. Xiao, D. Kalashnikov, J. Varley, A. Irpan, B. Eysenbach, R. Julian, C. Finn, and S. Levine.  Actionable models: Unsupervised offline reinforcement learning of robotic skills.  _arXiv preprint arXiv:2104.07749_ , 2021.
  * Shah et al. [2021] D. Shah, B. Eysenbach, N. Rhinehart, and S. Levine.  RECON: Rapid exploration for open-world navigation with latent goal models.  _arXiv preprint arXiv:2104.05859_ , 2021.