FRB 20190718A | 973.4 | $2.0\times 10^{54}$ | $2.1\times 10^{54}$ | $5.1\times 10^{54}$ | $2.0\times 10^{55}$ | $1.9\times 10^{56}$ | $1.4\times 10^{57}$ | $5.8\times 10^{57}$ | $2.3\times 10^{58}$
FRB 20190722A | 1129.9 | $9.4\times 10^{54}$ | $1.7\times 10^{55}$ | $6.7\times 10^{55}$ | $4.4\times 10^{56}$ | $7.2\times 10^{57}$ | $5.3\times 10^{58}$ | $2.2\times 10^{59}$ | $1.3\times 10^{60}$
FRB 20190812A | 1362.0 | $2.0\times 10^{54}$ | $2.2\times 10^{54}$ | $5.3\times 10^{54}$ | $2.3\times 10^{55}$ | $2.3\times 10^{56}$ | $2.0\times 10^{57}$ | $8.7\times 10^{57}$ | $3.1\times 10^{58}$
FRB 20190903A | 925.4 | $1.7\times 10^{55}$ | $1.9\times 10^{55}$ | $9.6\times 10^{55}$ | $8.4\times 10^{56}$ | $1.0\times 10^{58}$ | $1.4\times 10^{59}$ | $6.5\times 10^{59}$ | $1.8\times 10^{60}$
FRB 20190912A | 1090.5 | $1.5\times 10^{55}$ | $2.5\times 10^{55}$ | $9.9\times 10^{55}$ | $5.8\times 10^{56}$ | $1.2\times 10^{58}$ | $1.0\times 10^{59}$ | $4.7\times 10^{59}$ | $2.2\times 10^{60}$
FRB 20190912B | 485.0 | $3.2\times 10^{53}$ | $4.1\times 10^{53}$ | $7.7\times 10^{53}$ | $3.7\times 10^{54}$ | $3.1\times 10^{55}$ | $3.3\times 10^{56}$ | $1.8\times 10^{57}$ | $6.7\times 10^{57}$
FRB 20190922A | 959.6 | $1.1\times 10^{55}$ | $1.6\times 10^{55}$ | $6.6\times 10^{55}$ | $3.2\times 10^{56}$ | $5.0\times 10^{57}$ | $5.9\times 10^{58}$ | $3.2\times 10^{59}$ | $9.8\times 10^{59}$
FRB 20190928A | 510.3 | $6.1\times 10^{53}$ | $6.9\times 10^{53}$ | $1.5\times 10^{54}$ | $5.7\times 10^{54}$ | $6.6\times 10^{55}$ | $5.1\times 10^{56}$ | $2.3\times 10^{57}$ | $8.4\times 10^{57}$
FRB 20190929B | 1533.6 | $3.0\times 10^{54}$ | $4.1\times 10^{54}$ | $7.1\times 10^{54}$ | $3.6\times 10^{55}$ | $3.0\times 10^{56}$ | $2.8\times 10^{57}$ | $1.3\times 10^{58}$ | $4.2\times 10^{58}$
Table 7:  As for Table 5 but with distances based on the the upper bounds of 90% credible intervals on the luminosity distance. FRB | $D_{\mathrm{L}}$ | DS2P | DS2P | WNB | WNB | WNB | WNB
---|---|---|---|---|---|---|---
| [Mpc] | A | B | A | B | C | D
FRB 20190410A | 956.6 | $5.0\times 10^{58}$ | $4.5\times 10^{58}$ | $3.2\times 10^{55}$ | $2.4\times 10^{55}$ | $8.6\times 10^{56}$ | $2.5\times 10^{57}$
FRB 20190419B | 575.7 | $2.3\times 10^{58}$ | $1.6\times 10^{58}$ | $8.4\times 10^{56}$ | $9.5\times 10^{54}$ | $3.2\times 10^{56}$ | $9.0\times 10^{56}$
FRB 20190423B | 1704.6 | $2.1\times 10^{60}$ | $2.2\times 10^{60}$ | $2.4\times 10^{56}$ | $2.6\times 10^{56}$ | $1.2\times 10^{58}$ | $3.2\times 10^{58}$
FRB 20190425A | 385.9 | $1.6\times 10^{59}$ | $4.4\times 10^{57}$ | $4.5\times 10^{53}$ | $7.4\times 10^{53}$ | $1.3\times 10^{55}$ | $4.4\times 10^{55}$
FRB 20190517C | 1030.5 | $3.6\times 10^{58}$ | $3.2\times 10^{58}$ | $1.3\times 10^{55}$ | $1.7\times 10^{55}$ | $7.5\times 10^{56}$ | $4.0\times 10^{57}$
FRB 20190518D | 852.0 | $1.3\times 10^{58}$ | $2.1\times 10^{58}$ | $3.7\times 10^{54}$ | $4.9\times 10^{54}$ | $1.1\times 10^{56}$ | $3.2\times 10^{56}$
FRB 20190531B | 675.5 | $1.9\times 10^{57}$ | $2.1\times 10^{57}$ | $1.9\times 10^{54}$ | $2.9\times 10^{54}$ | $4.5\times 10^{55}$ | $1.8\times 10^{56}$
FRB 20190601C | 1736.9 | $4.2\times 10^{58}$ | $6.4\times 10^{58}$ | $9.4\times 10^{54}$ | $1.2\times 10^{55}$ | $2.6\times 10^{56}$ | $6.6\times 10^{56}$
FRB 20190604G | 1143.0 | $2.6\times 10^{59}$ | $2.2\times 10^{59}$ | $-$ | $6.8\times 10^{56}$ | $2.0\times 10^{58}$ | $4.7\times 10^{58}$
FRB 20190605C | 893.7 | $4.1\times 10^{58}$ | $2.9\times 10^{58}$ | $5.9\times 10^{55}$ | $2.7\times 10^{55}$ | $1.1\times 10^{57}$ | $3.0\times 10^{57}$
FRB 20190606B | 1465.7 | $4.3\times 10^{58}$ | $7.5\times 10^{58}$ | $2.7\times 10^{55}$ | $1.5\times 10^{55}$ | $3.6\times 10^{56}$ | $9.7\times 10^{56}$
FRB 20190612B | 922.2 | $1.3\times 10^{58}$ | $2.2\times 10^{60}$ | $2.7\times 10^{54}$ | $4.2\times 10^{54}$ | $7.5\times 10^{55}$ | $2.5\times 10^{56}$
FRB 20190613B | 780.1 | $4.9\times 10^{57}$ | $8.9\times 10^{57}$ | $1.3\times 10^{54}$ | $2.0\times 10^{54}$ | $4.3\times 10^{55}$ | $1.3\times 10^{56}$
FRB 20190616A | 1125.8 | $2.4\times 10^{59}$ | $3.0\times 10^{59}$ | $1.2\times 10^{56}$ | $8.0\times 10^{55}$ | $2.6\times 10^{57}$ | $1.6\times 10^{58}$
FRB 20190617A | 872.9 | $7.1\times 10^{57}$ | $1.0\times 10^{58}$ | $6.4\times 10^{54}$ | $5.3\times 10^{54}$ | $7.7\times 10^{55}$ | $3.2\times 10^{56}$
FRB 20190618A | 964.0 | $6.7\times 10^{57}$ | $1.1\times 10^{58}$ | $1.5\times 10^{54}$ | $2.7\times 10^{54}$ | $5.4\times 10^{55}$ | $1.8\times 10^{56}$
FRB 20190621A | 978.1 | $1.8\times 10^{59}$ | $7.6\times 10^{58}$ | $-$ | $1.5\times 10^{56}$ | $4.4\times 10^{57}$ | $9.2\times 10^{57}$
FRB 20190624B | 822.5 | $5.9\times 10^{58}$ | $1.1\times 10^{59}$ | $8.5\times 10^{54}$ | $1.4\times 10^{55}$ | $3.1\times 10^{56}$ | $1.1\times 10^{57}$
FRB 20190710A | 997.6 | $4.8\times 10^{57}$ | $5.4\times 10^{57}$ | $2.1\times 10^{54}$ | $3.2\times 10^{54}$ | $6.9\times 10^{55}$ | $2.0\times 10^{56}$
FRB 20190713A | 1436.5 | $2.4\times 10^{59}$ | $3.8\times 10^{59}$ | $3.2\times 10^{55}$ | $5.2\times 10^{55}$ | $1.6\times 10^{57}$ | $4.5\times 10^{57}$
FRB 20190718A | 973.4 | $6.7\times 10^{57}$ | $1.1\times 10^{58}$ | $3.2\times 10^{54}$ | $4.2\times 10^{54}$ | $8.4\times 10^{55}$ | $2.6\times 10^{56}$
FRB 20190722A | 1129.9 | $1.5\times 10^{59}$ | $1.1\times 10^{59}$ | $1.2\times 10^{58}$ | $3.6\times 10^{56}$ | $1.1\times 10^{58}$ | $2.3\times 10^{58}$
FRB 20190812A | 1362.0 | $1.4\times 10^{58}$ | $2.8\times 10^{58}$ | $4.4\times 10^{54}$ | $6.1\times 10^{54}$ | $1.3\times 10^{56}$ | $3.8\times 10^{56}$
FRB 20190903A | 925.4 | $2.1\times 10^{59}$ | $1.4\times 10^{59}$ | $9.6\times 10^{55}$ | $7.0\times 10^{55}$ | $3.3\times 10^{57}$ | $8.6\times 10^{57}$
FRB 20190912A | 1090.5 | $2.5\times 10^{59}$ | $1.8\times 10^{59}$ | $9.8\times 10^{55}$ | $8.2\times 10^{55}$ | $3.7\times 10^{57}$ | $1.1\times 10^{58}$
FRB 20190912B | 485.0 | $3.5\times 10^{57}$ | $6.4\times 10^{57}$ | $6.5\times 10^{53}$ | $7.9\times 10^{53}$ | $2.0\times 10^{55}$ | $5.3\times 10^{55}$
FRB 20190922A | 959.6 | $4.7\times 10^{59}$ | $6.7\times 10^{59}$ | $3.2\times 10^{56}$ | $8.8\times 10^{55}$ | $3.1\times 10^{57}$ | $8.2\times 10^{57}$
FRB 20190928A | 510.3 | $3.8\times 10^{57}$ | $7.1\times 10^{57}$ | $1.1\times 10^{54}$ | $1.6\times 10^{54}$ | $2.6\times 10^{55}$ | $1.1\times 10^{56}$
FRB 20190929B | 1533.6 | $1.5\times 10^{58}$ | $3.1\times 10^{58}$ | $7.0\times 10^{54}$ | $7.8\times 10^{54}$ | $1.9\times 10^{56}$ | $5.0\times 10^{56}$

Table 8 lists the repeating bursts that were analyzed in the generic transient
search. The most sensitive counterpart to a repeating FRB was for CHIME/FRB
event FRB20190825A. The SG injection centered at 1600 Hz (which most closely
models an f-mode) was recovered $90\%$ of the time at
$h_{\mathrm{\mathrm{rss}}}=2.62\times~{}10^{-22}$. The distance to this event
is $148.1$ Mpc to $149.9$ Mpc. This corresponds to an energy upper limit range
of $5.83\,\times\,10^{55}$ erg to $5.98\,\times\,10^{55}$ erg.

Table 8:  Details of the 3 repeating FRBs analyzed in the generic transient search and their various repeating episodes. The TNS name is provided in the first column. The Network column lists the GW detector network used: H1 = LIGO Hanford, L1 = LIGO Livingston, V1 = Virgo. The total DM for each FRB is listed in the DM column and the 90% credible intervals on the luminosity distance are provided in columns $D_{L}$-low and $D_{L}$-High. 11 total events were analyzed for the three different FRB repeaters considered. For FRB 20190518A and its associated repeats, we list only the distance of Marcote et al. (2020) obtained by galaxy localization. FRB Name | UTC Time | R.A. | Dec. | Network | DM |  $D_{L}$-Low |  $D_{L}$-high
---|---|---|---|---|---|---|---
| [s] |  |  |  | [pc cm-3] | [Mpc] | [Mpc]
FRB20190817A | 14:39:52 | $4^{\mathrm{h}}21^{\mathrm{m}}08^{\mathrm{s}}$ | $73^{\circ}47^{\prime}$ | H1L1V1 | 189.5 | 19.5 | 539.2
FRB20190929C | 11:58:29 | $4^{\mathrm{h}}22^{\mathrm{m}}25^{\mathrm{s}}$ | $73^{\circ}40^{\prime}$ | H1L1V1 | 191.6 | 20.8 | 550.1
FRB20190518A | 18:13:33 | $1^{\mathrm{h}}58^{\mathrm{m}}14^{\mathrm{s}}$ | $65^{\circ}46^{\prime}$ | L1V1 | 350.5 | 148.1 | 149.9
FRB20190518E | 18:20:57 | $1^{\mathrm{h}}57^{\mathrm{m}}50^{\mathrm{s}}$ | $65^{\circ}43^{\prime}$ | L1V1 | 350.0 | 148.1 | 149.9
FRB20190519A | 17:50:16 | $1^{\mathrm{h}}43^{\mathrm{m}}44^{\mathrm{s}}$ | $65^{\circ}48^{\prime}$ | H1V1 | 350.0 | 148.1 | 149.9
FRB20190519C | 18:10:41 | $1^{\mathrm{h}}58^{\mathrm{m}}00^{\mathrm{s}}$ | $65^{\circ}47^{\prime}$ | H1V1 | 348.8 | 148.1 | 149.9
FRB20190809A | 12:50:40 | $1^{\mathrm{h}}58^{\mathrm{m}}16^{\mathrm{s}}$ | $65^{\circ}43^{\prime}$ | H1L1 | 356.2 | 148.1 | 149.9
FRB20190825A | 11:48:18 | $1^{\mathrm{h}}58^{\mathrm{m}}07^{\mathrm{s}}$ | $65^{\circ}42^{\prime}$ | H1L1 | 349.6 | 148.1 | 149.9
FRB20190825B | 11:51:54 | $1^{\mathrm{h}}58^{\mathrm{m}}04^{\mathrm{s}}$ | $65^{\circ}23^{\prime}$ | H1L1 | 349.9 | 148.1 | 149.9
FRB20190421A | 08:00:04 | $13^{\mathrm{h}}51^{\mathrm{m}}57^{\mathrm{s}}$ | $48^{\circ}10^{\prime}$ | H1L1V1 | 225.9 | 125.1 | 1260.8
FRB20190702B | 03:14:36 | $13^{\mathrm{h}}52^{\mathrm{m}}25^{\mathrm{s}}$ | $48^{\circ}15^{\prime}$ | L1V1 | 224.4 | 125.8 | 1257.5

These estimates are well above predictions of the GW emissions by the NS’s
fundamental f-mode. For example Corsi & Owen (2011) have suggested
$E_{\mathrm{GW}}\sim 10^{48}-10^{49}$ erg in GW energy emitted at around $1-2$
kHz, although predictions in (Levin & van Hoven, 2011; Zink et al., 2012) span
a much lower range $E_{\mathrm{GW}}\sim 10^{28}-10^{38}$ erg based on studies
that suggest lower effective energy conversion to GWs.

##  6 The M81 repeater FRB 20200120E

A repeater, FRB 20200120E, which was discovered by CHIME/FRB on 20 Jan 2020,
overlaps with the second part of the third observing run of Advanced LIGO and
Advanced Virgo (O3b). This burst is at 3.6 Mpc, the closest extragalactic FRB
so far discovered (Bhardwaj et al., 2021b). This event was shown to be
conclusively associated with a globular cluster in the M81 galactic system
(Kirsten et al., 2021) which supports the possibility that it was formed from
an evolved stellar population such as a compact binary system. Due to the
proximity and significance of this burst, we discuss it in this paper, despite
it being discovered after O3a.

The burst FRB 20200120E was shown to repeat at least 4 times. Two of the
repeats occurred after O3b; another episode, despite being consistent with the
localization of the other associated bursts, had no intensity data saved.
Therefore, we discuss here only the initial burst FRB 20200120E, for which GW
data exists.

At the time of FRB 20200120E, only H1 data was available, thus a generic
transient search was not conducted. Likewise, since this is a repeating event,
it does not pass our criteria for conducting a modelled search. Due to these
restrictions, only a RAVEN coincidence search was conducted within a
[$-6000,+6000$] s time window. No coincidences were found with sufficient
significance as determined by the coincident false-alarm rate. Given the
relative close proximity of this burst, further repeat emissions will be of
interest for GW follow-up during the fourth observing run of Advanced LIGO,
Advanced Virgo and Kagra (O4) (Abbott et al., 2020).

##  7 Conclusions

We performed a targeted search for GWs associated with FRBs detected by the
CHIME/FRB project during O3a. As the sources of non-repeating FRBs are
currently not known, we ran both a modelled search for BNS and NSBH signals
(Harry & Fairhurst, 2011; Williamson et al., 2014) and a generic transient
search for generic GW transient signals (Sutton et al., 2010; Was et al.,
2012).

Our searches found no significant GW event candidates in association with the
analyzed FRBs. We set 90% confidence lower bounds on the distances to FRB
progenitors for several different emission models. Additionally, we present
90% credible intervals on the luminosity distance, $D_{L}$, inferred from the
DM measurement of each FRB source.

The $D_{L}$ information can be used to test models based on the simulated
injections used for calculating the $D_{90}$ values of each FRB. However, the
significant uncertainties in the relative contributions to the total DM for
each FRB produce relatively wide credible intervals for the $D_{L}$
posteriors. We find no FRB event can be fully excluded from any of the models
we have considered due to some posterior support on $D_{L}$ existing for the
FRB outside the detection range of the analyzes performed.

The results however, as illustrated in Fig. 6, indicate that the GW network’s
detection range is advancing into cosmological volumes where FRB emissions are
expected. This is encouraging as we look forward to future GW searches at
higher sensitivity. Furthermore, the redshifts obtained from the ongoing
efforts to localize host galaxies (there are currently 18 FRBs with an
associated host galaxy (see http://frbhosts.org/) could significantly improve
the chances of constraining progenitor populations (Heintz et al., 2020;
Bhandari et al., 2021).

The distance estimates for each FRB allowed us to place 90% upper limits on
the GW energy emitted by each FRB source, $E_{\mathrm{GW}}^{90\%}$. For each
non-repeating FRB analyzed with a generic transient search, we provided limits
on $E_{\mathrm{GW}}^{90\%}$ for a range of emission models. Repeating FRBs
were also analyzed to determine 90% upper limits on the energy emitted through
GWs. For the most sensitive repeating FRB analysis in our sample we find an
energy upper limit range of $5.83\,\times\,10^{54}$ erg to
$5.98\,\times\,10^{55}$ erg, well above the predictions for GW emissions from
the fundamental f-modes of NSs. Based on Equation 5, an FRB event such as that
associated with SGR 1935+2154 occurring during O3a would have allowed the
search to probe the more optimistic of these estimates allowing limits,
$E_{\mathrm{GW}}\,\sim\,10^{47}$ erg, assuming a generic burst waveform
emitting at roughly $1$ kHz at 10 kpc.

We also analyzed the repeater, FRB 20200120E, discovered on 20 Jan 2020 during
O3b. A RAVEN (Urban, 2016; Cho, 2019) coincidence search for any previously
detected compact binary coalescence GW events was conducted within a [$-6000$,
$+6000$] s time window around the first burst of this repeater. No
coincidences were found with sufficient significance to be distinguished from
random coincidences, as determined by the computed joint false-alarm rate from
the two samples.

This material is based upon work supported by NSF’s LIGO Laboratory which is a
major facility fully funded by the National Science Foundation. The authors
also gratefully acknowledge the support of the Science and Technology
Facilities Council (STFC) of the United Kingdom, the Max-Planck-Society (MPS),
and the State of Niedersachsen/Germany for support of the construction of
Advanced LIGO and construction and operation of the GEO 600 detector.
Additional support for Advanced LIGO was provided by the Australian Research
Council. The authors gratefully acknowledge the Italian Istituto Nazionale di
Fisica Nucleare (INFN), the French Centre National de la Recherche
Scientifique (CNRS) and the Netherlands Organization for Scientific Research
(NWO), for the construction and operation of the Virgo detector and the
creation and support of the EGO consortium. The authors also gratefully
acknowledge research support from these agencies as well as by the Council of
Scientific and Industrial Research of India, the Department of Science and
Technology, India, the Science & Engineering Research Board (SERB), India, the
Ministry of Human Resource Development, India, the Spanish Agencia Estatal de
Investigación (AEI), the Spanish Ministerio de Ciencia e Innovación and
Ministerio de Universidades, the Conselleria de Fons Europeus, Universitat i
Cultura and the Direcció General de Política Universitaria i Recerca del
Govern de les Illes Balears, the Conselleria d’Innovació, Universitats,
Ciència i Societat Digital de la Generalitat Valenciana and the CERCA
Programme Generalitat de Catalunya, Spain, the National Science Centre of
Poland and the European Union – European Regional Development Fund; Foundation
for Polish Science (FNP), the Swiss National Science Foundation (SNSF), the
Russian Foundation for Basic Research, the Russian Science Foundation, the
European Commission, the European Social Funds (ESF), the European Regional
Development Funds (ERDF), the Royal Society, the Scottish Funding Council, the
Scottish Universities Physics Alliance, the Hungarian Scientific Research Fund
(OTKA), the French Lyon Institute of Origins (LIO), the Belgian Fonds de la
Recherche Scientifique (FRS-FNRS), Actions de Recherche Concertées (ARC) and
Fonds Wetenschappelijk Onderzoek – Vlaanderen (FWO), Belgium, the Paris Île-
de-France Region, the National Research, Development and Innovation Office
Hungary (NKFIH), the National Research Foundation of Korea, the Natural
Science and Engineering Research Council Canada, Canadian Foundation for
Innovation (CFI), the Brazilian Ministry of Science, Technology, and
Innovations, the International Center for Theoretical Physics South American
Institute for Fundamental Research (ICTP-SAIFR), the Research Grants Council
of Hong Kong, the National Natural Science Foundation of China (NSFC), the
Leverhulme Trust, the Research Corporation, the Ministry of Science and
Technology (MOST), Taiwan, the United States Department of Energy, and the
Kavli Foundation. The authors gratefully acknowledge the support of the NSF,
STFC, INFN and CNRS for provision of computational resources. This work was
supported by MEXT, JSPS Leading-edge Research Infrastructure Program, JSPS
Grant-in-Aid for Specially Promoted Research 26000005, JSPS Grant-in-Aid for
Scientific Research on Innovative Areas 2905: JP17H06358, JP17H06361 and
JP17H06364, JSPS Core-to-Core Program A. Advanced Research Networks, JSPS
Grant-in-Aid for Scientific Research (S) 17H06133 and 20H05639 , JSPS Grant-
in-Aid for Transformative Research Areas (A) 20A203: JP20H05854, the joint
research program of the Institute for Cosmic Ray Research, University of
Tokyo, National Research Foundation (NRF), Computing Infrastructure Project of
KISTI-GSDC, Korea Astronomy and Space Science Institute (KASI), and Ministry
of Science and ICT (MSIT) in Korea, Academia Sinica (AS), AS Grid Center
(ASGC) and the Ministry of Science and Technology (MoST) in Taiwan under
grants including AS-CDA-105-M06, Advanced Technology Center (ATC) of NAOJ, and
Mechanical Engineering Center of KEK. We acknowledge that CHIME is located on
the traditional, ancestral, and unceded territory of the Syilx/Okanagan
people. We thank the Dominion Radio Astrophysical Observatory, operated by the
National Research Council Canada, for gracious hospitality and expertise.
CHIME is funded by a grant from the Canada Foundation for Innovation (CFI)
2012 Leading Edge Fund (Project 31170) and by contributions from the provinces
of British Columbia, Québec and Ontario. The CHIME/FRB Project is funded by a
grant from the CFI 2015 Innovation Fund (Project 33213) and by contributions
from the provinces of British Columbia and Québec, and by the Dunlap Institute
for Astronomy and Astrophysics at the University of Toronto. Additional
support was provided by the Canadian Institute for Advanced Research (CIFAR),
McGill University and the McGill Space Institute via the Trottier Family
Foundation, and the University of British Columbia. The Dunlap Institute is
funded through an endowment established by the David Dunlap family and the
University of Toronto. Research at Perimeter Institute is supported by the
Government of Canada through Industry Canada and by the Province of Ontario
through the Ministry of Research & Innovation. The National Radio Astronomy
Observatory is a facility of the National Science Foundation (NSF) operated
under cooperative agreement by Associated Universities, Inc. FRB research at
UBC is supported by an NSERC Discovery Grant and by the Canadian Institute for
Advanced Research. The CHIME/FRB baseband system is funded in part by a CFI
John R. Evans Leaders Fund award to IHS. We would like to thank all of the
essential workers who put their health at risk during the COVID-19 pandemic,
without whom we would not have been able to complete this work.

## References

  * Aasi et al. (2014) Aasi, J., et al. 2014, Phys. Rev. D, 89, 122004
  * Aasi et al. (2015) Aasi, J., et al. 2015, Class. Quant. Grav., 32, 074001
  * Abadie et al. (2012) Abadie, J., Abbott, B. P., Abbott, R., et al. 2012, ApJ, 760, 12
  * Abbott et al. (2016) Abbott, B., et al. 2016, Phys. Rev. D, 93, 122008
  * Abbott et al. (2017) Abbott, B. P., et al. 2017, Astrophys. J., 848, L13
  * Abbott et al. (2017a) Abbott, B. P., Abbott, R., Abbott, T. D., et al. 2017a, Phys. Rev. Lett., 119, 161101
  * Abbott et al. (2017b) —. 2017b, ApJ, 841, 89
  * Abbott et al. (2018) Abbott, B. P., et al. 2018, Phys. Rev. Lett., 121, 161101
  * Abbott et al. (2019a) Abbott, B. P., Abbott, R., Abbott, T. D., et al. 2019a, Phys. Rev. D, 100, 024017
  * Abbott et al. (2019b) —. 2019b, ApJ, 886, 75
  * Abbott et al. (2019c) —. 2019c, ApJ, 874, 163
  * Abbott et al. (2020) Abbott, B. P., Abbott, R., Abbott, T. D., et al. 2020, Living Reviews in Relativity, 23, 3
  * Abbott et al. (in preparation) Abbott, R., LIGO Scientific Collaboration, et al. in preparation, Search for gravitational wave transients associated with magnetar bursts during the third Advanced LIGO and Advanced Virgo observing run, , , in preparation
  * Abbott et al. (2021) Abbott, R., Abbott, T. D., Abraham, S., et al. 2021, ApJ, 915, 86
  * Accadia et al. (2012) Accadia, T., et al. 2012, JINST, 7, P03012
  * Acernese et al. (2015) Acernese, F., et al. 2015, Class. Quant. Grav., 32, 024001
  * Amiri et al. (2019a) Amiri, M., et al. 2019a, Nature, 566, 235
  * Amiri et al. (2019b) —. 2019b, Nature, 566, 230
  * Anderson et al. (2018) Anderson, M. M., Hallinan, G., Eastwood, M. W., et al. 2018, ApJ, 864, 22
  * Arun et al. (2009) Arun, K. G., Buonanno, A., Faye, G., & Ochsner, E. 2009, Phys. Rev., D79, 104023, [Erratum: Phys. Rev.D84,049901(2011)]
  * Babak et al. (2017) Babak, S., Taracchini, A., & Buonanno, A. 2017, Phys. Rev., D95, 024010
  * Bandura et al. (2014) Bandura, K., Addison, G. E., Amiri, M., et al. 2014, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9145, Ground-based and Airborne Telescopes V, ed. L. M. Stepp, R. Gilmozzi, & H. J. Hall, 914522
  * Bannister et al. (2017) Bannister, K. W., Shannon, R. M., Macquart, J.-P., et al. 2017, ApJ, 841, L12
  * Bannister et al. (2019) Bannister, K. W., Deller, A. T., Phillips, C., et al. 2019, Science, 365, 565
  * Bhandari et al. (2020) Bhandari, S., Sadler, E. M., Prochaska, J. X., et al. 2020, ApJ, 895, L37
  * Bhandari et al. (2021) Bhandari, S., Heintz, K. E., Aggarwal, K., et al. 2021, arXiv e-prints, arXiv:2108.01282
  * Bhardwaj et al. (2021a) Bhardwaj, M., Kirichenko, A. Y., Michilli, D., et al. 2021a, ApJ, 919, L24
  * Bhardwaj et al. (2021b) Bhardwaj, M., Gaensler, B. M., Kaspi, V. M., et al. 2021b, ApJ, 910, L18
  * Blanchet et al. (1996) Blanchet, L., Iyer, B. R., Will, C. M., & Wiseman, A. G. 1996, Class. Quant. Grav., 13, 575
  * Bohé et al. (2015) Bohé, A., Faye, G., Marsat, S., & Porter, E. K. 2015, Class. Quant. Grav., 32, 195010
  * Bohé et al. (2013) Bohé, A., Marsat, S., & Blanchet, L. 2013, Class. Quant. Grav., 30, 135009
  * Brown et al. (2012) Brown, D. A., Harry, I., Lundgren, A., & Nitz, A. H. 2012, Physical Review D, 86, doi:10.1103/physrevd.86.084017
  * Caleb et al. (2017) Caleb, M., Flynn, C., Bailes, M., et al. 2017, MNRAS, 468, 3746
  * Capano et al. (2016) Capano, C., Harry, I., Privitera, S., & Buonanno, A. 2016, PhRvD, 93, 124007
  * Champion et al. (2016) Champion, D. J., Petroff, E., Kramer, M., et al. 2016, MNRAS, 460, L30
  * Chatterjee et al. (2017) Chatterjee, S., Law, C. J., Wharton, R. S., et al. 2017, Nature, 541, 58
  * CHIME/FRB Collaboration (2020) CHIME/FRB Collaboration. 2020, The Canadian Hydrogen Intensity Mapping Experiment is a revolutionary new Canadian radio telescope designed to answer major questions in astrophysics and cosmology., https://chime-experiment.ca/, ,
  * CHIME/FRB Collaboration et al. (2020) CHIME/FRB Collaboration, Andersen, B. C., Bandura, K. M., et al. 2020, Nature, 587, 54
  * CHIME/FRB Collaboration et al. (2019) CHIME/FRB Collaboration, Andersen, B. C., et al. 2019, CHIME/FRB Detection of Eight New Repeating Fast Radio Burst Sources, , , arXiv:1908.03507
  * CHIME/FRB Collaboration et al. (2018) CHIME/FRB Collaboration, Amiri, M., Bandura, K., Berger, P., et al. 2018, ApJ, 863, 48
  * CHIME/FRB Collaboration et al. (2021) CHIME/FRB Collaboration, Amiri, M., et al. 2021, arXiv e-prints, arXiv:2106.04352
  * Cho (2019) Cho, M.-A. 2019, PhD thesis, University of Maryland
  * Cordes & Chatterjee (2019) Cordes, J. M., & Chatterjee, S. 2019, ARA&A, 57, 417
  * Cordes & Lazio (2002) Cordes, J. M., & Lazio, T. J. W. 2002, arXiv e-prints, astro
  * Corsi & Owen (2011) Corsi, A., & Owen, B. J. 2011, Phys. Rev. D, 83, 104014
  * Dal Canton & Harry (2017) Dal Canton, T., & Harry, I. W. 2017, arXiv:1705.01845
  * Dolag et al. (2015) Dolag, K., Gaensler, B. M., Beck, A. M., & Beck, M. C. 2015, MNRAS, 451, 4277
  * Falcke & Rezzolla (2014) Falcke, H., & Rezzolla, L. 2014, A&A, 562, A137
  * Foreman-Mackey et al. (2013) Foreman-Mackey, D., Hogg, D. W., Lang, D., & Goodman, J. 2013, PASP, 125, 306
  * Gao et al. (2016) Gao, H., Zhang, B., & Lü, H.-J. 2016, Phys. Rev. D, 93, 044065
  * Goodman & Weare (2010) Goodman, J., & Weare, J. 2010, Communications in Applied Mathematics and Computational Science, 5, 65
  * Gourdji et al. (2020) Gourdji, K., Rowlinson, A., Wijers, R. A. M. J., & Goldstein, A. 2020, MNRAS, 497, 3131
  * Grote (2010) Grote, H. 2010, Class. Quant. Grav., 27, 084003
  * Harry & Fairhurst (2011) Harry, I. W., & Fairhurst, S. 2011, Phys. Rev., D83, 084002
  * Harry et al. (2008) Harry, I. W., Fairhurst, S., & Sathyaprakash, B. S. 2008, Class. Quant. Grav., 25, 184027
  * Harry et al. (2014) Harry, I. W., Nitz, A. H., Brown, D. A., et al. 2014, Physical Review D, 89, doi:10.1103/physrevd.89.024010
  * Heintz et al. (2020) Heintz, K. E., Prochaska, J. X., Simha, S., et al. 2020, ApJ, 903, 152
  * Hessels et al. (2006) Hessels, J. W. T., Ransom, S. M., Stairs, I. H., et al. 2006, Science, 311, 1901
  * Ho et al. (2020) Ho, W. C. G., Jones, D. I., Andersson, N., & Espinoza, C. M. 2020, Phys. Rev. D, 101, 103009
  * Husa et al. (2016) Husa, S., Khan, S., Hannam, M., et al. 2016, Phys. Rev., D93, 044006
  * Ioka (2001) Ioka, K. 2001, MNRAS, 327, 639
  * James et al. (2019) James, C. W., Anderson, G. E., Wen, L., et al. 2019, MNRAS, 489, L75
  * Khan et al. (2016) Khan, S., Husa, S., Hannam, M., et al. 2016, Phys. Rev., D93, 044007
  * Kirsten et al. (2021) Kirsten, F., Marcote, B., Nimmo, K., et al. 2021, arXiv e-prints, arXiv:2105.11445
  * Kiziltan et al. (2013) Kiziltan, B., Kottas, A., De Yoreo, M., & Thorsett, S. E. 2013, ApJ, 778, 66
  * Kokkotas et al. (2001) Kokkotas, K. D., Apostolatos, T. A., & Andersson, N. 2001, MNRAS, 320, 307–315
  * Kreidberg et al. (2012) Kreidberg, L., Bailyn, C. D., Farr, W. M., & Kalogera, V. 2012, Astrophys. J., 757, 36
  * Kumar et al. (2019) Kumar, P., Shannon, R. M., Osłowski, S., et al. 2019, The Astrophysical Journal, 887, L30
  * Levin & van Hoven (2011) Levin, Y., & van Hoven, M. 2011, MNRAS, 418, 659
  * Li et al. (2019) Li, B.-A., Krastev, P. G., Wen, D.-H., & Zhang, N.-B. 2019, European Physical Journal A, 55, 117
  * Li & Zhang (2020) Li, Y., & Zhang, B. 2020, ApJ, 899, L6
  * LIGO Scientific Collaboration (2018) LIGO Scientific Collaboration. 2018, LIGO Algorithm Library, , , doi:10.7935/GT1W-FZ16
  * Lorimer et al. (2007) Lorimer, D. R., Bailes, M., McLaughlin, M. A., Narkevic, D. J., & Crawford, F. 2007, Science, 318, 777–780
  * Luo et al. (2020) Luo, R., Men, Y., Lee, K., et al. 2020, MNRAS, 494, 665
  * Lyons et al. (2010) Lyons, N., et al. 2010, MNRAS, 402, 705
  * Macquart et al. (2020) Macquart, J. P., Prochaska, J. X., McQuinn, M., et al. 2020, Nature, 581, 391
  * Marcote et al. (2020) Marcote, B., Nimmo, K., Hessels, J. W. T., et al. 2020, Nature, 577, 190
  * Margalit et al. (2019) Margalit, B., Berger, E., & Metzger, B. D. 2019, ApJ, 886, 110
  * Masui et al. (2015) Masui, K., Lin, H.-H., Sievers, J., et al. 2015, Nature, 528, 523
  * Metzger et al. (2017) Metzger, B. D., Berger, E., & Margalit, B. 2017, ApJ, 841, 14
  * Mikoczi et al. (2005) Mikoczi, B., Vasuth, M., & Gergely, L. A. 2005, Phys. Rev., D71, 124043
  * Miller & Miller (2014) Miller, M. C., & Miller, J. M. 2014, Phys. Rept., 548, 1
  * Mishra et al. (2016) Mishra, C. K., Kela, A., Arun, K. G., & Faye, G. 2016, Phys. Rev., D93, 084054
  * Moortgat & Kuijpers (2005) Moortgat, J., & Kuijpers, J. 2005, in 22nd Texas Symposium on Relativistic Astrophysics, ed. P. Chen, E. Bloom, G. Madejski, & V. Patrosian, 326–331
  * Newburgh et al. (2014) Newburgh, L. B., Addison, G. E., Amiri, M., et al. 2014, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 9145, Ground-based and Airborne Telescopes V, ed. L. M. Stepp, R. Gilmozzi, & H. J. Hall, 91454V
  * Nitz et al. (2020) Nitz, A., Harry, I., Brown, D., et al. 2020, gwastro/pycbc: PyCBC, Zenodo, doi:10.5281/zenodo.3961510
  * Nitz et al. (2017) Nitz, A. H., Dent, T., Dal Canton, T., Fairhurst, S., & Brown, D. A. 2017, ApJ, 849, 118
  * Owen & Sathyaprakash (1999) Owen, B. J., & Sathyaprakash, B. S. 1999, Phys. Rev. D, 60, 022002
  * Özel et al. (2010) Özel, F., Psaltis, D., Narayan, R., & McClintock, J. E. 2010, Astrophys. J., 725, 1918
  * Pace et al. (2012) Pace, A., Prestegard, T., Moe, B., & Stephens, B. 2012, Gravitational-Wave Candidate Event Database, https://gracedb.ligo.org, ,
  * Pan et al. (2014) Pan, Y., Buonanno, A., Taracchini, A., et al. 2014, Phys. Rev., D89, 084006
  * Petroff et al. (2019) Petroff, E., Hessels, J. W. T., & Lorimer, D. R. 2019, Astron. Astrophys. Rev., 27, 4
  * Petroff et al. (2016) Petroff, E., Barr, E. D., Jameson, A., et al. 2016, PASA, 33, e045
  * Planck Collaboration et al. (2016) Planck Collaboration, Ade, P. A. R., Aghanim, N., et al. 2016, A&A, 594, A13
  * Platts et al. (2019) Platts, E., Weltman, A., Walters, A., et al. 2019, Physics Reports, 821, 1–27
  * Popov & Postnov (2013) Popov, S. B., & Postnov, K. A. 2013, arXiv:1307.4924 [astro-ph], arXiv: 1307.4924
  * Prochaska & Zheng (2019) Prochaska, J. X., & Zheng, Y. 2019, MNRAS, 485, 648
  * Pshirkov & Postnov (2010) Pshirkov, M. S., & Postnov, K. A. 2010, Ap&SS, 330, 13
  * Quitzow-James et al. (2017) Quitzow-James, R., Brau, J., Clark, J. A., et al. 2017, Class. Quant. Grav., 34, 164002
  * Rafiei-Ravandi et al. (2021) Rafiei-Ravandi, M., et al. 2021, arXiv:2106.04354
  * Ravi (2019) Ravi, V. 2019, Nature Astronomy, 3, 928
  * Ravi & Lasky (2014) Ravi, V., & Lasky, P. D. 2014, MNRAS, 441, 2433
  * Rowlinson & Anderson (2019) Rowlinson, A., & Anderson, G. E. 2019, MNRAS, 489, 3316
  * Rowlinson et al. (2013) Rowlinson, A., O’Brien, P. T., Metzger, B. D., Tanvir, N. R., & Levan, A. J. 2013, MNRAS, 430, 1061
  * Rowlinson et al. (2010) Rowlinson, A., et al. 2010, MNRAS, 408, 383
  * Rowlinson et al. (2020) Rowlinson, A., Starling, R. L. C., Gourdji, K., et al. 2020, arXiv e-prints, arXiv:2008.12657
  * Sagiv & Waxman (2002) Sagiv, A., & Waxman, E. 2002, ApJ, 574, 861
  * Sarin & Lasky (2021) Sarin, N., & Lasky, P. D. 2021, General Relativity and Gravitation, 53, 59
  * Sathyaprakash & Dhurandhar (1991) Sathyaprakash, B. S., & Dhurandhar, S. V. 1991, Phys. Rev., D44, 3819
  * Shannon et al. (2018) Shannon, R. M., Macquart, J.-P., Bannister, K. W., et al. 2018, Nature, 1
  * Shibata et al. (2009) Shibata, M., Kyutoku, K., Yamamoto, T., & Taniguchi, K. 2009, Phys. Rev. D, 79, 044030
  * Spitler et al. (2014) Spitler, L. G., Cordes, J. M., Hessels, J. W. T., et al. 2014, ApJ, 790, 101
  * Sutton (2013) Sutton, P. J. 2013, arXiv e-prints, arXiv:1304.0210
  * Sutton et al. (2010) Sutton, P. J., Jones, G., Chatterji, S., et al. 2010, New Journal of Physics, 12, 053034
  * Taracchini et al. (2014) Taracchini, A., Buonanno, A., Pan, Y., et al. 2014, Phys. Rev. D, 89, 061502
  * Thornton et al. (2013) Thornton, D., Stappers, B., Bailes, M., et al. 2013, Science, 341, 53
  * Totani (2013) Totani, T. 2013, Publ Astron Soc Jpn Nihon Tenmon Gakkai, 65, doi:10.1093/pasj/65.5.L12
  * Troja et al. (2007) Troja, E., Cusumano, G., O’Brien, P. T., et al. 2007, ApJ, 665, 599
  * Urban (2016) Urban, A. L. 2016, PhD thesis
  * Usov & Katz (2000) Usov, V. V., & Katz, J. I. 2000, A&A, 364, 655
  * Wang et al. (2016) Wang, J.-S., Yang, Y.-P., Wu, X.-F., Dai, Z.-G., & Wang, F.-Y. 2016, ApJ, 822, L7
  * Was et al. (2012) Was, M., Sutton, P. J., Jones, G., & Leonor, I. 2012, Phys. Rev. D, 86, 022003
  * Wen et al. (2019) Wen, D.-H., Li, B.-A., Chen, H.-Y., & Zhang, N.-B. 2019, Physical Review C, 99, doi:10.1103/physrevc.99.045806
  * Williamson et al. (2014) Williamson, A. R., Biwer, C., Fairhurst, S., et al. 2014, Phys. Rev., D90, 122004
  * Yamasaki & Totani (2020) Yamasaki, S., & Totani, T. 2020, ApJ, 888, 105
  * Yamasaki et al. (2018) Yamasaki, S., Totani, T., & Kiuchi, K. 2018, PASJ, 70, 39
  * Yao et al. (2017) Yao, J. M., Manchester, R. N., & Wang, N. 2017, ApJ, 835, 29
  * Zhang (2014) Zhang, B. 2014, ApJ, 780, L21
  * Zhang (2016) —. 2016, ApJ, 827, L31
  * Zhang (2018) —. 2018, ApJ, 867, L21
  * Zhang (2019) —. 2019, arXiv e-prints, arXiv:1901.11177
  * Zhang (2020a) —. 2020a, ApJ, 890, L24
  * Zhang (2020b) —. 2020b, Nature, 587, 45
  * Zink et al. (2012) Zink, B., Lasky, P. D., & Kokkotas, K. D. 2012, Phys. Rev. D, 85, 024030

  *[CHIME]: the Canadian Hydrogen Intensity Mapping Experiment
  *[GW]: gravitational wave
  *[O3a]: the first part of the third observing run of Advanced LIGO and Advanced Virgo
  *[BNS]: binary neutron star
  *[NS]: neutron star
  *[GRB]: gamma-ray burst
  *[NSBH]: neutron star-black hole
  *[NSs]: neutron star
  *[BH]: black hole
  *[DM]: dispersion measure
  *[RA]: right ascension
  *[Dec]: declination
  *[GRBs]: gamma-ray burst
  *[GWs]: gravitational wave
  *[BHs]: black hole
  *[SNR]: signal-to-noise ratio
  *[CBC]: Compact Binary Coalescence
  *[SNRs]: signal-to-noise ratio
  *[SG]: sine–Gaussian
  *[O3]: the third observing run of Advanced LIGO and Advanced Virgo
  *[ADI]: accretion disk instability
  *[H1]: LIGO Hanford
  *[L1]: LIGO Livingston
  *[O3b]: the second part of the third observing run of Advanced LIGO and Advanced Virgo
  *[O4]: the fourth observing run of Advanced LIGO, Advanced Virgo and Kagra
  *[FRBs]: fast radio burst


# Guiding Online Reinforcement Learning with Action-Free Offline Pretraining

###### Abstract

Offline RL methods have been shown to reduce the need for environment
interaction by training agents using offline collected episodes. However,
these methods typically require action information to be logged during data
collection, which can be difficult or even impossible in some practical cases.
In this paper, we investigate the potential of using action-free offline
datasets to improve online reinforcement learning, name this problem
Reinforcement Learning with Action-Free Offline Pretraining (AFP-RL). We
introduce Action-Free Guide (AF-Guide), a method that guides online training
by extracting knowledge from action-free offline datasets. AF-Guide consists
of an Action-Free Decision Transformer (AFDT) implementing a variant of
Upside-Down Reinforcement Learning. It learns to plan the next states from the
offline dataset, and a Guided Soft Actor-Critic (Guided SAC) that learns
online with guidance from AFDT. Experimental results show that AF-Guide can
improve sample efficiency and performance in online training thanks to the
knowledge from the action-free offline dataset. Code is available at
https://github.com/Vision-CAIR/AF-Guide. ††Preprint

Machine Learning, ICML

Deyao Zhu1 Yuhui Wang1 Jüergen Schmidhuber1,2,3 Mohamed Elhoseiny1
1King Abdullah University of Science and Technology
2The Swiss AI Lab IDSIA/USI/SUPSI 3NNAISENSE
{deyao.zhu, yuhui.wang, juergen.schmidhuber<EMAIL_ADDRESS>

##  1 Introduction

Training a reinforcement learning agent directly in the environment from
scratch can be a challenging task, as it usually requires a large number of
time-consuming and costly interaction steps to explore the environment. As a
result, improving sample efficiency in RL has become one of the most important
directions in the reinforcement learning community. Offline reinforcement
learning methods use offline collected episodes only to train RL agents. After
the offline RL training, RL agents can be further online finetuned in the
environment with much fewer interaction steps. To apply offline RL methods,
actions need to be logged when collecting the offline episodes. However,
recording actions (e.g., motor torques) can be difficult or even impossible in
certain practical cases, like learning from large-scale internet videos.
Despite the lack of actions, these data (e.g., a video of a robot making a
pizza) still hold valuable information about agents’ movements and
environments’ transitions, which can be utilized to instruct the RL agent on
the distinctions between advantageous and detrimental movements.

In this paper, we explore the potential of utilizing action-free offline
reinforcement learning datasets to guide online Reinforcement Learning. We
name this setting Reinforcement Learning with Action-Free Offline Pretraining
(AFP-RL). We propose Action-Free Guide (AF-Guide), a method that improves
online training by learning to plan good target states from the action-free
offline dataset. AF-Guide comprises two main components that we introduce: an
Action-Free Decision Transformer (AFDT), and a Guided Soft Actor-Critic
(Guided SAC). AFDT, a variant of the Upside Down Reinforcement Learning
(Schmidhuber, 2019) model Decision Transformer (Chen et al., 2021), is trained
on an offline dataset without actions to plan the next states based on the
past states and the desired future returns. Guided SAC, a variation of SAC
(Haarnoja et al., 2018a, b), follows the planning of AFDT by maintaining an
additional Q function that fits an intrinsic reward built from the discrepancy
between the planned state and the achieved state with zero discount factor. An
overview of our method is summarized in Fig.1. Our experimental results
demonstrate that AF-Guide can significantly improve sample efficiency during
online training by utilizing action-free offline datasets.

Figure 1: An overview of AF-Guide. Action-Free Decision Transformer (AFDT) is
trained on the action-free offline dataset to plan the next state
$\tilde{s}_{t+1}$ given previous states and the desired return-to-go
$\hat{R}_{t}$. The guiding reward $r_{g}$ is formed based on the negative L2
distance between the planned state $\tilde{s}_{t+1}$ and the real state
$s_{t+1}$. In addition to SAC’s original Q function denoted as $Q_{e}$ that
fits the environment reward $r_{e}$, Guided SAC has an additional Q function
$Q_{g}$ to fit the guiding reward $r_{g}$ with zero discount factor to discard
the future return. The policy is trained by the weighted sum over the two Q
functions.

Our contribution can be summarized as follows:

  * •

We propose Reinforcement Learning with Action-Free Offline Pretraining (AFP-
RL), a novel setting to study how to guide online Reinforcement Learning with
action-free offline datasets.

  * •

We present Action-Free Guide (AF-Guide), a method that pretrains a model which
can extract knowledge from the action-free offline dataset and conduct state-
space planning to guide online policy learning.

  * •

Experimental results show that AF-Guide can benefit from the action-free
offline dataset to improve sample efficiency and performance during online
training.

##  2 Related Work

#### Offline Reinforcement Learning

Offline reinforcement learning methods learn policies using pre-collected
episodes from unknown behavior policies. Many offline RL methods, such as CQL
(Kumar et al., 2020), IQL (Kostrikov et al., 2021), AWAC (Nair et al., 2020),
BCQ, and COMBO (Yu et al., 2021), have been developed from off-policy
algorithms, with additional constraints to avoid out-of-distribution actions
that are not covered by the dataset. Recently, Decision Transformer (Chen et
al., 2021) and Trajectory Transformer (Janner et al., 2021) convert the
offline RL problem as a context-conditioned sequential modeling problem and
generate good actions by either conditioning on desired future return
following the Upside Down Reinforcement Learning framework (Schmidhuber, 2019)
or searching for a good rollout with a high future return. In our AFP-RL
setting, datasets do not contain explicitly labeled actions (although the
dataset may contain videos of acting agents). In this case, learning an
offline policy directly is infeasible. Our method AFDT-Guide instead leverages
action-free data to plan good target states and guide online training for
improved performance.

#### Imitation Learning from Observation

The target of imitation learning from observation is to learn a policy through
state-only action-free demonstrations from experts. imitation Learning from
observation methods can be broadly classified into different categories.
Methods like GSP (Pathak et al., 2018) and BCO (Torabi et al., 2018a) train an
inverse dynamic model to infer the expert actions given state transitions.
Intrinsic-reward-based methods like DeepMimic (Peng et al., 2018), Context-
Aware Translation (Liu et al., 2018), and Lee et al. (2021), create surrogate
reward functions to guide the online training. Other methods like GAIfO
(Torabi et al., 2018b), IDDM (Yang et al., 2019), MobILE (Kidambi et al.,
2021) employ adversarial learning. The difference between imitation learning
from observation and our setting AFP-RL is similar to the difference between
imitation learning and offline RL. In imitation learning from observation, the
dataset is collected by an expert policy, and agents are trained to directly
imitate the collected episodes. In contrast, episodes in AFP-RL are collected
by behavior policies that may be suboptimal. As a result, directly imitating
these episodes would lead to suboptimal performance.

#### Motion Forecasting

Motion forecasting is the task of predicting the future motion of agents given
past and context information. It helps autonomous systems like autonomous
driving and robotics to foresee and avoid potential risks like collisions in
advance. Recent methods for motion forecasting have explored various
architectural designs. For example, Social-LSTM (Alahi et al., 2016) and
Trajectron++ (Salzmann et al., 2020) are based on RNN. Social-GAN (Gupta et
al., 2018) and HalentNet (Zhu et al., 2021) benefit from generative
adversarial training. Social-STGCNN (Mohamed et al., 2020) and Social-Implicit
(Mohamed et al., 2022) predict the future via spatial-temporal convolution.
AgentFormer (Yuan et al., 2021), mmTransformer (Liu et al., 2021), and ST-
Transformer (Aksan et al., 2021) are models based on Transformer architecture
(Vaswani et al., 2017) designed for pedestrian or vehicle trajectory
prediction. Our state-planner AFDT is a Transformer model. Instead of simply
predicting the future states conditioned on the past, AFDT plans the future
states by additionally conditioning on the desired future return in the UDRL
framework (Schmidhuber, 2019).

##  3 Background

#### Soft Actor-Crtic (SAC)

SAC is an actor-critic RL approach based on the maximum entropy framework
(Haarnoja et al., 2018a, b), which involves optimizing a Q network $Q_{e}$
111We use the subscript $e$ to denote notations related to the environment
reward, and will use $g$ to differentiate the notations related to the guiding
reward (see Section 4.2). and the policy network $\pi$. The Q function $Q_{e}$
is learned with the following objective

$\min_{Q_{e}}\mathbb{E}_{\mathcal{D}_{\mathrm{online}}}\|Q_{e}(s_{t},a_{t})-Q_{e,t}^{\rm
target}\|_{2}^{2}$ (1)

where
$\mathcal{D}_{\mathrm{online}}\triangleq\\{(s_{t},a_{t},r_{e,t},s_{t+1})\\}$
with state $s_{t}$, action $a_{t}$, environment reward $r_{e,t}$, and next
state $s_{t+1}$, is the online replay buffer. $Q_{e,t}^{\rm target}$ is the
target Q value computed as follows

$Q_{e,t}^{\rm
target}=r_{e,t}+\gamma\mathbb{E}_{\pi}\left[Q_{e}(s_{t+1},a_{t+1})-\alpha\log\pi(a_{t+1}|s_{t+1})\right]$
(2)

Here, $\gamma$ is the discount factor and $\alpha$ is the temperature
parameter to weight the entropy. The policy network is learned by the
following objective

$\min_{\pi}\mathbb{E}_{s_{t}\sim\mathcal{D}_{\mathrm{online}},a_{t}\sim\pi}\left[\alpha\log(\pi(a_{t}|s_{t}))-Q_{e}(s_{t},a_{t})\right]$
(3)

#### Upside Down Reinforcement Learning and Decision Transformer

Traditional Reinforcement Learning methods are trained to predict future
rewards (e.g., a Q function) first and convert the prediction into rewarding
actions. In contrast, Upside Down Reinforcement Learning (UDRL) (Schmidhuber,
2019) framework takes desired future rewards as inputs to generate actions. As
an instance of UDRL in the offline RL setting, Decision Transformer (DT) (Chen
et al., 2021) is trained in the offline dataset to regress the current action
$a_{t}$ conditioned on the past $K$ states $s_{t-k:t}$, actions $a_{t-k:t-1}$,
and the future returns (named Return-To-Go, RTG) $\hat{R}_{t-k:t}$, with
$\hat{R}_{t}=\sum_{t^{\prime}=t}^{T}r_{t^{\prime}}$. The architecture of DT is
based on the language model GPT (Radford et al., 2018). When evaluated in an
environment, the model is provided with an initial state $s_{0}$ and a desired
initial RTG $\hat{R}_{0}$ to generate the first action. After executing the
action $a_{t}$ in the environment and observing the reward $r_{t}$ and the
next state $s_{t+1}$, the RTG is updated by $\hat{R}_{t+1}=\hat{R}_{t}-r_{t}$.
The executed action $a_{t}$, the current return-to-go $\hat{R}_{t+1}$, and the
current state $s_{t+1}$ are then fed back into DT to infer the next action.
Given a high initial RTG $\hat{R}_{0}$, DT is able to generate good actions
that lead to high future returns. Due to the dependence of standard DT on
action labels, it can not be directly applied for Action Free Pretraining.

##  4 Action-Free Guide

#### Action-Free Offline Pretraining

In the setting of Reinforcement Learning with Action-Free Offline Pretraining
(AFP-RL), an action-free offline dataset,
$\mathcal{D}=\\{\tau_{1},\tau_{2},...,\tau_{N}\\}$, is provided to boost the
online training in the environment. The trajectories in the dataset have been
pre-collected in the environment by behavior policies that are unknown to the
agent. Each trajectory, $\tau$, contains states and rewards in the format
$\tau=(s_{0},r_{0},s_{1},r_{1},...,s_{T},r_{T})$, with $T$ time steps. Unlike
traditional offline RL, where the policy is learned directly from the offline
dataset, it is infeasible to learn a policy from an action-free offline
dataset as it lacks the necessary action information. However, such a dataset
still contains valuable information about the agent’s movements and the
environment’s dynamics. Our proposed setting, AFP-RL, aims to leverage this
information to improve online training.

#### Methodology Overview

Our method, Action-Free Guide (AF-Guide), utilizes knowledge from action-free
offline datasets by training an Action-Free Decision Transformer (AFDT) on
these datasets to plan the next states that lead to high future returns. Then,
the online agents, trained by Guided Soft Actor-Critic (Guided SAC), follow
the planning with an additional Q function optimized for an intrinsic reward
based on the planned states. AF-Guide is similar to the Learning-to-Think
framework (Schmidhuber, 2015, 2018): Guided SAC sends queries (in this case
desired future returns) into AFDT and learns to use the answers (in this case
good next states) to improve its own performance. The overall methodology is
illustrated in Fig.1.

###  4.1 Action-Free Decision Transformer

Figure 2: Action-Free Decision Transformer. The next state is planned given
previous states and a desired future return, named return-to-go.

Action-Free Guide(AF-Guide) can be considered as a variant of the UDRL model
Decision Transformer (DT) (Chen et al., 2021) that we designed to operate on
action-free offline datasets. Unlike DT, which predicts actions based on past
RTGs, states, and actions, AFDT plans the next state based on previous states
and RTGs only. The overall architecture of AFDT is illustrated in Fig.2. AFDT
takes $K$ steps of input, consisting of $2K$ tokens, where each step contains
a state and an RTG. Similar to DT, states and RTGs are first mapped to token
embedding via separate single-layer state and return-to-go encoders
$\text{Embed}_{s}$ and $\text{Embed}_{R}$. The positional embedding mapped
from time steps $t$ by a single-layer temporal encoder $\text{Embed}_{t}$ is
then added to the token embedding to include temporal information, followed by
layer normalization. These token embeddings are then processed by a GPT model
(Radford et al., 2018). The next states are generated from the processed RTG
tokens through a single-layer decoder $\text{Pred}_{s}$. Note that we don’t
predict the next state $s_{t+1}$ directly, but rather predict the state change
$\Delta s_{t+1}=s_{t+1}-s_{t}$ first and add it back to $s_{t}$ to obtain
$s_{t+1}$. This is a common practice in motion forecasting (e.g., Mohamed et
al. (2020); Salzmann et al. (2020)) to improve the prediction accuracy and has
been observed to improve the performance of our model in experiments. The
algorithm of AFDT is listed in Algo.1.

#### Training

At each training step, a batch of trajectories truncated to length $K$ is
randomly sampled from the dataset. Each trajectory contains states and
precomputed RTGs, represented as
$\tau=(s_{t-K+1},\hat{R}_{t-K+1},...,s_{t},\hat{R}_{t})$. The model is trained
autoregressively with L1 loss to predict the next state from the processed RTG
token at each time step, using a causal mask to mask out future information.

Algorithm 1 Action-Free Guide

Input: states $s$, returns-to-go $\hat{R}$, time steps $t$

# get positional embedding for each time step

$f_{t}$ = $\text{Embed}_{t}(t)$

# compute the state and return-to-go embeddings

$f_{s},~{}f_{\hat{R}}$ =
$\text{Embed}_{s}(s)+f_{t},~{}\text{Embed}_{R}(\hat{R})+f_{t}$

# send to transformer in the order ($s_{0},\hat{R}_{0},s_{1},\hat{R}_{1},...$)

$f_{output}$ = Transformer(stack($f_{s}$, $f_{\hat{R}}$))

# predict the state change

$\Delta s$ = $\text{Pred}_{s}$(unstack($f_{output}$.states))

Output: $\Delta s$ \+ s

###  4.2 Guided Soft Actor-Critic

Now we illustrate how to use the AFDT model to benefit the learning of Soft
Actor-Critic (SAC) (Haarnoja et al., 2018a, b). As AFDT can conduct planning
in the state space and infer the subsequent states that lead to a high future
return, our idea is to guide the agent to follow AFDT’s planning. Our method,
named _Guided SAC_ , contains the following three main procedures.

#### Guiding Reward

We first design an intrinsic reward $r_{g,t}$, named _guiding reward_ , which
is the discrepancy between the planned state $\widetilde{s}_{t+1}$ inferred by
AFDT and the actual state $s_{t+1}\sim\mathrm{P}(\cdot|s_{t},a_{t})$ achieved
by the agent:

$r_{g,t}=-\|\frac{1}{\sigma_{\mathcal{D}}}\odot(\widetilde{s}_{t+1}-s_{t+1})\|_{2}$
(4)

where $\sigma_{\mathcal{D}}$ is the standard deviation of the states over the
entire offline dataset ${\mathcal{D}}$ and is used to normalize the different-
scale state values on different dimensions. The process to compute the guiding
reward with the AFDT model is summarized in Algo.2.

Algorithm 2 Compute Guiding Reward

Input: states $s_{1:t}$, return-to-go $\hat{R}_{1:t}$, policy $\pi$, state
standard deviation $\sigma_{s}$, environment $env$, AFDT with context length
$K$

repeat

# get AFDT’s prediction of the next state

$\widetilde{s}_{t+1}$ = AFDT($s_{t-K+1:t}$, $\hat{R}_{t-K+1,t}$)

# apply the policy in the environment for one step

$a_{t}$ = $\pi(s_{t})$

$s_{t+1}$, $r_{e}$ = $env$.step($a_{t}$)

# compute current guiding reward using Eq.4

$r_{g}$= $-\|\frac{1}{\sigma_{s}}\odot(\widetilde{s}_{t+1}-s_{t+1})\|_{2}$

# update return-to-go (same as DT) and time step

$\hat{R}_{t+1}$ = $\hat{R}_{t}$ \- $r_{e}$

$t=t+1$

until Episode is finished

(a) Halfcheetah

(b) Hopper

(c) Walker2d

(d) Maze2d-Medium

(e) Maze2d-Large

(f) Antmaze-Umaze

Figure 3: Experimental results of our methods. Utilizing the knowledge learned
from the action-free offline dataset, AF-Guide outperforms SAC in all
evaluated locomotion and ball maze environments in terms of learning speed.
Furthermore, while SAC struggles to complete the task of Antmaze-Umaze due to
the challenging exploration, AF-Guide successfully solves it, owing to the
guidance signals provided by AFDT.

#### Guiding Q Function

We then use the guiding reward $r_{g}$ to learn the Q function. A common
practice is to combine the intrinsic reward and the environment reward by
$r_{t}=r_{e,t}+\beta r_{g,t}$ with a coefficient $\beta$, and use a single Q
network $Q$ to approximate the long-term future return (Schmidhuber, 1990,
1991; Houthooft et al., 2016; Pathak et al., 2017; Tao et al., 2020). However,
this is not the case for the guiding reward, where _the current action should
only be responsible for the next immediate result rather than all the future
results_. Assume a robot gets stuck at step $t+1$ due to a bad action $a_{t}$
at step $t$. A good AFDT will give the robot a low guidance reward at step $t$
and predict a static future, resulting in high future guidance rewards for
getting the robot stuck in the same state. More generaly, as AFDT replans the
target states at every timestep, an agent missing the planned state
$\tilde{s}_{t}$ due to a bad action $a_{t-1}$ can still reach the replanned
state $\tilde{s}_{t+1}$ at the next step and receive a high guiding reward
$r_{g,t}$, which is not desirable. Hence, the action $a_{t}$ should not be
rewarded by $r_{g,t+1}$ as it didn’t reach the original plan $\tilde{s}_{t}$.
Therefore, to prevent the guiding reward from misleading the agent, it is more
reasonable to discard the future return for the Q value calculation of the
current action.

Due to the reason above, we set up an additional independent Guiding Q
function $Q_{g}$ which is optimized in the same way as the original Q function
$Q_{e}$ (see Eq.1), but the target Q value only involve the immediate reward
$r_{g,t}$ without future information, which is computed as follows:

$Q_{g,t}^{\rm target}=r_{g,t}$ (5)

Compared to Eq.(2), here the Q target of the current action is removed from
the future information by setting the discount factor $\gamma$ to zero. Our
ablation study in Sec.5.2 demonstrates that the Guiding Q function is crucial
to effective guidance.

#### Combined Q function

We finally replace the Q function $Q_{e}$ in Eq.(3) with the following
combined Q function to guide the policy learning:

$Q(s_{t},a_{t})=Q_{e}(s_{t},a_{t})+\beta Q_{g}(s_{t},a_{t})$ (6)

where $\beta$ is the coefficient. Note that when $\beta=0$, Guided SAC
degenerates to a standard SAC trained using environment rewards $r_{e}$ and
the corresponding Q function $Q_{e}$ only.

##  5 Experiments

(a) Halfcheetah

(b) Walker2d

(c) Maze2d-Medium

Figure 4: Ablation study on the usage of guiding reward $r_{g}$. ‘AF-Guide
[SAC]’ denotes the variant adding guiding reward to the environment reward and
training with SAC. The results show that AF-Guide [SAC] performs similarly to
SAC in Maze2d-Medium, but does not work in Halfcheetah and Walker2d, which
indicates that simply adding the guiding reward is detrimental to the policy
training and verifies the effectiveness of our Guided SAC design.

In this section, we demonstrate the effectiveness of our approach AF-Guide for
utilizing action-free offline reinforcement learning datasets in online
reinforcement learning through experimental evaluation. Furthermore, we
provide evidence for the validity of our design choices for the two components
of AF-Guide, Action-Free Decision Transformer, and Guided SAC, through three
ablation studies.

#### Action-Free D4RL Benchmark

To evaluate methods on AFP-RL, we build on top of the widely-used offline
reinforcement learning benchmark, D4RL (Fu et al., 2020), and adapt it to the
action-free reinforcement learning setting. We denote the introduced benchmark
as Action-Free D4RL. The original D4RL benchmark provides offline datasets
collected using various strategies across different environments. These
episodes in the original D4RL datasets include state, action, and reward
sequences. To create our action-free offline RL datasets, we remove the action
labels from the original datasets. We evaluate six environments, including
three locomotion tasks (Hopper, Halfcheetah, Walker2d), two ball maze
environments (Maze2d-Medium, Maze2d-Large), and one robot ant maze environment
(Antmaze-Umaze). For each locomotion task, we test our method on three
different datasets: Medium, Medium-Replay, and Medium-Expert. For the
environment Antmaze-Umaze, we test on two datasets: Antmaze-Umaze and Antmaze-
Umaze-Diverse. There is only one dataset for each ball maze environment, where
the ball navigates to random goal locations. Details of the datasets can be
found in Appx.A.

#### Implementation Details

The training of AF-Guide contains two stages: an offline stage training AFDT
using the offline dataset and an online stage training Guided SAC in the
environment. For the architecture and the training of AFDT, we follow the
default hyperparameters used in DT paper (Chen et al., 2021). The context
length $K$ is set to 20. The batch size for AFDT training is 64 and the
learning rate is 1e-4 with AdamW optimizer. In the online training stage, we
set the return-to-go $\hat{R}$ to 6000, 3600, and 5000 for Halfcheetah,
Hopper, and Walker2d, respectively, the same as the values used in the
original DT paper. The robot ant maze environment and the ball maze
environments are not used in the original DT paper. We set $\hat{R}$ to 1 and
5000, separately. For the hyperparameters of Guided SAC, we follow the default
hyperparameters of SAC in the widely used Stable Baseline 3 (Raffin et al.,
2021) implementation. The training batch size is 256 and the learning rate is
3e-4 with Adam optimizer. The discount factor for the environment reward is
0.99. The coefficient of the Guided Q function $\beta$ in Eq.6 is set to 3.
More details of the hyperparameters can be found in Appx.B.

###  5.1 Main Experiments

(a) Halfcheetah

(b) Walker2d

(c) Maze2d-Large

Figure 5: Ablation study on the effectiveness of Action-Free Decision
Transformer (AFDT). We train a variant of AFDT by regressing the behavior
policy trajectories, and use this variant to guide the online training,
referred to as AF-Guide [Imi]. Compared to AF-Guide, AF-Guide [Imi] performs
worse in Walker2d and Maze2d-Large. This suggests that AFDT can infer the next
states better than those collected by the behavior policy.

#### Results Analysis

Experimental results are presented in Figure 3. We run each experiment four
times, using different random seeds, and report the average and the standard
deviation band. Our method AF-Guide , using knowledge learned from the action-
free offline dataset, outperforms SAC in all the evaluated environments. In
the tasks of Halfcheetah and Walker2d, AF-Guide shows a significant advantage
in learning speed compared to SAC across all the three datasets. In
Halfcheetah, AF-Guide demonstrates a significant improvement of 50% at 500k
steps, with an achieved performance of 6000 compared to the 4000 achieved by
SAC alone. Similarly, in Walker2d, AF-Guide improves the performance by 50% at
1M steps, from 2000 to 3000. Additionally, we observe that different offline
datasets do not result in significant performance differences. In the tasks of
Hopper, Maze2d-Medium, and Maze2d-Large, while both AF-Guide and SAC reach
similar performance at 500k steps, AF-Guide converges faster. In the task of
Antmaze-Umaze, SAC is unable to complete it in 1M steps, whereas AF-Guide has
an 80% success rate when pretrained in the dataset Antmaze-Umaze and a 60%
success rate in Antmaze-Umaze-Diverse. This is likely due to the exploration
challenge faced by SAC. The robot ant in Antmaze-Umaze has 4 legs with a total
of 8 joints and only receives rewards when reaching the target location,
resulting in large state/action spaces and sparse reward signals. As a result,
the agent trained by SAC does not receive any rewards during exploration and
does not know how to move. In contrast, our guiding reward learned from the
action-free offline dataset provides dense learning signals that guide the
agent’s motion towards the target. Therefore, agents trained by AF-Guide can
successfully solve the maze in this task.

###  5.2 Ablation Study

#### Do we really need Guided SAC?

In this ablation study, we investigate whether our Guided SAC with an
additional Q function is necessary to process the guiding reward $r_{g}$, or
if it can be simply added to the environment reward and processed by SAC,
referred to as ‘AF-Guide [SAC]’. This study is conducted in the locomotion
environments of Halfcheetah and Walker2d using the Medium dataset and the maze
environment of Maze2d-Medium. The results, shown in Fig.4, reveal that AF-
Guide [SAC] has a similar performance as SAC in Maze2d-Medium and does not
work at all in Halfcheetah and Walker2d, indicating that the guiding reward
$r_{g}$ does not help or even hinders the training of SAC. In contrast, the
original AF-Guide with Guided SAC benefits from the guiding reward $r_{g}$ by
ignoring guiding rewards in future steps and setting the corresponding
discount factor to zero. This is in line with our explanation in Sec.4.2,
where we stated that high future guiding rewards are unrelated to the action
quality at the current step and therefore should be ignored in the Q function.
Experimental results verify the effectiveness of our Guided SAC design.

#### Does AFDT plan better states than that from behavior policy?

In this ablation study, we investigate whether the states planned by Action-
Free Decision Transformer (AFDT) are superior to the original states collected
by the behavior policy. To do this, we trained an AFDT in an ‘imitation’ style
by directly regressing future states based solely on past states without any
return-to-go information in the offline dataset. We denote this AFDT variant
as AF-Imitation. In the online training stage, AF-Imitation plans the next
state based on past states only without returns-to-go. We refer to the method
trained with the guidance of AF-Imitation as ‘AF-Guide [Imi]’ and evaluate it
in the locomotion environments of Halfcheetah and Walker2d using the Medium-
Replay dataset and the maze environment of Maze2d-Large. The results, shown in
Fig.5, demonstrate that AF-Guide [Imi] performs worse than the original
version in Walker2d and Maze2d-Large and is close to the original version in
Halfcheetah. This verifies that AFDT can plan the next states that lead to
higher future returns than the behavior policy when conditioned on a proper
return-to-go. Additionally, AF-Guide [Imi] performs better than SAC in
Halfcheetah and slightly better in Walker2d. While the predicted state from
AF-Guide [Imi] may not be optimal, it can still benefit policy training in
some cases.

Table 1: Training time comparison of AF-Guide [TT] and AF-Guide for 500k
environment steps in locomotion tasks. AF-Guide [TT] increases the training
time dramatically, due to the huge planning cost in the original Trajectory
Transformer design. Experiments are done using a single A100 GPU.
Env. | AF-Guide [TT] | AF-Guide
---|---|---
Halfcheetah |  $\sim$14 hours |  $\sim$1 hour
Hopper |  $\sim$10 hours |  $\sim$1 hour
Walker2d |  $\sim$20 hours |  $\sim$1 hour

(a) Halfcheetah

(b) Hopper

(c) Walker2d

Figure 6: Ablation study on using Action-Free Trajectory Transformer to guide
the training (AF-Guide [TT]). The results showed that AF-Guide [TT] had better
performance in the Hopper and Walker2d tasks, but performed worse in the
Halfcheetah task. These results suggest that our pipeline is compatible with
different sequential-modeling-based offline RL methods, but the choice of
method may impact performance depending on the specific task.

#### Can Action-Free Trajectory Transformer replace Action-Free Decision
Transformer?

Guided SAC is built on the predictions of AFDT, our action-free variant of the
Decision Transformer (DT). In theory, AFDT can be replaced by any other
sequential-modeling-based offline RL method after removing the action
information. In this experiment, we replaced AFDT with an action-free variant
of Trajectory Transformer (TT) (Janner et al., 2021), and evaluated its
performance on locomotion tasks using the Medium dataset. We denote this
variant as AF-Guide [TT]. Details of AF-Guide [TT] can be found in Appx.C.
Compared to DT, which generates the future in a UDRL style, TT directly
rollouts future predictions via beam search and selects the predictions with
the highest predicted future returns. Additionally, TT discretizes the state
and action spaces dimension-by-dimension for improved prediction accuracy.
Experimental results are shown in Fig.6. AF-Guide [TT] performs worse than AF-
Guide in Halfcheetah, but it showed a clear advantage over AF-Guide in Hopper
and Walker2d. This advantage of AF-Guide [TT] in Hopper and Walker2d may be
due to the better prediction quality from the dimension-wise discretization.
However, the training time of AF-Guide [TT] is much longer than AF-Guide. As
TT discretizes the state dimension-by-dimension, predicting one step of the
state requires multiple forward passes of the transformer model, and to pick
the best prediction, TT needs to generate multiple rollouts via beam-search.
This dramatically increases the computational cost of future reasoning and
slows down the training. A brief comparison of the training time between AF-
Guide and AF-Guide [TT] is shown in Tab.1. AF-Guide [TT] is at least 10 times
slower than AF-Guide in our experiments. The more dimensions the state space
has, the slower AF-Guide [TT] is. Therefore, we select Decision Transformer in
our method. The experiments show that our pipeline is compatible with
different sequential-modeling-based offline RL methods.

###  5.3 Limitations

As an attempt to utilize action-free offline datasets for improved online
learning, our method has some limitations in its current form. Firstly, our
current design of AF-Guide is based on Decision Transformer following UDRL
framework (Schmidhuber, 2019). Therefore, our planning ability is also limited
by Decision Transformer’s design and UDRL’s drawback: Štrupl et al. (2022)
show that UDRL may diverge from the optimal policy in an episodic setting with
stochastic environments. Note that the framework of our method AF-Guide is
agnostic to the sequential planning model as we show in the ablation study
with Trajectory Transformer based on beam-search instead of UDRL in Sec.5.2.
Secondly, our current guiding reward is based on L2 distance, which may not be
optimal in some state spaces where L2 distance doesn’t represent the state
similarities well, such as images We believe that combining AF-Guide with more
semantically meaningful similarity metrics can extend its applications in the
future for vision, language, and other multimodal problems. We leave this for
future research to explore.

##  6 Conclusion

In this paper, we explore the potential of utilizing action-free offline
datasets to guide online reinforcement learning, and denote this setting by
Reinforcement Learning with Action-Free Offline Pretraining (AFP-RL). We
propose Action-Free Guide (AF-Guide), a method that learns to plan the target
state from offline datasets to guide the online learning of an SAC agent. Our
experimental results demonstrate that AF-Guide has better sample efficiency
than SAC in various locomotion and maze environments, highlighting the
benefits of incorporating action-free offline datasets. We hope our work may
encourage further research in other areas where action-free offline
pretraining can be an effective learning approach. These applications may
include combining video prediction models with semantically meaningful
similarity metrics to build guidance rewards learned from large-scale Internet
data.

## References

  * Aksan et al. (2021) Aksan, E., Kaufmann, M., Cao, P., and Hilliges, O.  A spatio-temporal transformer for 3d human motion prediction.  In _2021 International Conference on 3D Vision (3DV)_ , pp. 565–574. IEEE, 2021.
  * Alahi et al. (2016) Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., and Savarese, S.  Social lstm: Human trajectory prediction in crowded spaces.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 961–971, 2016.
  * Barhate (2022) Barhate, N.  Minimal implementation of decision transformer.  https://github.com/nikhilbarhate99/min-decision-transformer, 2022\.
  * Chen et al. (2021) Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I.  Decision transformer: Reinforcement learning via sequence modeling.  _Advances in neural information processing systems_ , 34:15084–15097, 2021.
  * Fu et al. (2020) Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S.  D4rl: Datasets for deep data-driven reinforcement learning.  _arXiv preprint arXiv:2004.07219_ , 2020.
  * Gupta et al. (2018) Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., and Alahi, A.  Social gan: Socially acceptable trajectories with generative adversarial networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2255–2264, 2018.
  * Haarnoja et al. (2018a) Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International conference on machine learning_ , pp. 1861–1870. PMLR, 2018a.
  * Haarnoja et al. (2018b) Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., et al.  Soft actor-critic algorithms and applications.  _arXiv preprint arXiv:1812.05905_ , 2018b.
  * Houthooft et al. (2016) Houthooft, R., Chen, X., Duan, Y., Schulman, J., De Turck, F., and Abbeel, P.  Vime: Variational information maximizing exploration.  _Advances in neural information processing systems_ , 29, 2016.
  * Janner et al. (2021) Janner, M., Li, Q., and Levine, S.  Offline reinforcement learning as one big sequence modeling problem.  _Advances in neural information processing systems_ , 34:1273–1286, 2021.
  * Kidambi et al. (2021) Kidambi, R., Chang, J., and Sun, W.  Mobile: Model-based imitation learning from observation alone.  _Advances in Neural Information Processing Systems_ , 34:28598–28611, 2021.
  * Kostrikov et al. (2021) Kostrikov, I., Nair, A., and Levine, S.  Offline reinforcement learning with implicit q-learning.  _arXiv preprint arXiv:2110.06169_ , 2021.
  * Kumar et al. (2020) Kumar, A., Zhou, A., Tucker, G., and Levine, S.  Conservative q-learning for offline reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33:1179–1191, 2020.
  * Lee et al. (2021) Lee, Y., Szot, A., Sun, S.-H., and Lim, J. J.  Generalizable imitation learning from observation via inferring goal proximity.  _Advances in Neural Information Processing Systems_ , 34:16118–16130, 2021.
  * Liu et al. (2018) Liu, Y., Gupta, A., Abbeel, P., and Levine, S.  Imitation from observation: Learning to imitate behaviors from raw video via context translation.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pp. 1118–1125. IEEE, 2018.
  * Liu et al. (2021) Liu, Y., Zhang, J., Fang, L., Jiang, Q., and Zhou, B.  Multimodal motion prediction with stacked transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 7577–7586, 2021.
  * Mohamed et al. (2020) Mohamed, A., Qian, K., Elhoseiny, M., and Claudel, C.  Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 14424–14432, 2020.
  * Mohamed et al. (2022) Mohamed, A., Zhu, D., Vu, W., Elhoseiny, M., and Claudel, C.  Social-implicit: Rethinking trajectory prediction evaluation and the effectiveness of implicit maximum likelihood estimation.  In _European Conference on Computer Vision_ , pp. 463–479. Springer, 2022.
  * Nair et al. (2020) Nair, A., Gupta, A., Dalal, M., and Levine, S.  Awac: Accelerating online reinforcement learning with offline datasets.  _arXiv preprint arXiv:2006.09359_ , 2020.
  * Nikulin (2022) Nikulin, A.  Faster trajectory transformer.  https://github.com/Howuhh/faster-trajectory-transformer, 2022.
  * Pathak et al. (2017) Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T.  Curiosity-driven exploration by self-supervised prediction.  In _International conference on machine learning_ , pp. 2778–2787. PMLR, 2017.
  * Pathak et al. (2018) Pathak, D., Mahmoudieh, P., Luo, G., Agrawal, P., Chen, D., Shentu, Y., Shelhamer, E., Malik, J., Efros, A. A., and Darrell, T.  Zero-shot visual imitation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition workshops_ , pp. 2050–2053, 2018.
  * Peng et al. (2018) Peng, X. B., Abbeel, P., Levine, S., and Van de Panne, M.  Deepmimic: Example-guided deep reinforcement learning of physics-based character skills.  _ACM Transactions On Graphics (TOG)_ , 37(4):1–14, 2018.
  * Radford et al. (2018) Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.  Improving language understanding by generative pre-training.  2018\.
  * Raffin et al. (2021) Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N.  Stable-baselines3: Reliable reinforcement learning implementations.  _Journal of Machine Learning Research_ , 22(268):1–8, 2021.  URL http://jmlr.org/papers/v22/20-1364.html.
  * Salzmann et al. (2020) Salzmann, T., Ivanovic, B., Chakravarty, P., and Pavone, M.  Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data.  In _European Conference on Computer Vision_ , pp. 683–700. Springer, 2020.
  * Schmidhuber (1990) Schmidhuber, J.  Making the world differentiable: On using fully recurrent self-supervised neural networks for dynamic reinforcement learning and planning in non-stationary environments.  _Institut für Informatik, Technische Universität München. Technical Report FKI-126_ , 90, 1990.
  * Schmidhuber (1991) Schmidhuber, J.  A possibility for implementing curiosity and boredom in model-building neural controllers.  In _Proc. of the international conference on simulation of adaptive behavior: From animals to animats_ , pp. 222–227, 1991.
  * Schmidhuber (2015) Schmidhuber, J.  On learning to think: Algorithmic information theory for novel combinations of reinforcement learning controllers and recurrent neural world models.  _arXiv preprint arXiv:1511.09249_ , 2015.
  * Schmidhuber (2018) Schmidhuber, J.  One big net for everything.  _arXiv preprint arXiv:1802.08864_ , 2018.
  * Schmidhuber (2019) Schmidhuber, J.  Reinforcement learning upside down: Don’t predict rewards–just map them to actions.  _arXiv preprint arXiv:1912.02875_ , 2019.
  * Štrupl et al. (2022) Štrupl, M., Faccio, F., Ashley, D. R., Schmidhuber, J., and Srivastava, R. K.  Upside-down reinforcement learning can diverge in stochastic environments with episodic resets.  _arXiv preprint arXiv:2205.06595_ , 2022.
  * Tao et al. (2020) Tao, R. Y., François-Lavet, V., and Pineau, J.  Novelty search in representational space for sample efficient exploration.  _Advances in Neural Information Processing Systems_ , 33:8114–8126, 2020.
  * Torabi et al. (2018a) Torabi, F., Warnell, G., and Stone, P.  Behavioral cloning from observation.  _arXiv preprint arXiv:1805.01954_ , 2018a.
  * Torabi et al. (2018b) Torabi, F., Warnell, G., and Stone, P.  Generative adversarial imitation from observation.  _arXiv preprint arXiv:1807.06158_ , 2018b.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Yang et al. (2019) Yang, C., Ma, X., Huang, W., Sun, F., Liu, H., Huang, J., and Gan, C.  Imitation learning from observations by minimizing inverse dynamics disagreement.  _Advances in neural information processing systems_ , 32, 2019.
  * Yu et al. (2021) Yu, T., Kumar, A., Rafailov, R., Rajeswaran, A., Levine, S., and Finn, C.  Combo: Conservative offline model-based policy optimization.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Yuan et al. (2021) Yuan, Y., Weng, X., Ou, Y., and Kitani, K. M.  Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 9813–9823, 2021.
  * Zhu et al. (2021) Zhu, D., Zahran, M., Li, L. E., and Elhoseiny, M.  Halentnet: Multimodal trajectory forecasting with hallucinative intents.  In _International Conference on Learning Representations_ , 2021.

##  Appendix A Datasets

For locomotion environments, the Medium dataset is collected using a policy
trained to approximately 1/3 the performance of an expert. Medium-Replay uses
the training replay buffer of the ’Medium’ policy. The Medium-Expert dataset
contains 50% of data from Medium and the remaining data is collected by an
expert policy. For the ant robot maze environment, in the Antmaze-Umaze
dataset, the robot ant always goes from a fixed start position to a fixed
target location, while in Antmaze-Umaze-Diverse, the robot ant goes to random
target locations.

##  Appendix B Hyperparameters

For the architecture of AFDT, we follow the default hyperparameters of DT. In
detail, we use three transformer blocks for most of the environments and one
for ball maze environments. Each block has one attention head. The embedding
dimension is set to 128. Dropout rate is set to 0.1. We train AFDT for 50000
gradient steps and selected the best checkpoint from 3000 steps, 5000 steps,
10000 steps, 15000 steps, 30000 steps and 50000 steps. The implementation of
AFDT is based on the repository ‘minimal decision transformer’(Barhate, 2022).
For the architecture of Guided SAC, the environment Q function, the guided Q
function, and the policy net are all three-layer MLPs with ReLU activation
function and 256 hidden dimensions. The implementation of Guided SAC is based
on the repository ‘Stable Baseline 3’(Raffin et al., 2021).

##  Appendix C Action-Free Trajectory Transformer

Our implementation of AFTT is based on the repository ‘faster-trajectory-
transformer’ (Nikulin, 2022), which has an improved inference speed compared
to the original implementation. We follow all the default hyperparameters and
model architectures in the repository, but remove the action-related component
in TT to build AFTT. We use uniform strategy to discretize the state space.