#  Simulating Electron Transfer in a Molecular Triad within an Optical Cavity
Using NISQ Computers

Ningyi Lyu  Department of Chemistry, Yale University, New Haven, CT 06520,
U.S.A.  Pouya Khazaei  Department of Chemistry, University of Michigan, Ann
Arbor, MI 48109, U.S.A.  Eitan Geva  Department of Chemistry, University of
Michigan, Ann Arbor, MI 48109, U.S.A<EMAIL_ADDRESS>Victor S. Batista
Department of Chemistry, Yale University, New Haven, CT 06520, U.S.A.
<EMAIL_ADDRESS>
Yale Quantum Institute, Yale University, New Haven, CT 06511, U.S.A.

##  1 Abstract

We present a quantum algorithm based on the Tensor-Train Thermo-Field Dynamics
(TT-TFD) method to simulate the open quantum system dynamics of intramolecular
charge transfer modulated by an optical cavity on noisy intermediate-scale
quantum (NISQ) computers. We apply our methodology to a model that describes
the $\pi\pi^{*}$ to CT1 intermolecular charge transfer within the carotenoid-
porphyrin-C60 molecular triad solvated in tetrahydrofuran (THF) and placed
inside an optical cavity. We find how the dynamics is influenced by the cavity
resonance frequency and strength of the light-matter interaction, showcasing
the NISQ-based simulations to capture these effects. Furthermore, we compare
the approximate predictions of Fermi’s Golden Rule (FGR) rate theory and Ring-
Polymer Molecular Dynamics (RPMD) to numerically exact calculations, showing
the capabilitis of quantum computing methods to assess the limitations of
approximate methods.

##  2 Introduction

Exploring the potential of light-matter interactions to influence chemical
reactions has captivated the attention of physical chemists for many years.1,
2, 3, 4, 5, 6, 7, 8, 9, 10, 11 Recent experimental advancements have showcased
the possibility of utilizing these interactions to control various chemical
and physical phenomena, including energy and charge transfer, photochemical
processes, photocatalysis. 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24,
25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43,
44, 45, 46, 47, 48, 49, 50, 51, 52 The proposed control mechanisms involve
both strong and ultra-strong coupling of electromagnetic microcavity optical
modes with the electronic and vibrational degrees of freedom (DOF) of
molecules within these cavities. These experimental advances call for the
development of accurate computational methods for simulating the dynamics of
molecular matter inside cavities to reveal fundamental mechanisms and predict
various experimental outcomes such as reaction yields and transfer rates. So
far, most dynamical simulations have utilized mixed quantum classical (MQC)
methods which might not describe nuclear quantum effects properly since they
treat part of the system classically. 49 On the other hand, numerically exact
methods are now developed to treat vibronic systems with a few tens to a few
hundreds nuclear degrees of freedom (DoF),53 while rapidly advancing quantum
computers are expected to push that limit further. In this work, we implement
the Tensor-Train Thermo-Field Dynamics (TT-TFD) method54, 55, 56 for
numerically exact quantum dynamics simulations of molecular systems in optical
cavities. Utilizing the Sz-Nagy dilation scheme, 57, 58 we demonstrate the TT-
TFD method as applied to quantum computing simulations on currently available
IBM Noisy Intermediate-Scale Quantum (NISQ) computers.

Our simulations investigate electron transfer in the carotenoid-porphyrin-C60
molecular triad dissolved in tetrahydrofuran (THF) inside an optical cavity,
as illustrated in Fig 1. We investigate the intramolecular electron transfer
dynamics in the triad under various cavity setups. We find that the dynamics
can be controlled significantly by changing the resonance frequency of the
cavity, or the strength of the light-matter interaction. Our numerically exact
results are compared to calculations based on approximate methodologies,
including the Fermi Golden Rule (FGR), 59 and Ring-Polymer Molecular Dynamics
(RPMD) simulations, 49 providing valuable insights on the capabilities and
limitations of approximate methods. Therefore, it is natural to expect that
our quantum computing methodology should also be useful for other studies of
molecular systems in polaritonic cavities.

Figure 1: Illustration of the THF-solvated molecular triad in cavity.

##  3 Methodology

###  3.1 Hamiltonian for cavity-modified electron transfer

We consider the following light-matter Hamiltonian,

$\hat{H}=-\epsilon|A\rangle\langle A|+\Delta(|D\rangle\langle
A|+|A\rangle\langle D|)+\hat{H}_{B}+\hat{H}_{F},$ (1)

where $|D\rangle$ and $|A\rangle$ denote the $\pi\pi^{*}$ and CT1 diabatic
states of the solvated triad, respectively, with $\epsilon=532~{}\text{meV}$
defining the energy gap between the two states and $\Delta=24~{}\text{meV}$
the electronic coupling.60

The bath Hamiltonian $H_{B}$, corresponding to the mass-weighted bath
coordinates $R_{j}$, is described by a set of harmonic oscillators coupled to
the two electronic states, as follows:

$\hat{H}_{B}=\sum_{j=1}^{50}\bigg{(}\frac{\hat{P}_{j}^{2}}{2}+\frac{\omega_{j}^{2}}{2}\hat{R}_{j}^{2}\bigg{)}|D\rangle\langle
D|+\sum_{j=1}^{50}\bigg{(}\frac{\hat{P}_{j}^{2}}{2}+\frac{\omega_{j}^{2}}{2}(\hat{R}_{j}-\frac{c_{j}}{\omega_{j}^{2}})^{2}\bigg{)}|A\rangle\langle
A|,$

where the bath frequencies $\\{\omega_{j}\\}$ and coupling constants
$\\{c_{j}\\}$ are obtained by discretizing the following Ohmic spectral
density:

$J(\omega)=\frac{\pi}{2}\sum_{j=1}^{50}\frac{c_{j}^{2}}{m_{j}\omega_{j}}\delta(\omega-\omega_{j})=\eta\omega
e^{-\omega/\omega_{c}},$ (2)

where $\omega_{c}=6.57~{}\text{meV}$, and $\eta=1.066~{}\times
10^{6}~{}\text{au}$ are chosen so that the discretized vibrational frequencies
roughly cover the range of normal mode frequencies for this system as
calculated by molecular dynamics simulations. 61 The couplings $\\{c_{j}\\}$
are then scaled with a constant $a$, such that the values of the scaled
$\tilde{c}_{j}=c_{j}a$ correspond to the calculated molecular reorganization
energy of the bent triad molecule $\lambda=515~{}\text{meV}$:

$\frac{1}{2}\sum_{j=1}^{50}\omega_{j}^{2}\tilde{c}_{j}^{2}=\lambda.$ (3)

The coupling of the optical cavity mode with the molecule is described, as
follows:

$\hat{H}_{F}=\hbar\omega_{p}\hat{a}_{p}^{\dagger}\hat{a}_{p}+\hbar
g_{p}(|D\rangle\langle A|+|A\rangle\langle
D|)(\hat{a}_{p}^{\dagger}+\hat{a}_{p}),$ (4)

where $\hat{a}_{p}$ and $\hat{a}_{p}^{\dagger}$ are the photonic annihilation
and creation operators, respectively. The values of the light-matter coupling
strength ($g_{p}$) and the cavity frequency ($\omega_{p}$) determine the
effect of the cavity and will be varied to evaluate their effect on the
intramolecular electron transfer dynamics.

###  3.2 Tensor-Train Thermo-Field Dynamics Simulations

The evolution of the quantum state $\rho(t)$ is described by the quantum
Liouville equation,

$\frac{d}{dt}{\rho}(t)=-\frac{i}{\hbar}[\hat{H},{\rho}(t)],$ (5)

which is the equation of motion of the time-evolving density matrix
${\rho}(t)$. The TT-TFD method solves this equation in vectorized form, as
follows:

$\frac{d}{dt}{|\psi(\beta,t)\rangle}=-\frac{i}{\hbar}\bar{H}|\psi(\beta,t)\rangle,$
(6)

with $\bar{H}=\hat{H}\otimes\tilde{I}$. Here, $|\psi(\beta,t)\rangle$ is the
thermal wavefunction that defines the density matrix, as follows:

$\rho(t)=\text{Tr}_{f}\\{|\psi(\beta,t)\rangle\langle\psi(\beta,t)|\\},$ (7)

with $\rho(0)$ obtained from Eq. (7) with the initial thermal distribution:

$|\psi(0,\beta)\rangle=Z^{-1/2}\sum_{n}e^{-\beta E_{n}/2}|n,\tilde{n}\rangle.$
(8)

Such a transformation effectively recasts Eq. (5) into Eq. (6) which is the
time-dependent Schrödinger equation (TDSE) for the thermal wavefunction. Note
that according to Eq. (8), the thermal wavefunction is expanded in a double
Hilbert space $\mathcal{H}\otimes\tilde{\mathcal{H}}$, including states
$|\tilde{n}\rangle$ that are identical copies of states $|n\rangle$ in a
Hilbert space $\tilde{\mathcal{H}}$ that is an exact copy of the physical
Hilbert space $\mathcal{H}$ of states $|n\rangle$.

The preparation of the initial state $|\psi(\beta,0)\rangle$ as in Eq. (8)
requires imaginary time propagation, which could be computationally
burdensome. However, when the bath Hamiltonian is harmonic, we can define the
Hamiltonian, as follows:

$\displaystyle\bar{H}=\hat{H}\otimes\tilde{I}-\hat{I}\otimes\sum_{j}\omega_{j}\tilde{a}_{j}^{\dagger}\tilde{a}_{j}.$
(9)

This task is circumvented by utilizing the thermal Bogoliubov transformation,

$\displaystyle|0(\beta)\rangle=e^{-i\hat{G}}|0,\tilde{0}\rangle,$ (10)

where

$\hat{G}=-i\sum_{j}\theta_{j}\left(\hat{a}_{j}\tilde{a}_{j}-\hat{a}^{\dagger}_{j}\tilde{a}^{\dagger}_{j}\right),$
(11)

with $\theta_{j}=\text{arctanh}(e^{-\beta\omega_{j}/2})$ and
${\hat{a}_{j},\hat{a}_{j}^{\dagger}}$
(${\tilde{a}_{j},\tilde{a}_{j}^{\dagger}}$) the physical (fictional) bosonic
creation and annihilation operators for the $j$-th degree of freedom.

Substituting Eqs. (10) and (11) into Eq. (6), we obtain:

$\frac{\mathrm{d}|\psi_{\theta}(\beta,t)\rangle}{\mathrm{d}t}=-\frac{i}{\hbar}\bar{H}_{\theta}|\psi_{\theta}(\beta,t)\rangle,$
(12)

where:

$\begin{split}&\bar{H}_{\theta}=e^{i\hat{G}}\bar{H}e^{-i\hat{G}},\\\
&|\psi_{\theta}(\beta,0)\rangle=e^{i\hat{G}}|\psi(\beta,0)\rangle=|0,\tilde{0}\rangle.\end{split}$
(13)

For a vibronic system that involves many nuclear degrees of freedom, the
thermal wavepacket is a high-dimensional tensor which full representation
faces the curse of dimensionality. The tensor train (TT) decomposition62, 63,
64 is a numerically exact data compression strategy that represents moderately
entangled wavefunction with a train of 2- or 3-dimensional tensors, such that
they can be stored and computed efficiently. TT-TFD expresses
$|0,\tilde{0}\rangle$ as a rank-1 tensor train, and propagates Eq. (12) to
obtain $|\psi_{\theta}(\beta,t)\rangle$. The propagation can be carried out
with the TT-KSL integrator. 56, 55 The expectation value of physical
observables $\hat{A}$ are obtained, as follows:

$\displaystyle\langle
A(t)\rangle=\langle\psi_{\theta}(\beta,t)|\bar{A}_{\theta}|\psi_{\theta}(\beta,t)\rangle,$
(14)

where $\bar{A}_{\theta}=e^{i\hat{G}}\bar{A}e^{-i\hat{G}}$ and
$\bar{A}=\hat{A}\otimes\tilde{I}$.

###  3.3 Quantum computation for population dynamics

Numerically exact benchmark calculations based on TT-TFD simulations of the
model carotenoid-porphyrin-C60 molecular triad solvated in tetrahydrofuran
(THF) in the optical cavity are performed for the evolution of populations of
the electronic states. In what follows we outline the quantum computing
procedure for obtaining cavity-modified population dynamics based on the TT-
TFD results.

Our propagation scheme is based on the so-called population-only Liouville
space superoperator $\mathcal{P}^{pop}(t)$ that satisfies the following
equation:

$\hat{\sigma}^{\text{pop}}(t)=\mathcal{P}^{\text{pop}}(t)\hat{\sigma}^{\text{pop}}(0),$
(15)

where $\hat{\sigma}(t)=\text{Tr}_{n}[\hat{\rho}(t)]$ is the reduced density
matrix for the electronic DOFs, with $\hat{\rho}(t)$ the density matrix for
the full vibronic system. Here,
$\hat{\sigma}^{\text{pop}}(t)=(\sigma_{00}(t),\sigma_{11}(t))^{T}$ includes
only the diagonal elements of $\hat{\sigma}(t)$, necessary to describe the
electronic population dynamics. The preparation of the super-operator
$\mathcal{P}^{pop}(t)$ according to the TT-TFD generated population dynamics
is described in Appendix E of Ref. 65.

To perform quantum computing simulations based on Eq. (15), we first transform
$\mathcal{P}^{\text{pop}}(t)$ into a unitary matrix using the Sz.-Nagy
dilation theorem57, as follows: 58, 66

$\mathcal{U}_{\mathcal{P}^{\text{pop}}}(t)=\begin{pmatrix}\mathcal{P}^{\text{pop}}(t)&\sqrt{I-\mathcal{P}^{{\text{pop}}}(t)\mathcal{P}^{\text{pop}^{\dagger}}(t)}\\\
\sqrt{I-\mathcal{P}^{\text{pop}{{}^{\dagger}}}(t)\mathcal{P}^{\text{pop}}(t)}&-\mathcal{P}^{\text{pop}^{\dagger}}(t)\\\
\end{pmatrix}.$ (16)

The vectorized $v_{\sigma(0)}$ is dilated by appending ancillary zero
elements, as follows:

$\hat{\sigma}^{\text{pop}}(0)=(\sigma_{00}(0),\sigma_{11}(0))^{T}\rightarrow\tilde{\sigma}^{\text{pop}}(0)=(\sigma_{00}(0),\sigma_{11}(0),0,0)^{T}.$
(17)

The dilated time-updated population-only density matrix is obtained, as
follows:

$\tilde{\sigma}^{\text{pop}}(t)=\mathcal{U}_{\mathcal{P}^{\text{pop}}}(t)\tilde{\sigma}^{\text{pop}}(0).$
(18)

The dilation scheme thus provides the unitary matrix
$\mathcal{U}_{\mathcal{P}^{\text{pop}}}(t)$ governing the time-evolution of
$\tilde{\sigma}^{\text{pop}}(t)$, the first two digits of which agree with
those of $\hat{\sigma}^{\text{pop}}(t)$. Therefore, Eqs. (18) and (15)
describe the same dynamics, with Eq. (18) allowing for simulations on a NISQ
quantum computer. For the spin-boson model of interest,
$\mathcal{U}_{\mathcal{P}^{\text{pop}}}(t)$ is a $4\times 4$ unitary matrix
corresponding to a 2-qubit gate.

##  4 Results

Figure 2 shows the calculated short-time population dynamics of the model
carotenoid-porphyrin-C60 molecular triad solvated in tetrahydrofuran (THF) in
the optical cavity when prepared in a geometry relaxed bent configuration. The
results show that in the ultrafast timescale, significant population transfer
only happens with the aid of coupling to the cavity mode. Further, increasing
the strength of the coupling to the cavity mode changes the electron transfer
dynamics qualitatively by becoming highly oscillatory, while in the weaker
coupling regime the transfer dynamics exhibits a smooth decay as predicted by
rate kinetic theory. The observed oscillatory behavior is likely induced by to
Rabi oscillations due to the photonic coupling term with a frequency
approximately doubling by doubling $g_{p}$.

Figure 2: Time-dependent population of the ($\pi\pi^{*}$) state during the
early time relaxation after photoexcitation of the carotenoid-porphyrin-C60
molecular triad solvated in tetrahydro- furan (THF) prepared in a geometry
relaxed bent configuration in an optical cavity. The cavity frequency is set
as $\omega_{p}=$ 400 meV for all calculations, while the light-matter
interaction strength $g_{p}=$10, 20 and 40 meV. The temperature is set at
$T=300~{}K$. The calculations are based TT-TFD simulations with a propagation
time-step $\tau=1~{}a.u.$

Figure 3 shows the effect of changing the cavity resonance frequency on the
intramolecular electron transfer dynamics. While light of all frequencies
enhance the transfer, the ultrafast transfer efficiency reaches its peak when
$\hbar\omega_{p}$ is around 400 meV. It is worth noting that this optimal
cavity efficiency matches relatively well with that predicted by the FGR rate
theory59 (around 510 meV), despite the fact that the ultrafast dynamics does
not exactly follow the exponential relaxation predicted by the FGR rate
theory.

Figure 3: Time-dependent population of the ($\pi\pi^{*}$) state during the
early time relaxation after photoexcitation of the carotenoid-porphyrin-C60
molecular triad solvated in tetrahydro- furan (THF) prepared in a geometry
relaxed bent configuration in an optical cavity. The cavity frequency is
changed in the $\omega_{p}=$ 200–400 meV range as indicated in the inset. The
light-matter interaction strength is set at $g_{p}=$20 meV. The temperature is
set at $T=300~{}K$. The calculations are based TT-TFD simulations with a
propagation time-step $\tau=1~{}a.u.$

Figure 4 shows the comparison of the population dynamics described by
numerically exact TT-TFD calculations and approximate results based on Ring-
Polymer Molecular Dynamics (RPMD) simulations, reported in Ref. 49. Figure 4
shows that RPMD agrees qualitatively with numerically exact calculations
although exhibits noticeable quantitative deviations.

Figure 4: Comparison between population dynamics generated by TT-TFD and by
RPMD for Model III in Ref. 49. (a): donor population with a cavity, (b): donor
population without a cavity.

Figure 5 shows the results of calculations of intramolecular electron transfer
dynamics performed with the IBM Osaka quantum computer. The results show that
the quantum computing scheme is able to produce accurate results within about
1 percent error.

Figure 5: Time-dependent population of the ($\pi\pi^{*}$) state during the
early time relaxation after photoexcitation of the carotenoid-porphyrin-C60
molecular triad solvated in tetrahydro- furan (THF) prepared in a geometry
relaxed bent configuration in an optical cavity obtained with the IBM Osaka
quantum computer. The cavity frequency is fixed at $\omega_{p}=$ 500 meV. The
light-matter interaction strength is set at $g_{p}=$20 meV. Matrix-free
measurement mitigation is performed for readout error mitigation, as
implemented in Qiskit. The blue dots are averages over 10 calculations
following Eq. (18), each obtained with 2000 measurements. The error bar shows
the highest and lowest values among the 10 runs.

##  5 Conclusions

We have studied the optical cavity-modulated intermolecular electron transfer
dynamics using a rigorous methodology that combines the numerically exact TT-
TFD method with open quantum simulations based on dilation. We have shown that
the method offers a practical and accurate pathway for quantum computing
simulations of cavity modulated quantum reaction dynamics in molecular
systems. The simulations are based on a single model molecular triad in the
infinitely dilute limit, and therefore neglect collective effects that become
important at higher concentration. Nevertheless, our simulation results have
shown that the cavity modulation can be significant even in the infinitely
dilute limit, strongly modulating the charge transfer dynamics in the
ultrafast timescale.

##  6 Acknowledgments

The authors acknowledge support from the NSF grant 2124511 [CCI Phase I: NSF
Center for Quantum Dynamics on Modular Quantum Devices (CQD-MQD)].

## References

  * Tannor and Rice 1988 Tannor, D. J.; Rice, S. A. Coherent Pulse Sequence Control of Product Formation in Chemical Reactions. _Adv. Chem. Phys._ 1988, _70_ , 441–523
  * Kosloff et al. 1989 Kosloff, R.; Rice, S. A.; Gaspard, P.; Tersigni, S.; Tannor, D. J. Wave Packet Dancing: Achieving Chemical Selectivity By Shaping Light Pulses. _Chem. Phys._ 1989, _139_ , 201–220
  * Gordon and Rice 1997 Gordon, R. J.; Rice, S. A. Active Control of The Dynamics of Atoms and Molecules. _Ann. Rev. Phys. Chem._ 1997, _48_ , 601–641
  * Assion et al. 1998 Assion, A.; Baumert, T.; Bergt, M.; Brixner, T.; Kiefer, B.; Seyfried, V.; Strehle, M.; Gerber, G. Control of Chemical Reactions by Feedback-Optimized Phase-Shaped Femtosecond Laser Pulses. _Science_ 1998, _282_ , 919–922
  * Rabitz and Zhu 2000 Rabitz, H.; Zhu, W. Optimal Control of Molecular Motion: Design Implementation and Inversion. _Acc. Chem. Res._ 2000, _33_ , 572–578
  * Rice and Zhao 2000 Rice, S. A.; Zhao, M. _Optical Control of Molecular Dynamics_ ; Wiley: New York, 2000
  * Levis et al. 2001 Levis, R. J.; Menkir, G. M.; Rabitz, H. Selective Bond Dissociation and Rearrangement With Optimally Tailored Strong Field Laser Pulses. _Science_ 2001, _292_ , 709–713
  * Pearson et al. 2001 Pearson, B. J.; White, J. L.; Weinacht, T. C.; Bucksbaum, P. H. Coherent Control Using Adaptive Learning Algorithms. _Phys. Rev. A_ 2001, _63_ , 063412
  * Rice and Shah 2002 Rice, S. A.; Shah, S. P. Active Control of Product Selection in A Chemical Reaction: A View of The Current Scene. _Phys. Chem. Chem. Phys._ 2002, _4_ , 1683–1700
  * Shapiro and Brumer 2002 Shapiro, M.; Brumer, P. _Principles of The Quantum Control of Molecular Processes_ ; Wiley: Hoboken, New Jersey, 2002
  * McRobbie and Geva 2016 McRobbie, P.; Geva, E. Coherent Control of Population Transfer via Linear Chirp in Liquid Solution: The Role of Motional Narrowing. _J. Phys. Chem. A_ 2016, _120_ , 3015–3022
  * Andrew and Barnes 2000 Andrew, P.; Barnes, W. L. Förster energy transfer in an optical microcavity. _Science_ 2000, _290_ , 785–788
  * Schwartz et al. 2011 Schwartz, T.; Hutchison, J.; Genet, C.; Ebbesen, T. Reversible Switching of Ultrastrong Light-Molecule Coupling. _Phys. Rev. Lett._ 2011, _106_ , 196405–4
  * Hutchison et al. 2012 Hutchison, J.; Schwartz, T.; Genet, C.; Devaux, E.; Ebbesen, T. Modifying Chemical Landscapes by Coupling to Vacuum Fields. _Angew. Chem. Int._ 2012, _51_ , 1592–1596
  * Hutchison et al. 2013 Hutchison, J.; Liscio, A.; Schwartz, T.; Canaguier-Durand, A.; Genet, C.; Palermo, V.; Samorì, P.; Ebbesen, T. Tuning the Work-Function Via Strong Coupling. _Adv. Mater._ 2013, _25_ , 2481–2485
  * Törmä and Barnes 2014 Törmä, P.; Barnes, W. Strong coupling between surface plasmon polaritons and emitters: A review. _Rep. Prog. Phys._ 2014, _78_ , 013901–35
  * Flick et al. 2015 Flick, J.; Ruggenthaler, M.; Appel, H.; Rubio, A. Kohn–Sham approach to quantum electrodynamical density-functional theory: Exact time-dependent effective potentials in real space. _Proc. Natl. Acad. Sci. USA_ 2015, _112_ , 15285–15290
  * Feist and Garcia-Vidal 2015 Feist, J.; Garcia-Vidal, F. Extraordinary Exciton Conductance Induced by Strong Coupling. _Phys. Rev. Lett._ 2015, _114_ , 196402–5
  * Schachenmayer et al. 2015 Schachenmayer, J.; Genes, C.; Tignone, E.; Pupillo, G. Cavity-Enhanced Transport of Excitons. _Phys. Rev. Lett._ 2015, _114_ , 196403–6
  * Shalabney et al. 2015 Shalabney, A.; George, J.; Hutchison, J.; Pupillo, G.; Genet, C.; Ebbesen, T. Coherent coupling of molecular resonators with a microcavity mode. _Nat. Commun._ 2015, _6_ , 217–6
  * Orgiu et al. 2015 Orgiu, E.; George, J.; Hutchison, J.; Devaux, E.; Dayen, J.; Doudin, B.; Stellacci, F.; Genet, C.; Schachenmayer, J.; Genes, C.; Pupillo, G.; Samorì, P.; Ebbesen, T. Conductivity in organic semiconductors hybridized with the vacuum field. _Nat. Mater._ 2015, _14_ , 1123–1129
  * Long and Simpkins 2015 Long, J.; Simpkins, B. Coherent Coupling between a Molecular Vibration and Fabry–Perot Optical Cavity to Give Hybridized States in the Strong Coupling Limit. _ACS Photonics_ 2015, _2_ , 130–136
  * Ebbesen 2016 Ebbesen, T. Hybrid Light-Matter States in a Molecular and Material Science Perspective. _Acc. Chem. Res._ 2016, _49_ , 2403–2412
  * Thomas et al. 2016 Thomas, A.; George, J.; Shalabney, A.; Dryzhakov, M.; Varma, S.; Moran, J.; Chervy, T.; Zhong, X.; Devaux, E.; Genet, C.; Hutchison, J.; Ebbesen, T. Ground-State Chemical Reactivity under Vibrational Coupling to the Vacuum Electromagnetic Field. _Angew. Chem._ 2016, _128_ , 11634–11638
  * Zhong et al. 2016 Zhong, X.; Chervy, T.; Wang, S.; George, J.; Thomas, A.; Hutchison, J.; Devaux, E.; Genet, C.; Ebbesen, T. Non-Radiative Energy Transfer Mediated by Hybrid Light-Matter States. _Angew. Chem._ 2016, _128_ , 6310–6314
  * Herrera and Spano 2016 Herrera, F.; Spano, F. Cavity-Controlled Chemistry in Molecular Ensembles. _Phys. Rev. Lett._ 2016, _116_ , 238301–6
  * Casey and Sparks 2016 Casey, S.; Sparks, J. Vibrational Strong Coupling of Organometallic Complexes. _J. Phys. Chem. C_ 2016, _120_ , 28138–28143
  * Sanvitto and Kéna-Cohen 2016 Sanvitto, D.; Kéna-Cohen, S. The road towards polaritonic devices. _Nat. Mater._ 2016, _15_ , 1061–1073
  * Kowalewski et al. 2016 Kowalewski, M.; Bennett, K.; Mukamel, S. Cavity Femtochemistry: Manipulating Nonadiabatic Dynamics at Avoided Crossings. _J. Phys. Chem. Lett._ 2016, _7_ , 2050–2054
  * Kowalewski et al. 2016 Kowalewski, M.; Bennett, K.; Mukamel, S. Non-adiabatic dynamics of molecules in optical cavities. _J. Chem. Phys._ 2016, _144_ , 054309–9
  * Flick et al. 2017 Flick, J.; Ruggenthaler, M.; Appel, H.; Rubio, A. Atoms and molecules in cavities, from weak to strong coupling in quantum-electrodynamics (QED) chemistry. _Proc. Natl. Acad. Sci. USA_ 2017, _114_ , 3026–3034
  * Zhong et al. 2017 Zhong, X.; Chervy, T.; Zhang, L.; Thomas, A.; George, J.; Genet, C.; Hutchison, J.; Ebbesen, T. Energy Transfer between Spatially Separated Entangled Molecules. _Angew. Chem. Int. Ed._ 2017, _56_ , 9034–9038
  * Martínez-Martínez et al. 2017 Martínez-Martínez, L.; Ribeiro, R.; Campos-González-Angulo, J.; Yuen-Zhou, J. Can Ultrastrong Coupling Change Ground-State Chemical Reactions? _ACS Photonics_ 2017, _5_ , 167–176
  * Fregoni et al. 2018 Fregoni, J.; Granucci, G.; Coccia, E.; Persico, M.; Corni, S. Manipulating azobenzene photoisomerization through strong light-molecule coupling. _Nat. Commun._ 2018, _9_ , 2613–9
  * Sáez-Blázquez et al. 2018 Sáez-Blázquez, R.; Feist, J.; Fernández-Domínguez, A.; García-Vidal, F. Organic polaritons enable local vibrations to drive long-range energy transfer. _Phys. Rev. B_ 2018, _97_ , 241407–5
  * Flick et al. 2019 Flick, J.; Welakuh, D.; Ruggenthaler, M.; Appel, H.; Rubio, A. Light–Matter Response in Nonrelativistic Quantum Electrodynamics. _ACS Photonics_ 2019, _6_ , 2757–2778
  * Galego et al. 2019 Galego, J.; Climent, C.; Garcia-Vidal, F.; Feist, J. Cavity Casimir-Polder Forces and Their Effects in Ground-State Chemical Reactivity. _Phys. Rev. X_ 2019, _9_ , 021057–22
  * Lather et al. 2019 Lather, J.; Bhatt, P.; Thomas, A.; Ebbesen, T.; George, J. Cavity Catalysis by Cooperative Vibrational Strong Coupling of Reactant and Solvent Molecules. _Angew. Chem. Int. Ed._ 2019, _58_ , 10635–10638
  * Schäfer et al. 2019 Schäfer, C.; Ruggenthaler, M.; Appel, H.; Rubio, A. Modification of excitation and charge transfer in cavity quantum-electrodynamical chemistry. _Proc. Natl. Acad. Sci. USA_ 2019, _116_ , 4883–4892
  * Hoffmann et al. 2019 Hoffmann, N.; Schäfer, C.; Rubio, A.; Kelly, A.; Appel, H. Capturing vacuum fluctuations and photon correlations in cavity quantum electrodynamics with multitrajectory Ehrenfest dynamics. _Phys. Rev. A_ 2019, _99_ , 063819–9
  * Hoffmann et al. 2019 Hoffmann, N.; Schäfer, C.; Säkkinen, N.; Rubio, A.; Appel, H.; Kelly, A. Benchmarking semiclassical and perturbative methods for real-time simulations of cavity-bound emission and interference. _J. Chem. Phys._ 2019, _151_ , 244113–14
  * Lacombe et al. 2019 Lacombe, L.; Hoffmann, N.; Maitra, N. Exact Potential Energy Surface for Molecules in Cavities. _Phys. Rev. Lett._ 2019, _123_ , 083201–6
  * Mandal and Huo 2019 Mandal, A.; Huo, P. Investigating New Reactivities Enabled by Polariton Photochemistry. _J. Phys. Chem. Lett._ 2019, _10_ , 5519–5529
  * Semenov and Nitzan 2019 Semenov, A.; Nitzan, A. Electron transfer in confined electromagnetic fields. _The Journal of Chemical Physics_ 2019, _150_ , 174122
  * Hoffmann et al. 2020 Hoffmann, N.; Lacombe, L.; Rubio, A.; Maitra, N. Effect of many modes on self-polarization and photochemical suppression in cavities. _J. Chem. Phys._ 2020, _153_ , 104103–10
  * Flick et al. 2020 Flick, J.; Rivera, N.; Narang, P. Strong light-matter coupling in quantum chemistry and quantum photonics. _Nanophotonics_ 2020, _7_ , 1479–1501
  * Gu and Mukamel 2020 Gu, B.; Mukamel, S. Manipulating nonadiabatic conical intersection dynamics by optical cavities. _Chem. Sci._ 2020, _11_ , 1290–1298
  * Mandal et al. 2020 Mandal, A.; Krauss, T.; Huo, P. Polariton-Mediated Electron Transfer via Cavity Quantum Electrodynamics. _J. Phys. Chem. B_ 2020, _124_ , 6321–6340
  * Chowdhury et al. 2021 Chowdhury, S.; Mandal, A.; Huo, P. Ring polymer quantization of the photon field in polariton chemistry. _J. Chem. Phys._ 2021, _154_ , 044109–11
  * Saller et al. 2021 Saller, M.; Kelly, A.; Geva, E. Benchmarking Quasiclassical Mapping Hamiltonian Methods for Simulating Cavity-Modified Molecular Dynamics. _J. Phys. Chem. Lett._ 2021, _12_ , 3163–3170
  * Saller et al. 2022 Saller, M. A. C.; Lai, Y.; Geva, E. An Accurate Linearized Semiclassical Approach for Calculating Cavity-Modified Charge Transfer Rate Constants. _J. Phys. Chem. Lett._ 2022, _13_ , 2330–2337
  * Basov et al. 2021 Basov, D.; Asenjo-Garcia, A.; Schuck, P.; Zhu, X.; Rubio, A. Polariton panorama. _Nanophotonics_ 2021, _10_ , 549–577
  * Yan et al. 2021 Yan, Y.; Xu, M.; Li, T.; Shi, Q. Efficient propagation of the hierarchical equations of motion using the Tucker and hierarchical Tucker tensors. _The Journal of Chemical Physics_ 2021, _154_ , 194104
  * Borrelli and Gelin 2021 Borrelli, R.; Gelin, M. F. Finite temperature quantum dynamics of complex systems: Integrating thermo-field theories and tensor-train methods. _Wiley Interdisciplinary Reviews: Computational Molecular Science_ 2021, _11_ , e1539
  * Lyu et al. 2023 Lyu, N.; Mulvihill, E.; Soley, M. B.; Geva, E.; Batista, V. S. Tensor-Train Thermo-Field Memory Kernels for Generalized Quantum Master Equations. _Journal of Chemical Theory and Computation_ 2023, _19_ , 1111–1129
  * Wang et al. 2023 Wang, Y.; Mulvihill, E.; Hu, Z.; Lyu, N.; Shivpuje, S.; Liu, Y.; Soley, M. B.; Geva, E.; Batista, V. S.; Kais, S. Simulating Open Quantum System Dynamics on NISQ Computers with Generalized Quantum Master Equations. _Journal of Chemical Theory and Computation_ 2023, _19_ , 4851–4862
  * Nagy and Foias 1970 Nagy, B. S.; Foias, C. _Harmonic analysis of operators on Hilbert space_ ; Holland Publishing Co., Amsterdam-London, 1970
  * Hu et al. 2020 Hu, Z.; Xia, R.; Kais, S. A quantum algorithm for evolving open quantum dynamics on quantum computing devices. _Sci. Rep._ 2020, _10_ , 3301
  * Saller et al. 2023 Saller, M. A. C.; Lai, Y.; Geva, E. Cavity-Modified Fermi’s Golden Rule Rate Constants from Cavity-Free Inputs. _The Journal of Physical Chemistry C_ 2023, _127_ , 3154–3164
  * Sun et al. 2018 Sun, X.; Zhang, P.; Lai, Y.; Williams, K.; Cheung, M.; Dunietz, B.; Geva, E. Computational Study of Charge-Transfer Dynamics in the Carotenoid-Porphyrin-$C_{60}$ Molecular Triad Solvated in Explicit Tetrahydrofuran and Its Spectroscopic Signature. _J. Phys. Chem. C_ 2018, _122_ , 11288–11299
  * Tong et al. 2020 Tong, Z.; Gao, X.; Cheung, M.; Dunietz, B.; Geva, E.; Sun, X. Charge transfer rate constants for the carotenoid-porphyrin-C60 molecular triad dissolved in tetrahydrofuran: The spin-boson model vs the linearized semiclassical approximation. _J. Chem. Phys._ 2020, _153_ , 044105–10
  * Oseledets and Tyrtyshnikov 2010 Oseledets, I.; Tyrtyshnikov, E. TT-cross approximation for multidimensional arrays. _Linear Algebra and Its Applications_ 2010, _432_ , 70–88
  * Oseledets 2011 Oseledets, I. V. Tensor-train decomposition. _SIAM Journal on Scientific Computing_ 2011, _33_ , 2295–2317
  * Lyu et al. 2022 Lyu, N.; Soley, M. B.; Batista, V. S. Tensor-Train Split-Operator KSL (TT-SOKSL) Method for Quantum Dynamics Simulations. _Journal of Chemical Theory and Computation_ 2022, _18_ , 3327–3346
  * Lyu et al. 2023 Lyu, N.; Miano, A.; Tsioutsios, I.; Cortiñas, R. G.; Jung, K.; Wang, Y.; Hu, Z.; Geva, E.; Kais, S.; Batista, V. S. Mapping Molecular Hamiltonians into Hamiltonians of Modular cQED Processors. _Journal of Chemical Theory and Computation_ 2023, _19_ , 6564–6576
  * Levy and Shalit 2014 Levy, E.; Shalit, O. M. Dilation theory in finite dimensions: the possible, the impossible and the unknown. _Rocky Mt. J. Math._ 2014, _44_ , 203–221


# BeTAIL: Behavior Transformer Adversarial Imitation Learning from Human
Racing Gameplay

Catherine Weaver1, Chen Tang1,2, Ce Hao1,3, Kenta Kawamoto4, Masayoshi
Tomizuka1, Wei Zhan1 1Department of Mechanical Engineering, University of
California Berkeley, CA, USA. Contact<EMAIL_ADDRESS>of
Computer Science, University of Texas, Austin, TX, USA.3 School of Computing,
National University of Singapore, Singapore4Sony Research, Tokyo, Japan.

###### Abstract

Imitation learning learns a policy from demonstrations without requiring hand-
designed reward functions. In many robotic tasks, such as autonomous racing,
imitated policies must model complex environment dynamics and human decision-
making. Sequence modeling is highly effective in capturing intricate patterns
of motion sequences but struggles to adapt to new environments or distribution
shifts that are common in real-world robotics tasks. In contrast, Adversarial
Imitation Learning (AIL) can mitigate this effect, but struggles with sample
inefficiency and handling complex motion patterns. Thus, we propose BeTAIL:
Behavior Transformer Adversarial Imitation Learning, which combines a Behavior
Transformer (BeT) policy from human demonstrations with online AIL. BeTAIL
adds an AIL residual policy to the BeT policy to model the sequential
decision-making process of human experts and correct for out-of-distribution
states or shifts in environment dynamics. We test BeTAIL on three challenges
with expert-level demonstrations of real human gameplay in Gran Turismo Sport.
Our proposed residual BeTAIL reduces environment interactions and improves
racing performance and stability, even when the BeT is pretrained on different
tracks than downstream learning. Videos and code available at:
https://sites.google.com/berkeley.edu/BeTAIL/home.

##  I Introduction

Autonomous racing is of growing interest to inform controller design at the
limits of vehicle handling and provide a safe alternative to racing with human
drivers. An autonomous racer’s driving style should resemble human-like racing
behavior in order to behave in safe and predictable ways that align with the
written and social rules of the race [1, 2]. Deep reinforcement learning (RL)
outperforms expert human players but requires iterative tuning of dense
rewards policy selection [1]. In fact, reward shaping in RL is often performed
with ad hoc trial and error [3]. Imitation learning (IL) is a potential
solution that mimics the behavior of experts using offline demonstrations [4].
In this work, we examine the challenges with IL in human racing gameplay, and
we propose a novel solution to capture non-Markovian decision-making and learn
a high-performing online policy in the Gran Turismo Sport (GTS) racing game.

Modeling environmental dynamics and human decision-making is essential for
learning human racing strategies [5]. However, the performance of Markovian
policies can deteriorate with human demonstrations [6]. Sequence-based
transformer architectures [7, 8], which are remarkably successful in language
models [9], accurately model the complex dynamics of human thought [10] while
still relying on the environment reward for finetuning policies. The Behavior
Transformer (BeT) [11], and Trajectory Transformer [8], which do not require
rewards, are casually conditioned on the past, so they are better at modeling
long-term environment dynamics [8]. Policies are trained via supervised
learning to autoregressively maximize the likelihood of trajectories in the
offline dataset. Thus, policies are limited by dataset quality [12] and are
sensitive to variations in environment dynamics and states out of the dataset
distribution.

Adversarial Imitation Learning (AIL) [13] overcomes these issues with offline
learning by leveraging adversarial training and reinforcement learning. A
trained discriminator network encourages the agent to match the state
occupancy of online rollouts and expert trajectories, reducing susceptibility
to distribution shift problems when encountering unseen states [4] or minor
variations in environment dynamics. However, AIL requires extensive
environmental interactions when training a policy from scratch, and its
performance can deteriorate significantly when learning from human
demonstrations [6]. Thus, AIL training of racing policies is unstable and
sample inefficient. The resulting policies also exhibit shaky steering
behavior and are prone to spinning off the track, as the policy does not
capture the non-Markovian decision-making strategy of humans.

In this work, we propose Behavior Transformer Adversarial Imitation Learning
(BeTAIL), which leverages both offline sequential modeling and online
occupancy-matching fine-tuning to 1.) capture the sequential decision-making
process of human demonstrators and 2.) correct for out-of-distribution states
or minor shifts in environment dynamics. First, a BeT policy is learned from
offline human demonstrations. Then, we examine an AIL mechanism to finetune
the policy to match the state occupancy of the demonstrations. BeTAIL adds a
residual policy, e.g. [14], to the BeT action prediction; the residual policy
refines the agent’s actions while remaining near the action predicted by the
BeT.

Our contributions are as follows:

  1. 1.

We propose Behavior Transformer Adversarial Imitation learning (BeTAIL) to
pre-train a BeT and fine-tune it with a residual AIL policy to learn complex,
non-Markovian behavior from human demonstrations.

  2. 2.

We show that when learning a racing policy from real human gameplay in Gran
Turismo Sport, BeTAIL outperforms BeT or AIL alone while closely matching non-
Markovian patterns in human demonstrations.

  3. 3.

We show BeTAIL when pre-training on a library of demonstrations from multiple
tracks to improve sample efficiency and performance when fine-tuning on an
unseen track with a single demonstration trajectory.

In the following, we first discuss preliminaries (Section III) and then
introduce our BeTAIL method (Section IV). In Section VI, we describe our three
experimental challenges to model human players in GT Sport, followed by the
conclusion in Section VII.

##  II Related Works

###  II-A Behavior Modeling

Our BeTAIL method relates to Behavior Modeling by learning a control policy
solely from human demonstrations. Behavior modeling aims to capture the
behavior of human demonstrators, which is important for robots and automated
vehicles that operate in close proximity to humans [15]. AIL for behavior
modeling overcomes the problem of cascading errors with traditional techniques
like behavioral cloning and parametric models [16]. Latent variable space
further allows researchers to model multiple distinct driving behaviors [17,
18, 19, 20]. However, sample efficiency and training stability are common
problems with learning AIL policies from scratch, which is exacerbated when
using human demonstrations [6] or complicated environments such as autonomous
racing [1]. Augmented rewards [17, 19] or negative demonstrations [21] can
accelerate training but share the same pitfalls as reward shaping in RL.
BeTAIL overcomes the sample efficiency and training stability pitfalls in AIL
by pretraining a BeT policy on human demonstrations to guide downstream
learning of an AIL residual policy.

###  II-B Curriculum Learning and Guided Learning

In RL, rather than training a policy from scratch, structured training
regimens can accelerate policy learning. Curriculum learning gradually
increases the difficulty of tasks [22] and can be automated with task phasing,
which gradually transitions a dense imitation-based reward to a sparse
environment reward [23]. “Teacher policies” can accelerate RL by providing
good but not expert actions. Policy intervention prevents unsafe actions [24,
25]. In guided policy search [26], the teacher demonstrates and guides the
objective of the agent; thus, large offline policies can be distilled into
lightweight RL policies [10]. BeTAIL employs residual RL policies [27, 14,
28], which can adapt to variations in the task by adding a helper policy to
the teacher. Thus, the agent’s action can be kept close to the action
predicted by the original controller with a small residual action space [29,
30].

###  II-C Sequence Modeling

Sequence-based modeling in offline RL method predicts the next action in a
trajectory sequence, which contains states, actions, and optionally goals.
Commonly, goals are set to the return-to-go, i.e. the sum of future rewards
from that time step [7, 8], but advantage conditioning improves performance in
stochastic environments [31]. Goal-conditioned policies can be fine-tuned on
online trajectories by exploiting automatic goal labeling [12]. Sequence
models can be distilled during online rollouts into lightweight policies by
using offline RL algorithms with guidance from the environment reward [10].
When goals or rewards are not available, such as in imitation learning [11,
8], the performance of offline models trained with supervised learning can
suffer under distribution shift and poor dataset quality [12]. BeTAIL
leverages a Behavior Transformer [11] sequence model to accelerate imitation
learning, without requiring an environment reward or structure goal states.

##  III Preliminaries

###  III-A Problem Statement

We model the learning task as a Markov Process (MP) defined by
$\\{\mathcal{S},\mathcal{A},T\\}$ of states $s\in\mathcal{S}$, actions
$a\in\mathcal{A}$, and the transition probability
$T(s_{t},a_{t},s_{t+1}):\mathcal{S}\times\mathcal{A}\times\mathcal{S}\mapsto[0,1]$.
Note that, unlike RL, we do not have access to an environment reward. We have
access to human expert demonstrations in the training environment consisting
of a set of trajectories,
${D_{E}}=(\tau_{0}^{E},\tau_{1}^{E}...,\tau_{M}^{E})$, of states and actions
at every time step $\tau=(s_{t},a_{t},...)$. The underlying expert policy,
$\pi_{E}$, is unknown. The goal is to learn the agent’s policy $\pi$, that
best approximates the expert policy $\pi_{E}$. Each expert trajectory
$\tau^{E}$, consists of states and action pairs:

$\tau^{E}=(s_{0},a_{0},s_{1},a_{1},\ldots,s_{N},a_{N}).$ (1)

The human decision-making process of the expert is unknown and likely non-
Markovian [32]; thus, imitation learning performance can deteriorate with
human trajectories [6].

###  III-B Unimodal Decision Transformer

The Behavior Transformer (BeT) processes the trajectory $\tau_{E}$ as a
sequence of 2 types of inputs: states and actions. The original BeT
implementation [11] employed a mixture of Gaussians to model a dataset with
multimodal behavior. For simplicity and to reduce the computational burden, we
instead use an unimodal BeT that uses a deterministic similar to the one
originally used by the Decision Transformer [7]. However, since residual
policies can be added to black-box policies [14], BeTAIL’s residual policy
could be easily added to the k-modes present in the original BeT
implementation.

At timestep $t$, the BeT uses the tokens from the last $K$ timesteps to
generate the action $a_{t}$, where $K$ is referred to as the context length.
Notably, the context length during evaluation can be shorter than the one used
for training. The BeT learns a deterministic policy
$\pi_{\mathrm{BeT}}(a_{t}|\mathbf{s}_{-K,t})$, where $\mathbf{s}_{-K,t}$
represents the sequence of $K$ past states $\mathbf{s}_{\max(1,t-K+1):t}$. The
policy is parameterized using the minGPT architecture [33], which applies a
causal mask to enforce the autoregressive structure in the predicted action
sequence.

A notable strength of BeT is that the policy can model non-Markovian behavior;
in other words, rather than modeling the action probability as
$P(a_{t}|s_{t})$, the policy models the probability
$P(a_{t}|s_{t},s_{t-1},...,s_{t-h+1})$. However, the policy is trained using
only the offline dataset, and even minor differences between the training data
and evaluation environment can lead to large deviations in the policies’
performance [4].

##  IV Behavior Transformer-Assisted Adversarial Imitation Learning

We now present Behavior Transformer-Assisted Adversarial Imitation Learning
(BeTAIL), summarized in Fig. 1. First, an unimodal version of a causal
Behavior Transformer (BeT) [11] is trained on the offline demonstrations to
capture the sequential decision-making of the human experts. Then, the BeT
policy is used as the deterministic base policy for Adversarial Imitation
Learning (AIL) fine-tuning. BeTAIL uses AIL to train a residual policy [29],
which is added to the base policy’s action to form the agent’s policy. Thus,
BeTAIL combines offline sequential modeling and online occupancy-matching
fine-tuning to capture the decision-making process of human demonstrators and
adjust for out-of-distribution states or environment changes.

###  IV-A Behavior Transformer (BeT) Pretraining

A unimodal BeT policy $\hat{a}=\pi_{\mathrm{BeT}}(a_{t}|\textbf{s}_{-K,t})$ is
used to predict a base action $\hat{a}$. Following [7, 12], sub-trajectories
of length $K$ in the demonstrations are sampled uniformly and trained to
minimize the difference between the action predicted by the BeT and the next
action in the demonstration sequence. While the action predicted by the BeT,
$\hat{a}$, considers the previous state history $\textbf{s}_{-K,t}$, the
action may not be ideal in stochastic environments [31]. Thus, we correct the
base action by adding a residual action [14] that can correct the base action
based on how well the agent performs in the environment. Since an environment
reward is not available, and we wish to replicate the human demonstrations, we
employ AIL to train the residual policy.

Figure 1: BeTAIL rollout collection. The pre-trained BeT predicts action
$\hat{a}_{t}$ from the last $H$ state-actions. Then the residual policy
specifies action $\tilde{a}_{t}$ from the current state and $\hat{a}_{t}$, and
the agent executes $a_{t}=\hat{a}_{t}+\tilde{a}_{t}$ in the environment.

###  IV-B Residual Policy Learning for Online Fine-tuning

The agent’s action is the sum of the action specified by the BeT and the
action from a residual policy [14]. The residual policy corrects or improves
the actions from the BeT base policy [14]. First, we define an augmented MP:
$\tilde{\mathcal{M}}=\\{\tilde{\mathcal{S}},\tilde{\mathcal{A}},\tilde{T}\\}$.
The state space $\tilde{\mathcal{S}}$ is augmented with the base action:
$\tilde{s}_{t}\doteq\begin{bmatrix}s_{t}&a_{t}\end{bmatrix}^{\intercal}$. The
action space, $\tilde{A}$, is the space of the residual action $\tilde{a}$.
The transition probability,
$\tilde{T}(\tilde{s}_{t},\tilde{a}_{t},\tilde{s}_{t+1})$, includes both the
dynamics of the original environment, i.e. $T(s_{t},a_{t},s_{t+1})$, and also
the dynamics of the base action
$\hat{a}_{t}=\pi_{\mathrm{BeT}}(\cdot|\mathbf{s}_{-K,t})$.

We define the Gaussian $\tilde{a}\sim
f_{\textrm{res}}(\tilde{a}|s_{t},\hat{a}_{t})=\mathcal{N}(\mu,\sigma)$ as the
residual policy. The residual policy uses the current state and the action
predicted by the base policy to select the residual action. Then, this
residual action is added to the base action. This sum, $a_{t}$, is the action
that is taken in the environment:

$a_{t}=\hat{a}_{t}+\operatorname{clip}(\tilde{a}_{t},-\alpha,\alpha).$ (2)

The residual policy is constrained to be between $[-\alpha,\alpha]$, which
allows constraint on how much the environment action $a_{t}$ is allowed to
deviate from the base action $\hat{a}$ [29, 14]. In the case where $\alpha$ is
small, the environment action must still be close to the base action,
$\hat{a}_{t}$, predicted by the BeT.

Assuming the action space of $f_{\theta}(\cdot|s_{t},\hat{a}_{t})$ is
restricted to the range $[-\alpha,\alpha]$, we can define the policy
$\pi_{A}(a|\mathbf{s}_{-k,t})$:

$\displaystyle\pi_{A}$ $\displaystyle(a|\mathbf{s}_{-k,t})=$ (3)
$\displaystyle\tilde{a}_{t}|_{\tilde{a}_{t}\sim
f_{\mathrm{res}}(\cdot|s_{t},\pi_{\mathrm{BeT}}(\cdot|\mathbf{s}_{-k,t}))}+\pi_{\mathrm{BeT}}(\cdot|\mathbf{s}_{-k,t}).$

Eq. 3 follows the notation of [34]. Because the BeT policy is non-Markovian,
the agent’s policy $\pi_{A}$ is also non-Markovian. However, using the
definition of the augmented state, $\tilde{s}$, the input to the AIL residual
policy is the current state and the base action predicted by the BeT. Thus,
$\pi_{A}$ is in fact Markovian with respect to the augmented state
$\tilde{s}$:

$\pi_{A}(a|s_{t},\hat{a}_{t})=\tilde{a}_{t}|_{\tilde{a}_{t}\sim
f_{\mathrm{res}}(\cdot|s_{t},\hat{a}_{t})}+\hat{a}_{t}.$ (4)

Following previous works with residual policies [14, 34], during online
training, the agent’s policy during rollouts is given by $\pi_{A}$ in (4).
Namely, the agent’s policy is the sum of the action given by the base policy
$\pi_{\mathrm{BeT}}$ and the action given by the residual policy
$f_{\mathrm{res}}$ according to Eq. (3). After initial training (Section
IV-A), the deterministic base policy, $\pi_{\mathrm{BeT}}$, is frozen during
online training, and the agent’s policy is improved by updating the residual
policy $f_{\mathrm{res}}$ with AIL.

###  IV-C Residual Policy Training with AIL

To train the residual policy, we adapt Adversarial Imitation Learning (AIL)
[35] so that the agent’s policy in online rollouts solves the AIL objective.
However, AIL is typically defined in terms of a single Markovian policy that
is not added to a base policy [13]. In this section, we detail our updates to
the AIL formulation to employ AIL to update the residual policy
$f_{\mathrm{res}}$. Given the definition of $\pi_{A}$ (4), we define the AIL
objective for residual policy learning as

$\underset{f_{\mathrm{res}}}{\operatorname{minimize}}\quad
D_{\mathrm{JS}}\left(\rho_{\pi_{A}},\rho_{\pi_{E}}\right)-\lambda
H(f_{\mathrm{res}}).$ (5)

Similar to the standard AIL objective [35], we still aim to minimize the
distance between the occupancy measures of the expert’s and agent’s policies,
denoted as $\rho_{\pi_{E}}$ and $\rho_{\pi_{A}}$ respectively in Eq. (5). The
minimization with respect to $f_{\mathrm{res}}$ is valid since Eq. (4) simply
defines a more restrictive class of policies than standard single-policy AIL.
Since the contribution from the base policy is deterministic, we regularize
the problem using only the entropy of the residual policy, and the policy
update step is replaced with

$\max_{f_{\mathrm{res}}}\underset{\tilde{\tau}\sim
f_{\mathrm{res}}}{\mathbb{E}}\left[\sum_{t=0}^{\infty}\gamma^{t}\Bigg{(}\tilde{R}^{E}\left(\tilde{s}_{t},\tilde{a}_{t}\right)+\lambda
H\left(f_{\mathrm{res}}\left(\cdot\mid\tilde{s}_{t}\right)\right)\Bigg{)}\right],$
(6)

where $\tilde{\tau}$ represents the trajectory
$\tilde{\tau}=(\tilde{s}_{0},\tilde{a}_{0},...)$ collected using
$f_{\mathrm{res}}$ on $\tilde{\mathcal{M}}$. Thus, the residual policy
$f_{\mathrm{res}}$ is updated using the augmented state and residual action.
We define $\tilde{R}^{E}$ as the proxy environment reward on $\tilde{M}$.
Following previous works with residual policy learning for RL [14], the proxy
reward is calculated using the action taken in the environment. To similarly
adapt AIL’s proxy environment reward, we define

$\displaystyle\tilde{R}^{E}(\tilde{s}_{t},\tilde{a}_{t})$
$\displaystyle=-\log\left(1-D_{\omega}^{E}(s,\hat{a}+\tilde{a})\right),$ (7)

where $D_{\omega}^{E}(s,\hat{a}+\tilde{a})$ is a binary classifier trained to
minimize

$\displaystyle\mathcal{L}_{D,\mathcal{D}_{E}}(\omega)=-$ $\displaystyle
E_{\tau_{E}\sim\mathcal{D}_{E}}\left[\log\left(D^{E}_{\omega}(s,a)\right)\right]$
(8) $\displaystyle-E_{\tilde{\tau}\sim
f_{\mathrm{res}}}\left[\log\left(1-D^{E}_{\omega}(s,\hat{a}+\tilde{a})\right)\right],$

which is equivalent to

$\displaystyle\mathcal{L}_{D,\mathcal{D}_{E}}(\omega)=-$ $\displaystyle
E_{\tau_{E}\sim\mathcal{D}_{E}}\left[\log\left(D^{E}_{\omega}(s,a)\right)\right]$
(9) $\displaystyle-
E_{\tau\sim\pi_{a}}\left[\log\left(1-D^{E}_{\omega}(s,a)\right)\right].$

Eq. (7) implies that give the pair $(\tilde{s},\tilde{a})$, the environment
reward is the probability that the state-action pair
$(s,a)=(s,\hat{a}+\tilde{a})$ comes from the expert policy, according to the
discriminator. Thus, by iterating between the policy learning objective (6)
and the discriminator loss (9), AIL minimizes (5) and determines the
$f_{\mathrm{res}}$ that allows $\pi_{A}$ to closely match the occupancy
measure of the expert $\pi_{E}$.

##  V Imitation of Human Racing Gameplay

We describe the method to learn from human gameplay in the GTS racing game,
including the state features, environment, and training regime for three IL
challenges.

###  V-A State Feature Extraction and Actions

Our selected features include features that were previously shown as important
in RL and are available in the demonstrations. We include the following state
features exactly as described in [36]: 1.) The linear velocity,
$\mathbf{v}_{t}\in\mathbb{R}^{3}$, and linear acceleration
$\dot{\mathbf{v}}_{t}\in\mathbb{R}^{3}$ with respect to the inertial frame of
the vehicle. 2.) The Euler angle $\theta_{t}\in(-\pi,\pi]$ between the 2D
vector that defines the agent’s rotation in the horizontal plane and the unit
tangent vector that is tangent to the centerline at the projection point. 3.)
A binary flag with $w_{t}=1$ indicating wall contact 4.) N sampled curvature
measurement of the course centerline in the near future
$\mathbf{c}_{t}\in\mathbb{R}^{N}$.

Additionally, we select features similar to those used in [1]: 5.) The cosine
and sine of the vehicle’s current heading, $\cos(\psi)$ and $\sin(\psi)$. 6.)
The relative 2D distance from the vehicle’s current position to the left,
$\mathbf{e}_{LA,l}$, right, $\mathbf{e}_{LA,r}$, and center,
$\mathbf{e}_{LA,c}$, of the track at 5 “look-ahead points.” The look-ahead
points (blue dots in Fig. 2) are placed evenly using a look-ahead time of 2
seconds, i.e., they are spaced evenly over the next 2 seconds of track,
assuming the vehicle maintains its current speed. Thus the full state is a
vector composed of
$s_{t}=\left[\boldsymbol{v}_{t},\dot{\boldsymbol{v}}_{t},\theta_{t},w_{t},\boldsymbol{c}_{t},\cos(\psi),\sin(\psi),\mathbf{e}_{LA,l},\mathbf{e}_{LA,r},\mathbf{e}_{LA,c}\right]$.
Finally, the state is normalized using the mean and standard deviation of each
state feature in the demonstrations.

GTS receives the steering command $\delta\in[-\pi/6,\pi/6]$ rad and a
throttle-brake signal $\omega_{\tau}\in[-1,1]$ where $\omega_{\tau}=1$ denotes
full throttle and $\omega_{\tau}=-1$ denotes full brake [36, 1]. For all
baseline comparisons, the steering command in the demonstrations is scaled to
be between $[-1,1]$. The agent specifies steering actions between $[-1,1]$,
which are scaled to $\delta\in[-\pi/6,\pi/6]$ before being sent to GTS.

In the case where a residual policy is learned (see BeTAIL and BCAIL in the
next section), the residual policy network predicts $\tilde{a}\in[-1,1]$, and
then $\tilde{a}$ is scaled111The choice to scale the residual policy between
[-1,1] is made based on the practice of using scaled action spaces with SAC
[37] to be between $[-\alpha,\alpha]$ and then added to $a=\hat{a}+\tilde{a}$.
Since $\hat{a}\in[-1,1]$, it is possible for $a$ to be outside the bounds of
$[-1,1]$, so we clip $a$ before sending the action to GTS. The choice of the
hyperparameter $\alpha$ determines the maximum magnitude of the residual
action. Depending on the task and the strength of the base policy, $\alpha$
can be relatively large to allow the residual policy to correct for bad
actions [34], or $\alpha$ can be relatively small to ensure the policy does
not deviate significantly from the base policy [14, 30]. In this work, we set
$\alpha$ as small as possible to achieve high performance for each task so
that the policy remains as close as possible to the offline BeT, which
captures non-Markovian racing behavior.

###  V-B Environment and Data Collection

####  V-B1 Gran Turismo Sport Racing Simulator

We conduct experiments in the high-fidelity PlayStation (PS) game Gran Turismo
Sport (GTS) (https://www.gran-turismo.com/us/), developed by Polyphony
Digital, Inc. GTS takes two continuous inputs: throttle/braking and steering.
The vehicle positions, velocities, accelerations, and pose are observed. The
agent’s and demonstrator’s state features and control inputs are identical.
Deep RL recently achieved super-human performance against human opponents in
GTS [1]. However, this success required tuned, dense reward functions in order
for the racing agent to behave “well” with human opponents. Rather than
carefully crafting a reward function, we explore whether demonstrations from
expert human drivers can inform a top-performing agent.

Figure 2: Track Lookahead Points on Lago Maggiore

####  V-B2 Agent Rollout and Testing Environments

To collect rollouts and evaluation episodes, the GTS simulator runs on a
PlayStation, while the agent runs on a separate desktop computer. The desktop
computer communicates with GTS over a dedicated API via an Ethernet connection
similar to the one described in [36]. The API provides the current state of up
to 20 simulated cars and accepts car control commands, which are active until
the next command is received. While the GTS state is updated every 60Hz, we
limit the control frequency of our agent to 10Hz to reduce the load on the PS
and the desktop computers following previous works in RL [36, 1]. Unlike
previous works that trained a single policy network and collected rollouts in
parallel, we opt to train and collect rollouts sequentially to reduce the
computational burden on the desktop PC, given our framework has two networks:
the BeT policy and the residual policy.

(a) Lago Maggiore Challenge: Training Scheme

(b) Dragon Tail Challenge: Training Scheme

(c) Mount Panorama Challenge: Training Scheme

(d) Training on Lago Maggiore (mean $\pm$ std)

(e) Training on Dragon Tail (mean $\pm$ std)

(f) Training on Mount Panorama (mean $\pm$ std)

Lap Steering Change Algorithm Time (s) $|\delta_{t}-\delta_{t-1}|$ (rad)
BeTAIL(0.05) 129.2$\pm$0.7 0.014$\pm$0.011 BeTAIL(1.0) 127.7$\pm$0.7
0.068$\pm$0.085 AIL 131.6$\pm$4.1 0.081$\pm$0.091 BCAIL 140.7$\pm$8.6
0.022$\pm$0.021 BeT 205.2 $\pm$38 0.0058$\pm$0.010 Human 121.5 $\pm$0.4
0.0067$\pm$0.011

(g) Results on Lago Maggiore Track

Lap Steering Change Algorithm Time (s) $|\delta_{t}-\delta_{t-1}|$ (rad)
BeTAIL(0.10) 112.1$\pm$0.8 0.024$\pm$0.019 BeTAIL(1.0) 109.2$\pm$0.3
0.083$\pm$0.094 AIL 109.1$\pm$0.7 0.077$\pm$0.075 BeT unfinished
0.011$\pm$0.019 Human 103.9 $\pm$0.5 0.0018$\pm$0.0061

(h) Results on Dragon Tail Track

Lap Steering Change Algorithm Time (s) $|\delta_{t}-\delta_{t-1}|$ (rad)
BeTAIL(1-trk) 146.0$\pm$1.9 0.044$\pm$0.036 BeTAIL(4-trk) 140.9$\pm$1.2
0.039$\pm$0.033 AIL 141.8$\pm$1.4 0.087$\pm$0.088 BeT(4-trk) nan $\pm$0.0
0.014$\pm$0.024 Human 133.3 $\pm$0.0 0.0067$\pm$0.013

(i) Best Results on Mount Panorama Track ($\alpha$=0.2)

Figure 3: Experimental results on three racing challenges. (a) Lago Maggiore
challenges pretrains the BeT on the same demonstrations and downstream
environments. (b) Dragon Tail transfers the BeT policy to a new track with
BeTAIL finetuning. (c) The Mount Panorama challenge pretrains the BeT on a
library of 4 tracks, and BeTAIL finetunes on an unseen track. (d)-(f)
evaluation of mean (std) success rate to finish laps and mean (std) of lap
times. (g)-(i) Best policy’s mean $\pm$ std lap time and change in steering
from previous time step.

###  V-C Training Scheme for Imitation Learning

We evaluate BeTAIL on three challenges with different demonstration
availability and environment setups, as indicated in the top of Fig. 3. For a
more detailed description of each challenge, see the corresponding part in
Section VI.

####  V-C1 Algorithms and Baselines

For each challenge, the BeT policy,
$\pi_{\textrm{BeT}}(\cdot|\mathbf{s}_{-K,t})$, is pretrained on offline human
demonstrations from one or more tracks. Then our BeTAIL fine-tunes control by
training an additive residual policy, $f_{\mathrm{res}}$, with AIL on the
downstream environment and human demonstrations. The residual policy is
constricted between $[-\alpha,\alpha]$, where $\alpha$ is specified
individually for each challenge. In Fig. 3a-c, red corresponds to
demonstrations for BeT policy and green corresponds to the downstream
environment and demonstrations for the residual AIL policy. The BeT baseline
uses the BeT policy, $\pi_{\textrm{BeT}}(\cdot|\mathbf{s}_{-K,t})$, with the
pretraining demonstrations red. Conversely, the AIL baseline consists of a
single policy, $\pi_{A}(a|s)$, trained via AIL using only the downstream
demonstrations and environment in green. The BCAIL baseline trains a BC
policy, $\pi_{A}(a|s)$, on the offline demonstrations (in red) and then fine-
tunes a residual policy in the downstream environment (in green) following the
same training scheme as BeTAIL. However, given the underperformance of both BC
and BCAIL in the first challenge, we exclude the comparison of BCAIL in the
other, more difficult challenges.

Agents use soft actor-critic (SAC) [37] for the reinforcement learning
algorithm in adversarial learning (i.e. for AIL, BCAIL, and BeTAIL). Since SAC
is off-policy, this implies that the replay buffer can include historical
rollouts from multiple iterations prior. However, the discriminator network,
and thus the reward associated with a state-action pair may have changed.
Thus, when sampling data from the replay buffer for policy and Q-network
training, we re-calculate the reward for the associated state-action pair
using the current discriminator network. Finally, discriminator overfitting
can cause AIL to fail to learn a meaningful policy if the discriminator is too
powerful. For all AIL methods, we employ two regulators when training the
discriminator: gradient penalty and discriminator entropy regularization [6].

####  V-C2 Demonstration Overview

All demonstrations were recorded from different human players GTS
participating in real gameplay, “time-trial” competitions with the Audi TT Cup
vehicle. All trajectories are high-quality and performed by an expert-level
racer. Each demonstration contains a full trajectory of states and actions,
around a single lap of the track. The trajectories were recorded at a
frequency of 60Hz, which we downsample to 10Hz (the agent’s control
frequency). Each trajectory (i.e. lap) contains approximately 7000 timesteps.
As shown in Fig. 3a-c, we test three training schemes to test if BeTAIL can
improve performance on the same track as pretraining, transfer a single-track
BeT policy to a new track, and transfer a multi-track BeT policy to an unseen
track (see Section VI for more information).

##  VI Racing Experiment Results

We introduce three challenges with distinct pretraining demonstrations and
downstream environments to test our BeTAIL formulation. For evaluation, 20
cars are randomly placed evenly on the track. Each car’s initial speed is set
to the expert’s speed at the nearest position on a demonstration trajectory.
Each car has 500 seconds to complete a full lap. In Fig. 3d-f, we provide two
evaluation metrics during training: the proportion of cars that finish a lap
(top) and the average lap times of cars that finish. Error bars show the total
standard deviation across the 20 cars (evaluation episodes) and 3 seeds. Fig.
3g-i provide the mean$\pm$standard deviation of the best policy’s lap time and
absolute change in steering $|\delta_{t}-\delta_{t-1}|$, as RL policies can
exhibit undesirable shaky steering behavior [1] not present in human experts.
Videos of the trajectories and GTS game environment are provided at
https://sites.google.com/berkeley.edu/BeTAIL/home. In the following, we detail
the setup of each challenge and analyze the results.

###  VI-A Lago Maggiore Challenge: Fine-tuning in the same environment with
BeTAIL

The first experiment tests how well BeTAIL can fine-tune the BeT policy when
the BeT policy is learned from demonstrations on the same track as the
downstream environment (Fig. 3a). Since the BeT policy is pretrained on the
same environment, BeTAIL(0.05) employs a small residual policy, $\alpha=0.05$.
The ablation BeTAIL(1.0) uses a large residual policy, $\alpha=1.0$. There are
49 demonstrations from different human players on the Lago Maggiore track. The
BeT policy is trained on the 49 trajectories, then we run BeTAIL on the same
49 trajectories and the same environment as the demonstrations. BCAIL follows
the same training scheme ($\alpha=0.05$). AIL is trained on the same 49
trajectories.

Figure 4: Agent trajectories on Lago Maggiore Challenge. We deliberately
initialize AIL and BeTAIL to start at a lower initial speed than the human
demonstration. Car drawing is placed at the vehicle’s location and heading
every 0.4s. See the website for the animated version.

Fig. 3d shows the evaluation of each agent during training. BeTAIL performs
the best of all the methods and rapidly learns a policy that can consistently
finish a full lap and achieve the lowest lap time. The AIL baseline can
eventually learn a policy that navigates the track and achieves a similarly
low lap time; however, even towards the end of the training, the AIL policies
are less consistent, and multiple cars may fail to finish a full lap during
evaluation, as evidenced by the higher standard deviation in the number of
finished laps in the top of Fig. 3d. The remaining baselines perform poorly on
this task. Using a residual BC policy (BCAIL) actually worsens performance,
likely because the BC policy performs poorly in an unseen environment due to
distribution shifts. Impressively, the BeT policy can finish some laps even
though it is trained exclusively on offline data; however, the lap time is
poor compared to BeTAIL and AIL, which are able to use online rollouts to
correct any issues in the policy.

In Fig. 3g, BeTAIL(1.0) can achieve a better lap time than BeTAIL(0.05), since
larger $\alpha$ allows the residual policy to apply more correction to the BeT
base policy. However, BeTAIL(1.0) is prone to oscillating steering back and
forth, as indicated by the higher deviation in the steering command between
timesteps. We hypothesize that it is because the non-Markovian human-like
behavior diminishes as $\alpha$ becomes larger. It is supported by the
observation that the Markovian AIL policy also tends to have extreme changes
in the steering command. In Fig. 4, We provide further insight into how the
extreme steering rates of the AIL policy harm its robustness. We deliberately
initialize the race cars at a significantly lower speed than the human
demonstrator before the corner for AIL and BeTAIL. We aim to test the
robustness of learned agents by monitoring their stability at unseen states.
The heading of the AIL agent oscillates due to its tendency to rapidly steer
back and forth, which causes the AIL agent to lose control of the vehicle and
collide with the corner. Conversely, BeTAIL smoothly accelerates and brakes
into the corner.

###  VI-B Dragon Tail Challenge: Transferring the BeT policy to another track
with BeTAIL

The second experiment tests how well BeTAIL can fine-tune the BeT policy when
the downstream environment is different from the BeT demonstrations (Fig. 3b).
The same BeT policy is used in the Lago Maggiore Challenge; however, the
downstream environment is a different track, and there are only 12
demonstrations for the new Dragon Trail track. For BeTAIL(0.10), we allow the
residual policy to be slightly larger, $\alpha=0.10$, since the downstream
environment is different than the BeT pretraining demonstrations. The vehicle
dynamics are unchanged.

The results for training and evaluating the policies on the Dragon Tail track
are given in Fig. 3e/h. Again, BeTAIL can employ BeT to guide policy learning;
as a result, BeTAIL quickly learns to consistently navigate the track at a
high speed. Additionally, small $\alpha$ can ensure that non-Markovian human
behavior is preserved, resulting in smooth steering. Conversely, AIL can learn
policies that are capable of achieving low lap times; however, they are
significantly more prone to rapid changes in steering and much more prone to
fail to finish a lap (top in Fig. 3e). The pretrained BeT, which was trained
from demonstrations on a different track, is unable to complete any laps.

###  VI-C Mount Panorama Challenge: Learning a multi-track BeT policy and
solving an unseen track with BeTAIL

Finally, a BeT policy is trained on a library of trajectories on four
different tracks: Lago Maggiore GP (38 laps), Autodromo de Interlagos (20
laps), Dragon Tail - Seaside (28 laps), and Brands Hatch GP (20 laps). For
BeTAIL fine-tuning, BeTAIL is trained on a single demonstration on the Mount
Panorama Circuit. Due to the availability of trajectories, there is a slight
change in vehicle dynamics from the first two challenges due to the use of
different tires (Racing Hard); the vehicle dynamics in downstream training and
evaluation employ the same Racing Hard tires as the library of trajectories.
The Mount Panorama circuit has a significantly different course geometry with
more hills and sharp banked turns than the pretraining tracks; thus, we allow
the residual policy to be relatively larger to correct for errors in the
offline BeT policy, $\alpha=0.2$. In Fig. 3f, BeTAIL(4-track) indicates the
BeT is trained on the library of trajectories (Fig. 3c) with $\alpha=0.2$. As
an ablation, we also compare BeTAIL(1-track) with $\alpha=0.2$, where the
pretrained BeT policy is the one used in Fig. 3a/b with demonstrations
collected with Racing Medium tires, which have a lower coefficient of friction
than the Racing Hard tires in the downstream environment.

As with previous challenges, both BeTAIL(1-track) and BeTAIL(4-track) can
learn a policy that learns to consistently navigate the track significantly
faster than AIL alone. BeTAIL(4-track) achieves faster lap times and smoother
steering than AIL, indicating that BeT pre-training on a library of tracks can
assist when encountering new tracks. BeTAIL(1-track) exhibits a slight
performance drop when transferring between different vehicle dynamics compared
to BeTAIL(4-track). However, BeTAIL(1-track) still accelerates training to
achieve a high success rate faster than AIL alone. Videos are provided on our
website. While all agents struggle at the beginning of the track, BeTAIL and
AIL can navigate the track rapidly despite the complicated track geometry and
hills. However, AIL still exhibits undesirable shaking behavior leading into
corners while BeTAIL navigates each corner smoothly and with the highest
speed.

The results of the challenges indicate that BeTAIL is a powerful method for
accelerating learning with AIL. For all challenges and ablations, BeTAIL can
consistently achieve high finished laps success rates early in training. We
contrast this to AIL’s training, which is slower and unstable as indicated by
the large deviations and drops in mean in tops of Fig. 3d-f. Thus, BeTAIL can
outperform AIL and BeT alone when the BeT policy is trained on the same track
or a library of many tracks.

##  VII Conclusion

We proposed BeTAIL, a Behavior Transformer that is assisted with a residual
Adversarial Imitation Learning policy, to leverage sequence modeling and
online imitation learning to learn racing policies from human demonstrations.
In three experiments in the high-fidelity racing simulator Gran Turismo Sport,
we show that BeTAIL can leverage both demonstrations on a single track and a
library of tracks to accelerate downstream AIL on unseen tracks with limited
demonstrations. BeTAIL policies exhibit smoother steering, which is
reminiscent of human steering behavior, and additionally result in consistent
policies that reliably complete racing laps. Furthermore, in a small ablation,
we demonstrate that BeTAIL can even accelerate learning under minor dynamics
shifts when the BeT is trained with different tires and tracks than the
downstream residual AIL.

Limitations and Future Work: BeTAIL formulates the online imitation learning
as a separate pre-trained BeT network and a residual AIL policy network, which
each require individual training. Further, residual AIL formulation assumes
that the residual policy and the discriminator network are applied to single-
time steps in a Markovian fashion, rather than exploiting the sequence
modeling in the BeT. In future work, we are interested in an alternate
theoretical framework that improves the BeT action predictions themselves
using online rollouts. Additionally, we are interested in how sequence
modeling could be introduced into AIL frameworks to match the policy’s and
expert’s trajectory sequences instead of single-step state-action occupancy
measures. While BeTAIL accelerates learning and achieves lower lap times
compared to existing methods, there is still a small gap between BeTAIL’s lap
times and the lap times achieved in the expert demonstrations. Future work
will explore how to narrow this gap with improved formulations or longer
training and increased environment interactions.

## References

  * [1] P. R. Wurman, S. Barrett, K. Kawamoto, J. MacGlashan, K. Subramanian, T. J. Walsh, R. Capobianco, A. Devlic, F. Eckert, F. Fuchs, _et al._ , “Outracing champion gran turismo drivers with deep reinforcement learning,” _Nature_ , vol. 602, no. 7896, pp. 223–228, 2022.
  * [2] J. Betz, H. Zheng, A. Liniger, U. Rosolia, P. Karle, M. Behl, V. Krovi, and R. Mangharam, “Autonomous vehicles on the edge: A survey on autonomous vehicle racing,” _Open Journal Intell. Trans. Syst._ , vol. 3, pp. 458–488, 2022.
  * [3] S. Booth, W. B. Knox, J. Shah, S. Niekum, P. Stone, and A. Allievi, “The perils of trial-and-error reward design: Misdesign through overfitting and invalid task specifications,” _Proc. AAAI Conf. Artificial Intell._ , vol. 37, no. 5, pp. 5920–5929, Jun. 2023. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/25733
  * [4] M. Zare, P. M. Kebria, A. Khosravi, and S. Nahavandi, “A survey of imitation learning: Algorithms, recent developments, and challenges,” _arXiv preprint arXiv:2309.02473_ , 2023.
  * [5] E. L. Zhu, F. L. Busch, J. Johnson, and F. Borrelli, “A gaussian process model for opponent prediction in autonomous racing,” in _Int. Conf. Intell. Robots Systems (IROS)_. IEEE, 2023, pp. 8186–8191.
  * [6] M. Orsini, A. Raichuk, L. Hussenot, D. Vincent, R. Dadashi, S. Girgin, M. Geist, O. Bachem, O. Pietquin, and M. Andrychowicz, “What matters for adversarial imitation learning?” _Adv. Neural Inform. Processing Syst._ , vol. 34, pp. 14 656–14 668, 2021.
  * [7] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and I. Mordatch, “Decision transformer: Reinforcement learning via sequence modeling,” _Adv. Neural Inform. Processing Syst._ , vol. 34, pp. 15 084–15 097, 2021.
  * [8] M. Janner, Q. Li, and S. Levine, “Offline reinforcement learning as one big sequence modeling problem,” _Adv. Neural Inform. Processing Syst._ , vol. 34, pp. 1273–1286, 2021.
  * [9] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, _et al._ , “Improving language understanding by generative pre-training,” 2018.
  * [10] J. Li, X. Liu, B. Zhu, J. Jiao, M. Tomizuka, C. Tang, and W. Zhan, “Guided online distillation: Promoting safe reinforcement learning by offline demonstration,” _arXiv preprint arXiv:2309.09408_ , 2023.
  * [11] N. M. Shafiullah, Z. Cui, A. A. Altanzaya, and L. Pinto, “Behavior transformers: Cloning $k$ modes with one stone,” _Adv. Neural Inform. Processing Syst._ , vol. 35, pp. 22 955–22 968, 2022.
  * [12] Q. Zheng, A. Zhang, and A. Grover, “Online decision transformer,” in _Int. Conf. Machine Learning_. PMLR, 2022, pp. 27 042–27 059.
  * [13] J. Ho and S. Ermon, “Generative adversarial imitation learning,” _Adv. Neural Inform. Processing Syst._ , vol. 29, 2016.
  * [14] T. Silver, K. Allen, J. Tenenbaum, and L. Kaelbling, “Residual policy learning,” _arXiv preprint arXiv:1812.06298_ , 2018.
  * [15] K. Brown, K. Driggs-Campbell, and M. J. Kochenderfer, “A taxonomy and review of algorithms for modeling and predicting human driver behavior,” _arXiv preprint arXiv:2006.08832_ , 2020.
  * [16] A. Kuefler, J. Morton, T. Wheeler, and M. Kochenderfer, “Imitating driver behavior with generative adversarial networks,” in _Intell. Vehicles Sym. (IV)_. IEEE, 2017, pp. 204–211.
  * [17] Y. Li, J. Song, and S. Ermon, “Infogail: Interpretable imitation learning from visual demonstrations,” _Adv. Neural Inform. Processing Syst._ , vol. 30, 2017.
  * [18] R. Bhattacharyya, B. Wulfe, D. J. Phillips, A. Kuefler, J. Morton, R. Senanayake, and M. J. Kochenderfer, “Modeling human driving behavior through generative adversarial imitation learning,” _IEEE Trans. Intell. Transp. Syst._ , vol. 24, no. 3, pp. 2874–2887, 2022.
  * [19] T. Fernando, S. Denman, S. Sridharan, and C. Fookes, “Learning temporal strategic relationships using generative adversarial imitation learning,” _arXiv preprint arXiv:1805.04969_ , 2018.
  * [20] A. Sharma, M. Sharma, N. Rhinehart, and K. M. Kitani, “Directed-info gail: Learning hierarchical policies from unsegmented demonstrations using directed information,” _arXiv preprint arXiv:1810.01266_ , 2018.
  * [21] G. Lee, D. Kim, W. Oh, K. Lee, and S. Oh, “Mixgail: Autonomous driving using demonstrations with mixed qualities,” in _IEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS)_. IEEE, 2020, pp. 5425–5430.
  * [22] Y. Song, H. Lin, E. Kaufmann, P. Dürr, and D. Scaramuzza, “Autonomous overtaking in gran turismo sport using curriculum reinforcement learning,” in _Int. Conf. Robot. Autom. (ICRA)_. IEEE, 2021, pp. 9403–9409.
  * [23] V. Bajaj, G. Sharon, and P. Stone, “Task phasing: Automated curriculum learning from demonstrations,” in _Int. Conf. Automated Planning Scheduling_ , vol. 33, no. 1, 2023, pp. 542–550.
  * [24] Z. Xue, Z. Peng, Q. Li, Z. Liu, and B. Zhou, “Guarded policy optimization with imperfect online demonstrations,” in _Int. Conf. Learning Representations_ , 2022.
  * [25] X.-H. Liu, F. Xu, X. Zhang, T. Liu, S. Jiang, R. Chen, Z. Zhang, and Y. Yu, “How to guide your learner: Imitation learning with active adaptive expert involvement,” in _Pro. Int. Conf. Autonomous Agents Multiagent Syst._ , 2023, pp. 1276–1284.
  * [26] S. Levine and V. Koltun, “Guided policy search,” in _Pro. Int. Conf. Machine Learning_ , ser. Proceedings of Machine Learning Research, S. Dasgupta and D. McAllester, Eds., vol. 28, no. 3. Atlanta, Georgia, USA: PMLR, 17–19 Jun 2013, pp. 1–9. [Online]. Available: https://proceedings.mlr.press/v28/levine13.html
  * [27] R. Zhang, J. Hou, G. Chen, Z. Li, J. Chen, and A. Knoll, “Residual policy learning facilitates efficient model-free autonomous racing,” _IEEE Robot. Autom. Letters_ , vol. 7, no. 4, pp. 11 625–11 632, 2022.
  * [28] T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, M. Loskyll, J. A. Ojea, E. Solowjow, and S. Levine, “Residual reinforcement learning for robot control,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 6023–6029.
  * [29] K. Rana, M. Xu, B. Tidd, M. Milford, and N. Sünderhauf, “Residual skill policies: Learning an adaptable skill-based action space for reinforcement learning for robotics,” in _Conf. Robot Learning_. PMLR, 2023, pp. 2095–2104.
  * [30] J. Won, D. Gopinath, and J. Hodgins, “Physics-based character controllers using conditional vaes,” _Trans. Graphics (TOG)_ , vol. 41, no. 4, pp. 1–12, 2022.
  * [31] C. Gao, C. Wu, M. Cao, R. Kong, Z. Zhang, and Y. Yu, “Act: Empowering decision transformer with dynamic programming via advantage conditioning,” _arXiv preprint arXiv:2309.05915_ , 2023.
  * [32] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martín, “What matters in learning from offline human demonstrations for robot manipulation,” _arXiv preprint arXiv:2108.03298_ , 2021.
  * [33] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, _et al._ , “Language models are few-shot learners,” _Adv. Neural Inform. Processing Syst._ , vol. 33, pp. 1877–1901, 2020.
  * [34] R. Trumpp, D. Hoornaert, and M. Caccamo, “Residual policy learning for vehicle control of autonomous racing cars,” _arXiv preprint arXiv:2302.07035_ , 2023.
  * [35] I. Kostrikov, K. K. Agrawal, D. Dwibedi, S. Levine, and J. Tompson, “Discriminator-actor-critic: Addressing sample inefficiency and reward bias in adversarial imitation learning,” _arXiv preprint arXiv:1809.02925_ , 2018.
  * [36] F. Fuchs, Y. Song, E. Kaufmann, D. Scaramuzza, and P. Dürr, “Super-human performance in gran turismo sport using deep reinforcement learning,” _Robot. Autom. Letters_ , vol. 6, no. 3, pp. 4257–4264, 2021.
  * [37] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in _Int. Conf. Machine Learning_. PMLR, 2018, pp. 1861–1870.