# Floquet band engineering with Bloch oscillations

Xi Liu  NUS Graduate School - Integrative Sciences and Engineering Programme
(ISEP), National University of Singapore, Singapore 119077, Singapore  Senmao
Tan  Department of Physics, National University of Singapore, Singapore
117551, Singapore  Qing-hai Wang  Department of Physics, National University
of Singapore, Singapore 117551, Singapore  Longwen Zhou<EMAIL_ADDRESS>College of Physics and Optoelectronic Engineering, Ocean University of China,
Qingdao 266100, China  Jiangbin Gong<EMAIL_ADDRESS>Department of Physics,
National University of Singapore, Singapore 117551, Singapore  Center for
Quantum Technologies, National University of Singapore, Singapore 117543,
Singapore

###### Abstract

This work provides a convenient and powerful means towards the engineering of
Floquet bands via Bloch oscillations, by adding a tilted linear potential to
periodically driven lattice systems. The added linear field not only restricts
the spreading of a time-evolving wavepacket but also, depending on the ratio
between the Bloch oscillation frequency and the modulation frequency of the
periodic driving, dramatically modifies the band profile and topology.
Specifically, we consider a driven Aubry-André-Harper model as a working
example, in the presence of a linear field. Almost flat Floquet bands or
Floquet bands with large Chern numbers due to the interplay between the
periodic driving and Bloch oscillations can be obtained, with the band
structure and topology extensively tunable by adjusting the ratio of two
competing frequencies. To confirm our finding, we further execute the Thouless
pumping of one and two interacting bosons in such a lattice system and
establish its connection with the topological properties of single- and two-
particle Floquet bands.

††preprint: APS/123-QED

##  I Introduction

Topological Thouless pumping describes the quantized adiabatic transport of
particles in a one-dimensional (1D) lattice, which is induced by the slow and
time-periodic variation of the lattice potential Thouless (1983). It reveals
the deep connection among geometric phase, band topology and quantum adiabatic
process Thouless  _et al._ (1982); Niu and Thouless (1984); King-Smith and
Vanderbilt (1993); Xiao  _et al._ (2010); Gritsev and Polkovnikov (2012). The
observations of Thouless pumping in setups like cold atoms and photonics not
only demonstrate the topology of integer quantum Hall effect from a dynamical
perspective, but also provides a powerful tool for the detection of
topological invariants through quantum dynamics Wang  _et al._ (2013a); Lohse
_et al._ (2016); Nakajima  _et al._ (2016); Jürgensen  _et al._ (2021); You
_et al._ (2022).

In the conventional Thouless pump, the applied adiabatic driving field plays
the role of a second synthetic dimension Xiao  _et al._ (2010). To guarantee
the pumping quantization, the initial state of the system needs to be a
Wannier state or a uniformly filled Bloch band along the physical dimension,
which can be rather challenging to prepare in experiments Lohse  _et al._
(2016); Nakajima  _et al._ (2016). Recently, Bloch oscillations induced by a
linearly tilted potential were found to be able to facilitate the initial
state preparation in Thouless pump Ke  _et al._ (2020). Specially, the Bloch
oscillations assist in the uniform sampling of eigenstates at all quasimomenta
in the first Brillouin zone, so that nearly perfect pumping quantization can
be achieved for an arbitrary initial state selected in a band of interest.
Adding more interest to studies of Bloch oscillations versus topological
phases, Bloch oscillations have also been used in experiments to probe the
band topology Duca  _et al._ (2015); Aidelsburger  _et al._ (2014); Atala  _et
al._ (2013). For systems with many-body interactions, the introduction of a
linear potential is also of great interest, because a time-evolving state can
be localized by such a linear potential due to the so-called Stark many-body
localization Schulz  _et al._ (2019).

Beyond the adiabatic limit, time-periodic driving fields were found to induce
rich Floquet topological phases Oka and Aoki (2009); Lindner  _et al._ (2011);
Ho and Gong (2012); Cayssol  _et al._ (2013); Grushin  _et al._ (2014); Titum
_et al._ (2015); Leykam  _et al._ (2016); Zhou  _et al._ (2016); Roy and
Harper (2017); Bomantara and Gong (2018); Rodriguez-Vega and Seradjeh (2018);
Yap  _et al._ (2018); Zhou and Gong (2018a, b); Yang  _et al._ (2019);
Bomantara  _et al._ (2019); Rudner and Lindner (2020); Afzal  _et al._ (2020);
Wintersperger  _et al._ (2020); McIver  _et al._ (2020); Tan  _et al._ (2020);
Zhou and Du (2020); Bomantara and Gong (2020); Zhu  _et al._ (2021a, b); Chen
_et al._ (2022). An extension of the Thouless pump to Floquet systems was
introduced as a powerful dynamical framework to probe the large Chern number
phases of Floquet topological insulators Ho and Gong (2012); Zhou  _et al._
(2014); Xiong  _et al._ (2016). However, the Floquet-Thouless pump defined in
Ref. Zhou  _et al._ (2014) requires the system to be prepared in a localized
Wannier state or as a uniformly populated Floquet-Bloch band, which is even
more challenging to achieve in experiments due to the lack of a Fermi surface
in periodically driven systems Wang  _et al._ (2015); Raghava  _et al._
(2017); Ma  _et al._ (2018). One may then consider adding a linear potential
to a Floquet system, and await the resulting Bloch oscillations to assist the
initial state preparation in a Floquet Chern band and thus yielding the
quantized Thouless pumping. The actual situation is, however, more subtle and
complicated. Indeed, in Floquet systems, once a linear field is introduced,
the resulting effective Hamiltonian of the system associated with one whole
period of driving is NOT the original effective Hamiltonian plus the linear
field under considetation. Qualitatively, this is because of the interplay
between the linear field and the microscopic motion within one driving period.
Specifically, let $H[\theta_{1}(t)]+F=H[\theta_{1}(t+2\pi/\Omega)]+F$ be the
time-periodic Hamiltonian of a tight-binding lattice under a linearly tilted
potential $F$, where $T_{1}=2\pi/\Omega$ is the Floquet driving period and
$\Omega$ is the driving frequency. By transforming $H[\theta_{1}(t)]+F$ into a
rotating frame, the linear potential $F$ can be replaced by an oscillating
phase factor $\theta_{2}(t)=\omega_{F}t$ in the hopping amplitude of the
lattice, where $T_{2}=2\pi/\omega_{F}$ is the oscillation period and
$\omega_{F}$ is the field strength of the linear potential. The rotated
Hamiltonian $H_{\rm r}[\theta_{1}(t),\theta_{2}(t)]$ now recovers the
translational symmetry along the physical dimension, but as a cost carries two
separate time-dependencies through the parameters $\theta_{1}(t)$ and
$\theta_{2}(t)$. If the ratio between the two driving frequencies
$\Omega/\omega_{F}$ is irrational, the rotated Hamiltonian can no longer
describe a Floquet system. When the frequency ratio
$\Omega/\omega_{F}\in{\mathbb{Q}}$, $H_{\rm r}[\theta_{1}(t),\theta_{2}(t)]$
could still describe a Floquet model, but with a new driving period $T$ that
is equal to the least common multiple of $T_{1}$ and $T_{2}$. Nevertheless,
the two driving fields may couple and interplay strongly in this case,
yielding Floquet topological bands and Thouless pumping that are totally
different from those that one could expect in the system described by
$H[\theta_{1}(t)]$. Therefore, the impacts of a linearly tilted potential (and
its resulting Bloch oscillations) on the topology, initial state preparation
and adiabatic transport in Floquet systems are highly nontrivial and deserve
to be explored in detail.

In this work, we uncover how the Bloch oscillations induced by a linear
potential explicitly modify the topology of quasienergy bands and control the
Thouless pumping in a Floquet system. We find that the added linear field
could not only restrict the spreading of a time-evolving wavepacket, but also
dramatically modifies the profile and topology of Floquet bands when the
Floquet driving frequency and linear field strength are set at appropriate
rational ratios. Focusing on a driven Aubry-André-Harper model in the presence
of a linear field as a working example, we find almost flat Floquet bands or
Floquet bands with large Chern numbers due to the interplay between the
periodic driving and Bloch oscillations. The band structures and topology are
further found to be highly tunable by adjusting the ratio of two competing
frequencies. To confirm our findings, we further execute the Thouless pumpings
of one and two interacting bosons in the system, and establish their
connections with the topological properties of single- and two-particle
Floquet quasienergy bands.

The rest of the manuscript is structured as follows. In Sec. II, we introduce
a theoretical framework to describe the band structure and dynamics of bosons
in a 1D superlattice subject to time-periodic driving fields, particle-
particle interactions and an onsite linear potential. In Secs. III and IV, we
apply our theory to reveal the Floquet band topology and quantized Thouless
pumping in one- and two-particle systems with an emphasis on the role played
by the linear potential and the resulting Bloch oscillations. In Sec. V, we
summarize our results and discuss potential future directions. Some further
calculation details and results are presented in the Appendices A–C.

##  II Theory

Here we introduce the system that we are going to explore and outline the
theoretical framework that will be used to study its spectral, topological and
transport features.

###  II.1 Hamiltonian and pumping dynamics

We start with a harmonically driven AAH model Zhou  _et al._ (2014) plus a
linear onsite potential and an interaction term. The resulting Hamiltonian
reads

$\displaystyle H$
$\displaystyle=\sum_{j}\left[\frac{J}{2}(\hat{a}_{j}^{\dagger}\hat{a}_{j+1}+{\rm
H.c.})+V\cos(\lambda j-\beta)\cos(\Omega t)\hat{n}_{j}\right]$
$\displaystyle+\sum_{j}\frac{U}{2}\hat{n}_{j}(\hat{n}_{j}-1)+\sum_{j}\omega_{F}j\hat{n}_{j}.$
(1)

Here $\hat{a}_{j}^{\dagger}$ ($\hat{a}_{j}$) creates (annihilates) a boson on
the sublattice site $j$. $J$ is the hopping amplitude. $V$ is the driving
strength. $\Omega$ is the driving frequency. $\omega_{F}$ is the strength of
linear potential. $\lambda=2\pi p/q$, with $p$ and $q$ being coprime integers.
We choose $p/q=1/3$ in this work, so that the spectrum of $H$ possesses three
bands in the absence of the linear potential and many-body interactions.
$\beta\in[0,2\pi]$ is a phase shift of the superlattice potential, which will
be tuned adiabatically in the Floquet Thouless pumping. More precisely, we
consider the evolution over a time duration $MT$ with $M\gg 1$ and $T$ the
driving period in the pumping dynamics. Within the $m$th driving period
($m=1,2,...,M-1,M$), we let $\beta=2\pi(m-1)/M$ and evolve the state using the
Hamiltonian in Eq. (1) from $t=(m-1)T$ to $t=mT$. The pumping is revealed by
the expectation value of unit-cell position operator
$\hat{x}=\frac{1}{qN}\sum_{j}j\hat{n}_{j}$ at the end of each driving period,
with $N$ the particle number.

The linear potential in Eq. (1) breaks the translational symmetry of $H$.
Instead of directly studying the Hamiltonian in the lab frame, we apply a
rotation to eliminate the linear potential term, so that the system can be
made translationally invariant in real space Eckardt (2017). The rotation is
defined as

$R=\sum_{\ket{\mathbf{n}}}e^{i\omega_{F}t\sum_{j}j{n}_{j}}\ket{\mathbf{n}}\bra{\mathbf{n}},$
(2)

following which we get the Hamiltonian in the rotating frame, i.e.,

$\displaystyle H_{r}$
$\displaystyle=\sum_{j}\frac{J}{2}(e^{-i\omega_{F}t}\hat{a}_{j}^{\dagger}\hat{a}_{j+1}+{\rm
H.c.})$ $\displaystyle+\sum_{j}\left[V\cos(\lambda j-\beta)\cos(\Omega
t)\hat{n}_{j}+\frac{U}{2}\hat{n}_{j}(\hat{n}_{j}-1)\right].$ (3)

As shown in Eq. (3), $H_{r}$ is subject to drivings with two frequencies
$\omega_{F}$ and $\Omega$. In order to apply the Floquet formalism, we assume
that $\omega_{F}$ and $\Omega$ are commensurate, i.e., letting their ratio be
a rational number,

$\omega_{F}/\Omega=a/b,$ (4)

where $a,b$ are coprime integers. Let $T_{1}={2\pi}/\Omega$ and
$T_{2}=2\pi/\omega_{F}$. The smallest common period of the two driving fields
is then

$T=aT_{2}=bT_{1}.$ (5)

Let $\mathcal{T}$ be the time ordering operator, Floquet operator of the
system at each fixed $\beta$ reads

$U(\beta)=\mathcal{T}\exp\left[-i\int_{0}^{T}dtH_{r}(t,\beta)\right].$ (6)

$U(\beta)$ governs the dynamics of the system over each common driving period
of the two-color field. In the next subsection, we discuss how to use the
translational invariance of the system over a unit cell to block diagonalize
$U(\beta)$ in such a system under the periodic boundary condition (PBC).

###  II.2 Multiparticle Floquet-Bloch bands

We now generalize the approaches of Refs. Zhou  _et al._ (2014); Ke  _et al._
(2017) to construct multiparticle Floquet-Bloch bands. We consider an
$N$-particle system with $L$ unit cells. For each cell, we focus on the case
with three sublattices. The situations with other numbers of sublattices can
be treated in a similar fashion. To simplify the demonstration and without
loss of generality, we require $N$ and $L$ to be coprime for $N>1$. Since
$\lambda=2\pi/3$, the onsite potential term in $H_{r}$ is a periodic function
of period $3$. Thus there are $3$ sublattices in each unit cell. We next
define a co-translation operator $\widehat{T}_{3}$ that acts on Fock states as

$\displaystyle\widehat{T}_{3}\ket{n_{1},n_{2},\ldots,n_{3L}}$ $\displaystyle=$
$\displaystyle\ket{n_{3L-2},n_{3L-1},n_{3L},n_{1},\dots,n_{3(L-1)}},$ (7)

i.e., $\widehat{T}_{3}$ translates the center-of-mass of particles over three
sublattices to the left under the PBC. One can readily verify that
$\widehat{T}_{3}$ reduces to the familiar translation operator in the one-
particle case. For the rotated Hamiltonian in Eq. (3) and under the PBC, we
have

$\widehat{T}_{3}H_{r}(t,\beta)\widehat{T}_{3}^{\dagger}=H_{r}(t,\beta).$ (8)

For two arbitrary Fock states $\ket{\mathbf{n}_{1}}$ and
$\ket{\mathbf{n}_{2}}$, there may not exist a $j\in\mathbb{Z}$ such that
$\ket{\mathbf{n}_{2}}=\widehat{T}_{3}^{j}\ket{\mathbf{n}_{1}}$. We define a
set $S$ comprising of the so-called seed states Ke  _et al._ (2017). The
elements of $S$ include all the Fock states such that any two of them cannot
be transformed into each other by $\widehat{T}_{3}^{j}$ for any
$j\in\mathbb{Z}$. Note that the set $S$ is in fact an equivalence relation.
For example, for one-particle case, with a particle on either sublattice of
the first unit cell, we get one $S$ comprised of $3$ seed states as
$S=\\{\ket{1},\ket{2},\ket{3}\\}$. For a general $N$, the dimension of Fock
space is $D=\binom{3L+N-1}{N}$. Under the assumption that $L$ and $N$ are
coprime, each seed state will go back to itself after being acted by
$\widehat{T}_{3}$ over $L$ times. If $L$ and $N$ are not coprime, the number
of actions of $\widehat{T}_{3}$ to recover a seed state is not unique, and we
will not consider this case. With $L$ and $N$ being coprime, the number of
seed states, i.e., the cardinality of set $S$ is $D_{S}=D/L$. Because of Eq.
(8), $\widehat{T}_{3}$ also commutes with $U(\beta)$ and they have common
eigenstates $\ket{\psi}$, i.e.,

$\displaystyle U(\beta)\ket{\psi}$ $\displaystyle=e^{i\epsilon}\ket{\psi},$
(9) $\displaystyle\widehat{T}_{3}^{-1}\ket{\psi}$
$\displaystyle=e^{i\phi}\ket{\psi}.$ (10)

After some algebra (see Appendix A), we find for any $\ket{\mathbf{m}}\in S$
that

$e^{i\epsilon}\braket{\mathbf{m}}{\psi}=\sum_{\ket{\mathbf{n}}\in
S}\bra{\mathbf{m}}U(\beta)\sum_{j=0}^{L-1}e^{ij\phi}\widehat{T}_{3}^{j}\ket{\mathbf{n}}\braket{\mathbf{n}}{\psi}.$
(11)

Therefore, we can reduce the full matrix $U(\beta)$ of dimension $D\times D$
to a reduced Floquet operator $\widetilde{U}(\beta,\phi)$ of dimension
$D_{S}\times D_{S}$, whose matrix elements are

$\bra{\mathbf{m}}U(\beta)\sum_{j=0}^{L-1}e^{ij\phi}\widehat{T}_{3}^{j}\ket{\mathbf{n}},\quad\ket{\mathbf{m}},\ket{\mathbf{n}}\in
S.$ (12)

The Floquet states of $\widetilde{U}(\beta,\phi)$ are given by

$\ket{\widetilde{\psi}(\beta,\phi)}=[\braket{\mathbf{m}}{\psi}]_{\ket{\mathbf{m}}\in
S}^{\top}.$ (13)

Here $\top$ means transpose, so that $\ket{\widetilde{\psi}(\beta,\phi)}$ is a
column vector. Note that
$\ket{\phi,\mathbf{n}}=\frac{1}{\sqrt{L}}\sum_{j=0}^{L-1}e^{ij\phi}\widehat{T}_{3}^{j}\ket{\mathbf{n}}$
is just a Bloch eigenstate, and
$\ket{\widetilde{\psi}(\beta,\phi)}=[\braket{\mathbf{m}}{\psi}]_{\ket{\mathbf{m}}\in
S}^{\top}$ is the eigenvector projected over seed states. By solving this
eigenvalue problem, we get $D_{S}$ quasienergy bands. Note that there is a
distinction between one-particle and multi-particle cases. For the one-
particle case, the number of bands is always three, i.e., $D_{S}=3$,
regardless of the value of $L$. We can use a splitting operator scheme to
calculate the $3\times 3$ reduced Floquet operator $\widetilde{U}(\beta,\phi)$
for one-particle case, which is much faster than evaluating
$\widetilde{U}(\beta,\phi)$ for the two-particle case (see the Appendix A of
Ref. Wang  _et al._ (2013b) for more derivation details).

For the two-particle case, the value of $D_{S}$ changes with $L$. As long as
$D_{S}\gg 1$, the exact number of bands $D_{S}$ does not generate a difference
in the overall band structure. In the subsequent calculation, we take $L=21$.
The number of bands is thus $D_{S}=96$ for $N=2$. We can then exponentiate the
full $D\times D$ Hamiltonian, and multiply these exponentials according to
their time ordering. For the one- or two-particle case, the exponential of
Hamiltonian is manageable. If we have more particles, the computation will be
more challenging as the dimension of the Hamiltonian increases exponentially
and quickly becomes computationally prohibitive for exact diagonalization
treatments. Note that our scheme of evaluating the reduced Floquet operator in
Eq. (12) is equivalent to the Eq. (3) in Ref. Ke  _et al._ (2017).

Meanwhile, our scheme has one computational advantage. That is, in order to
get each matrix element of $\widetilde{U}$, we do not need to compute matrix-
vector products. Instead, we only need to compute the inner product between
vectors. Since $\bra{\mathbf{m}}U(\beta)$ is a row of $U(\beta)$, we only need
to select this row to perform the inner product with
$\sqrt{L}\ket{\phi,\mathbf{n}}$. Once we have obtained an eigenvector
$\ket{\widetilde{\psi}(\beta,\phi)}$ over seed states, we can recover the full
eigenstate through the relation

$\displaystyle\ket{\psi(\beta,\phi)}$ $\displaystyle=\sum_{\ket{\mathbf{n}}\in
S}\sum_{j=0}^{L-1}\widehat{T}_{3}^{j}\ket{\mathbf{n}}\bra{\mathbf{n}}\widehat{T}_{3}^{-j}\ket{\psi(\beta,\phi)}$
$\displaystyle=\sum_{\ket{\mathbf{n}}\in
S}\sum_{j=0}^{L-1}\widetilde{\psi}_{\mathbf{n}}e^{ij\phi}\widehat{T}_{3}^{j}\ket{\mathbf{n}}.$
(14)

Our pumping scheme uses Wannier or Gaussian states as initial states. Their
(unnormalized) explicit expressions are given by (i) Wannier state:

$\displaystyle\ket{W(R_{0},\beta)}$ $\displaystyle=\sum_{\phi}e^{-i\phi
R_{0}}\ket{\psi(\beta,\phi)}$
$\displaystyle=\sum_{j=0}^{L-1}\sum_{\ket{\mathbf{n}}\in
S}\sum_{\phi}\widetilde{\psi}_{\mathbf{n}}e^{i(j-R_{0})\phi}\widehat{T}_{3}^{j}\ket{\mathbf{n}},$
(15)

and (ii) Gaussian state:

$\displaystyle\ket{G(R_{0},\beta)}$
$\displaystyle=\sum_{j=0}^{L-1}\sum_{\ket{\mathbf{n}}\in
S}\sum_{\phi}\widetilde{\psi}_{\mathbf{n}}e^{i\phi(j-R_{0})-\frac{1}{4\sigma^{2}}\phi^{2}}\widehat{T}_{3}^{j}\ket{\mathbf{n}}.$
(16)

In the study of adiabatic pumping, we impose PBC for the rotated Hamiltonian
$H_{r}$. We also obtain the Floquet bands with respect to the phase shift
$\beta$ and center-of-mass quasimomentum $\phi\in[0,2\pi]$ by diagonalizing
the Floquet operator under the PBC.

##  III Continuously driven AAH model with one boson

Figure 1: Floquet spectrum under the PBC, the pumping of Wannier and Gaussian
initial states in real space for $T_{1}=2$, $a=3$ and $b=1$. (a) The three
quasienergy bands with Chern numbers -2, 4, -2. (b) Pumping of Wannier states
prepared in the three Floquet bands over an adiabatic cycle ($t=2000T$). (c)
Pumping of Gaussian states initialized in the three Floquet bands over an
adiabatic cycle ($t=2000T$). For each Wannier or Gaussian initial state, the
shift of wavepacket center over an adiabatic cycle yields the Chern number of
the corresponding Floquet band.

We first demonstrate the Floquet band engineering in the single-particle case.
In this case, we find topological transitions induced by changing the
commensurate ratio $\omega_{F}/\Omega=a/b$ between the two driving
frequencies. Moreover, the change of linear potential strength $\omega_{F}$
leads to new topological phases with large Chern numbers and many chiral edge
states. We further employ the quantized adiabatic pumping of both Wannier and
Gaussian states to reveal the rich Floquet topological properties induced by
Bloch oscillations in the system from a dynamical perspective. Throughout this
work, we work in dimensionless units and set the system parameters $J=V=2.5$.

Figure 2: Floquet spectrum under the PBC, Wannier state pumping and Floquet
spectrum under the OBC for $(T_{1}=4,\omega_{F}=0)$ in (a)–(c) and
$T_{1}/T_{2}=(2/5,5/2)$ in (d)–(i).

We first present the Floquet bands under the PBC and the adiabatic pumping of
Wannier and Gaussian states $\ket{W(R_{0},0)}$ and $\ket{G(R_{0},0)}$ in Figs.
1 and 2, where we take $T_{1}=2$, $a=3$ and $b=1$. We observe that the
quasienergy bands are flat along the direction of quasimomentum $\phi$, as
shown in Fig. 1(a) and Fig. 2(g). This holds true in other cases with
$T_{1}/T_{2}>1$. The flat dispersion is originated from Bloch oscillations
induced by the linear potential. It could lead to the uniform sampling of
initial states in momentum space, which is essential for the realization of
quantized pumping for an arbitrary single-band state. Regardless of the
initial momentum distribution, each state is linearly swept in the momentum
space multiple times in a single Floquet period when $T_{1}/T_{2}>1$, as shown
in the Appendix B (see Fig. 5), leading to the flat band in the
$\phi$-direction. The easiness of the flat-band generation is one of the
benefits of employing the tilted potential, as it allows us to perform the
Thouless pumping with arbitrary single-band initial states. To demonstrate the
availability of quantized pumping with an arbitrary single-band initial state,
we use the Gaussian state as an example. In Figs. 1(b) and 1(c), we show the
pumping of Wannier and Gaussian states prepared in the three Floquet bands
over an adiabatic cycle consisting of $2000$ driving periods. For each case,
we observe that the pumping is quantized and the net drift of wavepacket
center coincides with the Chern number of the corresponding Floquet band.
Importantly, the quantized pumping of Gaussian initial state is made possible
due to the flat quasienergy band along $\phi$-direction. Compared with the
Wannier state, a Gaussian wavepacket is easier to prepare in experiments.
Therefore, the linear potential provides us with a flexible route to achieve
Thouless pumping and measure Chern numbers of Floquet bands in quantum
simulators like ultracold atoms in optical lattices.

By changing the strength of the linear potential, the band topology of the
system is modified. We consider a group of spectrum and pumping results with
other frequency ratios including $\omega_{F}=0$, $T_{1}/T_{2}=2/5$ and $5/2$.
The Floquet spectrum under the PBC without linear potential is shown in Fig.
2(a) Zhou  _et al._ (2014). Here, we observe three dispersive bands with Chern
numbers $4$, $-8$, $4$. In Fig. 2(b), the pumping of Wannier states indeed
gives the Chern numbers of different Floquet bands. In Figs. 2(d) and 2(e), we
turn on the linear potential $\omega_{F}$ such that $T_{1}/T_{2}=2/5$.
Interestingly, we see that the curvature of Floquet bands changes dramatically
in response to the linear potential, and their Chern numbers could reach
values as large as $-20$, $40$, $-20$. Therefore, we could get topological
phases with huge Chern numbers by simply changing the value of linear
potential strength $\omega_{F}$. Physically, the Floquet bands with large
Chern numbers are originated from the interplay between the Bloch oscillations
induced by linear potential and the external periodic driving field. In Fig.
2(g), we take $T_{1}/T_{2}$ as the inverse of $2/5$. Although the Floquet
bands in this case have the same Chern numbers as those in Fig. 2(a), their
local band curvatures are totally different. Fig. 2(g) has flat band along the
$\phi$-direction, while along $\phi$ the band in Fig. 2(a) is curved. The
pumping results of Wannier states in Fig. 2(h) are consistent with the Chern
numbers of the corresponding Floquet bands. We present three more groups of
Floquet bands and pumping results by adjusting the ratio of
$\omega_{F}/\Omega$ in Fig. 6. They indeed show that we can generate rich
Floquet topological phases with different frequency ratios
$\omega_{F}/\Omega$. For completeness, we show the Floquet spectrum of the
system under the OBC in Figs. 2(c), 2(f) and 2(i), and observe that the band
Chern numbers correctly predict the number of chiral edge modes traversing the
bulk gap at each edge.

Putting together, we conclude that the competition between the periodic
driving $e^{\pm i\omega_{F}t}$ in the hopping terms (due to the linear
potential) and the harmonic driving term $\cos(\Omega t)$ (in the onsite
superlattice potential) determines the spectral topology and also has a clear
impact on the wavepacket dynamics. The interplay between the two driving
fields further induces rich topological structures in Floquet bands, leading
to quantized adiabatic transport of wavepackets over a long spatial range in
the lattice due to the large band Chern numbers. The unique feature of this
system is that with the driving frequency $\Omega$ fixed, by applying a linear
field $\sum_{j}\omega_{F}j\hat{n}_{j}$ that breaks the translational
invariance of the lattice, the motion of the wavepacket can be strongly
affected/tuned by Bloch oscillations and exhibits highly nontrivial transport
signatures.

Some necessary discussions are in order. First, although the drift in the
rotated frame of reference is the same as in the lab frame, there is also one
subtlety. That is, working with a rotated Hamiltonian via the aforementioned
gauge transformation with translational invariance is valid only for states
far from the boundary. For this reason, in actual simulations done in real
space, we take a sufficiently long lattice and make sure that the wavepacket
never reaches the boundary of the system. The translational invariance of the
rotated Hamiltonian thus becomes valid, yielding the Bloch momentum $\phi$ to
block diagonalize the Floquet operator and to compute the spectrum. Second,
although our work and Ref. Ke  _et al._ (2020) have both demonstrated
quantized adiabatic pumping under a linear potential, there are notable
differences. In Ref. Ke  _et al._ (2020), the linear potential assists the
uniform sampling in the Brillouin zone, and the quantization of pumping is
independent of the strength of linear potential. By contrast, here the
strength of the linear field is crucial as it controls the Floquet band
topology, leading to rather different pumping results depending on the
frequency of the Bloch oscillations induced by the linear field.

##  IV Continuously driven AAH model with two bosons

Figure 3: Floquet spectrum under the PBC and adiabatic pumping of two
interacting bosons. The driving period $T_{1}=2$ and interaction strength
$U=20$ are chosen for all panels. (a) Two-particle quasienergy spectrum for
$T_{1}/T_{2}=3/2$ along $\phi=0$. (b) Pumping of a Wannier initial state for
$T_{1}/T_{2}=3/2$. (c) Pumping of a Gaussian initial state for
$T_{1}/T_{2}=3/2$. (d) Quasienergy spectrum for $\omega_{F}=0$ along $\phi=0$.
(e) Pumping of two Wannier initial states for $\omega_{F}=0$. Figure 4:
Floquet spectrum under the PBC along $\phi=0$ in (a), (d), (g), (j), Wannier
state pumping in (b), (e), (h), (k) and Floquet spectrum under the OBC in (c),
(f), (i), (l) for interaction strengths $U=2,10$ and driving period ratios
$T_{1}/T_{2}=1/1,1/2$.

In this section, we study the Floquet spectrum and Thouless pumping of two
interacting bosons in our system. We use the python package QuSpin Weinberg
and Bukov (2019) to treat the two-boson case, where exact diagonalization is
applied. With two bosons, we have a total of $D_{S}=96$ bands under the PBC
(still for a unit cell there are three sublattice sites), and it is tedious to
visualize all Floquet bands using three-dimensional plots. Therefore, we focus
on the Floquet bands obtained along a cut at $\phi=0$. The band structures
obtained at other values of $\phi$ do not show noteworthy differences. We
shall see that rich topological structures of Floquet bands, topological phase
transitions and pumping dynamics can be generated by changing the ratio
$T_{1}/T_{2}$ between two driving periods and the interaction strength $U$.
The interplay among the periodic driving, interactions and Bloch oscillations
thus induce nontrivial many-body Floquet topological phases.

The main results of this section are presented in Figs. 3 and 4. In Fig. 3(a),
we show the two-particle Floquet spectrum under the PBC for $T_{1}=2$, $U=20$,
and $T_{1}/T_{2}=3/2$. The highest Floquet band (denoted by band $95$) is
found to be isolated from the rest of spectra. We hence choose to prepare the
Wannier/Gaussian initial states on this band. Here Gaussian states refer to a
Gaussian profile of the translational state of the center of mass of the
bipartite system. Results of our adiabatic pumping using Wannier and Gaussian
states as the initials states are shown in Figs. 3(b) and 3(c). The width of
the Gaussian state in momentum space is set as $\sigma=0.7$. We find that the
drift of wavepacket center over an adiabatic cycle is the same for the
Gaussian and Wannier initial states. They both reproduce the Chern number
$C=-2$ of band $95$. Therefore, we have successfully observed quantized
pumping for two types of initial wavepackets. Of particular interest, the
quantization of the pumping using only a two-particle Gaussian state is made
possible by the linear potential induced Bloch oscillations as in the single-
particle case. This indicates that the stage of initial state preparation in
Thouless pumping can be greatly simplified even in many-body interacting cases
thanks to the Bloch oscillations. As a comparison, we show in Figs. 3(d) and
3(e) the two-particle Floquet bands under the PBC without a linear potential,
the pumping of Wannier states prepared on the lowest band (named band $0$) and
the highest band (named band $95$) without the linear potential. The results
show that the presence of a linear potential leads to quantized pumping at
different integer values. As such, in the case of two interacting bosons,
topological phase transitions can be also induced by turning on a linear
potential $\omega_{F}$.

Next, we present four groups of results in Fig. 4 for two-body Floquet
spectrum under different boundary conditions, in connection with the quantized
two-particle pumping. We set $T_{1}=1$ in all panels of Fig. 4, while choosing
the interaction strengths to be $U=2,10$ and setting the ratios of
$T_{1}/T_{2}$ as $1/1$ and $1/2$. One goal here is to examine the impact of
interaction on the topological pumping. First, comparing the results presented
in Figs. 4(a) and 4(g), we see that the decreasing of the ratio $T_{1}/T_{2}$
from $1/1$ to $1/2$ for $U=2$ makes the lowest band more dispersive.
Meanwhile, the number of groups of Floquet bands is kept to be the same as
seven. On the other hand, by comparing Figs. 4(d) and 4(j), we see that for
$U=10$ the decreasing of ratio $T_{1}/T_{2}$ from $1/1$ to $1/2$ leads to more
apparent differences. Both cases have the highest band (band $95$) as an
isolated band, while three more isolated bands appear at the bottom in Fig.
4(j), as denoted by bands $0$, $1$ and $2$. The change of interaction strength
$U$ also generates significant differences in the spectrum, topological and
transport nature of the system. With the same ratio $T_{1}/T_{2}=1/1$, Figs.
4(a) and 4(d) show the same number of seven groups of bands, but the isolated
band goes from the bottom to the top. We can also observe that Fig. 4(a) is
similar to the image being turned upside down of Fig. 4(d). Likewise, by
comparing the results shown in Figs. 4(g) and 4(j), we see that with the same
ratio $T_{1}/T_{2}=1/2$, the number of isolated bands increases from one to
four by increasing $U$ from $2$ to $10$, and the number of groups of bands
increases from seven to nine. We conclude that not only the change of the
ratio $T_{1}/T_{2}$, but also the change of the interaction strength $U$ will
lead to topological phase transitions in the two-particle system. The Chern
numbers in each case of Figs. 4(a), 4(d), 4(g), and 4(j) are obtained by
calculating Berry curvatures of Floquet bands under the PBC, and are validated
by the quantized two-particle pumpings in Figs. 4(b), 4(e), 4(h), and 4(k).
Therefore, we provide concrete examples of topological phase transitions and
quantized transport in two-particle Floquet systems that are induced by the
interplay among a periodic driving field, interactions and Bloch oscillations.

In Appendix C, we have presented more computational examples. There it is seen
that Floquet bands with very high Chern numbers can be obtained by simply
adjusting the ratio of the Bloch oscillation frequency to that of the periodic
driving. These additional results further strongly indicate that the
introduction of a linear field to periodically driven lattices is a powerful
means towards Floquet-band engineering.

##  V Discussion and summary

Experimentally, our model and the pumping dynamics may be realized in cold
atom systems. A 1D static AAH model can be realized by superimposing two
optical lattices with different lattice constants, which result from standing
waves created by retro-reflected single-frequency laser beams Schreiber  _et
al._ (2015). To realize the continuously driven AAH model, we may periodically
modulate the strength of lattice lasers. The tunable linear field can be
generated by a magnetic field gradient along the lattice direction Geiger  _et
al._ (2018). Different magnetic field gradients lead to different band
topology of the system, thus generating different quantized pumping results.

We have shown that a linear potential applied to a Floquet system can induce
quasienergy bands with large Chern numbers and rich topological phase
transitions. This is verified by calculating the topological invariants as
well as adiabatic Thouless pumping of Wannier states. Moreover, we have shown
that we can use Gaussian initial states on relatively flat bands to achieve
quantized pumping, which is easier to implement in experiments. It is the
Bloch oscillation that facilitates the uniform sweeping of eigenstates for
such flat bands. Certainly, in Floquet systems, there is no longer uniform
sampling of momentum by the Bloch oscillations, as evidenced by the existence
of non-flat bands. In these cases, the linear potential can even induce new
topological phases. This is the main message from this work, and we hence
expect that the introduction of a linear field can be generally useful to
multi-color engineering of Floquet bands Zhang and Gong (2019). For a two-
particle interacting system, not only the ratio of the two driving periods but
also the interaction strength determines the topological phases and hence
changes the result of Floquet Thouless pumping. In future work, we may
consider situations with more bosons or interacting fermions Mu  _et al._
(2019), in connection with many-body stark localization and quantum pre-
thermalization.

###### Acknowledgements.

L.Z. is supported by the National Natural Science Foundation of China (Grant
No. 11905211), the Young Talents Project at Ocean University of China (Grant
No. 861801013196), and the Applied Research Project of Postdoctoral Fellows in
Qingdao (Grant No. 861905040009). J.G. acknowledges support from Singapore
National Research Foundation Grant No. NRF-NRFI2017-04 (WBS No.
A-0004162-00-00). The computational work for this article was fully performed
on resources of the National Supercomputing Centre, Singapore
(https://www.nscc.sg).

##  Appendix A Derivation of the eigenvalue problem

In this section, we give derivation details of Eq. (11) in the main text. For
common eigenstates of the Floquet operator $U(\beta)$ and co-translation
operator $\widehat{T}_{3}^{-1}$, we have

$\displaystyle U(\beta)\ket{\psi}$ $\displaystyle=e^{i\epsilon}\ket{\psi},$
(17) $\displaystyle\widehat{T}_{3}^{-1}\ket{\psi}$
$\displaystyle=e^{i\phi}\ket{\psi}.$ (18)

For the set $S$ of all seed states, the identity operator is

$\mathds{1}=\sum_{\ket{\mathbf{n}}\in
S}\sum_{j=0}^{L-1}\widehat{T}_{3}^{j}\ket{\mathbf{n}}\bra{\mathbf{n}}\widehat{T}_{3}^{-j}.$
(19)

Then we obtain

$\displaystyle e^{i\epsilon}\ket{\psi}$ $\displaystyle=U(\beta)\ket{\psi}$
$\displaystyle=U(\beta)\sum_{\ket{\mathbf{n}}\in
S}\sum_{j=0}^{L-1}\widehat{T}_{3}^{j}\ket{\mathbf{n}}\bra{\mathbf{n}}\widehat{T}_{3}^{-j}\ket{\psi}$
$\displaystyle=\sum_{\ket{\mathbf{n}}\in
S}U(\beta)\sum_{j=0}^{L-1}e^{ij\phi}\widehat{T}_{3}^{j}\ket{\mathbf{n}}\braket{\mathbf{n}}{\psi}.$
(20)

Multiplying $\ket{\mathbf{m}}\in S$ from the left, we get

$e^{i\epsilon}\braket{\mathbf{m}}{\psi}=\sum_{\ket{\mathbf{n}}\in
S}\bra{\mathbf{m}}U(\beta)\sum_{j=0}^{L-1}e^{ij\phi}\widehat{T}_{3}^{j}\ket{\mathbf{n}}\braket{\mathbf{n}}{\psi}.$
(21)

this is Eq. (11) in the main text.

##  Appendix B Evolution of particle density in momentum space

In this appendix, we provide details for the time evolution of the single-
particle density distribution in the momentum space. It allows us to directly
observe the effect of Bloch oscillations in Floquet systems. The density
distribution is given by

$\displaystyle|\psi_{k}|^{2}=|\phi_{0,k}|^{2}+|\phi_{1,k}|^{2}+|\phi_{2,k}|^{2},$
(22)

where

$\displaystyle\phi_{i,k}=\frac{1}{\sqrt{L}}\sum_{j=1}^{L}e^{-ikj}\psi_{j,i}.$
(23)

In numerical calculations, we choose $L=500$ as the number of unit cells. In a
single Floquet period, the momentum is linearly swept through the Brillouin
zone multiple times when $T_{1}/T_{2}>1$, leading to flat quasienergy bands
along the direction of Bloch quasimomentum.

Figure 5: Time evolution of single-particle density distribution in the
momentum space. The chosen parameters are $T=T_{1}=2,T_{2}=0.5,J=V=2.5.$

##  Appendix C More examples for the single-particle case

Figure 6: Floquet bands under the PBC, Wannier state pumping and Floquet bands
under the OBC for the single-particle case with $T_{1}=4$ and
$T_{1}/T_{2}=5/7$ [in (a), (b), (c)], $7/10$ [in (d), (e), (f)], $8/11$ [in
(g), (h), (i)], respectively.

In this appendix, we present $3$ more examples of Floquet quasienergy spectrum
and Thouless pumping in Fig. 6. We can see that by changing the ratio of
$T_{1}/T_{2}$, one can easily obtain topological phases with very high Chern
numbers. It is also observed that the actual Chern numbers obtained can be
rather sensitive to the strength of the applied linear field. This provides us
with more flexibility to control topological phase transitions in Floquet
systems.

## References

  * Thouless (1983) D. J. Thouless, Phys. Rev. B 27, 6083 (1983).
  * Thouless  _et al._ (1982) D. J. Thouless, M. Kohmoto, M. P. Nightingale, and M. den Nijs, Phys. Rev. Lett. 49, 405 (1982).
  * Niu and Thouless (1984) Q. Niu and D. J. Thouless, Journal of Physics A: Mathematical and General 17, 2453 (1984).
  * King-Smith and Vanderbilt (1993) R. D. King-Smith and D. Vanderbilt, Phys. Rev. B 47, 1651 (1993).
  * Xiao  _et al._ (2010) D. Xiao, M.-C. Chang, and Q. Niu, Rev. Mod. Phys. 82, 1959 (2010).
  * Gritsev and Polkovnikov (2012) V. Gritsev and A. Polkovnikov, Proceedings of the National Academy of Sciences 109, 6457 (2012).
  * Wang  _et al._ (2013a) L. Wang, M. Troyer, and X. Dai, Phys. Rev. Lett. 111, 026802 (2013a).
  * Lohse  _et al._ (2016) M. Lohse, C. Schweizer, O. Zilberberg, M. Aidelsburger, and I. Bloch, Nature Physics 12, 350 (2016).
  * Nakajima  _et al._ (2016) S. Nakajima, T. Tomita, S. Taie, T. Ichinose, H. Ozawa, L. Wang, M. Troyer, and Y. Takahashi, Nature Physics 12, 296 (2016).
  * Jürgensen  _et al._ (2021) M. Jürgensen, S. Mukherjee, and M. C. Rechtsman, Nature 596, 63 (2021).
  * You  _et al._ (2022) O. You, S. Liang, B. Xie, W. Gao, W. Ye, J. Zhu, and S. Zhang, Phys. Rev. Lett. 128, 244302 (2022).
  * Ke  _et al._ (2020) Y. Ke, S. Hu, B. Zhu, J. Gong, Y. Kivshar, and C. Lee, Phys. Rev. Research 2, 033143 (2020).
  * Duca  _et al._ (2015) L. Duca, T. Li, M. Reitter, I. Bloch, M. Schleier-Smith, and U. Schneider, Science 347, 288 (2015).
  * Aidelsburger  _et al._ (2014) M. Aidelsburger, M. Lohse, C. Schweizer, M. Atala, J. T. Barreiro, S. Nascimbène, N. R. Cooper, I. Bloch, and N. Goldman, Nature Physics 11, 162 (2014).
  * Atala  _et al._ (2013) M. Atala, M. Aidelsburger, J. T. Barreiro, D. Abanin, T. Kitagawa, E. Demler, and I. Bloch, Nature Physics 9, 795 (2013).
  * Schulz  _et al._ (2019) M. Schulz, C. Hooley, R. Moessner, and F. Pollmann, Physical Review Letters 122 (2019), 10.1103/physrevlett.122.040606.
  * Oka and Aoki (2009) T. Oka and H. Aoki, Phys. Rev. B 79, 081406 (2009).
  * Lindner  _et al._ (2011) N. H. Lindner, G. Refael, and V. Galitski, Nature Physics 7, 490 (2011).
  * Ho and Gong (2012) D. Y. H. Ho and J. Gong, Phys. Rev. Lett. 109, 010601 (2012).
  * Cayssol  _et al._ (2013) J. Cayssol, B. Dóra, F. Simon, and R. Moessner, physica status solidi (RRL) - Rapid Research Letters 7, 101 (2013).
  * Grushin  _et al._ (2014) A. G. Grushin, A. Gómez-León, and T. Neupert, Phys. Rev. Lett. 112, 156801 (2014).
  * Titum  _et al._ (2015) P. Titum, N. H. Lindner, M. C. Rechtsman, and G. Refael, Phys. Rev. Lett. 114, 056801 (2015).
  * Leykam  _et al._ (2016) D. Leykam, M. C. Rechtsman, and Y. D. Chong, Phys. Rev. Lett. 117, 013902 (2016).
  * Zhou  _et al._ (2016) L. Zhou, C. Chen, and J. Gong, Phys. Rev. B 94, 075443 (2016).
  * Roy and Harper (2017) R. Roy and F. Harper, Phys. Rev. B 96, 155118 (2017).
  * Bomantara and Gong (2018) R. W. Bomantara and J. Gong, Phys. Rev. Lett. 120, 230405 (2018).
  * Rodriguez-Vega and Seradjeh (2018) M. Rodriguez-Vega and B. Seradjeh, Phys. Rev. Lett. 121, 036402 (2018).
  * Yap  _et al._ (2018) H. H. Yap, L. Zhou, C. H. Lee, and J. Gong, Phys. Rev. B 97, 165142 (2018).
  * Zhou and Gong (2018a) L. Zhou and J. Gong, Phys. Rev. A 97, 063603 (2018a).
  * Zhou and Gong (2018b) L. Zhou and J. Gong, Phys. Rev. B 97, 245430 (2018b).
  * Yang  _et al._ (2019) K. Yang, L. Zhou, W. Ma, X. Kong, P. Wang, X. Qin, X. Rong, Y. Wang, F. Shi, J. Gong, and J. Du, Phys. Rev. B 100, 085308 (2019).
  * Bomantara  _et al._ (2019) R. W. Bomantara, L. Zhou, J. Pan, and J. Gong, Phys. Rev. B 99, 045441 (2019).
  * Rudner and Lindner (2020) M. S. Rudner and N. H. Lindner, Nature Reviews Physics 2, 229 (2020).
  * Afzal  _et al._ (2020) S. Afzal, T. J. Zimmerling, Y. Ren, D. Perron, and V. Van, Phys. Rev. Lett. 124, 253601 (2020).
  * Wintersperger  _et al._ (2020) K. Wintersperger, C. Braun, F. N. Ünal, A. Eckardt, M. D. Liberto, N. Goldman, I. Bloch, and M. Aidelsburger, Nature Physics 16, 1058 (2020).
  * McIver  _et al._ (2020) J. W. McIver, B. Schulte, F.-U. Stein, T. Matsuyama, G. Jotzu, G. Meier, and A. Cavalleri, Nature Physics 16, 38 (2020).
  * Tan  _et al._ (2020) S. Tan, R. W. Bomantara, and J. Gong, Physical Review A 102, 022608 (2020).
  * Zhou and Du (2020) L. Zhou and Q. Du, Phys. Rev. A 101, 033607 (2020).
  * Bomantara and Gong (2020) R. W. Bomantara and J. Gong, Phys. Rev. B 101, 085401 (2020).
  * Zhu  _et al._ (2021a) W. Zhu, M. Umer, and J. Gong, Phys. Rev. Research 3, L032026 (2021a).
  * Zhu  _et al._ (2021b) W. Zhu, Y. D. Chong, and J. Gong, Phys. Rev. B 104, L020302 (2021b).
  * Chen  _et al._ (2022) C. Chen, X. Ding, J. Qin, J. Wu, Y. He, C.-Y. Lu, L. Li, X.-J. Liu, B. C. Sanders, and J.-W. Pan, Phys. Rev. Lett. 129, 046401 (2022).
  * Zhou  _et al._ (2014) L. Zhou, H. Wang, D. Y. Ho, and J. Gong, The European Physical Journal B 87 (2014), 10.1140/epjb/e2014-50465-9.
  * Xiong  _et al._ (2016) T.-S. Xiong, J. Gong, and J.-H. An, Phys. Rev. B 93, 184306 (2016).
  * Wang  _et al._ (2015) H. Wang, L. Zhou, and J. Gong, Phys. Rev. B 91, 085420 (2015).
  * Raghava  _et al._ (2017) G. N. Raghava, L. Zhou, and J. Gong, The European Physical Journal B 90 (2017), 10.1140/epjb/e2017-80195-3.
  * Ma  _et al._ (2018) W. Ma, L. Zhou, Q. Zhang, M. Li, C. Cheng, J. Geng, X. Rong, F. Shi, J. Gong, and J. Du, Phys. Rev. Lett. 120, 120501 (2018).
  * Eckardt (2017) A. Eckardt, Rev. Mod. Phys. 89, 011004 (2017).
  * Ke  _et al._ (2017) Y. Ke, X. Qin, Y. S. Kivshar, and C. Lee, Phys. Rev. A 95, 063630 (2017).
  * Wang  _et al._ (2013b) H. Wang, D. Y. H. Ho, W. Lawton, J. Wang, and J. Gong, Physical Review E 88 (2013b), 10.1103/physreve.88.052920.
  * Weinberg and Bukov (2019) P. Weinberg and M. Bukov, SciPost Physics 7 (2019), 10.21468/scipostphys.7.2.020.
  * Schreiber  _et al._ (2015) M. Schreiber, S. S. Hodgman, P. Bordia, H. P. Lüschen, M. H. Fischer, R. Vosk, E. Altman, U. Schneider, and I. Bloch, Science 349, 842 (2015).
  * Geiger  _et al._ (2018) Z. A. Geiger, K. M. Fujiwara, K. Singh, R. Senaratne, S. V. Rajagopal, M. Lipatov, T. Shimasaki, R. Driben, V. V. Konotop, T. Meier, _et al._ , Physical review letters 120, 213201 (2018).
  * Zhang and Gong (2019) S. Zhang and J. Gong, Phys. Rev. B 100, 235452 (2019).
  * Mu  _et al._ (2019) S. Mu, D.-J. Zhang, L. Zhou, and J. Gong, Phys. Rev. B 100, 144303 (2019).


# Towards Robust Online Dialogue Response Generation

Leyang Cui♡, Fandong Meng♠, Yijin Liu♠, Jie Zhou♠, Yue Zhang♡◇
♡School of Engineering, Westlake University
♠Pattern Recognition Center, WeChat AI, Tencent
◇Institute of Advanced Technology, Westlake Institute for Advanced Study
{leyangcui, fandongmeng, yijinliu<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Work was done when Leyang Cui was interning at Pattern
Recognition Center, WeChat AI, Tencent.

###### Abstract

Although pre-trained sequence-to-sequence models have achieved great success
in dialogue response generation, chatbots still suffer from generating
inconsistent responses in real-world practice, especially in multi-turn
settings. We argue that this can be caused by a discrepancy between training
and real-world testing. At training time, chatbot generates response with the
golden context, while it has to generate based on the context consisting of
both user utterances and the model predicted utterances during real-world
testing. With the growth of the number of utterances, this discrepancy becomes
more serious in the multi-turn settings. In this paper, we propose a
hierarchical sampling-based method consisting of both utterance-level sampling
and semi-utterance-level sampling, to alleviate the discrepancy, which
implicitly increases the dialogue coherence. We further adopt reinforcement
learning and re-ranking methods to explicitly optimize the dialogue coherence
during training and inference, respectively. Empirical experiments show the
effectiveness of the proposed methods for improving the robustness of chatbots
in real practice.

##  1 Introduction

Sequence-to-sequence neural models Vinyals and Le (2015) serve as a foundation
for dialogue response generation Roller et al. (2020); Zhang et al. (2020a),
where typical models adopt the auto-regressive framework Sutskever et al.
(2014). During training, models are optimized to maximize the token-level
likelihood of the golden response given the golden dialogue history context as
input; during inference, the dialogue response generation model is required to
predict the response token by token based on the golden multi-turn dialogue
context.

(a) Training.  (b) Offline Test.  (c) Online Test.

Figure 1: The illustration of how Blender-bot generates responses in different
settings. The prompt utterance is sampled from MuTual Cui et al. (2020).
Blender-bot uses golden context in both training and offline test settings.
The blue part indicates the discrepancy utterances in the context of real-
world testing (online test). Blender-bot generates an incoherent response in
human-bot conversation (Red utterance in Figure 1(c)).

With advance in large-scale pre-training Zhang et al. (2020b); Roller et al.
(2020); Lewis et al. (2020) and the availability of high-quality
conversational datasets Li et al. (2017); Dinan et al. (2019b), models are
able to generate fluent and informative responses Shum et al. (2018). On the
other hand, despite achieving promising performance on the standard evaluation
metrics (e.g., F-1, BLEU, PPL), dialogue response generation models still
suffer from unsatisfactory user experience in practice Welleck et al. (2020);
Ram et al. (2018). Previous work shows that chatbots generate repetition Li et
al. (2020a) and contradictory responses Nie et al. (2021); Li et al. (2021a).
One possible reason is that current research focuses on the offline evaluation
settings, where the golden context is used as input. However, the golden
context cannot be accessed in online settings. Figure 1(c) shows a human-bot
conversation in practice. The golden context in Figure 1(a) and Figure 1(b) is
replaced with a system-generated context in Figure 1(c). In this real-world
setting, the multi-turn context consists of both previous chatbot generated
utterance ($r$) and human response ($u$), which is inconsistent with the
training settings.

Such utterance-level discrepancy between offline training and online testing
is reminiscent of the exposure bias problem Bengio et al. (2015); Ranzato et
al. (2016). Recent research has made solid strides towards alleviating the
exposure bias problem in various generation tasks, such as image captioning
Bengio et al. (2015), speech recognition Bengio et al. (2015), and neural
machine translation Zhang et al. (2019); Mihaylova and Martins (2019). They
simulate the inference stage by replacing golden target input tokens with the
model predictions during training. Intuitively, it can be applied to dialogue
generation also. However, the unique challenge in multi-turn dialogue response
generation is the existence of both the utterance-level and token-level
discrepancy in a hierarchical manner, which is more severe compared to the
above tasks. Given the golden context, 93.3% of generated utterances are
coherent with the context after 10 turns in our experiments. However, when it
comes to the predicted context, the coherence rate drops to less than 30%
(Figure 2).

To alleviate the inconsistency between training and real-world testing, we
propose both utterance-level and semi-utterance-level sampling-based methods
to improve the performance for the online setting. In particular, we sample
whole utterances with a scheduled probability and use model generated
utterances to replace golden utterances. We schedule our sampling in a
hierarchy way. Utterance-level sampling method generates the utterance based
on the previous context, which simulates the online-testing scene during
training. Semi-utterance-level sampling generates an utterance by using both
the previous context and the first few tokens in the sampled utterance, for
keeping the semantic similarity between the generated utterance and the golden
utterance. To further boost the performance, we adopt reinforcement learning
and re-ranking to directly optimize the dialogue coherence between the context
and the response in the simulated online setting, by consulting an external
natural language inference (NLI) based coherence classifier during training
and inference, respectively.

Figure 2: We fine-tune BART on Wizard Dinan et al. (2019b) and report the
coherence rate against number of utterances on test set. Coherence rate (Eq 6)
measures the percentage of responses is coherence with the corresponding
contexts.

We conduct our experiments on Wizard of Wikipedia Dinan et al. (2019b) and
human-bot conversation. Empirical results show that our hierarchical sampling
approach improves the abilities of dialogue models on generating coherent and
less repetitive responses without introducing external training signals. We
further demonstrate that an external coherence classifier can be used in both
training and inference to help models produce more coherent responses.
Finally, we demonstrate that these methods make chatbots more robust in real-
word testing. We release our code and models at https://anonymous.

##  2 Related Work

#### Alleviating Discrepancy.

To bridge the gap between training and inference in auto-regressive models,
Bengio et al. (2015) first attempted to randomly sample the previous generated
token to replace the ground-truth token during training. Zhang et al. (2019)
extended the work of Bengio et al. (2015) by sampling candidates using beam
search. Mihaylova and Martins (2019) considered scheduled sampling for
transformer-based model. Liu et al. (2021a) and Liu et al. (2021b) further
designed sampling strategy based on the model confidence and decode steps,
respectively. Xu et al. (2021) introduced scheduled sampling in the one-to-
many generation scenario. All these method are designed for mitigating the
token-level exposure bias problem. To our knowledge, we are the first to
improve the utterance-level discrepancy between training and real-world
testing.

#### Dialogue Coherence.

Welleck et al. (2019) modeled dialogue coherence as natural language inference
and released the dialogue NLI dataset based on persona Zhang et al. (2018). Li
et al. (2020b) leveraged NLI as supervision to reduce incoherent and
repetition response via unlikelihood training. Nie et al. (2021) extended
dialogue NLI by releasing a human-written multi-domain dataset. Qin et al.
(2021) further introduced dialogue NLI in task-oriented dialogue system.
Khandelwal (2021) used reinforcement learning to optimize semantic coherence
and consistent flow. Li et al. (2021b) proposed a dynamic flow mechanism to
model the context flow. We use coherence as a measure of online dialogue
quality. In contrast, existing work all consider the offline setting where the
input is a golden history.

##  3 Definition

###  3.1 Task

Given a dialogue context
$\mathbf{U}=\\{\mathbf{u}_{1},\dots,\mathbf{u}_{l-1}\\}$, where
$\mathbf{u}_{i}=\\{\mathbf{x}^{\mathbf{u}_{i}}_{1},\dots,\mathbf{x}^{\mathbf{u}_{i}}_{|\mathbf{u}_{i}|}\\}$
represents the $i$-th utterance. $\mathbf{U}$ can be formed as
$\mathbf{U}=\\{\mathbf{x}_{1},\dots,\mathbf{x}_{T}\\}$ by concatenating all
utterances as a token sequence, where $\mathbf{x}_{i}$ denotes the $i$-th
token in $\mathbf{U}$. The corresponding response can be denoted as
$\mathbf{r}=\mathbf{u}_{l}=\\{y_{1},y_{2},\dots,y_{T^{\prime}}\\}$. Given a
training context-response pair $\\{\mathbf{U},\mathbf{r}\\}$, the probability
$P(\mathbf{r}|\mathbf{U})$ can be computed by:

$p(\mathbf{r}|\mathbf{U})=\prod_{t=1}^{T^{\prime}}p(y_{t}|\mathbf{U},y_{1:t-1})$
(1)

which can be estimated by a sequence-to-sequence neural network (i.e.,
transformers) with parameters $\theta$. Our goal is to learn a dialogue
generation model $P_{\theta}(\mathbf{r}|\mathbf{U})$, which is able to
generate response $\mathbf{r}$ based on the context $\mathbf{U}$.

###  3.2 Model

We adopt a standard Transformer Vaswani et al. (2017) seq2seq model in a
dialogue response generation setting.

The dialogue context $\mathbf{U}$ is first fed into the transformer encoder,
yielding a sequence of hidden representations.

$\mathbf{h}^{enc}=\textsc{Transformer\\_Encoder}(\mathbf{U})$ (2)

At the $t$ th step of the decoder, $\mathbf{h}^{enc}$ and the previous output
tokens $y_{1:t-1}$ are then used as inputs, yielding an output representation

$\mathbf{h}^{dec}_{t}=\textsc{Transformer\\_Decoder}(\mathbf{h}^{enc},y_{1:t-1})$
(3)

The generative probability distribution of $y_{t}$ is given by a linear
projection of the hidden vector $\mathbf{h}^{dec}_{t}$ followed by a softmax
transformation

$p(y_{t}|\mathbf{U},y_{1:t-1})=softmax(\mathbf{W}^{o}\mathbf{h}^{dec}_{t}+\mathbf{b}^{o})$
(4)

where $\mathbf{W}^{o}$ and $\mathbf{b}^{o}$ are trainable parameters.

The standard cross-entropy loss is used to optimize the parameters $\theta$.
Given a training pair $(\mathbf{U},\mathbf{r})$, the objective is to minimize:

$\mathcal{L}_{dialogue}=-\sum_{t=1}^{T^{\prime}}\log
p(y_{t}|\mathbf{U},y_{1:t-1})$ (5)

During inference, model auto-regressive generates the response
$\mathbf{\hat{r}}$ based on the context $\mathbf{U}$.

###  3.3 Evaluation

#### Offline Evaluation.

A conventional practice for evaluating dialogue generation model is formed as
a lexical similarity task. In particular, the dialogue generation model is
first required to generate response $\hat{\mathbf{r}}$ based on the golden
dialogue context $\mathbf{U}$. And then the lexical similarity (i.e., F1,
BLEU) between the golden response $\mathbf{r}$ and the generated response
$\hat{\mathbf{r}}$ is calculated to measure the performance.

#### Online Evaluation.

In real practice, chatbot is used to communicate with human users online. As
an example for the $l$-th turn, the dialogue context consists of both human
utterances and chatbot utterances generated in previous turns, formed as
$\mathbf{\hat{U}}=\\{\mathbf{u}_{1},\hat{\mathbf{r}}_{2},\mathbf{u}_{3},\hat{\mathbf{r}}_{4},\dots,\mathbf{u}_{l-1}\\}$,
where $\mathbf{u}_{i}$ represents the $i$-th user utterances and
$\hat{\mathbf{r}}_{i}$ represents the chatbot prediction based on
$\mathbf{\hat{U}}_{1}^{i-1}$. In this setting, the golden context $\mathbf{U}$
does not exist, because the context has been dynamically generated. An
intuitive method for online evaluation is to employ a human to talk with
chatbot naturally. However this evaluation method is high-cost Li et al.
(2021a) and relative subjective Dinan et al. (2019a), which cannot be adopted
in large-scale evaluation. Following Deriu et al. (2020), we use bot-bot
conversations (self-talk) to simulate human-bot conversation, and conduct a
NLI-based classifier $f_{c}(\mathbf{\hat{U}},\mathbf{\hat{r}})$ to estimate
whether the generated response is in line with the context. In particular,
given a prompt utterance $\mathbf{u}_{1}$, we conduct $K$ turns self-talk
conversations, yielding a list of utterances
$\mathbf{\hat{U}}=\\{\mathbf{u}_{1},\hat{\mathbf{r}}_{2},\hat{\mathbf{r}}_{3},\dots,\hat{\mathbf{r}}_{K}\\}$.
At turn $k\in[1,K]$, the coherence rate $c_{k}$ is calculated by:

$c_{k}=\sum_{i=1}^{D}\frac{\mathbb{1}(f_{c}(\mathbf{\hat{U}}_{1}^{i-1},\mathbf{\hat{r}}_{i})=1)}{D}$
(6)

where $D$ represents the number of instances for evaluation,
$\mathbb{1}(\cdot)$ returns 1 if $\cdot$ is true and 0 otherwise.

##  4 Method

We take sampling-based methods to simulate online consentaneous (Section 4.1),
and introduce a reinforcement learning method and a re-ranking method to
optimize the dialogue coherence explicitly (Section 4.2).

###  4.1 Hierarchical Sampling

The main difference between training and inference in real world practice when
generating $\mathbf{\hat{r}}$ is whether we use the golden context
$\mathbf{U}$ or the predicted context $\mathbf{\hat{U}}$ partly predicted by
the model. We address this by introducing the hierarchical sampling to
optimize dialogue coherence implicitly.

#### Utterance Level Sampling.

Our utterance-level sampling mechanism is shown in Figure 3. Given a golden
context $\mathbf{U}_{1}^{l-1}$, we sample an utterance $\mathbf{u}_{i},\
i\in[1,l-1]$ from geometric distribution $\sim Geo(p)$ (with $p=0.2$ and max
clip $i_{max}=10$), which tends to sample previous utterance to be replaced.
After obtaining the utterance $\mathbf{u}_{i}$, we first ask the model to
predict the response $\mathbf{\hat{r}}_{i}$ based on the previous context
$\mathbf{U}_{1}^{{}^{\prime}i-1}$, and then we use the predicted utterance
$\mathbf{\hat{r}_{i}}$ to replace the golden utterance $\mathbf{u}_{i}$ in the
golden context
$\mathbf{U}_{1}^{l-1}=\\{\mathbf{u}_{1},\dots,\mathbf{u}_{i},\dots,\mathbf{u}_{l-1}\\}$,
yielding the mixed context
$\mathbf{U}_{1}^{{}^{\prime}l-1}=\\{\mathbf{u}_{1},\dots,\mathbf{\hat{r}}_{i},\dots,\mathbf{u}_{l-1}\\}$.
Finally, $\mathbf{U}_{1}^{{}^{\prime}l-1}$ are fed into the encoder.
Accordingly, equation 5 is modified as below:

$\mathcal{L}_{dialogue}=-\sum_{t=1}^{T^{\prime}}\log
p(y_{t}|\mathbf{U}_{1}^{{}^{\prime}l-1},y_{1:t-1})$ (7)

#### Semi-utterance Level Sampling.

Our semi-utterance-level sampling method generates the response based on both
the previous context and the first few tokens in the sampled utterance. In
particular, after obtaining the sampled utterance $\mathbf{u}_{i}$, we further
keep the first $j$ tokens in $\mathbf{u}_{i}$ as additional cues to generate
$\mathbf{\hat{r}}^{\prime}_{i}$. Intuitively, a larger $j$ increase both
semantic-level and lexical-level overlap between the
$\mathbf{\hat{r}}^{\prime}_{i}$ and $\mathbf{u}_{i}$. A smaller $j$ to
simulate more accumulate errors along with the inference steps. The same as
utterance level sampling in Section 4.1, $\mathbf{\hat{r}}^{\prime}_{i}$ is
used to replace $\mathbf{u}_{i}$.

Figure 3: Training with proposed sampling-based methods.

###  4.2 Explicit Coherence Optimization

#### Training.

Inspired by optimizing the generation towards high BLEUYang et al. (2018), we
introduce a reinforcement learning method, which explicitly optimizes the
coherence between the context and the generated response. We fine-tune the
dialogue model $P_{\theta}$ to optimize the reward model $P^{RL}_{\theta}$.

As shown in Figure 4(a), we first ask the model to generate a response
$\mathbf{\hat{r}}$ based on the context $\mathbf{U}$. Then an external
coherence classifier $f_{c}$ is used to justify whether the response is
coherent with the context. We adopt the logits of $f_{c}$ corresponding to the
coherent label as the reward. In particular, the input of $f_{c}$ is a
context-response pair $(\mathbf{U},\mathbf{r})$ and the output is whether the
response is coherent with the context. For training $f_{c}$, we turn context-
response pair $(\mathbf{U},\mathbf{r})$ to [CLS] $\mathbf{U}$ [SEP]
$\mathbf{r}$ [SEP], and feed it into the RoBERTa model. The hidden state of
the [CLS] token is used for MLP followed by a softmax scoring function to
obtain the coherence score. We train $f_{c}$ on DialoguE COntradiction
DEtection (DECODE) Nie et al. (2021), which is a human annotated corpus
labeled with “contradiction (non-coherent)” and “non-contradiction
(coherent)”. The classifier achieves 94.24 on DECODE dev.

Following Ziegler et al. (2019) and Jaques et al. (2020), we additionally
introduce a Kullback–Leibler (KL) divergence term to prevent $P^{RL}_{\theta}$
from drifting too far from $P_{\theta}$ (Figure 4(b)). Formally, given the
context $\mathbf{U}$, we calculate the KL-divergence between two models’
output probabilities

$KL(\mathbf{U})=\sum_{t=1}^{T^{\prime}}\log\frac{p^{RL}_{\theta}(\mathbf{x}_{t}|\mathbf{U},\mathbf{x}_{1:t-1})}{p_{\theta}(\mathbf{x}_{t}|\mathbf{U},\mathbf{x}_{1:t-1})}$
(8)

$KL(\mathbf{U})$ can be considered as a KL-divergence for the language model
task.

Finally, we optimize $P^{RL}_{\theta}$ using Proximal Policy Optimization
(PPO) Schulman et al. (2017) with the clipped reward:

$Reward(\mathbf{U,\mathbf{r}})=f_{c}(\mathbf{U},\mathbf{\hat{r}})-\beta
KL(\mathbf{U})$ (9)

where $\beta$ is a hyper-parameter to control the contribution of the KL term.
Intuitively, we use the classifier to encourage the model to generate coherent
responses, and rely on the KL term to ensure fluency. The inference stage can
be the same as the baseline methods in Section 3.2.

#### Inference with Re-ranking.

Another method to enhance dialogue coherence explicitly is inference with re-
ranking. In particular, we first adopt beam search to produce multiple
candidates responses, and then re-rank the utterances using the coherence
classifier $f_{c}$. At each turn, the candidate with the highest coherence
score is used as the response.

##  5 Experiments

We train our model based on the golden context - response pair on Wizard of
Wikipedia Dinan et al. (2019b), a chit-chat dialogue benchmark. Two annotators
are employed to chat based on an initial topic. The dataset contains 18,430
training dialogues with 1,365 topics.

###  5.1 Metrics

Following Dinan et al. (2019b) and Kim et al. (2020), the perplexity (PPL) of
the ground-truth response, given the golden context as input is taken as one
automatic metric. Additionally, coherence rate and non-repetition rate are
used as automatic metrics, and human evaluation is conducted.

#### Coherence Rate.

To evaluate online performance in real-world practice, we conduct self-talk to
simulate the human-bot conversation, and measure whether the generated
response is coherent with the previous context as one automatic metric. The
maximum interaction turn is set to 10. As model-based methods have been proved
efficient and reliable Nie et al. (2021); Cui et al. (2021); Li et al.
(2021a), and we evaluate the dialogue coherence by consulting $f_{c}$ in
Section 4.2.

#### Non-Repetition Rate.

Inspired by Li et al. (2016), we adopt non-repetition rate to quantify the
diversity of the generated sequence during self-talk as a second automatic
metric. We calculate distinct-1, distinct-2 and distinct-3 by counting the
diversity of uni-grams, bi-grams and tri-grams, respectively. For each context
$\mathbf{\hat{U}}$, the distinct-$n$ is calculated by:

$\text{distinct}-n=\frac{\textsc{count}(\textsc{unique}_{\text{$n$-gram}_{i}\in\mathbf{\hat{U}}}(\text{$n$-gram}_{i}))}{\textsc{count}(\textsc{total}_{\text{$n$-gram}_{i}\in\mathbf{\hat{U}}}(\text{$n$-gram}_{)})}$
(10)

where count(), unique() and total() denote count the item of a list, unique
items in a list and enumeration a list, respectively. A higher distinct-$n$
indicates a lower repetition rate during self-talk.

(a) Reward Calculation.  (b) Optimization.

Figure 4: Coherence-Oriented Reinforcement Learning.

| Online Evaluation | Offline
---|---|---
| $c_{1}$ | $c_{2}$ | $c_{3}$ | $c_{4}$ | $c_{5}$ | $c_{6}$ | $c_{7}$ | $c_{8}$ | $c_{9}$ | $c_{10}$ | avg_5 | avg_10 | PPL
BART w/ Golden context | 99.7 | 98.9 | 98.2 | 96.0 | 97.6 | 97.2 | 96.0 | 94.2 | 94.1 | 93.3 | 99.0 | 96.5 | -
Single-turn BART | 99.2 | 88.1 | 71.5 | 63.5 | 57.2 | 53.0 | 46.7 | 41.8 | 37.3 | 34.9 | 75.9 | 59.3 | 21.3
Multi-turn BART | 99.2 | 96.5 | 79.2 | 67.7 | 48.7 | 43.0 | 32.5 | 28.4 | 24.5 | 21.9 | 78.3 | 54.2 | 17.8
w/ Noise | 99.2 | 95.4 | 76.5 | 58.7 | 47.1 | 35.4 | 31.4 | 22.1 | 23.1 | 12.4 | 75.4 | 50.1 | 18.1
w/ Utterance | 98.4 | 97.0 | 89.3 | 76.7 | 71.6 | 59.1 | 60.5 | 45.7 | 49.8 | 35.6 | 86.6 | 68.4 | 17.2
w/ Semi-Utterance | 98.1 | 97.2 | 85.7 | 69.2 | 64.0 | 50.5 | 52.1 | 36.4 | 43.6 | 29.1 | 82.9 | 62.6 | 17.1
w/ Hierarchical | 99.2 | 97.6 | 91.2 | 78.5 | 72.3 | 60.7 | 57.8 | 45.5 | 44.3 | 33.0 | 87.8 | 68.0 | 17.4

Table 1: Test performance of self-talk given a prompt utterance on Wizard test
set.

#### Human Evaluation.

Following previous work Ritter et al. (2011), we conduct human evaluation on
self-talk to compare our hierarchical sampling-based methods with our baseline
multi-turn BART by randomly sampling 50 instances (including 500 utterances).
Following Wu et al. (2018), we employ three annotators to do a side-by-side
human evaluation.

In order to pursue more authentic evaluation in real practice, we further
adopt a human-bot conversation to online evaluate these two methods. In
particular, given a prompt utterance, we ask an annotator to chat with chatbot
10 turns. The final human-bot test set we derive contains 50 dialogues
(including 500 utterances) for each model. We define three metrics for human
evaluation, including fluency, non-repetitive and coherence. Each aspect is
scored into three grades (0, 1 and 2) representing “bad”, “normal” and “good”,
respectively. We further calculate the Pearson correlation between the human
annotated coherence rate and the model assigned coherence rate.

###  5.2 Baselines

We compare the proposed methods with the following BART-based baselines:

#### BART w/ Golden context.

We fine-tune BART on the Wizard training set. During inference at turn $k$,
the golden context $\mathbf{U}_{1}^{k-1}$ is used to produce the response
$\mathbf{\hat{r}}_{k}$. Because the golden context is unavailable in practice,
the performance can be considered as the ceiling performance for alleviating
the discrepancy between training and real-world testing.

#### Multi-turn BART.

During training, we fine-tune BART based on the golden context-response pair.
Different from BART w/ Golden context, we use the context
$\mathbf{\hat{U}}_{1}^{k-1}$ predicted by previous turns to generate the
response $\mathbf{\hat{r}}_{k}$ during inference.

#### Single-turn BART.

We fine-tune BART for the dialogue generation following the single-turn
setting Wang et al. (2013). Only the last predicted utterance
$\hat{\mathbf{r}}_{k-1}$ is fed to the encoder to generate
$\mathbf{\hat{r}_{k}}$ for both training and inference. Single-turn BART
ignores the history in previous utterances.

#### w/ Noise

After sample an utterance $\mathbf{u}_{i}$, we use a random noise
$\mathbf{u}_{random}$ randomly sampled from the training set to replace
$\mathbf{u}_{i}$.

###  5.3 Results

Table 1 reports the performance of coherence rate as well as PPL for various
methods, and Table 2 shows the distinct-$n$ for the predicted context
generated by these methods.

#### Predicted Context vs Golden Context.

We first compare whether the dialogue generation model is able to generate
coherence response based on the golden context and the predicted context. As
shown on the top of Table 1, the coherence rate of BART w/ Golden context does
not decrease significantly with the number of turns increasing. The
performance drops by only 5.6 points coherence rate from 2 turns to 10 turns.
However, given the predicted context, the coherence rate decreases sharply as
the number of turns increase, with only 21.9 $c_{10}$. This shows the severity
of the discrepancy problem in real-world multi-turn dialogue generation.

#### Single-turn vs Multi-turn.

In offline evaluation, multi-turn BART achieves 17.8 PPL, which significantly
outperforms single-turn BART. This indicates that context information is
important for response generation. However, we have mixed results in online
evaluation. For example, multi-turn BART outperforms single-turn BART when the
number of utterances in the context is less than four in Table 1. When the
number of utterances becomes larger, single-turn BART surprisingly gives
better results compared with multi-turn BART. The reason can be that the
mismatch between the golden context and the predicted context hinders the
model performance as the number of utterances grows for multi-turn model.

#### Sampling vs w/o Sampling.

In Table 1, the proposed sampling-based approach performs slightly better on
PPL compared to the multi-turn BART, which shows our methods also work well in
general offline settings. When it comes to online settings, our sampling-based
methods outperform multi-turn BART significantly in all metrics, although
there is no direct supervision signal on coherence. For example, when measured
in context corresponding to 5 turns, multi-turn BART w/ hierarchical sampling
gives a $c_{5}$ of 72.3%, as compared to 48.7% by multi-turn BART.
Furthermore, multi-turn BART w/ Noise do not work well, since sampled noises
are difficult to accurately simulate errors of the inference scene during
training.

Model | Dis-1 | Dis-2 | Dis-3
---|---|---|---
Multi-turn BART | 24.37 | 32.30 | 36.35
w/ Hierarchical sampling | 36.29 | 49.77 | 55.29
Table 2: Non-Repetition Rate (%) for $n$-gram. ‘Dis-$n$’ means ‘Distinct-$n$’. Model | Fluency | Rep | Coh
---|---|---|---
Self-talk
Multi-turn BART | 1.93 | 0.89 | 0.74
w/ Hierarchical sampling | 1.91 | 1.37 | 1.45
Human-bot Conversation
Multi-turn BART | 1.89 | 0.96 | 0.63
w/ Hierarchical sampling | 1.90 | 1.53 | 1.32
Table 3: Human Evaluation. ‘Rep’ and ‘Coh’ indicate non-repetition and
coherence, respectively.

(a) Multi-turn BART.

(b) Multi-turn BART w/ Hierarchical Sampling.

Figure 5: Coherence rate with explicit optimization.

#### Utterance vs Hierarchical.

In Table 1, semi-utterance level sampling underperforms utterance-level
sampling in online evaluation. This is because semi-utterance level sampling
cannot accurately simulate errors of the inference scene during training. For
instance, the dialogue model tends to generate the response beginning with the
word “I”. While semi-utterance level sampling keeps the first few tokens in
the sampled utterance. When integrating utterance-level and semi-utterance
level sampling, hierarchical sampling gives the best coherence rate when
context less than six turns, which achieves 87.8% on $avg_{5}$. This shows the
effectiveness of sampling in a hierarchy way, which simulates the errors on
both utterance-level and token-level.

#### Repetition.

Table 2 reports the non-repetition rate of our sampling-based methods, drawing
multi-turn BART as a reference. We find that our methods give higher
distinct-$n$ measured by uni-gram, bi-gram and tri-gram, which shows the
effect of introducing hierarchical sampling to reduce copying and repetition
in model generated context. This also provides support for the effectiveness
of sampling-based methods to increase the robustness of multi-turn models.

#### Human Evaluation.

Table 3 compares the hierarchical sampling-based method with multi-turn BART
using human evaluation. All models are able to produce fluent responses due to
the power of pre-training, where fluency exceeds 1.89 for all models. Measured
in non-repetition and coherence, our hierarchical sampling method
significantly outperforms the baselines ($p$<0.01) on both self-talk and
human-bot conversation. In human-bot conversation, the coherence increases
largely from 0.96 to 1.53, showing that sampling enhances the robustness of
online multi-turn conversation. For self-talk, the pearson correlation between
the human annotated and the model assigned coherence rate is 0.78, which also
demonstrates the effectiveness of the model-based evaluation methods.

$\mathbf{u}_{1}$(Prompt) | My favorite video game is Quake. Have you ever played it?
---|---
Multi-turn BART
$\mathbf{\hat{r}}_{2}$ | I have not played it, but I know it was developed by the Quake team.
$\mathbf{\hat{r}}_{3}$ | Oh, Quake was developed by the Quake team. It’s a great game!
$\mathbf{\hat{r}}_{4}$ | I know it was developed by the Quake team. It was the first video game to be released commercially.
Multi-turn BART w/ Hierarchical sampling
$\mathbf{\hat{r}}_{2}$ | I have not played it, but I have heard it is a very good game.
$\mathbf{\hat{r}}_{3}$ | Yes it is. It was developed by the Quake team. It’s a team-based game.
$\mathbf{\hat{r}}_{4}$ | That sounds like a fun game to play with friends. What other games do you like?
Table 4: Examples of generated responses given a prompt utterance on the
Wizard of Wikipedia Test Seen.

#### Explicit Objectives.

Figure 5 shows the effect of the explicit coherence optimization method.
Training model with reinforcement learning outperforms with MLE measured by
coherence rate, showing the usefulness of optimizing the dialogue coherence
directly. We also find that the coherence rate improves significantly after
re-ranking in the inference scene for both multi-turn BART and multi-turn BART
w/ hierarchical sampling. Furthermore, as the number of candidate utterances
increases, the coherence rate increases. Multi-turn BART w/ beam=20 even
achieves 86.42 $c_{10}$ compared with 21.9 $c_{10}$ for multi-turn BART. This
indicates that the dialogue model can give coherent response candidates, which
can be re-ranked by an external coherence classifier to produce a coherent
response. Our hierarchical sampling-based methods also consistently perform
better than multi-turn BART by introducing coherence re-ranking.

Figure 6: Contradiction rate across different turn. Contradiction rate defined
by (1 $-$ coherence rate) $\times 100$%. Figure 7: Coherence rate across the
number of golden utterances at the beginning.

##  6 Analysis

#### The Number of Golden Turns.

We investigate whether a larger number of golden turns at the start is able to
help model to produce more coherent responses during inference. Figure 7 shows
the coherence rate against the number of golden utterances at the beginning
during the self-talk, drawing using the golden context as a reference. It can
be seen that a larger number of golden utterance at the beginning yields a
larger coherence rate in the first few turns. However, the coherence rate
decreases sharply with the number of turns increasing, which shows that simply
increasing beginning golden turns cannot help to alleviate the discrepancy
between training and real-world testing.

#### Utterance-level Contradiction.

To understand which turns in the context leads to an incoherence response, we
introduce an utterance-based classifier to probe different utterances during
generating the response at $10$-th turn in self-talk. As shown in Figure 6,
both models tend to generate response that contradict with the early turns.
This shows that current models do not take full advantage of the long-range
dialogue context. Compared with the multi-turn BART, the proposed sampling-
based methods significantly decrease the contradiction rate in the early
turns, and achieves the similar results in the later turns, which shows our
hierarchical sampling-based methods are able to improve robustness of multi-
turn models by alleviating the error accumulation.

#### Case Study.

We present an example to better understanding of multi-turn BART and our model
in Table 4. We observe that both models are able to generate reasonable
response $\hat{\mathbf{r}_{2}}$. Because the context for generating
$\hat{\mathbf{r}_{2}}$ contains prompt utterance (golden context)
$\mathbf{u}_{1}$ only. However, when the model encounters the predicted
utterance as context, multi-turn BART tends to generate response with
repetition and contradiction. With hierarchical sampling, our model produces
coherence responses during self-talk.

##  7 Conclusion

We quantified online dialogue generation in practice, and proposed the
hierarchical sampling-based methods to alleviate the discrepancy between
training and real-world testing. We further introduce an external coherence
classifier on both training and inference to boost the performance.
Experiments demonstrate the effectiveness of our methods for generating robust
online response on both self-talk and human-bot conversation.

##  8 Acknowledge

We would like to thank Zhen Yang and Sen Yang their helpful discussion and
valuable feedback.

## References

  * Bengio et al. (2015) Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015.  Scheduled sampling for sequence prediction with recurrent neural networks.  In _Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1_ , NIPS’15, page 1171–1179, Cambridge, MA, USA. MIT Press.
  * Cui et al. (2021) Leyang Cui, Yu Wu, Shujie Liu, and Yue Zhang. 2021.  Knowledge enhanced fine-tuning for better handling unseen entities in dialogue generation.  In _EMNLP_.
  * Cui et al. (2020) Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020.  MuTual: A dataset for multi-turn dialogue reasoning.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1406–1416, Online. Association for Computational Linguistics.
  * Deriu et al. (2020) Jan Deriu, Don Tuggener, Pius von Däniken, Jon Ander Campos, Alvaro Rodrigo, Thiziri Belkacem, Aitor Soroa, Eneko Agirre, and Mark Cieliebak. 2020\.  Spot the bot: A robust and efficient framework for the evaluation of conversational dialogue systems.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3971–3984, Online. Association for Computational Linguistics.
  * Dinan et al. (2019a) Emily Dinan, Varvara Logacheva, Valentin Malykh, Alexander Miller, Kurt Shuster, Jack Urbanek, Douwe Kiela, Arthur Szlam, Iulian Serban, Ryan Lowe, and et al. 2019a.  The second conversational intelligence challenge (convai2).  _The Springer Series on Challenges in Machine Learning_ , page 187–208.
  * Dinan et al. (2019b) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019b.  Wizard of wikipedia: Knowledge-powered conversational agents.  In _International Conference on Learning Representations_.
  * Jaques et al. (2020) Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. 2020.  Way off-policy batch deep reinforcement learning of human preferences in dialog.
  * Khandelwal (2021) Anant Khandelwal. 2021.  WeaSuL: Weakly supervised dialogue policy learning: Reward estimation for multi-turn dialogue.  In _Proceedings of the 1st Workshop on Document-grounded Dialogue and Conversational Question Answering (DialDoc 2021)_ , pages 69–80, Online. Association for Computational Linguistics.
  * Kim et al. (2020) Byeongchang Kim, Jaewoo Ahn, and Gunhee Kim. 2020.  Sequential Latent Knowledge Selection for Knowledge-Grounded Dialogue.  In _ICLR_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2016) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016.  A diversity-promoting objective function for neural conversation models.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 110–119, San Diego, California. Association for Computational Linguistics.
  * Li et al. (2020a) Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2020a.  Don’t say that! making inconsistent dialogue unlikely with unlikelihood training.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4715–4728, Online. Association for Computational Linguistics.
  * Li et al. (2020b) Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2020b.  Don’t say that! making inconsistent dialogue unlikely with unlikelihood training.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4715–4728, Online. Association for Computational Linguistics.
  * Li et al. (2017) Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017.  Dailydialog: A manually labelled multi-turn dialogue dataset.
  * Li et al. (2021a) Zekang Li, Jinchao Zhang, Zhengcong Fei, Yang Feng, and Jie Zhou. 2021a.  Addressing inquiries about history: An efficient and practical framework for evaluating open-domain chatbot consistency.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 1057–1067, Online. Association for Computational Linguistics.
  * Li et al. (2021b) Zekang Li, Jinchao Zhang, Zhengcong Fei, Yang Feng, and Jie Zhou. 2021b.  Conversations are not flat: Modeling the dynamic information flow across dialogue utterances.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 128–138, Online. Association for Computational Linguistics.
  * Liu et al. (2021a) Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. 2021a.  Confidence-aware scheduled sampling for neural machine translation.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 2327–2337, Online. Association for Computational Linguistics.
  * Liu et al. (2021b) Yijin Liu, Fandong Meng, Yufeng Chen, Jinan Xu, and Jie Zhou. 2021b.  Scheduled sampling based on decoding steps for neural machine translation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , Online.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Mihaylova and Martins (2019) Tsvetomila Mihaylova and André F. T. Martins. 2019.  Scheduled sampling for transformers.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_ , pages 351–356, Florence, Italy. Association for Computational Linguistics.
  * Nie et al. (2021) Yixin Nie, Mary Williamson, Mohit Bansal, Douwe Kiela, and Jason Weston. 2021.  I like fish, especially dolphins: Addressing contradictions in dialogue modeling.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1699–1713, Online. Association for Computational Linguistics.
  * Qin et al. (2021) Libo Qin, Tianbao Xie, Shijue Huang, Qiguang Chen, Xiao Xu, and Wanxiang Che. 2021\.  Don’t be contradicted with anything! CI-ToD: Towards benchmarking consistency for task-oriented dialogue system.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2357–2367, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Ram et al. (2018) Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, Eric King, Kate Bland, Amanda Wartick, Yi Pan, Han Song, Sk Jayadevan, Gene Hwang, and Art Pettigrue. 2018.  Conversational ai: The science behind the alexa prize.
  * Ranzato et al. (2016) Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016.  Sequence level training with recurrent neural networks.  In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
  * Ritter et al. (2011) Alan Ritter, Colin Cherry, and William B. Dolan. 2011.  Data-driven response generation in social media.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 583–593, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  * Roller et al. (2020) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, and Jason Weston. 2020.  Recipes for building an open-domain chatbot.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017\.  Proximal policy optimization algorithms.
  * Shum et al. (2018) Heung-yeung Shum, Xiao-dong He, and Di Li. 2018.  From eliza to xiaoice: challenges and opportunities with social chatbots.  _Frontiers of Information Technology & Electronic Engineering_, 19(1):10–26.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  In _Advances in Neural Information Processing Systems_ , volume 27. Curran Associates, Inc.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30, pages 5998–6008. Curran Associates, Inc.
  * Vinyals and Le (2015) Oriol Vinyals and Quoc Le. 2015.  A neural conversational model.
  * Wang et al. (2013) Hao Wang, Zhengdong Lu, Hang Li, and Enhong Chen. 2013.  A dataset for research on short-text conversations.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 935–945, Seattle, Washington, USA. Association for Computational Linguistics.
  * Welleck et al. (2020) Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020.  Neural text generation with unlikelihood training.  In _International Conference on Learning Representations_.
  * Welleck et al. (2019) Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019.  Dialogue natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3731–3741, Florence, Italy. Association for Computational Linguistics.
  * Wu et al. (2018) Yu Wu, Furu Wei, Shaohan Huang, Yunli Wang, Zhoujun Li, and Ming Zhou. 2018.  Response generation by context-aware prototype editing.
  * Xu et al. (2021) Haoran Xu, Hainan Zhang, Yanyan Zou, Hongshen Chen, Zhuoye Ding, and Yanyan Lan. 2021.  Adaptive bridge between training and inference for dialogue.
  * Yang et al. (2018) Zhen Yang, Wei Chen, Feng Wang, and Bo Xu. 2018.  Improving neural machine translation with conditional sequence generative adversarial nets.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1346–1355, New Orleans, Louisiana. Association for Computational Linguistics.
  * Zhang et al. (2018) Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018.  Personalizing dialogue agents: I have a dog, do you have pets too? In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2204–2213, Melbourne, Australia. Association for Computational Linguistics.
  * Zhang et al. (2019) Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. 2019.  Bridging the gap between training and inference for neural machine translation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4334–4343, Florence, Italy. Association for Computational Linguistics.
  * Zhang et al. (2020a) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020a.  Dialogpt: Large-scale generative pre-training for conversational response generation.  _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_.
  * Zhang et al. (2020b) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b.  Dialogpt: Large-scale generative pre-training for conversational response generation.
  * Ziegler et al. (2019) Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019.  Fine-tuning language models from human preferences.

##  Appendix A Appendix

###  A.1 Setup

We implement our methods with transformers and choose bart-base as the pre-
trained transformer language model. AdamW Loshchilov and Hutter (2019) with a
batch size of 32 is used to optimize parameters. The initial learning is set
as 5e-5, which will be halved in each training iteration. Following Lewis et
al. (2020), we set the maximum input tokens as 512. The training time of our
methods is 0.6 times slower than the baseline method. Our inference time is
the same as that of the baseline. For the coherence-oriented reinforcement
learning method, we set $\beta$ in Equation 9 as 0.2. For computational
efficiency, we truncate the maximum decode length as 20 to calculate the KL-
divergence.