# Quench Dynamics of Thermal Bose Gases Across Wide and Narrow Feshbach

Xiaoyi Yang  MOE Key Laboratory for Nonequilibrium Synthesis and Modulation of
Condensed Matter, Shaanxi Province Key Laboratory of Quantum Information and
Quantum Optoelectronic Devices, School of Physics, Xi’an Jiaotong University,
Xi’an 710049, China  Ren Zhang<EMAIL_ADDRESS>MOE Key Laboratory for
Nonequilibrium Synthesis and Modulation of Condensed Matter, Shaanxi Province
Key Laboratory of Quantum Information and Quantum Optoelectronic Devices,
School of Physics, Xi’an Jiaotong University, Xi’an 710049, China

###### Abstract

Using high-temperature virial expansion, we study the quench dynamics of the
thermal Bose gases near a wide, narrow, and intermediate Feshbach resonance.
Our results show that the shallow bound state near Feshbach resonance leads to
interesting phenomena. Near the wide Feshbach resonance, the long-time
$\hat{n}_{\bf k}$ oscillates when the scattering length $a_{s}$ is quenched
from zero to large but with finite positive values. The oscillation frequency
$\omega=E_{\rm b}/\hbar$ with $E_{\rm b}$ being the binding energy. When
$a_{s}$ is quenched to infinity or negative value, the oscillation vanishes.
Near the narrow Feshbach resonance, the interaction should be characterized by
a two-channel model. When the background scattering length $a_{\rm
bg}\gtrsim\lambda$, there is an oscillation in the long-time dynamics, and the
frequency is determined by the energy of the shallow bound state in the open
channel. When $a_{\rm bg}<0$ or $0<a_{\rm bg}\ll\lambda$, there is no shallow
bound state in the open channel, hence no long-time oscillation. We check our
conclusion using some realistic systems, and the results are consistent with
our conclusion.

Introduction- Thanks to Feshbach resonances, the pairwise interaction between
atoms can be controlled flexibly by tuning external fields, and the
equilibrium properties in strongly interacting atomic gases have been
intensively studied Chin  _et al._ (2010); Köhler  _et al._ (2006). The
timescale of the Hamiltonian manipulation can be much smaller than the
relaxation time. As such, ultracold atomic gases have also become one of the
most ideal platforms to investigate non-equilibrium physics, including the
quench dynamics Makotyn  _et al._ (2014); Eigen  _et al._ (2017); Prüfer  _et
al._ (2018); Erne  _et al._ (2018); Eigen  _et al._ (2018); Saint-Jalm  _et
al._ (2019); Deng  _et al._ (2016).

Here, quench dynamics refers to the evolution of initial states under an
abruptly changed Hamiltonian. For instance, the $s$-wave scattering length
$a_{s}$ can be controlled by the magnetic field. When $a_{s}$ is modulated by
time-dependent external fields, the particle number on the non-condensation
mode will exponentially grow, i,e. Bose-Einstein condensation (BEC) is
depleted. If the modulation phase is suddenly changed by $\pi$, it was found
that the excited particle number decreases, i.e. BEC revives Hu  _et al._
(2019). Motivated by this phenomenon, a new kind of echo theory has also been
raised in BEC, which can be realized by quenching $a_{s}$ or the trapping
potential Lv  _et al._ (2020); Chen  _et al._ (2020). In the same spirit,
there are many other studies on quench dynamics via quenching parameters of
the Hamiltonian. Many-body localization and thermalization can be
distinguished by the quench dynamics of the entanglement entropy Abanin  _et
al._ (2019). The topology of Hamiltonian of band insulators can be extracted
in the quench dynamics of linking number Wang  _et al._ (2017); Tarnowski  _et
al._ (2019); Sun  _et al._ (2018). The dynamical fractal has been established
in quantum gases with discrete scaling symmetry Gao  _et al._ (2019).

In the seminal experiment by Cambridge group Eigen  _et al._ (2018), a series
of universal quench dynamics of Bose gas have been revealed by quenching the
interaction from zero to unitary. Both degenerate and thermal Bose gases
composited of 39K are studied near a Feshbach resonance located at $\sim
402.7G$. This is a resonance of intermediate width, $s_{\rm res}\sim 2.1$ Chin
_et al._ (2010). In the follow-up theoretical studies, it has been treated as
a wide one, and the comparison of theoretical and experimental results are
satisfactory for both degenerate and thermal gases Gao  _et al._ (2020); Sun
_et al._ (2020). A natural question arises: what are the effects induced by
resonance width? To address this question, we focus on the dynamics of thermal
Bose gas near a Feshbach resonance with varying width.

The virial expansion builds a connection between the few-body and the many-
body physics Rupak (2007); Liu  _et al._ (2009, 2010); Liu (2013); Peng  _et
al._ (2011a, 2014); Yan and Blume (2015, 2016a); Sun and Cui (2017); Nishida
(2019); Hou and Drut (2020a); Hou  _et al._ (2021a); Sun  _et al._ (2017);
Kaplan and Sun (2011). It works well when comparing with experiments Bourdel
_et al._ (2003); Stewart  _et al._ (2008); Nascimbène  _et al._ (2010); Kuhnle
_et al._ (2011); Feld  _et al._ (2011); Ku  _et al._ (2012); Mukherjee  _et
al._ (2019); Carcy  _et al._ (2019). The control parameter is the fugacity
$z=e^{\mu/(k_{B}T)}$, where $\mu$ is the chemical potential and $k_{B}$ is the
Boltzmann constant. At high temperature, $\mu$ is large and negative.
Therefore, $z<1$. This method has been applied to equilibrium quantum gases
near wide Feshbach resonances Ho and Mueller (2004); Yan and Blume (2016b); Yu
_et al._ (2015); Marcelino  _et al._ (2014); Hou and Drut (2020b) and narrow
Feshbach resonances Peng  _et al._ (2011a); Ho  _et al._ (2012); Peng  _et
al._ (2011b); Hou  _et al._ (2021b); Tajima  _et al._ (2021); Hofmann (2020).
Recently, it has also been implemented to the quench dynamics of Bose gas near
a wide Feshbach resonance Sun  _et al._ (2020). As such, it is natural to
generalize the virial expansion for dynamics to the narrow Feshbach resonance,
and a crossover from a wide one to a narrow one. Our main results are
summarized in Table 1.

Table 1: Long-time behavior of the momentum distribution $n_{\bf k}$ dynamics near Feshbach resonance. Resonance width |  Oscillation |  Non-oscillation
---|---|---
$s_{\rm res}\gg 1$ (wide resonance ) |  $s$-wave scattering length $a_{s}>0$ |  $s$-wave scattering length $a_{s}<0$ or $a_{s}=\infty$
$s_{\rm res}\ll 1$ (narrow resonance) or $s_{\rm res}\sim 1$ (intermediate width) |  Background scattering length $a_{\rm bg}\gtrsim\lambda$ |  Background scattering length $a_{\rm bg}<0$ or $0<a_{\rm bg}\ll\lambda$

Virial expansion for dynamic- Let us first review the basics of virial
expansion for quench dynamics. We consider an equilibrium thermal Bose gas
with temperature $T$, then quench the interaction from zero to finite or
unitary. The Hamiltonian becomes $\hat{H}=\hat{H}_{0}+\hat{V}$ with $\hat{V}$
denoting the interaction. For later time $t>0$, the system evolve under the
full Hamiltonian $\hat{H}$, and eventually achieve a new equilibrium state.
Some universal physics can be revealed in prethermal process. To this end, we
could measure an observable ${\hat{W}}$, the expectation value ${\cal W}(t)$
of which can be written as Sun  _et al._ (2020)

$\displaystyle\mathcal{W}(t)=\frac{\mathrm{Tr}[e^{-\beta(\hat{H}_{0}-\mu\hat{N})}e^{i\hat{H}t}\hat{W}e^{-i\hat{H}t}]}{\mathrm{Tr}[e^{-\beta(\hat{H}_{0}-\mu\hat{N})}]},$
(1)

where $\beta=1/{k_{B}T}$ denotes the inverse temperature, $\hat{N}$ is the
total particle number of Bose gas. Here and after forth, we set
$\hbar=k_{B}=1$ for convenience. The exact evaluation of ${\cal W}(t)$ is
formidable in a many-body system because $\hat{W}$ does not commute with the
Hamiltonian.

At high temperature, we expand the observable ${\mathcal{W}}(t)$, instead of
the thermodynamic potential $\Omega$, in terms of the fugacity $z$. Up to the
order of $z^{2}$, $\mathcal{W}(t)$ is expressed as

$\displaystyle\mathcal{W}(t)=X_{1}z+\left(-Q_{1}X_{1}+X_{2}\right)z^{2}+{\cal
O}\left(z^{3}\right).$ (2)

Here, $X_{n}$ and $Q_{n}$ are defined as

$\displaystyle X_{n}$
$\displaystyle=\mathrm{Tr}_{n}[\Theta(t)e^{-\beta\hat{H}_{0}}e^{it\hat{H}}\hat{W}e^{-it\hat{H}}]$
$\displaystyle=\underset{\alpha,\beta,\gamma}{\sum}e^{-\beta
E_{\alpha}^{(n)}}G_{\beta\alpha}^{(n)*}(t)\langle\psi_{\beta}^{(n)}|\hat{W}|\psi_{\gamma}^{(n)}\rangle
G_{\gamma\alpha}^{(n)}(t),$ (3) $\displaystyle Q_{n}$
$\displaystyle=\mathrm{Tr}_{n}[e^{-\beta\hat{H}_{0}}],$ (4)

respectively. $n=1,2\cdots$ indicates the particle number, $E^{(n)}_{\alpha}$
and $\psi^{(n)}_{\alpha}$ represents the energy and wave function of
$n$-particle non-interaction state, respectively. $\Theta(t)$ is the step
function. $G^{(n)}(t)$ is retarded Green’s function of $n$-particle
interacting system, and it is defined as

$\displaystyle G_{\gamma\alpha}^{(n)}(t)$
$\displaystyle=\langle\psi_{\gamma}^{(n)}|\Theta(t)e^{-it\hat{H}}|\psi_{\alpha}^{(n)}\rangle$
$\displaystyle=\frac{i}{2\pi}\int_{-\infty}^{\infty}d\omega e^{-i\omega
t}G_{\gamma\alpha}^{(n)}(\omega+i0^{+}).$ (5)

As a result, by solving the $n$-particle problem, the evolution of the many-
body system can be obtained, and the accuracy can be improved by increasing
$n$.

The same as the experiment, we consider the dynamics of the particle number in
the ${\bf k}$-mode, i.e., $\hat{W}=\hat{n}_{\mathbf{k}}$. For a single
particle system, $\hat{H}=\hat{H}_{0}$, and

$\displaystyle
X_{1}=\mathrm{Tr}_{1}\left[\Theta(t)e^{-\beta\hat{H}_{0}}e^{it\hat{H}_{0}}\hat{n}_{\mathbf{k}}e^{-it\hat{H}_{0}}\right]=e^{-\beta\mathbf{k}^{2}/(2m)}.$
(6)

Therefore, $X_{1}$ and $Q_{n}$ is independent on time $t$, The evolution of
momentum distribution $\delta
n_{\mathbf{k}}=n_{\mathbf{k}}(t)-n_{\mathbf{k}}(0)$ only depends on $X_{2}$ in
Eq.(2). Specifically,

$\displaystyle\delta n_{\mathbf{k}}=\left[X_{2}(t)-X_{2}(0)\right]z^{2},$ (7)

which can be obtained by only solving the two-body problem. For the two-body
problem, the non-interacting wave function is labeled by
$|\psi_{a}^{(2)}\rangle=|{\bf P},{\bf q}\rangle$ with ${\bf P}$ and ${\bf q}$
being the total momentum and the relative momentum of two bosons,
respectively. The corresponding energy reads $E_{a}^{(2)}=P^{2}/(4m)+q^{2}/m$,
where $m$ is the reduced mass. According to the Lippman-Schwinger function,
the retarded Green’s function can be written as

$\displaystyle G_{\alpha\beta}^{(2)}(s)$
$\displaystyle=G_{\alpha\beta}^{(0)}(s)+G_{\alpha\beta}^{(0)}(s)T_{2}(s)G_{\alpha\beta}^{(0)}(s)$
$\displaystyle=\left[\frac{\langle\mathbf{q_{\alpha}}|\mathbf{q_{\beta}}\rangle}{s-\varepsilon_{\mathbf{q_{\alpha}}}}+\frac{T_{2}(s)}{(s-\varepsilon_{\mathbf{q_{\alpha}}})(s-\varepsilon_{\mathbf{q_{\beta}}})}\right]\delta_{\mathbf{P_{\alpha},P_{\beta}}},$
(8)

where $s=\omega+i0^{+}$,
$\varepsilon_{\mathbf{q_{\alpha}}}=\mathbf{q}^{2}_{\alpha}/m$ is the kinetic
energy of the relative motion, $T_{2}(s)$ denotes the T-matrix of the two-body
scattering. We have also used the free Green’s function for the relative
motion $G_{0}(s)=(s-\mathbf{q}^{2}/m)^{-1}$ in the second line of Eq.(8).

Wide and narrow resonances- Before embarking on the difference between wide
and narrow resonance, let us recall the two-body scattering theory. The
generic relation between scattering amplitude $f({\bf
k}^{\prime}\leftarrow{\bf k})$ and scattering T-matrix $T({\bf
k}^{\prime},{\bf k};E)$ is $f({\bf k}^{\prime}\leftarrow{\bf
k})=-\frac{m}{4\pi}T_{2}({\bf k}^{\prime},{\bf k};E)$. For the partial wave
scattering, the scattering amplitude $f_{\ell}(k)$ is defined by the partial
wave scattering matrix $s_{\ell}=e^{2i\delta_{\ell}}$,

$\displaystyle
f_{\ell}(k)=\frac{s_{\ell}-1}{2ik}=-\frac{1}{ik-k/\tan\delta_{\ell}(k)},$ (9)

where $\delta_{\ell}(k)$ is the energy-dependent phase shift of the $\ell$-th
partial wave. In this manuscript, we consider the $s$-wave scattering, and the
corresponding T-matrix is then written as

$\displaystyle T_{2}(s)=\frac{4\pi/m}{-\sqrt{ms}/\tan\delta_{0}-\sqrt{-ms}}$
(10)

In the effective field theory, $\sqrt{ms}/\tan\delta_{0}=-1/a_{s}+r_{\rm
eff}ms/2+\cdots$ with $a_{s}$ and $r_{\rm eff}$ denoting the $s$-wave
scattering length and the effective range, respectively. For wide resonance,
the effective range effect can be ignored, and $a_{s}$ is the only parameter
to characterize the pairwise interaction. For narrow resonance, one has to
include the effective range to incorporate the energy-dependent of phase
shift. For the van der Waals interaction between two atoms, one has to resort
to the complicated quantum defect theory to obtain the exact phase shift Gao
(1998); Gao  _et al._ (2005). To be simple, we consider a two-channel square
model to mimic the interaction between atoms. In the basis spanned by closed
and open channel, the interaction can be written as

$\displaystyle V(r)=\begin{cases}\begin{bmatrix}-V_{o}&W\\\
W&-V_{c}+\delta\mu_{B}B\end{bmatrix},&\text{for }r<r_{0};\\\
\begin{bmatrix}0&0\\\ 0&\infty\end{bmatrix},&\text{for }r>r_{0},\end{cases}$
(11)

where $V_{c}$ and $V_{o}$ represent the closed channel and open channel
potential, and $W$ is the inter-channel coupling strength. $r_{0}$ is the
potential range, and the corresponding energy scale is $E_{0}=1/(mr_{0}^{2})$.
$\delta\mu_{B}$ is the magnetic momentum difference between closed and open
channel. By tuning the magnetic field, a series of Feshbach resonances appear
(supplementary material). Although this toy model does not quantitatively
capture the interaction potential detail, it is an insight model to present
the qualitative picture. By solving the two-channel model, the effective phase
shift can be analytically obtained. Upon substituting the phase shift into
Eq.(10) and (8), the evolution of momentum distribution $\delta n_{\bf k}(t)$
can be obtained. To precisely distinguish wide and narrow Feshbach resonance,
we define the parameter

$\displaystyle s_{\rm res}=\frac{a_{\rm
bg}}{r_{0}}\frac{\delta\mu_{B}\Delta}{E_{0}},$ (12)

where $a_{\rm bg}$ is the background scattering length determined by $V_{o}$.
$\Delta$ is the resonance width in the magnetic field aspect, and is
determined by the inter-channel coupling $W$. When $s_{\rm res}\gg 1$, it is a
wide resonance; when $s_{\rm res}\ll 1$, it is a narrow resonance; when
$s_{\rm res}\sim 1$, it is of intermediate width.

Figure 1: The phase shift $\sin^{2}\delta_{0}$ and momentum distribution
$\delta n_{\bf k}$ evolution of thermal Bose gas near wide and narrow
resonance. The interaction is quenched by abruptly changing the magnetic field
to resonance. $s_{\rm res}\sim 260$ for the wide resonance (a) and $s_{\rm
res}\sim 0.04$ for the narrow resonance (b). (c): Near the wide resonance, the
low-momentum $n_{\bf k}$ monotonically decreases and the high-momentum $n_{\bf
k}$ monotonically increases. The critical momentum is around $k\lambda=4.5$.
(d): Near the narrow resonance, $n_{\bf k}$ shows damped oscillation when it
decreases or increases.

In Fig. 1, we depict $\sin^{2}\delta_{0}$ as a function of the magnetic field
and incident energy for both wide resonance (a) and narrow resonance (b). The
$s_{\rm res}\sim 260$ and $s_{\rm res}\sim 0.04$ for the wide and narrow
resonance. When $\delta_{0}=\pi/2$, i.e., $\sin^{2}\delta_{0}=1$, the
resonance happens. It is clear that the phase shift of wide resonance almost
does not depend on the incident energy, as shown in (a). In contrast, the
phase shift of narrow resonance strongly depends on the incident energy, as
shown in (b). For both cases, we quench the interaction by abruptly changing
the magnetic field to $B_{\rm res}$, the position of resonance, and measure
the dynamics of momentum distribution. Near the wide resonance, the low-
momentum ($k\lambda<4.5$) $n_{\bf k}$ decreases monotonically after quenching
and tends to a stable value after a long time evolution; the high-momentum
($k\lambda>4.5$) $n_{\bf k}$ increases monotonically and tends to a stable
value after a long time evolution. There is a critical momentum
($k\lambda=4.5$), where $n_{\bf k}$ goes up and down, and tends to its initial
value. This observation is consistent with experimental results Eigen  _et
al._ (2018) and theoretical results given by the zero-range potential Sun  _et
al._ (2020). Here $\lambda=\sqrt{2\pi/(mT)}$ denotes the thermal de Broglie
wavelength, and we define a time unit $t_{\lambda}=1/T$. However, near the
narrow resonance, the momentum distribution dynamics show very different
behavior in contrast to their wide resonance counterpart. Although the
tendency of $n_{\bf k}$ for low-momentum and high-momentum remains, there is
oscillation with damping amplitude, which means that there must be an
intrinsic energy scale near the narrow resonance. The critical momentum shifts
slightly.

Zero-range model- To understand this phenomenon, let us turn to the zero-range
model. For the wide resonance, the two-channel model in Eq.(11) can be
approximated by a zero-range single-channel model. The two-body scattering
T-matrix $T_{2}(s)$ reduces to

$\displaystyle T_{2}(s)=\frac{4\pi/m}{a_{s}^{-1}-\sqrt{-ms}}.$ (13)

Figure 2: The evolution of the momentum distribution when the quenched
interaction deviates from the resonance position. (a): Final
$a_{s}=\lambda/\sqrt{2\pi}$. (b): Final $a_{s}=-\lambda/\sqrt{2\pi}$. (c):
$n_{\bf k}$ evolution for three particular momentum; Blue solid line
($k\lambda=6$); Black dashed-dotted (cross momentum); Red dashed line
($k\lambda=3$). (d): The shallow bound state energy $E_{\rm b}=1/(ma_{s}^{2})$
and oscillation frequency $\omega$ of the momentum dynamics. $\omega$
collapses to the bound state energy. Inset: The spectrum ${\cal A}(\omega)$
for the final $a_{s}=\lambda/\sqrt{2\pi}$.

The pole of $T_{2}(s)$ gives the energy of the shallow bound state, $E_{\rm
b}=-1/(ma_{s}^{2})$. When the interaction is quenched to unitary, i.e.,
$a_{s}=\infty$, the bound state energy vanishes. The Hamiltonian is scale-
invariant, and the only relevant length scales are the inter-particle spacing
and the thermal de Broglie wavelength. As such, the dynamics driven by the
scale-invariant Hamiltonian are universal. Nevertheless, when $a_{s}$ is
quenched to a positive finite value, this extra length scale would exhibit
itself in the dynamics.

In Fig. 2, we show the momentum distribution dynamics when the quenched
interaction deviates from the resonance position. For final $a_{s}>0$, we find
a long-time oscillation in the momentum distribution dynamics, as shown in
(a), similar to that near the narrow resonance. The oscillation frequency for
different momentum is the same, as depicted in (c). We extract the oscillation
frequency for varying $a_{s}$. Our results show that the frequency collapses
to the binding energy $|E_{\rm b}|=1/(ma_{s}^{2})$, as shown in (d).
Therefore, we conclude that the oscillation in dynamics originates from the
shallow bound state. When the final $a_{s}<0$, there is no shallow bound
state, hence no oscillation in the dynamics, as shown in (b).

Now we turn to the dynamics near the narrow or intermediate Feshbach
resonance, where the single-channel model is not sufficient to characterize
the interaction. As such, we need to adopt the two-channel zero range model
Zhai (2021), the two-body scattering T-matrix $T_{2}$ of which can be written
as (supplementary material)

$\displaystyle T_{2}(s)=\frac{4\pi/m}{\frac{s-\delta\mu_{B}(B-B_{\rm
res})}{a_{\rm bg}\left[s+\delta\mu_{B}\Delta-\delta\mu_{B}(B-B_{\rm
res})\right]}-\sqrt{-ms}},$ (14)

where $B_{\rm res}$ denotes the magnetic field at resonance. The pole of
$T_{2}(s)$ in Eq.(14) gives two bound states. By substituting Eq.(14) into
Eq.(8), we can evaluate the momentum distribution $n_{\bf k}$.

Figure 3: (a) Dynamics of $n_{\bf k}$ near the narrow and intermediate
Feshbach resonance. The numbers in the legend represent $(s_{\rm res},a_{\rm
bg})$. Solid (dashed) curves represent two-channel square (zero-range) model.
$k\lambda=7(3)$ for narrow (intermediate) resonance. (b) Binding energy given
two-channel square (solid lines) model and zero-range (dashed lines) model.
The blue circle and red triangle are the oscillation frequency extracted from
the dynamics of $n_{\bf k}$.

Fig. 3(a) shows the momentum distribution dynamics of $n_{\bf k}$ after
quenching the interaction to resonance. We compare the results of the two-
channel square model (solid curves) and zero range model (dashed curves) for
some particular momentum. Near the narrow resonance ($s_{\rm res}=0.003$),
both results show there is a long-time oscillation, but the frequencies are
different. Near resonance of intermediate width ($s_{\rm res}=2$), the
dynamics are almost the same as that near the wide resonance. This explains
why could the single-channel model give consistent results with the
experiment. When $a_{\rm bg}<0$, $n_{\bf k}$ also monotonically decays near
the intermediate resonance ($s_{\rm res}=1$), as shown by the black curves.
This is because the shallow bound state in open channel is absent in this
case.

Near the narrow resonance, the oscillation frequency $\omega$ is determined by
the binding energy of the bound state in the open channel instead of the
shallow bound state near the threshold. In Fig. 3(b), we present the binding
energy given by the two-channel square model and zero-range model. The binding
energy at the threshold of these two models is the same, as illustrated by the
blue dashed and red solid lines. However, the other bound states originate
from the open channel and binding energies given by these two models are
different, as shown by the green dashed and black solid lines. We extract the
oscillation frequency near narrow resonance in Fig. 3(a). It collapses to the
binding energy of the bound state in the open channel, instead of that at the
threshold.

Figure 4: The dynamics of $n_{\bf k}$ for realistic systems near Feshbach
resonance. (a): 133Cs near the wide resonance with $s_{\rm res}=560$. (b):
133Cs near the intermediate resonance with $s_{\rm res}=0.67$ and $a_{\rm
bg}=926a_{0}$. (c): 7Li near the intermediate resonance with $s_{\rm res}=0.8$
and $a_{\rm bg}=-25a_{0}$. (d): 133Cs near the narrow resonance with $s_{\rm
res}=0.002$ and $a_{\rm bg}=160a_{0}$. $a_{0}$ is the Bohr’s radius.

Application to realistic systems- We show the quench dynamics of some
realistic systems. Here, we consider four different systems including wide,
narrow, and intermediate resonance Chin  _et al._ (2010). (I): We choose
${}^{133}\text{Cs}$ near a wide resonance with $s_{\rm res}=560$. When the
interaction is quenched to unitary, the evolution of the momentum distribution
is as same as that near the wide resonance, as shown in Fig.4 (a). (II): In
contrast, for ${}^{133}\text{Cs}$ near the narrow resonance with $s_{\rm
res}=0.002$ and $a_{\rm bg}=160a_{0}$ ($a_{0}$ is the Bohr’s radius), shown in
Fig.4(d), we see that the momentum distribution oscillates, which originates
from the bound state in the open channel. (III): For quench dynamics near
resonance of intermediate width, we choose two systems, ${}^{133}\text{Cs}$
near resonance with $s_{\rm res}=0.67$ (b) and ${}^{7}\text{Li}$ near
resonance with $s_{\rm res}=0.8$ (c). However, the background scattering
length are different for these two systems. For ${}^{133}\text{Cs}$, $a_{\rm
bg}=926a_{0}$ implies a shallow bound state in the open channel, and we see
that the momentum distribution oscillates. Nevertheless, for
${}^{7}\text{Li}$, $a_{\rm bg}=-25a_{0}$, there is no shallow bound state in
the open channel, thus no oscillation in the momentum distribution.

In summary, we use virial expansion to study quench dynamics across the wide
and narrow Feshbach resonances. Taking the dynamics of momentum distribution
as an example, we show the dynamics can be affected by the bound state, the
frequency of oscillation is proportional to the energy of the bound state.
Near wide resonance, the relevant bound state is the shallow bound state near
the threshold, while near the narrow or intermediate resonance, the relevant
bound state is the bound state of the open channel. We check our conclusion
using some realistic systems.

###### Acknowledgements.

We are grateful to Mingyuan Sun and Xin Chen for helpful discussion. The work
was supported by the National Nature Science Foundation of China (Grant No.
12074307), the National Key R$\&$D Program of China (Grant No. 2018YFA0307601)
and the Fundamental Research Funds for the Central Universities (Grant No.
71211819000001).

## References

  * Chin  _et al._ (2010) C. Chin, R. Grimm, P. Julienne, and E. Tiesinga, Feshbach resonances in ultracold gases, Rev. Mod. Phys. 82, 1225 (2010).
  * Köhler  _et al._ (2006) T. Köhler, K. Góral, and P. S. Julienne, Production of cold molecules via magnetically tunable feshbach resonances, Rev. Mod. Phys. 78, 1311 (2006).
  * Makotyn  _et al._ (2014) P. Makotyn, C. E. Klauss, D. L. Goldberger, E. A. Cornell, and D. S. Jin, Universal dynamics of a degenerate unitary bose gas, Nature Physics 10, 116 (2014).
  * Eigen  _et al._ (2017) C. Eigen, J. A. P. Glidden, R. Lopes, N. Navon, Z. Hadzibabic, and R. P. Smith, Universal scaling laws in the dynamics of a homogeneous unitary bose gas, Phys. Rev. Lett. 119, 250404 (2017).
  * Prüfer  _et al._ (2018) M. Prüfer, P. Kunkel, H. Strobel, S. Lannig, D. Linnemann, C.-M. Schmied, J. Berges, T. Gasenzer, and M. K. Oberthaler, Observation of universal dynamics in a spinor bose gas far from equilibrium, Nature 563, 217 (2018).
  * Erne  _et al._ (2018) S. Erne, R. Bücker, T. Gasenzer, J. Berges, and J. Schmiedmayer, Universal dynamics in an isolated one-dimensional bose gas far from equilibrium, Nature 563, 225 (2018).
  * Eigen  _et al._ (2018) C. Eigen, J. A. P. Gliden, R. Lopes, E. A. Cornel, R. P. Smith, and Z. Hadzibabic, Universal prethermal dynamics of bose gases quenched to unitarity, Natrue 563, 221 (2018).
  * Saint-Jalm  _et al._ (2019) R. Saint-Jalm, P. C. M. Castilho, E. Le Cerf, B. Bakkali-Hassani, J.-L. Ville, S. Nascimbene, J. Beugnon, and J. Dalibard, Dynamical symmetry and breathers in a two-dimensional bose gas, Phys. Rev. X 9, 021035 (2019).
  * Deng  _et al._ (2016) S. Deng, Z.-Y. Shi, P. Diao, Q. Yu, H. Zhai, R. Qi, and H. Wu, Observation of the efimovian expansion in scale-invariant fermi gases, Science 353, 371 (2016).
  * Hu  _et al._ (2019) J. Hu, L. Feng, Z. Zhang, and C. Chin, Quantum simulation of unruh radiation, Nature Physics 15, 785 (2019).
  * Lv  _et al._ (2020) C. Lv, R. Zhang, and Q. Zhou, $su(1,1)$ echoes for breathers in quantum gases, Phys. Rev. Lett. 125, 253002 (2020).
  * Chen  _et al._ (2020) Y.-Y. Chen, P. Zhang, W. Zheng, Z. Wu, and H. Zhai, Many-body echo, Phys. Rev. A 102, 011301 (2020).
  * Abanin  _et al._ (2019) D. A. Abanin, E. Altman, I. Bloch, and M. Serbyn, Colloquium: Many-body localization, thermalization, and entanglement, Rev. Mod. Phys. 91, 021001 (2019).
  * Wang  _et al._ (2017) C. Wang, P. Zhang, X. Chen, J. Yu, and H. Zhai, Scheme to measure the topological number of a chern insulator from quench dynamics, Phys. Rev. Lett. 118, 185701 (2017).
  * Tarnowski  _et al._ (2019) M. Tarnowski, F. N. Ünal, N. Fläschner, B. S. Rem, A. Eckardt, K. Sengstock, and C. Weitenberg, Measuring topology from dynamics by obtaining the chern number from a linking number, Nature Communications 10, 1728 (2019).
  * Sun  _et al._ (2018) W. Sun, C.-R. Yi, B.-Z. Wang, W.-W. Zhang, B. C. Sanders, X.-T. Xu, Z.-Y. Wang, J. Schmiedmayer, Y. Deng, X.-J. Liu, S. Chen, and J.-W. Pan, Uncover topology by quantum quench dynamics, Phys. Rev. Lett. 121, 250403 (2018).
  * Gao  _et al._ (2019) C. Gao, H. Zhai, and Z.-Y. Shi, Dynamical fractal in quantum gases with discrete scaling symmetry, Phys. Rev. Lett. 122, 230402 (2019).
  * Gao  _et al._ (2020) C. Gao, M. Sun, P. Zhang, and H. Zhai, Universal dynamics of a degenerate bose gas quenched to unitarity, Phys. Rev. Lett. 124, 040403 (2020).
  * Sun  _et al._ (2020) M. Sun, P. Zhang, and H. Zhai, High temperature virial expansion to universal quench dynamics, Phys. Rev. Lett. 125, 110404 (2020).
  * Rupak (2007) G. Rupak, Universality in a 2-component fermi system at finite temperature, Phys. Rev. Lett. 98, 090403 (2007).
  * Liu  _et al._ (2009) X.-J. Liu, H. Hu, and P. D. Drummond, Virial expansion for a strongly correlated fermi gas, Phys. Rev. Lett. 102, 160401 (2009).
  * Liu  _et al._ (2010) X.-J. Liu, H. Hu, and P. D. Drummond, Three attractively interacting fermions in a harmonic trap: Exact solution, ferromagnetism, and high-temperature thermodynamics, Phys. Rev. A 82, 023619 (2010).
  * Liu (2013) X.-J. Liu, Virial expansion for a strongly correlated fermi system and its application to ultracold atomic fermi gases, Physics Reports 524, 37 (2013), virial expansion for a strongly correlated Fermi system and its application to ultracold atomic Fermi gases.
  * Peng  _et al._ (2011a) S.-G. Peng, S.-Q. Li, P. D. Drummond, and X.-J. Liu, High-temperature thermodynamics of strongly interacting $s$-wave and $p$-wave fermi gases in a harmonic trap, Phys. Rev. A 83, 063618 (2011a).
  * Peng  _et al._ (2014) S.-G. Peng, S.-H. Zhao, and K. Jiang, Virial expansion of a harmonically trapped fermi gas across a narrow feshbach resonance, Phys. Rev. A 89, 013603 (2014).
  * Yan and Blume (2015) Y. Yan and D. Blume, Energy and structural properties of $n$-boson clusters attached to three-body efimov states: Two-body zero-range interactions and the role of the three-body regulator, Phys. Rev. A 92, 033626 (2015).
  * Yan and Blume (2016a) Y. Yan and D. Blume, Path-integral monte carlo determination of the fourth-order virial coefficient for a unitary two-component fermi gas with zero-range interactions, Phys. Rev. Lett. 116, 230401 (2016a).
  * Sun and Cui (2017) M. Sun and X. Cui, Enhancing the efimov correlation in bose polarons with large mass imbalance, Phys. Rev. A 96, 022707 (2017).
  * Nishida (2019) Y. Nishida, Viscosity spectral functions of resonating fermions in the quantum virial expansion, Annals of Physics 410, 167949 (2019).
  * Hou and Drut (2020a) Y. Hou and J. E. Drut, Virial expansion of attractively interacting fermi gases in one, two, and three dimensions, up to fifth order, Phys. Rev. A 102, 033319 (2020a).
  * Hou  _et al._ (2021a) Y. Hou, K. J. Morrell, A. J. Czejdo, and J. E. Drut, Fourth- and fifth-order virial expansion of harmonically trapped fermions at unitarity, Phys. Rev. Res. 3, 033099 (2021a).
  * Sun  _et al._ (2017) M. Sun, H. Zhai, and X. Cui, Visualizing the efimov correlation in bose polarons, Phys. Rev. Lett. 119, 013401 (2017).
  * Kaplan and Sun (2011) D. B. Kaplan and S. Sun, New field-theoretic method for the virial expansion, Phys. Rev. Lett. 107, 030601 (2011).
  * Bourdel  _et al._ (2003) T. Bourdel, J. Cubizolles, L. Khaykovich, K. M. F. Magalhães, S. J. J. M. F. Kokkelmans, G. V. Shlyapnikov, and C. Salomon, Measurement of the interaction energy near a feshbach resonance in a ${}^{6}\mathrm{L}\mathrm{i}$ fermi gas, Phys. Rev. Lett. 91, 020402 (2003).
  * Stewart  _et al._ (2008) J. T. Stewart, J. P. Gaebler, and D. S. Jin, Using photoemission spectroscopy to probe a strongly interacting fermi gas, Nature 454, 744 (2008).
  * Nascimbène  _et al._ (2010) S. Nascimbène, N. Navon, K. J. Jiang, F. Chevy, and C. Salomon, Exploring the thermodynamics of a universal fermi gas, Nature 463, 1057 (2010).
  * Kuhnle  _et al._ (2011) E. D. Kuhnle, S. Hoinka, P. Dyke, H. Hu, P. Hannaford, and C. J. Vale, Temperature dependence of the universal contact parameter in a unitary fermi gas, Phys. Rev. Lett. 106, 170402 (2011).
  * Feld  _et al._ (2011) M. Feld, B. Fröhlich, E. Vogt, M. Koschorreck, and M. Köhl, Observation of a pairing pseudogap in a two-dimensional fermi gas, Nature 480, 75 (2011).
  * Ku  _et al._ (2012) M. J. H. Ku, A. T. Sommer, L. W. Cheuk, and M. W. Zwierlein, Revealing the superfluid lambda transition in the universal thermodynamics of a unitary fermi gas, Science 335, 563 (2012).
  * Mukherjee  _et al._ (2019) B. Mukherjee, P. B. Patel, Z. Yan, R. J. Fletcher, J. Struck, and M. W. Zwierlein, Spectral response and contact of the unitary fermi gas, Phys. Rev. Lett. 122, 203402 (2019).
  * Carcy  _et al._ (2019) C. Carcy, S. Hoinka, M. G. Lingham, P. Dyke, C. C. N. Kuhn, H. Hu, and C. J. Vale, Contact and sum rules in a near-uniform fermi gas at unitarity, Phys. Rev. Lett. 122, 203401 (2019).
  * Ho and Mueller (2004) T.-L. Ho and E. J. Mueller, High temperature expansion applied to fermions near feshbach resonance, Phys. Rev. Lett. 92, 160404 (2004).
  * Yan and Blume (2016b) Y. Yan and D. Blume, Path-integral monte carlo determination of the fourth-order virial coefficient for a unitary two-component fermi gas with zero-range interactions, Phys. Rev. Lett. 116, 230401 (2016b).
  * Yu  _et al._ (2015) Z. Yu, J. H. Thywissen, and S. Zhang, Universal relations for a fermi gas close to a $p$-wave interaction resonance, Phys. Rev. Lett. 115, 135304 (2015).
  * Marcelino  _et al._ (2014) E. Marcelino, A. Nicolai, I. Roditi, and A. LeClair, Virial coefficients for trapped bose and fermi gases beyond the unitary limit: An $s$-matrix approach, Phys. Rev. A 90, 053619 (2014).
  * Hou and Drut (2020b) Y. Hou and J. E. Drut, Virial expansion of attractively interacting fermi gases in one, two, and three dimensions, up to fifth order, Phys. Rev. A 102, 033319 (2020b).
  * Ho  _et al._ (2012) T.-L. Ho, X. Cui, and W. Li, Alternative route to strong interaction: Narrow feshbach resonance, Phys. Rev. Lett. 108, 250401 (2012).
  * Peng  _et al._ (2011b) S.-G. Peng, X.-J. Liu, H. Hu, and S.-Q. Li, Non-universal thermodynamics of a strongly interacting inhomogeneous fermi gas using the quantum virial expansion, Physics Letters A 375, 2979 (2011b).
  * Hou  _et al._ (2021b) Y. Hou, K. J. Morrell, A. J. Czejdo, and J. E. Drut, Fourth- and fifth-order virial expansion of harmonically trapped fermions at unitarity, Phys. Rev. Res. 3, 033099 (2021b).
  * Tajima  _et al._ (2021) H. Tajima, S. Tsutsui, T. M. Doi, and K. Iida, Unitary $p$-wave fermi gas in one dimension, Phys. Rev. A 104, 023319 (2021).
  * Hofmann (2020) J. Hofmann, High-temperature expansion of the viscosity in interacting quantum gases, Phys. Rev. A 101, 013620 (2020).
  * Gao (1998) B. Gao, Quantum-defect theory of atomic collisions and molecular vibration spectra, Phys. Rev. A 58, 4222 (1998).
  * Gao  _et al._ (2005) B. Gao, E. Tiesinga, C. J. Williams, and P. S. Julienne, Multichannel quantum-defect theory for slow atomic collisions, Phys. Rev. A 72, 042719 (2005).
  * Zhai (2021) H. Zhai,  _Ultracold Atomic Physics_ (Cambridge University Press, 2021).

##  I Supplementary material on “Quench Dynamics of Thermal Bose Gases Across
Wide and Narrow Feshbach”

###  I.1 1\. Two-channel square well model

We here solve the two-channel square model to show Feshbach resonances and use
a spherical box potential $V$ with the interaction range of $r_{0}$ to
describe the interaction between two atoms. We consider the $s$-wave
scattering, so that the radial wave function is given by $\Psi=\chi/r$ with
$\Psi$ being the wave function. $\chi$ satisfies the Schrödinger equation,

$\displaystyle-\frac{1}{m}\frac{d^{2}\chi}{dr^{2}}{\mathbb{I}}+V(r)\chi=E\chi,$
(S1) $\displaystyle V(r)=\begin{cases}\begin{bmatrix}-V_{o}&W\\\
W&-V_{c}+\delta\mu_{B}B\end{bmatrix},&\text{for }r<r_{0};\\\
\begin{bmatrix}0&0\\\ 0&\infty\end{bmatrix},&\text{for }r>r_{0}.\end{cases}$
(S2)

Here, $m$ is the mass of particles. $V_{c}$ and $V_{o}$ represent the closed
channel $|c\rangle$ and open channel $|o\rangle$ potential, and $W$ is the
inter-channel coupling strength. $\delta\mu_{B}$ is the magnetic momentum
difference between closed and open channel. ${\mathbb{I}}$ denotes the
$2\times 2$ identity.

At the distance $r>r_{0}$, the wave function is written as

$\displaystyle\chi=A\sin(kr+\delta_{0})|o\rangle.$ (S3)

While, at a distance $r<r_{0}$, those two channels are coupled. We define a
new set of bases $|+\rangle$ and $|-\rangle$ such that the wave function and
the Hamiltonian are diagonalized. Thus, the wave function can be rewritten as
$\chi=\chi_{+}|+\rangle+\chi_{-}|-\rangle$, and the bases $|\pm\rangle$ are
superposition of $|o\rangle$ and $|c\rangle$,

$\displaystyle\begin{bmatrix}|+\rangle\\\
|-\rangle\end{bmatrix}=\begin{bmatrix}\cos\theta&\sin\theta\\\
-\sin\theta&\cos\theta\end{bmatrix}\begin{bmatrix}|o\rangle\\\
|c\rangle\end{bmatrix}.$ (S4)

In the region $r<r_{0}$, the wave function satisfies the boundary condition
$\chi(r=0)=0$, and the solution is given by

$\displaystyle\chi_{\pm}=C\sin\left(\sqrt{m(E-V_{\pm})}r\right),$ (S5)

where
$V_{\pm}=\frac{-V_{c}-V_{o}+\delta\mu_{B}B}{2}\pm\frac{1}{2}\sqrt{(V_{c}-V_{o}-\delta\mu_{B}B)^{2}+4W^{2}}$.

Considering the boundary condition at $r=r_{0}$, the close channel wave
function vanishes and the open channel wave function keeps continuum. With the
solution given in Eq.(S3) and (S5), we obtian

$\displaystyle\frac{k}{\tan\delta_{0}}=\sqrt{m(E-V_{+})}\cot\left(\sqrt{m(E-V_{+})}r_{0}\right)\cos^{2}\theta+\sqrt{m(E-V_{-})}\cot\left(\sqrt{m(E-V_{-})}r_{0}\right)\sin^{2}\theta.$
(S6)

The relation between the scattering amplitude and the T-matrix reads

$\displaystyle
f_{0}(k)=-\frac{1}{ik-k\cot\delta_{0}}=-\frac{m}{4\pi}T_{2}({\bf
k}^{\prime},{\bf k};E),$ (S7)

thus we obtain the two-body scattering T-matrix with the phase shift given in
Eq.(S6).

Next, we consider the bound state energy. At the distance $r>r_{0}$, the wave
function of bound state is written as

$\displaystyle\chi=e^{-k_{\rm b}r}|o\rangle.$ (S8)

At the distance $r<r_{0}$, the same as Eq.(S5), the solution is given by

$\displaystyle\chi_{\pm}=D\sin\left(\sqrt{m(-k_{\rm b}^{2}-V_{\pm})}r\right).$
(S9)

By matching the boundary condition at $r=r_{0}$, we obtain

$\displaystyle-k_{\rm b}=$ $\displaystyle\sqrt{m(-k_{\rm
b}^{2}-V_{+})}\cot\left(\sqrt{m(-k_{\rm
b}^{2}-V_{+})}r_{0}\right)\cos^{2}\theta+\sqrt{m(-k_{\rm
b}^{2}-V_{-})}\cot\left(\sqrt{m(-k_{\rm
b}^{2}-V_{-})}r_{0}\right)\sin^{2}\theta.$ (S10)

The solution $k_{\rm b}$ of Eq.(S10) should be a positive real.

Figure S1:  Feshbach resonances scattering length and the binding energy. (a)
and (b) show the scattering length $a_{s}$ change with magnetic field strength
$B$ under the wide and narrow resonances, respectively. (c) and (e) show
$\sin^{2}\delta_{0}$ change with energy $E$. (d) and (f) show the energy of
bound state, the soild line represent the binding energy given by Eq.(S10),
the red dot-dashed line represent the binding energy given by
$E=-\hbar^{2}/(ma_{s}^{2})$, the blue dashed line in (f) represents another
binding energy under narrow resonances. Here, we choose $V_{c}=20k_{B}T$,
$V_{o}=2.5k_{B}T$, and the coupling strength $W=0.5k_{B}T$ as wide resonances
in (a), (c) and (d), choose $V_{c}=20k_{B}T$, $V_{o}=4.68k_{B}T$,
$W=0.35k_{B}T$ as narrow resonances in (b), (e) and (f).

In order to distinguish wide and narrow resonances, we can define the
parameter $s_{\rm res}$ as

$\displaystyle s_{\rm res}=\frac{a_{\rm
bg}}{r_{0}}\frac{\delta\mu_{B}\Delta}{E_{0}},$ (S11)

where $a_{\rm bg}$ is the background scattering length, and $\Delta$ is the
resonances width. $E_{0}=1/(mr_{0}^{2})$. The background scattering length
$a_{\rm bg}$ is determined by the open channel,

$\displaystyle a_{\rm bg}=\sqrt{-mV_{o}}\cot\left(\sqrt{-mV_{o}}r_{0}\right).$
(S12)

Fig. S1(a) shows the scattering length $a_{s}$ of resonances with $s_{\rm
res}=260$, it diverges at the position of resonance $\delta\mu_{B}B_{\rm
res}=5.2k_{B}T$. Fig. S1(c) shows a broad region of phase shift near the
$B_{\rm res}$ where $\sin^{2}\delta_{0}\approx 1$, and remains stable when the
incoming energy increases. Fig. S1(d) shows the binding energy near threshold,
The solid line represents the binding energy given by Eq.(S10), which is
consistent well with the energy $E=-\hbar^{2}/(ma_{s}^{2})$. In this
resonances, we can also find another bound state energy, but those two energy
difference is 4 or 5 orders of magnitude.

Fig. S1(b) shows the scattering length of narrow resonances with $s_{\rm
res}=0.04$. Fig. S1(f) shows the binding energy near this resonances. First,
we can see the energy given by the exact solution no longer matches the energy
given by the single-channel model. This means that when dealing with a narrow
resonance-related problems, we should use the two-channel model. Second, we
obtain two bound states near the position of resonance, and one of the
energies is neat threshold(solid line). The deeper energy is mainly
contributed by the open channel, which is represented by the dashed line.
Here, we choose $V_{o}=4.68k_{B}T$, thus the bare bound state energy supported
by the open channel is $E_{\rm b}=0.75k_{B}T$, which is close to the deeper
binding energy.

###  I.2 2\. T-matrix of two-channel zero range model

In this section, we derive the T-matrix of two-channel zero range model. With
a contact potential, the second-quantized Hamiltonian in the momentum space
can be written as

$\displaystyle\mathcal{\hat{H}}=$
$\displaystyle\underset{\mathbf{k}\sigma}{\sum}\frac{\mathbf{k}^{2}}{2m}\hat{\Psi}_{\mathbf{k}\sigma}\hat{\Psi}_{\mathbf{k}\sigma}+\underset{\mathbf{k}}{\sum}\left(\frac{\mathbf{k}^{2}}{4m}+\nu\right)\hat{b}^{\dagger}_{\mathbf{k}}\hat{b}_{\mathbf{k}}+\frac{g}{V}\underset{\mathbf{k},\mathbf{k}_{1},\mathbf{k}_{2}}{\sum}\Psi^{\dagger}_{\frac{\mathbf{k}}{2}+\mathbf{k}_{1,\uparrow}}\Psi^{\dagger}_{\frac{\mathbf{k}}{2}-\mathbf{k}_{1,\downarrow}}\Psi_{\frac{\mathbf{k}}{2}-\mathbf{k}_{2,\downarrow}}\Psi_{\frac{\mathbf{k}}{2}+\mathbf{k}_{2,\uparrow}}$
(S13)
$\displaystyle+\frac{\alpha}{\sqrt{V}}\underset{\mathbf{k},\mathbf{k}_{1}}{\sum}\Psi^{\dagger}_{\frac{\mathbf{k}}{2}+\mathbf{k}_{1,\uparrow}}\Psi^{\dagger}_{\frac{\mathbf{k}}{2}-\mathbf{k}_{1,\downarrow}}\hat{b}_{\mathbf{k}}+\hat{b}^{\dagger}_{\mathbf{k}}\Psi_{\frac{\mathbf{k}}{2}-\mathbf{k}_{1,\downarrow}}\Psi_{\frac{\mathbf{k}}{2}+\mathbf{k}_{1,\uparrow}}.$

Here, $g$ is the bare interaction between open channel atoms themselves,
$\Psi^{\dagger}_{\sigma}$ and $\Psi_{\sigma}$ are the creation and
annihilation operators for scattering states in the open channels and $V$ is
the volume of the system. $\hat{b}^{\dagger}$ and $\hat{b}$ are the creation
and annihilation operators of the two-body bound state in the closed channel,
and $\nu$ is the detuning of the molecular state in the closed channel. The
last term denotes the conversion between the open channel scattering states
and the closed channel molecular state, with the strength given by $\alpha$.
The ladder diagram for the two-channel model is shown in Fig. S2. The
summation of the ladder diagram leads to the Schwinger-Dyson equation

Figure S2:  Ladder diagrams for two-body scattering T-matrix of the two-
channel model.

$\displaystyle
T_{2}(E)=g+\frac{|\alpha|^{2}}{E-\nu}+\left(g+\frac{|\alpha|^{2}}{E-\nu}\right)\frac{1}{V}\underset{\mathbf{k}}{\sum}\frac{1}{E-\mathbf{k}^{2}/m}T_{2}(E),$
(S14)

which leads to

$\displaystyle
T_{2}(E)=\frac{g+\frac{|\alpha|^{2}}{E-\nu}}{1-\left(g+\frac{|\alpha|^{2}}{E-\nu}\right)\frac{1}{V}\underset{\mathbf{k}}{\sum}\frac{1}{E-\mathbf{k}^{2}/m}}.$
(S15)

Notice that the T-matrix given in Eq.(S15) is renormalizable. This two-body
T-matrix should be related to the $s$-wave scattering amplitude, therefore, we
have

$\displaystyle T_{2}(E=0)=\frac{4\pi}{m}f_{0}(k=0)=\frac{4\pi}{m}a_{\rm
bg}\left(1-\frac{\Delta}{B-B_{\rm res}}\right).$ (S16)

We also define $\nu=\delta\mu_{B}(B-B_{\rm res})+\nu_{p}$. When detuning the
magnetic field away from the resonance position $|B-B_{\rm res}|\gg|\Delta|$,
we have

$\displaystyle T_{2}(E=0)=\frac{4\pi}{m}a_{\rm
bg}=\frac{1}{\frac{1}{g}+\frac{1}{V}\underset{\mathbf{k}}{\sum}\frac{1}{\mathbf{k}^{2}/m}}.$
(S17)

Hence, we reach the renormalization identity that relates $g$ to physical
quantity $a_{\rm bg}$,

$\displaystyle\frac{1}{g}=\frac{m}{4\pi a_{\rm bg}}-\Lambda,$ (S18)

where $\Lambda$ denotes

$\displaystyle\Lambda=\frac{1}{V}\underset{\mathbf{k}}{\sum}\frac{1}{\mathbf{k}^{2}/m}.$
(S19)

By comparing Eq.(S15) with Eq.(S16), and using the relation between $g$ and
$a_{\rm bg}$, the remainder renormalization conditions can be given by

$\displaystyle\frac{1}{\alpha}=\left(1-\frac{4\pi a_{\rm
bg}}{m}\Lambda\right)\sqrt{\frac{m}{4\pi a_{\rm bg}\delta\mu_{B}\Delta}},$
(S20) $\displaystyle\nu=\delta\mu_{B}(B-B_{\rm
res})+\frac{\Lambda}{1-\frac{4\pi a_{\rm bg}}{m}\Lambda}\frac{4\pi a_{\rm
bg}\delta\mu_{B}\Delta}{m}.$ (S21)

By substituting Eq.(S18,S20,S21) into Eq.(S15), we have

$\displaystyle T_{2}=\frac{4\pi/m}{\frac{E-\delta\mu_{B}(B-B_{\rm
res})}{a_{\rm bg}\left[E+\delta\mu_{B}\Delta-\delta\mu_{B}(B-B_{\rm
res})\right]}-\sqrt{-mE}}.$ (S22)


# Generating Bug-Fixes Using Pretrained Transformers

Dawn Drain
Microsoft
<EMAIL_ADDRESS>Wu
Microsoft
<EMAIL_ADDRESS>Svyatkovskiy
Microsoft
<EMAIL_ADDRESS>Sundaresan
Microsoft
<EMAIL_ADDRESS>
(April 2021)

###### Abstract

Detecting and fixing bugs are two of the most important yet frustrating parts
of the software development cycle. Existing bug detection tools are based
mainly on static analyzers, which rely on mathematical logic and symbolic
reasoning about the program execution to detect common types of bugs. Fixing
bugs is typically left out to the developer. In this work we introduce
DeepDebug: a data-driven program repair approach which learns to detect and
fix bugs in Java methods mined from real-world GitHub repositories. We frame
bug-patching as a sequence-to-sequence learning task consisting of two steps:
(i) denoising pretraining, and (ii) supervised finetuning on the target
translation task. We show that pretraining on source code programs improves
the number of patches found by 33% as compared to supervised training from
scratch, while domain-adaptive pretraining from natural language to code
further improves the accuracy by another 32%. We refine the standard accuracy
evaluation metric into non-deletion and deletion-only fixes, and show that our
best model generates 75% more non-deletion fixes than the previous state of
the art. In contrast to prior work, we attain our best results when generating
raw code, as opposed to working with abstracted code that tends to only
benefit smaller capacity models. Finally, we observe a subtle improvement from
adding syntax embeddings along with the standard positional embeddings, as
well as with adding an auxiliary task to predict each token’s syntactic class.
Despite focusing on Java, our approach is language agnostic, requiring only a
general-purpose parser such as tree-sitter111https://tree-
sitter.github.io/tree-sitter/.

##  1 Introduction

Early results in automated bug-patching assumed a set of test functions and
took the approach of generating patches until one passed the test suite. The
edits were operationalized as abstract-syntax tree (AST) manipulations such as
deleting, inserting, or swapping nodes. The seminal work GenProg Le Goues et
al. (2012) combined these operations with a genetic algorithm and claimed to
fix 55 out of 105 bugs in 5.1m LOC across eight projects with over 10k test
cases. However, a followup paper Qi et al. (2015) found that 53 out of the 55
claimed fixes were merely test-suite adequate, and that most of them were
generated simply using the deletion operator.

More recent work has explored patch generation using a more stringent
criterion: produce exactly the same fix as the developer who originally fixed
the bug. We use the Java dataset from Patches in the Wild Tufano et al.
(2019), which is comprised of short methods extracted from commits whose
messages indicate they are bug-fixing. Copy That! Panthaplackel et al. (2020)
applies a novel span-copying mechanism on this dataset, achieving impressive
top-1 accuracy. SequenceR Chen et al. (2018) both narrows and expands on the
Patches in the Wild dataset by focusing on one-line changes only, while also
providing additional context beyond the buggy method by including other
methods’ signatures. This extended context gives a 15% relative boost. All
three approaches see large gains from copying, which is sensible given that
the fixed method has a large overlap with the buggy method. ENCORE Lutellier
et al. (2019) also looks at one-line changes, and in more languages, although
they give only the buggy line as input and do not provide any further context.

For certain bug-types it is possible to generate millions of synthetic bugs to
train on. Devlin et al. Devlin et al. (2017) train an RNN on Python to fix
incorrect comparison operators, the mistaken use of “is” vs. “is not”,
variable misuse, and forgotten “self” accessors. Overall, they achieve 86%
accuracy on synthetic bugs and 41% on real-life bugs. Kanade et al. Kanade et
al. (2019) pretrain a BERT-style model “CuBERT” on a larger Python dataset and
then finetune on a related suite of synthetic bugs, achieving over 90%
accuracy.

In the broader field of NLP, our task is closest in spirit to grammatical
error correction (GEC). Drawing from the CoNLL GEC leaderboard Ng et al.
(2014), we see several different approaches using transformers. Kiyano et al.
see large gains from data augmentation via noisy backtranslation, in addition
to more standard techniques like injecting spelling errors, applying left-to-
right reranking, and using a sentence-level error detection task Kiyono et al.
(2019). Kaneko et al. use a seq2seq model where the input embeddings are
concatenated with the embeddings produced by a BERT finetuned on grammatical
error detection (detecting whether a given token is faulty) Kaneko et al.
(2020). Zhao et al. Zhao et al. (2019) apply a copy-mechanism, in addition to
injecting artificial noise and using an auxiliary token-labeling task. Awasthi
et al. Awasthi et al. (2019) pursue an encoder-only approach, which maps token
$z_{i}$ to the edit operation to perform at the $i^{th}$ location. They see
large speed gains, and also investigate applying their model iteratively.

Expanding more broadly, still, various task-agnostic pretraining approaches
like BERT, BART, and T5 Devlin et al. (2017); Lewis et al. (2020); Raffel et
al. (2019) have seen large performance gains on composite benchmarks like GLUE
Wang et al. (2018). These models are typically pretrained using a denoising
objective on a large corpus of synthetically corrupted text.

We retain the scope of work trying to fix generic bugs while taking advantage
of techniques for pretraining powerful sequence-to-sequence transformers. We
first treat raw code as text and use a span-masking objective to pretrain a
400 million parameter encoder-decoder transformer on 67k Java repositories
mined from GitHub before finetuning on the Patches in the Wild benchmark. More
specifically, we consider three pretraining experiments: pretrain on Java
only; finetune directly from the strong baseline BART which was pretrained on
English; and further pretrain on Java with a warmstart from BART. Pretraining
on Java alone improves the number of patches found by a third, from 1049/12380
to 1392/12380, while pretraining with a warmstart further improves by another
third to 1839/12380. We refine the standard top-1 accuracy metric used for
this benchmark into non-deletion and deletion-only fixes, and show that our
best DeepDebug model generates 75% more non-deletion fixes than the previous
state of the art. In contrast to prior work, we attain our best results when
generating raw code, as opposed to working with abstracted code that tends to
benefit smaller models. Finally, we see a subtle improvement from using the
formal grammar unique to programming languages. Specifically, we experiment
with adding syntax embeddings along with the standard positional embeddings,
as well as with adding an auxiliary task to predict each token’s syntactic
class.

##  2 Problem overview

The task of patch generation is as follows: given a buggy Java method (found
by mining commit messages), produce the exact same sequence of tokens as the
developer who committed the fix.

Example Found Fix ⬇ double a_ods_light_Detected() {  double l_return = 0.0;
if ((v_sensor_ods) != null) {  v_sensor_ods.getLightDetected();  }  return
l_return;  }  ⬇ double a_ods_light_Detected() {  double l_return = 0.0;  if
((v_sensor_ods) != null) {  l_return |\colorbox{green}{=}|
v_sensor_ods.getLightDetected();  }  return l_return;  }  Figure 1: Our model
makes sure to capture the value returned by the getter method.

During finetuning, we train our sequence-to-sequence transformer so as to
minimize the expected log-probability of generating the fix given the buggy
method. Formally, let $B=\\{b_{1},\dots,b_{n}\\}$ and
$C=\\{c_{1},\dots,c_{n}\\}$ be the sets of buggy and correct functions
respectively, where we can tokenize $c_{i}=c_{i,1},\dots,c_{i,k_{i}}$. Let
$\theta$ denote the model parameters. The objective is thus:

$\begin{split}\text{argmin}_{\theta}\mathbb{E}_{b_{i}\in
B}\text{log}&[P(c_{i,1}|b_{i};\theta)\cdot P(c_{i,2}|b_{i},c_{i,1};\theta)\\\
&\cdots P(c_{i,k_{i}}|b_{i},c_{1},\dots,c_{k_{i}-1};\theta)].\end{split}$

More simply, we seek

$\text{argmin}_{\theta}\mathbb{E}_{b_{i}\in
B}\text{log}P(c_{i}|b_{i};\theta).$

Example Found Fix ⬇ private double normalizeTime(double time,
ec.graph.GraphInitializer init) {  if (((init.maxTime) - (init.minTime)) ==
0.0)  return 1.0;  else return |\colorbox{red}{((init.maxTime) - time)}|  /
((init.maxTime) - init.minTime);  }  ⬇ private double normalizeTime(double
time,  ec.graph.GraphInitializer init) {  if (((init.maxTime) -
(init.minTime)) == 0.0)  return 1.0;  else return |\colorbox{green}{(time -
(init.minTime))}|  / ((init.maxTime) - init.minTime));  }  Figure 2: A clever
fix found by our model. It manipulates the return value so as to normalize
minTime to 0 and maxTime to 1, whereas the developer originally had the
boundary values reversed.

##  3 Data

We evaluate on the Java dataset from Patches in the Wild Tufano et al. (2019).
Tufano et al. mined all the public, non-fork repositories on Github up until
2016 for commits to Java code that include a commit message containing at
least one of the following words: bug, error, issue or fix, patch, correct,
which comprises about 5% of all commits. From these commits they extracted all
modified methods with at most one hundred tokens, and broke them into two
datasets: a dataset of small methods containing less than 50 tokens and a
complementary dataset of medium methods containing up to 100 tokens.

To facilitate learning and to hasten convergence, Tufano et al. abstracted the
code snippets while preserving necessary syntactical information. An example
is given in figure 3. To abstract the code snippets, they replaced all tokens
outside a whitelist of approximately 500 commonly occurring tokens with
abstractions of the form METHODi, VARIABLEi, STRING_LITi, NUM_LITi, and TYPEi.
Finally, they introduce idioms – frequently occurring programming language
identifiers and literals, to which abstraction is not applied. Figure 3
illustrates differences between these code representations.

Figure 3: Raw code, fully abstracted code, and code abstracted with idioms. We
find that smaller models perform best on abstracted code, whereas higher-
capacity models are better able to use the semantics associated to concrete
code.

As part of normalization the authors of the dataset also stripped comments and
irrelevant whitespaces and indentation. In this paper, we present results both
on the concrete code, and abstracted code with idioms. We obtain superior
results on concrete code, especially after pretraining.

##  4 Baseline

Abstracting code removes the valuable semantics of variable and function
names, and insisting on the verbatim fix is very strict, but it is
nevertheless possible to obtain remarkably high accuracy under these
conditions. Using a 10 million parameter bidirectional LSTM with copy
mechanism and the default hyperparameters from Google’s Seq2Seq Tufano et al.
were able to get 9.2% exact accuracy on small methods with less than 50
tokens, and 3.2% accuracy on medium methods with between 50 and 100 tokens.
Even more impressively, Copy That! Panthaplackel et al. (2020) got 17.7% and
8.0% accuracy respectively using their bi-GRU implementation with a novel,
span-copying mechanism.

We speculate that 17.7% and 8.0% are not far from the best practical top-1
accuracy on this task. “Bug-fixing commits” mined based on their commit
message tend to be very diverse and noisy, in contrast to methods that are
truly known to contain a bug, especially from a small class of synthetic bugs.

##  5 Model

Our DeepDebug model uses the standard transformer Vaswani et al. (2017)
architecture. Copy mechanisms have been shown to give a dramatic boost to
small LSTM models Chen et al. (2018), and to give a small boost to
transformers for especially relevant tasks like summarization See et al.
(2017) and grammar correction Zhao et al. (2019). We use copy-attention when
training our 60 million parameter models from scratch and when replicating the
state of the art. We do not use copy attention in our full-scale pretraining
experiments, due primarily to technical limitations.

We use a Byte Pair Encoding (BPE) vocabulary of roughly 50k tokens for
concrete code and a vocabulary of all 433 unique tokens for abstracted code
Sennrich et al. (2016). In order to form the BPE vocabulary, we take the
existing byte-level BPE tokenizer utilized by each of the GPT-2 Radford and Wu
(2019), BART, and RoBERTa, and append whitespace tokens in the order that they
are learned by a tokenizer trained from scratch on raw code. Adding whitespace
tokens effectively increases the context window length and processing speed by
around 40% during pretraining, although the improvement is smaller during
finetuning as we restrict our focus to left-adjusted methods. Extending the
pre-existing tokenizer allows us to reuse public checkpoints trained on
English while efficiently adapting them to source code.

We experiment with adding syntactic embeddings along with the standard
vocabulary and positional embeddings, as demonstrated in figure 4. In this way
we immediately distinguish whether BPE tokens like “return” or “static” are
key words. In general we relieve the model of some of the difficulty of
parsing the language’s formal grammar, which is disproportionately difficult
for transformers Hahn (2020). We can also think of this tactic as subsuming
both the approach of using raw code and the approach of using abstracted code.

Figure 4: We experiment with adding syntax embeddings along with the standard
positional and vocabulary embeddings during encoding. We find that this gives
a small benefit when training from scratch, but no benefit when the syntax
embeddings are added only during finetuning.

##  6 Pretraining

We use the same span-masking objective used by T5 Raffel et al. (2019). That
is, we randomly select spans of three consecutive tokens and replace each span
with a single mask token. The goal is to reproduce the masked tokens in order,
reusing the mask token in the target by putting it before every span of three
tokens. We take advantage of the fact that code is lower-entropy than English.
For instance, GPT-C attains perplexities in the range of 1.3 – 2 for several
programming languages, whereas a comparably sized GPT-2 model attains a
perplexity of 26 on Wikitext-3 Svyatkovskiy et al. (2020); Radford and Wu
(2019). Thus we mask a larger fraction of 30% of all tokens. Masking a larger
fraction also has the added advantage of increasing the effective context
length during pretraining, by about 50% in our case.

Span-masking Input and Output ⬇ private double
normalizeTime(<mask>.GraphInitializer init) {  if (((init.maxTime) -
<mask>0.0)  return 1.0;  else <mask> (init.minTime)) /  ((init.maxTime) -
(init.minTime));  }  ⬇ <mask>double time, ec.graph <mask> (init.minTime)) ==
<mask> return (time \-  Figure 5: Example of span-masking used when
pretraining our sequence-to-sequence model. We reuse a single mask token and
cover exactly three tokens with each mask.

We pretrain on all open-source Java repositories hosted on GitHub with at
least 10 stars that have been updated in the last five years, 67,000
repositories in total. We deduplicate files based on file hash. Finally, we
filter out files based on heuristics like if they contain data or have been
auto-generated, and clean files by removing licenses and non-ASCII characters.
This final stage of filtering and cleaning removes 10% of the remaining
tokens. After taking these steps, we are left with 67k repositories, 8 million
files, 1.3 billion lines of code, 13 billion tokens, or 54GB.

We pretrain for ten epochs, which takes one week on a DGX-2 box, which has
sixteen 32GB V100 GPUs. We estimate this is approximately 4% of the resources
used to pretrain the public English BART checkpoint released by Facebook,
which was pretrained for forty epochs on 160GB using a more compute-intensive
pretraining objective.

##  7 Experiments

We first examine training on concrete code as compared to abstracted code
(with the 433 most common tokens added as idioms) from scratch using a 60
million parameter transformer with a copy mechanism as implemented in the
OpenNMT framework Klein et al. (2017).

We then experiment with several pretraining strategies implemented in the
Fairseq framework Ott et al. (2019). Specifically, we consider three following
experiments: i) pretrain on Java only, ii) finetune directly from the strong
baseline BART which was pretrained on English; and iii) domain-adaptive
pretraining on Java with a warmstart from BART. We hypothesized that
warmstarting from English would help, since learning how to program is much
easier if one already knows English and has an intuitive understanding of
concepts like dictionaries, graphs, or lists, and can read out these roles
from self-documenting method and variable names.

We next investigate the effect of adding an auxiliary token-type labeling loss
on top of the standard next-token prediction loss, as demonstrated in Figure
6. We use the same token types used during abstraction i.e. METHOD, VARIABLE,
STRING_LIT, and NUM_LIT. Syntax highlighting is a ubiquitous developer tool,
and especially helpful during the early stages of learning a language before
the developer has internalized its formal grammar.

Figure 6: We experiment with adding an auxiliary token-type classification
loss for multitask training. In this condition, the loss is a weighted sum of
the standard next-token prediction loss and the auxiliary loss of predicting
whether each subword token is a part of a TYPE, METHOD, VARIABLE, STRING_LIT,
NUM_LIT, or other token.

Our final experiment more closely investigates syntax highlighting. During
encoding, we add a token-type embedding to the standard positional and
vocabulary embeddings, as demonstrated in figure 4. For both of these token-
type experiments we find a small improvement when training from scratch, but
no improvement when adding them during finetuning only. We also partially
replicate and extend the work from Copy That!, by evaluating on small methods
on both concrete and fully abstracted code without idioms. We use the code for
the specialized span-copying GRU model generously shared by the author.

##  8 Results

We found that many of the fixes fell into a few uninspiring patterns,
especially given the limited context of the methods: Simple deletions, as with
classic models like GenProg; swapping “protected”, “private”, and “public”;
and inserting “native”.

Indeed, roughly two-thirds of all the fixes we find on first attempt are
simply deletions. For our strongest models, 7% of the non-deletion fixes are
some kind of protected/private/public swapping or inserting the word “native”,
while the fraction is as high as 50% for the weakest models we consider. In
presenting our results, we thus also give the number of fixes that were simply
deletions and the complementary number that are non-deletion fixes.

Table 1: Transformer Results for Concrete vs. Abstract code. Contrary to the
results for smaller GRU-based models, we see that training on concrete code
helps.

| Non-deletion Fixes | All Fixes | All Methods | Deletion Fixes
---|---|---|---|---
| abstract with idioms medium,
---
from scratch,
60M parameters
11 (0.2%) | 178 (2.7%) | 6545 | 166
| concrete medium,
---
from scratch,
60M parameters
20 (0.3%) | 244 (3.7%) | 6545 | 224
| abstract with idioms small,
---
from scratch,
60M parameters
91 (1.6%) | 648 (11.1%) | 5835 | 557
| concrete small,
---
from scratch,
60M parameters
349 (6.0%) | 854 (14.6%) | 5835 | 505

Table 2: GRU Span-copying Results for Small Java Methods, our replication of
the previous state of the art Panthaplackel et al. (2020). Our pretrained
model finds a slightly higher total number of fixes, and 75% more non-deletion
fixes.

| Non-deletion Fixes | All Fixes | All Methods | Deletion Fixes
---|---|---|---|---
fully abstract small, |  |  |  |
from scratch, |  |  |  |
250K parameters | 207 (3.5%) | 991 (17.0%) | 5835 | 784
| concrete small,
---
from scratch,
250K parameters
109 (1.5%) | 537 (9.2%) | 5835 | 428

Table 3: GRU Classical Copy Mechanism Results for Small Java Methods, model
also provided by Panthaplackel et al. (2020). Unlike the transformers, the
smaller, GRU-based models did not benefit from training on concrete code; on
abstracted code they achieved both higher top-1 performance and a better ratio
of non-deletion to deletion fixes.

| Non-deletion Fixes | All Fixes | All Methods | Deletion Fixes
---|---|---|---|---
fully abstract small, |  |  |  |
from scratch, |  |  |  |
250K parameters | 125 (2.1%) | 672 (11.5%) | 5835 | 547
| concrete small,
---
from scratch,
250K parameters
50 (0.9%) | 397 (6.8%) | 5835 | 347

Example Found Fix ⬇ public void backward() {  switch (heading) {  case "N" :
|\colorbox{red}{(x)++}|;  default :  }  }  ⬇ public void backward() {  switch
(heading) {  case "N" :  |\colorbox{green}{(y)–}|;  break;  default :  break;
}  }  Figure 7: Our model infers from the function name “backward” that the
function should be decrementing rather than incrementing.

When considering fixes that our model did not find, the overriding theme is
that the task is impossible without more context. In theory, bug-fixing should
be nearly deterministic; code can be edited in nearly arbitrary ways while
implementing or optimizing additional functionality, but a bug is a glaring
error that needs to be addressed.

In practice, the correct behavior is often underspecified, especially since
our approach excludes useful information such as test cases, example usages,
and the class constructor. For example, consider the function pop() from
figure 8, which our model did not fix on its first attempt. Off-by-one errors
are common. However, given only the buggy method, it is ambiguous whether the
head index should be bounded below by zero or negative one.

Example Fix not Found on First Attempt ⬇ public G pop() {  G output =
storage[i_head];  if ((i_head) |\colorbox{red}{>}| 0) {  i_head = (i_head) -
1;  }else {  }  return output }  ⬇ public G pop() {  G output =
storage[i_head];  if ((i_head) |\colorbox{green}{>=}| 0) {  i_head = (i_head)
- 1;  }else {  }  return output }  Figure 8: It is ambiguous whether this off-
by-one error is a bug without seeing the definitions of storage or i_head.

##  9 Analysis

In this section, we draw several conclusions from the results of the
experiments presented in section 8.

First of all, we see that producing fixes for medium-sized methods is harder
than for short methods. This is unsurprising since longer methods are more
complicated and can be edited in more ways.

Our next observation is that pretraining helps, especially for medium methods.
Indeed, the number of medium fixes found increases by a factor of three when
adding pretraining.

We see that the smaller GRU models and the transformers trained on abstract
code with idioms are especially destructive, producing seven times as many
deletions as constructive fixes. The smaller GRU models benefit from
abstracting code, although the larger transformers do not.

Pretraining on English boosts performance of program repair, presumably due to
the semantics inherent in variable and function names. A function name like
getMinElement is intuitive even for a non-programmer. Indeed, the public BART
checkpoint, which was pretrained only on English, outperforms our model
pretrained only on Java, although our Java-only model was trained for only 4%
as long as the English-only checkpoint. Our best results are obtained by
taking the powerful English model and further pretraining it on Java.

We observe that adding an auxiliary syntax-labeling task helps somewhat when
training from scratch, but makes no difference when we finetune the pretrained
model. We hypothesize this is because the pretrained model already has a
strong understanding of the language’s syntax, and ample syntactical feedback
is already provided by the standard next-token prediction task.

We see underwhelming returns in scaling from 60 million to 400 million
parameters when training from scratch, especially in light of the dramatic and
predictable gains from scaling up transformers such as T5 or GPT Raffel et al.
(2019); Kaplan et al. (2020). We hypothesize that this is because of the small
scale of training data. Each of our training sets contains only around 50k
examples comprising mere tens of megabytes, whereas BART-large has 400 million
parameters and is nearly a full gigabyte when loaded in FP16.

Finally, similarly to adding the syntax-labeling task, we see a small benefit
from adding syntax highlighting in the form of adding token-type embeddings to
the vocabulary embeddings during encoding, but no benefit from adding those
embeddings during finetuning. It makes sense that there should be some small
benefit from syntax highlighting. We hypothesize that the lack of benefit from
adding highlighting during finetuning is due to the pretrained model already
having developed a way to represent different token types, which clashes with
the added embeddings.

Table 4: Transformer results for different pretraining workflows on medium
methods. The best result is obtained by first pretraining on English and then
on Java. Note that English pretraining used twenty-five times as much compute
as Java pretraining.

| Non-deletion Fixes | All Fixes | All Methods | Deletion Fixes
---|---|---|---|---
| concrete medium,
---
from scratch,
400M parameters
72 (1.1%) | 233 (3.6%) | 6545 | 161
| concrete medium,
---
Java pretrain only,
400M parameters
98 (1.5%) | 413 (6.3%) | 6545 | 315
| concrete medium,
---
English pretrain only,
400M parameters
143 (2.2%) | 439 (6.7%) | 6545 | 296
| concrete medium,
---
English plus Java,
400M parameters
259 (4.0%) | 749 (11.4%) | 6545 | 490

Table 5: Transformer results for different pretraining workflows on small
methods. The English BART baseline is very strong, although the best
performance on the standard top-1 accuracy metric is still obtained by further
pretraining on Java.

| Non-deletion Fixes | All Fixes | All Methods | Deletion Fixes
---|---|---|---|---
| concrete small,
---
from scratch,
400M parameters
328 (5.6%) | 816 (13.9%) | 5835 | 488
| concrete small,
---
Java pretrain only,
400M parameters
330 (5.7%) | 979 (16.8%) | 5835 | 649
| concrete small,
---
English pretrain only,
400M parameters
385 (6.6%) | 976 (16.7%) | 5835 | 591
| concrete small,
---
English plus Java,
400M parameters
361 (6.2%) | 1090 (18.7%) | 5835 | 729

Table 6: Effects of adding a syntax-type labeling task and of adding syntax-
type embeddings during encoding. There is a small improvement for multitask
training from scratch, but no improvement for adding the additional task
during finetuning, presumably because the labeling task is too easy for the
pretrained model. There is similarly a small benefit from adding syntax
embeddings when training from scratch, perhaps the lack of benefit during
finetuning is because the added embeddings clash with the pretrained model’s
representation of different token types.

| Non-deletion Fixes | All Fixes | All Methods | Deletion Fixes
---|---|---|---|---
| concrete small,
---
from scratch,
400M parameters
328 (5.6%) | 816 (13.9%) | 5835 | 488
| concrete small,
---
from scratch,
400M parameters,
syntax token embedding
343 (5.9%) | 847 (14.4%) | 5835 | 504
| concrete small,
---
from scratch,
400M parameters,
syntax token prediction
340 (5.8%) | 862 (14.7%) | 5835 | 522
| concrete small,
---
English plus Java,
400M parameters,
361 (6.2%) | 1090 (18.7%) | 5835 | 729
| concrete small,
---
English plus Java,
400M parameters,
syntax token embedding
355 (6.1%) | 1082 (18.5%) | 5835 | 727
| concrete small,
---
English plus Java,
400M parameters,
syntax token prediction
357 (6.1%) | 1097 (18.8%) | 5835 | 740

Table 7: Comparison bewteen prior SOTA results and our best model, which was
warmstarted from BART and then further pretrained on Java. In table 2 we
partially replicate Panthaplackel et al.’s small copy-span model and find that
they only produce 57% as many non-deletion fixes as our approach.

| Medium Fixes | Small Fixes | Overall | Parameters
---|---|---|---|---
| Tufano et al.
---
Tufano et al. (2019)
209 (3.2%) | 537 (9.2%) | 746 (6.0%) | 10M
| Panthaplackel et al.
---
Panthaplackel et al. (2020)
524 (8.0%) | 1033 (17.7%) | 1557 (12.6%) | 0.25M
| CodeBERT
---
Feng et al. (2020)
340 (5.2%) | 957 (16.4%) | 1297 (10.5%) | 180M
| BART
---
Lewis et al. (2020)
439 (6.7%) | 976 (16.7%) | 1415 (11.4%) | 400M
| DeepDebug
---
(Ours)
749 (11.4%) | 1090 (18.7%) | 1839 (14.9%) | 400M

##  10 Conclusions and Future Work

We introduced DeepDebug, a data-driven program repair approach which learns to
fix bugs in real-world code using pretrained transformers. We achieve new
state of the art by following a multi-stage pretraining pipeline, in which we
warmstart from a public English checkpoint, further pretrain on code, and
finally finetune on the target task. We show that these large transformers are
able to take advantage of meaningful function and variable names, unlike
smaller GRU models which perform better on abstracted code. We also
investigate the extent to which our fixes are constructive, rather than mere
deletions, and find that we create 75% more non-deletion fixes than the
previous state of the art.

In future work we are interested in expanding to multiple languages and
considering iterative fixes.

## References

  * Awasthi et al. (2019) Abhijeet Awasthi, Sunita Sarawagi, Rasna Goyal, Sabyasachi Ghosh, and Vihari Piratla. 2019.  Parallel iterative edit models for local sequence transduction.  _arXiv preprint arXiv:1910.02893_.
  * Chen et al. (2018) Zimin Chen, Steve Kommrusch, Michele Tufano, Louis-Noël Pouchet, Denys Poshyvanyk, and Martin Monperrus. 2018.  Sequencer: Sequence-to-sequence learning for end-to-end program repair.
  * Devlin et al. (2017) Jacob Devlin, Jonathan Uesato, Rishabh Singh, and Pushmeet Kohli. 2017.  Semantic code repair using neuro-symbolic transformation networks.
  * Feng et al. (2020) Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020.  CodeBERT: A pre-trained model for programming and natural languages.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1536–1547, Online. Association for Computational Linguistics.
  * Hahn (2020) Michael Hahn. 2020.  Theoretical limitations of self-attention in neural sequence models.  _Transactions of the Association for Computational Linguistics_ , 8:156–171.
  * Kanade et al. (2019) Aditya Kanade, Petros Maniatis, Gogul Balakrishnan, and Kensen Shi. 2019.  Pre-trained contextual embedding of source code.
  * Kaneko et al. (2020) Masahiro Kaneko, Masato Mita, Shun Kiyono, Jun Suzuki, and Kentaro Inui. 2020.  Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction.  _arXiv e-prints_ , page arXiv:2005.00987.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.
  * Kiyono et al. (2019) Shun Kiyono, Jun Suzuki, Masato Mita, Tomoya Mizumoto, and Kentaro Inui. 2019.  An empirical study of incorporating pseudo data into grammatical error correction.  _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
  * Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017\.  OpenNMT: Open-source toolkit for neural machine translation.  In _Proceedings of ACL 2017, System Demonstrations_ , pages 67–72, Vancouver, Canada. Association for Computational Linguistics.
  * Le Goues et al. (2012) Claire Le Goues, Michael Dewey-Vogt, Stephanie Forrest, and Westley Weimer. 2012\.  A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each.  In _Proceedings of the 34th International Conference on Software Engineering_ , ICSE ’12, page 3–13. IEEE Press.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lutellier et al. (2019) Thibaud Lutellier, Lawrence Pang, Viet Hung Pham, Moshi Wei, and Lin Tan. 2019.  Encore: Ensemble learning using convolution neural machine translation for automatic program repair.
  * Ng et al. (2014) Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014.  The CoNLL-2014 shared task on grammatical error correction.  In _Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task_ , pages 1–14, Baltimore, Maryland. Association for Computational Linguistics.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.
  * Panthaplackel et al. (2020) Sheena Panthaplackel, Miltiadis Allamanis, and Marc Brockschmidt. 2020.  Copy that! editing sequences by copying spans.  _arXiv preprint arXiv:2006.04771_.
  * Qi et al. (2015) Zichao Qi, Fan Long, Sara Achour, and Martin Rinard. 2015.  An analysis of patch plausibility and correctness for generate-and-validate patch generation systems.  In _Proceedings of the 2015 International Symposium on Software Testing and Analysis_ , ISSTA 2015, page 24–36, New York, NY, USA. Association for Computing Machinery.
  * Radford and Wu (2019) Alec Radford and Jeffrey Wu. 2019.  Rewon child, david luan, dario amodei, and ilya sutskever. 2019.  _Language models are unsupervised multitask learners_.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.
  * See et al. (2017) Abigail See, Peter J Liu, and Christopher D Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  _arXiv preprint arXiv:1704.04368_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Svyatkovskiy et al. (2020) Alexey Svyatkovskiy, Shao Kun Deng, Shengyu Fu, and Neel Sundaresan. 2020.  Intellicode compose: Code generation using transformer.  _Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_.
  * Tufano et al. (2019) Michele Tufano, Cody Watson, Gabriele Bavota, Massimiliano Di Penta, Martin White, and Denys Poshyvanyk. 2019.  An empirical study on learning bug-fixing patches in the wild via neural machine translation.  _ACM Trans. Softw. Eng. Methodol._ , 28(4).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30, pages 5998–6008. Curran Associates, Inc.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Zhao et al. (2019) Wei Zhao, Liang Wang, Kewei Shen, Ruoyu Jia, and Jingming Liu. 2019.  Improving grammatical error correction via pre-training a copy-augmented architecture with unlabeled data.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 156–165, Minneapolis, Minnesota. Association for Computational Linguistics.