# Fermions in the 6D Standing Wave Braneworld with Real Scalar Field

Pavle Midodashvili<EMAIL_ADDRESS>Ilia State University, 3/5
Cholokashvili Ave., Tbilisi 0162, Georgia

###### Abstract

In the article, we investigate localization problem for spinor fields within
the 6D standing wave braneworld with the bulk real scalar field, introduced
earlier in [30], and explicitly show that there is no normalizable fermion
field zero mode trapped on the brane. Keywords: brane; standing waves;
trapping of particles; zero modes

PACS numbers: 04.50.-h; 04.20.Jb; 11.25.-w

Braneworld models [1, 2, 3, 4, 5, 6] are very useful in addressing several
unsolved problems in high energy physics (e.g. the hierarchy problem, the
nature of flavor, etc.)(for reviews see [7, 8, 9, 10]). Along with many static
braneworld models, there are also dynamic braneworld models, appearing mainly
in cosmological studies, which involve time-dependent metrics and source
matter fields, as well as branes with tensions varying in time [11, 12, 13,
14].

Localization problem is the key issue in realization of a braneworld scenario.
For reasons of economy and stability it would be nice to have a universal
gravitational trapping mechanism for all fields. In the 5D and 6D braneworld
models proposed in [15, 16, 17, 18, 19, 20], the brane was generated by
standing waves of gravitational and phantom-like bulk scalar field, and the
universal gravitational trapping of zero modes of all kinds of matter fields
on the brane was provided by rapid oscillations of the waves [20, 21, 22, 23,
24, 25]. Later in [26] it was introduced another dynamic 5D anisotropic
braneworld generated by standing waves of the gravitational and real scalar
fields, instead of the phantom-like scalar fields of [15, 16, 17, 18, 19, 20].
The metric of that model had horizons in the bulk, and in the case of small
amplitudes of standing waves matter was actually trapped on the brane of the
width equal to the horizon size, while in the case of large amplitudes of
standing waves the zero modes of all kinds of matter fields were trapped on
the brane of the width much less than the horizon size [27, 28].

Recently we have introduced the new non-static 6D braneworld model generated
by standing gravitational waves coupled to bulk real scalar field [30], which
differs in underlying metric from that considered in [29], where authors also
studied the 6D standing-wave braneworld with normal matter as source.

In [30], the 6D standing wave braneworld with real scalar field is introduced
with the metric ansatz

$d{s^{2}}=\frac{{{e^{S}}}}{{{{f}^{{\frac{3}{4}}}}}}\left({d{t^{2}}-d{r^{2}}}\right)\\\
-{f^{\frac{1}{2}}}\left[{{e^{u}}\left({d{x^{2}}+d{y^{2}}+d{z^{2}}}\right)+{e^{-3u}}d{\theta^{2}}}\right],$
(1)

where the metric function $f$ is defined as

$f=1+a\left|r\right|,$ (2)

with parameter $a$ being a positive constant. The determinant of the metric
(1) is

$\sqrt{-g}={f^{\frac{1}{4}}}{e^{S}}.$ (3)

The metric functions $u$ and $S$ are solutions to 6D Einstein equations and
have the following form [30]:

$\displaystyle u\left({t,r}\right)=A\sin\left({\omega
t}\right){J_{0}}\left({\frac{\omega}{a}f}\right),$ $\displaystyle
S\left(r\right)=3A^{2}\left(\frac{\omega}{a}f\right)^{2}\left[J_{0}^{2}\left(\frac{\omega}{a}f\right)+J_{1}^{2}\left(\frac{\omega}{a}f\right)-\frac{J_{0}\left(\frac{\omega}{a}f\right)J_{1}\left(\frac{\omega}{a}f\right)}{\frac{\omega}{a}f}\right]-B,$
(4)

where $A$ and $\omega$ are some constants, $J_{0}$ and $J_{1}$ are Bessel
functions of the first kind, and the integration constant $B$ is fixed as

$B=3A^{2}\frac{\omega^{2}}{a^{2}}{{{J_{1}^{2}}\left({\frac{\omega}{a}}\right)}}.$
(5)

There is also the bulk real scalar field standing wave, the solution to the 6D
Klein-Gordon equations in the background metric (1) [30],

$\phi\left({t,r}\right)=\sqrt{3}{M^{2}}A\cos\left({\omega
t}\right){J_{0}}\left({\frac{\omega}{a}f}\right),$ (6)

where $M$ denotes the 6D fundamental scale, which is related to the 6D
Newton’s constant $G$ by the relation $G=\frac{1}{8\pi M^{4}}$. The scalar
field $\phi\left({t,r}\right)$ and the metric function $u\left({t,r}\right)$
are oscillating $\pi/2$ out of phase in time, but have similar dependence on
the spatial coordinate $r$.

Bulk standing gravitational wave $u\left({t,r}\right)$, given by (Fermions in
the 6D Standing Wave Braneworld with Real Scalar Field), has the frequency
$\omega$ and the amplitude $A{J_{0}}\left({\frac{\omega}{a}f}\right)$. In the
extra dimension $r$ the wave is bounded by the brane at $r=0$, and it decays
at infinities $r\to\pm\infty$. The standing wave amplitude vanishes at the
points where the Bessel function ${J_{0}}\left({\frac{\omega}{a}f}\right)$ is
zero, these points correspond to nodes of the bulk standing wave. In the model
it is also imposed the quantization condition

$\frac{\omega}{a}=Z_{n},$ (7)

where the constant $Z_{n}$ denotes the $n$-th zero of the Bessel function
$J_{0}$, and $n\geq 1$ is some fixed integer. This condition guarantees that
the 5D-brane is located at the origin $r=0$ of the infinite spacelike extra
dimension $r$ orthogonal to the brane [30]. Among four spatial coordinates of
the 5D-brane, three of them, $x$, $y$ and $z$, denote the ordinary dimensions
of our universe, while the remaining one, $\theta$, is assumed to be compact,
curled up to the unobservable sizes for the present energies (hybrid
compactification), and it is normalized as

$0\leq\frac{\theta}{\theta_{0}}\leq 2\pi,$ (8)

where $\theta_{0}$ is some fixed length scale.

Using (7) the parameter (5) can be written as

$B=3A^{2}{Z_{n}^{2}}{{{J_{1}^{2}}\left({Z_{n}}\right)}}.$ (9)

It was shown in [30] that in the case of large parameter $B$,

$B\gg 1,$ (10)

the metric functions $S$ and $u$ defined by (Fermions in the 6D Standing Wave
Braneworld with Real Scalar Field) obey the relation:

$\left|\frac{u\left(t,r\right)}{S\left(r\right)}\right|\ll 1,$ (11)

which holds for any $t$ and $r$. Then the width of the brane along the $r$
extra dimension,

$\Delta\sim\frac{1}{aB},$ (12)

which is located at the origin of the $r$ extra dimension, is determined by
the metric function $S\left(r\right)$ [30], and matter fields are dynamically
trapped on the brane by the bulk standing wave of gravity.

Close to the brane at $r=0$ the metric (1) has the following approximate form:

$\displaystyle g_{tt}$ $\displaystyle=$ $\displaystyle-g_{rr}\approx
1+\frac{3\left|r\right|}{\Delta},$ $\displaystyle g_{xx}$ $\displaystyle=$
$\displaystyle g_{yy}=g_{zz}\approx-1+O\left(\frac{\left|r\right|}{\Delta
B}\right),$ (13) $\displaystyle g_{\theta\theta}$ $\displaystyle\approx$
$\displaystyle-1+O\left(\frac{\left|r\right|}{\Delta B}\right).$

Then the zero mode wavefunction $\Phi\left(x^{A}\right)$ of a free brane
particle with the energy-momentum

$P_{\mu}\ll\frac{1}{\Delta},$ (14)

can be factorized as:

$\left.\Phi\left(x^{A}\right)\right|_{r\to 0}\approx
e^{-iP_{\mu}x^{\mu}}\varphi\left({r,\theta}\right),$ (15)

where Greek indices numerate 4D coordinates $t$, $x$, $y$ and $z$, and the
scalar function $\varphi\left({r,\theta}\right)$ is the extra dimension factor
of the wavefunction near the brane. It is obvious from (Fermions in the 6D
Standing Wave Braneworld with Real Scalar Field) that, in the case (10), the
Lorentz violation terms in $g_{xx}$, $g_{yy}$, $g_{zz}$ and $g_{\theta\theta}$
are much less than KK corrections, $\approx 1/\Delta$, in $g_{tt}$ and
$g_{rr}$.

In the article [30], it was shown that in the case of large parameter $B$,
(10), there are two different physical limits realizing the pure gravitational
trapping of classical particles and light and also of massless scalar fields
on the brane. In the first physical limit the standing wave amplitudes have to
be sufficiently large, while in the second one the amplitudes of standing
waves can be small enough. And it was explicitly shown in [30] that in both
cases particles and light, as well as massless scalar fields are dynamically
trapped on the brane by the pressure of bulk standing waves of gravity and
real scalar field.

Now let us consider the problem of localization of 6D spin $\frac{1}{2}$
fermion field in this braneworld model. The starting action is the 6D Dirac
action of free massless spin $\frac{1}{2}$ fermion

$S_{\frac{1}{2}}=\int
d^{6}x\sqrt{-g}i\overline{\Psi}\left(x^{A}\right)\Gamma^{M}D_{M}\Psi\left(x^{A}\right).$
(16)

The curved gamma matrices , $\Gamma^{A}$, which obey the relations

$\\{\Gamma^{A},\Gamma^{B}\\}=2g^{AB},~{}~{}(A,B,...=t,x,y,z,r,\theta),$ (17)

can be chosen as:

$\Gamma^{A}=h_{\bar{A}}^{A}\Gamma^{\bar{A}},~{}~{}~{}\Gamma^{\bar{A}}=\left(\gamma^{t},\gamma^{x},\gamma^{y},\gamma^{z},\gamma^{r},\gamma^{\theta}\right),$
(18)

where the indices $\bar{A}$, $\bar{B}$, … refer to 6D local Lorentz (tangent)
frame and $A$, $B$, … stand for 6D spacetime coordinates $t$, $x$, $y$, $z$,
$r$, $\theta$.

The vielbein for the metric (1),

$\displaystyle
h_{A}^{\bar{A}}=diag\left(\frac{e^{\frac{S}{2}}}{f^{\frac{3}{8}}},f^{\frac{1}{4}}e^{\frac{u}{2}},f^{\frac{1}{4}}e^{\frac{u}{2}},f^{\frac{1}{4}}e^{\frac{u}{2}},\frac{e^{\frac{S}{2}}}{f^{\frac{3}{8}}},f^{\frac{1}{4}}e^{-\frac{3u}{2}}\right),$
$\displaystyle
h^{\bar{A}A}=g^{AB}h_{B}^{\bar{A}},~{}~{}h_{\bar{A}}^{A}=\eta_{\bar{A}\bar{B}}h^{\bar{B}A},~{}~{}h_{\bar{A}A}=\eta_{\bar{A}\bar{B}}h_{A}^{\bar{B}},$

is introduced through the conventional definition:

$g_{AB}=\eta_{\bar{A}\bar{B}}h_{A}^{\bar{A}}h_{B}^{\bar{B}}.$ (20)

According to (18), the curved spacetime gamma matrices are related to
Minkowskian ones by the following expressions:

$\displaystyle\Gamma^{t}$ $\displaystyle=$ $\displaystyle
f^{\frac{3}{8}}e^{-\frac{S}{2}}\gamma^{t},$ $\displaystyle\Gamma^{x}$
$\displaystyle=$ $\displaystyle f^{-\frac{1}{4}}e^{-\frac{u}{2}}\gamma^{x},$
$\displaystyle\Gamma^{y}$ $\displaystyle=$ $\displaystyle
f^{-\frac{1}{4}}e^{-\frac{u}{2}}\gamma^{y},$ (21) $\displaystyle\Gamma^{z}$
$\displaystyle=$ $\displaystyle f^{-\frac{1}{4}}e^{-\frac{u}{2}}\gamma^{z},$
$\displaystyle\Gamma^{r}$ $\displaystyle=$ $\displaystyle
f^{\frac{3}{8}}e^{-\frac{S}{2}}\gamma^{r},$ $\displaystyle\Gamma^{\theta}$
$\displaystyle=$ $\displaystyle
f^{-\frac{1}{4}}e^{\frac{3u}{2}}\gamma^{\theta}.$

In the 6D Dirac action (16) the covariant derivatives are:

$D_{A}=\partial_{A}+\frac{1}{4}\Omega_{A}^{\bar{B}\bar{C}}\Gamma_{\bar{B}}\Gamma_{\bar{C}},$
(22)

with $\Omega_{M}^{\bar{M}\bar{N}}$ denoting the spin-connection,

$\displaystyle\Omega_{M}^{\bar{M}\bar{N}}=-\Omega_{M}^{\bar{N}\bar{M}}$
$\displaystyle=$
$\displaystyle\frac{1}{2}\left[h^{N\bar{M}}\left(\partial_{M}h_{N}^{\bar{N}}-\partial_{N}h_{M}^{\bar{N}}\right)-h^{N\bar{N}}\left(\partial_{M}h_{N}^{\bar{M}}-\partial_{N}h_{M}^{\bar{M}}\right)\right.$
(23)
$\displaystyle~{}~{}~{}~{}~{}~{}~{}~{}\left.-h_{M}^{\bar{A}}h^{P\bar{M}}h^{Q\bar{N}}\left(\partial_{P}h_{Q\bar{A}}-\partial_{Q}h_{P\bar{A}}\right)\right].$

The non-vanishing components of the spin-connection in the background (1) are:

$\displaystyle\Omega_{t}^{\bar{t}\bar{r}}$ $\displaystyle=$
$\displaystyle-\frac{S^{\prime}}{2}+\frac{3a\,\rm{sgn}\left(r\right)}{8f},$
$\displaystyle\Omega_{x}^{\bar{x}\bar{r}}$ $\displaystyle=$
$\displaystyle\Omega_{y}^{\bar{y}\bar{r}}=\Omega_{z}^{\bar{z}\bar{r}}=-\left(\frac{u^{\prime}}{2}f^{\frac{5}{8}}+\frac{a\,\rm{sgn}\left(r\right)}{4f^{\frac{3}{8}}}\right)e^{-\frac{S}{2}+\frac{u}{2}},$
$\displaystyle\Omega_{x}^{\bar{x}\bar{t}}$ $\displaystyle=$
$\displaystyle\Omega_{y}^{\bar{y}\bar{t}}=\Omega_{z}^{\bar{z}\bar{t}}=\frac{\dot{u}}{2}f^{\frac{5}{8}}e^{-\frac{S}{2}+\frac{u}{2}},$
(24) $\displaystyle\Omega_{\theta}^{\bar{\theta}\bar{t}}$ $\displaystyle=$
$\displaystyle-\frac{3\dot{u}}{2}f^{\frac{5}{8}}e^{-\frac{S}{2}-\frac{3u}{2}},$
$\displaystyle\Omega_{\theta}^{\bar{\theta}\bar{r}}$ $\displaystyle=$
$\displaystyle\left(\frac{3u^{\prime}}{2}f^{\frac{5}{8}}-\frac{a\,\rm{sgn}\left(r\right)}{4f^{\frac{3}{8}}}\right)e^{-\frac{S}{2}-\frac{3u}{2}},$

where dots and primes denote derivatives with respect to the time $t$ and the
extra coordinate $r$ respectively.

Using (Fermions in the 6D Standing Wave Braneworld with Real Scalar Field),
the covariant derivatives (22) can be written as:

$\displaystyle D_{t}$ $\displaystyle=$
$\displaystyle\partial_{t}+\frac{1}{2}\left[\frac{S^{\prime}}{2}-\frac{3a\,\rm{sgn}\left(r\right)}{8f}\right]\gamma_{r}\gamma_{t},$
(25) $\displaystyle D_{x}$ $\displaystyle=$
$\displaystyle\partial_{x}+f^{\frac{5}{8}}e^{-\frac{S}{2}+\frac{u}{2}}\left[\left(\frac{u^{\prime}}{4}+\frac{a\,\rm{sgn}\left(r\right)}{8f}\right)\gamma_{r}-\frac{\dot{u}}{4}\gamma_{t}\right]\gamma_{x},$
(26) $\displaystyle D_{y}$ $\displaystyle=$
$\displaystyle\partial_{y}+f^{\frac{5}{8}}e^{-\frac{S}{2}+\frac{u}{2}}\left[\left(\frac{u^{\prime}}{4}+\frac{a\,\rm{sgn}\left(r\right)}{8f}\right)\gamma_{r}-\frac{\dot{u}}{4}\gamma_{t}\right]\gamma_{y},$
(27) $\displaystyle D_{z}$ $\displaystyle=$
$\displaystyle\partial_{z}+f^{\frac{5}{8}}e^{-\frac{S}{2}+\frac{u}{2}}\left[\left(\frac{u^{\prime}}{4}+\frac{a\,\rm{sgn}\left(r\right)}{8f}\right)\gamma_{r}-\frac{\dot{u}}{4}\gamma_{t}\right]\gamma_{z},$
(28) $\displaystyle D_{r}$ $\displaystyle=$ $\displaystyle\partial_{r},$ (29)
$\displaystyle D_{\theta}$ $\displaystyle=$
$\displaystyle\partial_{\theta}+f^{\frac{5}{8}}e^{-\frac{S}{2}-\frac{3u}{2}}\left[\left(-\frac{3u^{\prime}}{4}+\frac{a\,\rm{sgn}\left(r\right)}{8f}\right)\gamma_{r}+\frac{3\dot{u}}{4}\gamma_{t}\right]\Gamma_{\theta}.$
(30)

The 6D Dirac equation ,

$i\gamma^{A}D_{A}\Psi=0,$ (31)

reduces to

$\left[\gamma^{{t}}\partial_{t}+\frac{e^{\frac{S}{2}-\frac{u}{2}}}{f^{\frac{5}{8}}}\left(\gamma^{{i}}\partial_{i}+e^{2u}\gamma^{\theta}\partial_{\theta}\right)+\gamma^{{r}}\left(\partial_{r}+\frac{S^{\prime}}{4}+\frac{5a\,\rm{sgn}\left(r\right)}{16f}\right)\right]\Psi=0,$
(32)

where $\gamma^{{i}}\partial_{i}$ denotes the sum over $i=x,y,z$:

$\gamma^{{i}}\partial_{i}=\gamma^{{x}}\partial_{x}+\gamma^{{y}}\partial_{y}+\gamma^{{z}}\partial_{z}~{}.$
(33)

Using (11), the function $u$ can be ignored in comparison with the function
$S$ and the equation gets the form:

$\left[\gamma^{{t}}\partial_{t}+\frac{e^{\frac{S}{2}}}{f^{\frac{5}{8}}}\left(\gamma^{{i}}\partial_{i}+\gamma^{\theta}\partial_{\theta}\right)+\gamma^{{r}}\left(\partial_{r}+\frac{S^{\prime}}{4}+\frac{5a\,\rm{sgn}\left(r\right)}{16f}\right)\right]\Psi=0.$
(34)

In general the solution to this equation has non-trivial dependence on 4D, $r$
and $\theta$ coordinates. We look for the solution to this equation in the
form:

$\Psi\left({t,x,y,z,r,\theta}\right)=\psi\left({{x^{\mu}}}\right)\sum\limits_{l,m}{{\alpha_{m}}\left(r\right){e^{il\frac{\theta}{{{\theta_{0}}}}}}},$
(35)

where 4D factor $\psi\left({{x^{\mu}}}\right)$ of the spinor field
wavefunction obeys the equation:

$\gamma^{{\mu}}\partial_{\mu}\psi\left({{x^{\mu}}}\right)=0.$ (36)

Let us use this decomposition of the 6D spinor wavefunction for two limiting
regions: close to the brane, $r\to 0$, where the zeroth mode must be located,
and far from the brane, $r\to\pm\infty$.

Close to the brane, $r\to 0$, we assume that 4D part of (35),
$\psi\left(x^{\mu}\right)$, corresponds to the 4D zero mode Dirac spinor, and

$\gamma^{{\mu}}\partial_{\mu}\psi\left({{x^{\mu}}}\right)=0.$ (37)

In this region the equation (34) reduces to the following asymptotic form

$\left(\partial_{r}+\frac{S^{\prime}}{4}+\frac{5a\,\rm{sgn}\left(r\right)}{16f}\right)\alpha_{0}\left(r\right)=0~{},$
(38)

which has the solution:

$\alpha_{0}\left(r\right)=\frac{Ce^{-\frac{S}{4}}}{f^{\frac{5}{16}}}~{},$ (39)

where $C$ is the integration constant.

Far from the brane, $r\to\pm\infty$, using the factorization (35), and
assuming that at the infinities $r\to\pm\infty$

$\psi\left(x^{\mu}\right)=const~{},$ (40)

for the extra dimension factor $\alpha_{0}\left(r\right)$ we get the same
asymptotic equation (38) with the same solution (39).

To have a localized field on a brane, its 4D effective Lagrangian, which
appears upon integration of corresponding 6D action over extra coordinates,
must be non-vanishing and finite. But in our case this condition is not true
for the zero mode spinor field action (16). Indeed, putting the found solution
$\alpha_{0}\left(r\right)$ (39) into the 6D Dirac action (16) and performing
integration over extra coordinate $\theta$, the corresponding 4D effective
action gets the following form:

$S_{\frac{1}{2}}^{\left(0\right)}=2\pi\theta_{0}C^{2}\int
d^{4}xi\overline{\psi}\left(x^{\mu}\right)\left[I_{1}\gamma^{t}\partial_{t}+I_{2}\gamma^{i}\partial_{i}\right]\psi\left(x^{\mu}\right),$
(41)

where

$I_{1}=\int\limits_{-\infty}^{\infty}{{f^{\frac{3}{8}}}{e^{-\frac{S}{2}}}dr},~{}~{}~{}~{}~{}~{}~{}I_{2}=\int\limits_{-\infty}^{\infty}{{f^{\frac{1}{4}}}dr}.$
(42)

Taking into account (2) and the asymptotics of the metric function
$S\left(r\right)$ [30],

${\left.S\right|_{\left|r\right|\to\infty}}=3Ba\left|r\right|+O\left(\frac{1}{\left|r\right|}\right)~{},$
(43)

in (41), the first integral,
$I_{1}=\int\limits_{-\infty}^{\infty}{{f^{\frac{3}{8}}}{e^{-\frac{S}{2}}}dr}$,
is convergent (it gives some concrete finite real number), but the second
integral, $I_{2}=\int\limits_{-\infty}^{\infty}{{f^{\frac{1}{4}}}dr}$, is
divergent, which means that the zero mode is not localized on the brane.

To conclude, in this paper, we have investigated localization problem for
fermion fields within the 6D standing wave braneworld with the bulk real
scalar field, introduced in [30]. It was shown earlier, in [30], that the
braneworld model realizes pure gravitational trapping mechanism for particles
and light and for massless scalar fields on the brane. Despite that, in this
article we have explicitly shown that there is no normalizable fermion field
zero mode trapped on the brane.

## References

  * [1] N. Arkani-Hamed, S. Dimopoulos and G. Dvali, Phys. Lett. B 429 (1998) 263, arXiv: hep-ph/9803315.
  * [2] I. Antoniadis, N. Arkani-Hamed, S. Dimopoulos and G. Dvali, Phys. Lett. B 436 (1998) 257, arXiv: hep-ph/9804398.
  * [3] M. Gogberashvili, Int. J. Mod. Phys. D 11 (2002) 1635, arXiv: hep-ph/9812296.
  * [4] M. Gogberashvili, Mod. Phys. Lett. A 14 (1999) 2025, arXiv: hep-ph/9904383.
  * [5] L. Randall and R. Sundrum, Phys. Rev. Lett. 83 (1999) 3370, arXiv: hep-ph/9905221.
  * [6] L. Randall and R. Sundrum, Phys. Rev. Lett. 83 (1999) 4690, arXiv: hep-th/9906064.
  * [7] V.A. Rubakov, Phys. Usp. 44 (2001) 871 (Usp. Fiz. Nauk 171 (2001) 913).
  * [8] D. Langlois, Prog. Theor. Phys. Suppl. 148 (2003) 181, arXiv: hep-th/0209261.
  * [9] P.D. Mannheim, Brane-localized Gravity. World Scientific, Singapore 2005.
  * [10] R. Maartens and K. Koyama, Living Rev. Rel. 13 (2010) 5, arXiv: 1004.3962 [hep-th].
  * [11] M. Gutperle and A. Strominger, JHEP 0204 (2002) 018, arXiv: hep-th/0202210.
  * [12] M. Kruczenski, R.C. Myers and A.W. Peet, JHEP 0205 (2002) 039, arXiv: hep-th/0204144.
  * [13] V.D. Ivashchuk and D. Singleton, JHEP 0410 (2004) 061, arXiv: hep-th/0407224.
  * [14] C.P. Burgess, F. Quevedo, R. Rabadan, G. Tasinato and I. Zavala, JCAP 0402 (2004) 008, arXiv: hep-th/0310122.
  * [15] M. Gogberashvili and D. Singleton, Mod. Phys. Lett. A 25 (2010) 2131, arXiv: 0904.2828 [hep-th].
  * [16] M. Gogberashvili, A. Herrera-Aguilar and D. Malagon-Morejon, Class. Quant. Grav. 29 (2012) 025007, arXiv: 1012.4534 [hep-th].
  * [17] M. Gogberashvili, A. Herrera-Aguilar, D. Malagon-Morejon, R.R. Mora-Luna and U. Nucamendi, Phys. Rev. D 87 (2013) 084059, arXiv: 1201.4569 [hep-th].
  * [18] M. Gogberashvili, A. Herrera-Aguilar, D. Malagon-Morejon and R.R. Mora-Luna, Phys. Lett. B 725 (2013) 208, arXiv: 1202.1608 [hep-th].
  * [19] L.J.S. Sousa, J.E.G. Silva and C.A.S. Almeida, arXiv: 1209.2727 [hep-th].
  * [20] P. Midodashvili, Int.J.Theor.Phys. 53 (2014) 1174, arXiv: 1211.0206 [hep-th].
  * [21] M. Gogberashvili, P. Midodashvili and L. Midodashvili, Phys. Lett. B 702 (2011) 276, arXiv: 1105.1701 [hep-th].
  * [22] M. Gogberashvili, P. Midodashvili and L. Midodashvili, Phys. Lett. B 707 (2012) 169, arXiv: 1105.1866 [hep-th].
  * [23] M. Gogberashvili, P. Midodashvili and L. Midodashvili, Int. J. Mod. Phys. D 21 (2012) 1250081, arXiv: 1209.3815 [hep-th].
  * [24] M. Gogberashvili, JHEP 09 (2012) 056, arXiv: 1204.2448 [hep-th].
  * [25] M. Gogberashvili, O. Sakhelashvili and G. Tukhashvili, Mod. Phys. Lett. A 28 (2013) 1350092, arXiv: 1304.6079 [hep-th].
  * [26] M. Gogberashvili and P. Midodashvili, Adv.High Energy Phys. 2013 (2013) 873686, arXiv: 1310.1911 [hep-th].
  * [27] M. Gogberashvili and P. Midodashvili, EPL 104 (2013) 50002, arXiv: 1312.6241 [hep-th].
  * [28] M. Gogberashvili and P. Midodashvili, Int.J.Mod.Phys. A 29 (2014) 1450141, arXiv: 1409.4408 [physics.gen-ph].
  * [29] L.J.S. Sousa, W.T. Cruz and C.A.S. Almeida, Phys.Rev. D 89 (2014) 064006, arXiv: 1311.5848 [hep-th].
  * [30] P. Midodashvili and L. Midodashvili, Braz.J.Phys. 50 (2020) 6, 750.


[booktabs]abovesep=0pt, belowsep=0pt, rowsep=0.5pt [booktabs]cells = cmd=

#  Magicoder: Source Code Is All You Need

Yuxiang Wei1 Zhe Wang2 Jiawei Liu1 Yifeng Ding1 Lingming Zhang1
1University of Illinois at Urbana-Champaign 2Tsinghua University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
https://github.com/ise-uiuc/magicoder

###### Abstract

We introduce Magicoder, a series of _fully open-source_ (code, weights, and
data) Large Language Models (LLMs) for code that significantly closes the gap
with top code models while having no more than 7B parameters. Magicoder models
are trained on 75K synthetic instruction data using OSS-Instruct, a novel
approach to enlightening LLMs with open-source code snippets to generate high-
quality instruction data for code. Our main motivation is to mitigate the
inherent bias of the synthetic data generated by LLMs by empowering them with
a wealth of open-source references for the production of more _diverse_ ,
_realistic_ , and _controllable_ data. The orthogonality of OSS-Instruct and
other data generation methods like Evol-Instruct further enables us to build
an enhanced Magicoder$\mathcal{S}$. Both Magicoder and Magicoder$\mathcal{S}$
substantially outperform state-of-the-art code models with similar or even
larger sizes on a wide range of coding benchmarks, including Python text-to-
code generation, multilingual coding, and data-science program completion.
Notably, Magicoder$\mathcal{S}$-CL-7B based on CodeLlama even surpasses the
prominent ChatGPT on HumanEval+ (66.5 vs. 65.9 in pass@1). Overall, OSS-
Instruct opens a new direction for low-bias and high-quality instruction
tuning using abundant open-source references.

Figure 1: Overview of OSS-Instruct and the pass@1 results of different LLMs on
HumanEval (+)

##  1 Introduction

Code generation, also known as program synthesis Gulwani et al. [2017], is a
long-standing challenge in computer science. In the past few decades, a large
body of research has been studying symbolic approaches, such as abstraction-
based synthesis Wang et al. [2017], Feng et al. [2018] for general-purpose
synthesis problems and programming by examples Cambronero et al. [2023], Liu
et al. [2023a] for domain-specific tasks. Until recently, Large Language
Models (LLMs) trained on code Austin et al. [2021], Chen et al. [2021] has
shown outstanding breakthroughs in generating code that accurately satisfies
user intents, and they are widely deployed to assist real-world software
development Microsoft [2023b], Services [2023].

Initially, closed-source models such as GPT-3.5 Turbo OpenAI [2022] (_i.e.,_
ChatGPT) and GPT-4 OpenAI [2023] massively dominated various code generation
benchmarks and leaderboards Chen et al. [2021], Austin et al. [2021], Liu et
al. [2023b], Lai et al. [2022]. To further push the boundaries of code
generation with open source LLMs, _Self-Instruct_ Wang et al. [2023a] is
adopted to bootstrap the instruction-following ability of LLMs. In the realm
of code, practitioners commonly devise synthetic coding instructions using a
stronger teacher model (_e.g.,_ ChatGPT and GPT-4) and then finetune a weaker
student model (_e.g.,_ CodeLlama Rozière et al. [2023]) with the generated
data to distill the knowledge from the teacher Taori et al. [2023], Chaudhary
[2023].For example, Code Alpaca Chaudhary [2023] consists of 20K automatically
generated code instructions by applying Self-Instruct on ChatGPT using 21 seed
tasks. To further enhance the coding abilities of LLMs, Luo et al. [2023b]
proposes _Code Evol-Instruct_ that employs various heuristics to increase the
complexity of seed code instructions (Code Alpaca in this case), achieving
state-of-the-art (SOTA) results among open-source models.

While these data generation methods can effectively improve the instruction-
following capability of an LLM, they rely on a narrow range of predefined
tasks or heuristics under the hood.For example, on the one hand, Code Alpaca
that adopts Self-Instruct only relies on _21 seed tasks_ to generate new code
instructions using an identical prompt template. On the other hand, Code Evol-
Instruct takes Code Alpaca as seeds and merely depends on _5 heuristics_ to
evolve the dataset. As partly suggested by Yu et al. [2023] and Wang et al.
[2023a], such approaches may significantly inherit the system bias inherent in
the LLMs as well as the predefined tasks.

Therefore, in this paper, we propose OSS-Instruct to mitigate the inherent
bias of LLMs and to unleash their potential to craft high-quality and creative
code instructions via direct learning from the open source. As shown in Figure
1, OSS-Instruct leverages a powerful LLM to automatically generate new coding
problems by _drawing inspiration_ from any random code snippets collected from
the open source. In this example, the LLM gets inspired by two incomplete code
fragments from different functions and manages to relate them and craft a
realistic machine learning problem. Thanks to the “infinite” real-world open-
source code, OSS-Instruct can directly produce _diverse_ , _realistic_ , and
_controllable_ code instructions by providing distinct seed code snippets. In
the end, we generate 75K synthetic data to finetune CodeLlama-Python-7B,
resulting in Magicoder-CL. While being simple and effective, OSS-Instruct is
orthogonal to existing data generation methods, and they can be combined to
further push the boundaries of the models’ coding capabilities. Therefore, we
continually finetune Magicoder-CL on an open-source Evol-Instruct with 110K
entries, producing Magicoder$\mathcal{S}$-CL.

We evaluate Magicoder and Magicoder$\mathcal{S}$ on a wide range of coding
tasks, including HumanEval Chen et al. [2021] and MBPP Austin et al. [2021]
for Python text-to-code generation, MultiPL-E Cassano et al. [2022] for
multilingual code completion, and DS-1000 Lai et al. [2022] for solving data
science problems. We further adopt EvalPlus Liu et al. [2023b], which includes
the augmented HumanEval+ and MBPP+ datasets for more rigorous model
evaluation. Both Magicoder-CL and Magicoder$\mathcal{S}$-CL substantially
boost the base CodeLlama-Python-7B. Additionally, Magicoder-CL even
outperforms WizardCoder-CL-7B, WizardCoder-SC-15B, and all studied SOTA LLMs
with less than or equal to 16B parameters on all the benchmarks we
tested.Also, the pass@1 result of the enhanced Magicoder$\mathcal{S}$-CL is on
par with ChatGPT on HumanEval (70.7 vs. 72.6) and surpasses it on the more
rigorous HumanEval+ (66.5 vs. 65.9), indicating that Magicoder$\mathcal{S}$-CL
can generate more robust code. It also achieves SOTA results among all code
models at the same scale.

Additionally, we notice a very recent advancement in the development of the
DeepSeek-Coder series DeepSeek AI [2023] which has shown exceptional coding
performance. However, due to the limited technical details currently
disclosed, we only briefly discuss them in Section 4.4. Despite this, we
applied OSS-Instruct on DeepSeek-Coder-Base 6.7B, resulting in the creation of
Magicoder-DS and Magicoder$\mathcal{S}$-DS. In addition to the consistent
findings on the previous results with CodeLlama-Python-7B as the base model,
Magicoder-DS and Magicoder$\mathcal{S}$-DS benefit from the more powerful
DeepSeek-Coder-Base-6.7B. This advantage is demonstrated by
Magicoder$\mathcal{S}$-DS, which achieves a remarkable 76.8 pass@1 on
HumanEval. Magicoder$\mathcal{S}$-DS also outperforms DeepSeek-Coder-
Instruct-6.7B on HumanEval, HumanEval+, MBPP, and MBPP+ with 8$\times$ less
finetuning tokens.

To justify the design of OSS-Instruct, _i.e.,_ generating instruction-tuning
data from open-source references rather than using the reference directly, we
further demonstrate that finetuning the base models with semantically relevant
comment-function pairs directly extracted from open-source projects even
negatively impacts the model performance (Table 6).

In general, we make the following contributions:

  * •

We introduce OSS-Instruct, a pioneering approach to enlightening LLMs with
open-source code snippets to generate more diverse, realistic, and
controllable coding instruction data, which can be leveraged to substantially
boost the performance of various LLMs via instruction tuning. It opens a new
dimension for creating _low-bias_ and _high-quality_ instruction-tuning data
from the abundance of open-source references.

  * •

We build the Magicoder series trained with OSS-Instruct and
Magicoder$\mathcal{S}$ series trained on a combination of OSS-Instruct and
Evol-Instruct. Our evaluation across 6 benchmarks shows that all Magicoders
significantly improve the base LLMs. Notably, both Magicoder$\mathcal{S}$-CL
and Magicoder$\mathcal{S}$-DS outperform ChatGPT on HumanEval+ with only _7B
parameters_.

  * •

We fully open source the model weights, training data, and source code at
https://github.com/ise-uiuc/magicoder to facilitate future research.

##  2 OSS-Instruct: Instruction Tuning from Open Source

In this section, we elaborate on our OSS-Instruct approach. From a high level,
as shown in Figure 1, OSS-Instruct works by prompting an LLM (_e.g.,_ ChatGPT)
to generate a coding problem and its solution according to some seed code
snippet collected from the wild (_e.g.,_ from GitHub). The seed snippet offers
controllability of the generation and encourages the LLM to create diverse
coding problems that can reflect real-world programming scenarios.

###  2.1 Generating Coding Problems

Figure 2: The detailed prompt design for OSS-Instruct

OSS-Instruct is powered by seed code snippets that can be easily collected
from open source. In this work, we directly adopt starcoderdata as our seed
corpus, a filtered version of The Stack Kocetkov et al. [2022] dataset that
StarCoder is trained on, containing permissively licensed source code
documents in various programming languages.We chose starcoderdata because it
is widely adopted, includes massive high-quality code snippets, and is even
post-processed for data decontamination Li et al. [2023], Allal et al. [2023].
For each code document from the corpus, we randomly extract 1–15 consecutive
lines as the seed snippet for the model to gain inspiration from and produce
coding problems. In total, we collected 80K initial seed snippets from 80K
code documents, 40K from Python, and 5K from each of C++, Java, TypeScript,
Shell, C#, Rust, PHP, and Swift respectively. Then, each collected seed code
snippet is applied to the prompt template shown in Figure 2, which a teacher
model takes as input and outputs both a coding problem and its solution.

###  2.2 Data Cleaning and Decontamination

We perform data cleaning by excluding samples that are identical or share the
same seed code snippet” or linebreaks “\n”. While there exist other sorts of
noisiness (_e.g.,_ the solution is incomplete) in the generated data, inspired
by Honovich et al. [2023], they are not removed as we believe they still
contain valuable information for LLMs to learn. Finally, we apply the same
logic as StarCoder Li et al. [2023] to decontaminate our training data by
removing coding problems that contain docstrings or solutions from HumanEval
Chen et al. [2021] and MBPP Austin et al. [2021], docstrings from APPS
Hendrycks et al. [2021], prompts from DS-1000 Lai et al. [2022], or questions
from GSM8K Cobbe et al. [2021]. As part of our analysis, the decontamination
procedure only filters out 9 additional samples. Since the seed corpus
starcoderdata has already gone through rigorous data decontamination, this
observation suggests that OSS-Instruct is unlikely to introduce additional
data leakage beyond the seeds. The eventual OSS-Instruct dataset contains
about 75K entries.

###  2.3 Qualitative Examples of OSS-Instruct

Figure 3: Examples showing how OSS-Instruct generates problems and solutions
from seed code snippets. Details such as full problem requirements, complete
implementations, and explanations are omitted for brevity.

Figure 3 shows some qualitative examples of how OSS-Instruct can help LLM get
inspiration from a seed code snippet to create new coding problems and
solutions. For example, the _shell script_ example shows how an LLM crafts a
Python coding problem with just one line of shell script. The _library
imports_ example demonstrates how an LLM can create a realistic machine
learning problem using just a few import statements. Meanwhile, the _class
signature_ instance illustrates the ability of LLM to draw inspiration from an
incomplete class definition featuring annotations like SpringBootApplication
and keywords such as bank. From this, the LLM generates a problem that
requires implementing a complete banking system based on Spring Boot! Overall,
OSS-Instruct can inspire an LLM with distinct code structures and semantics to
create diverse coding tasks, including algorithmic challenges, realistic
issues, single-function code generation, library-based program completion,
whole-program development, and even whole-application construction.

Figure 4: The category constitution of OSS-Instruct

Figure 5: Token count distribution of OSS-Instruct-generated problems and
solutions

Figure 6: Cosine similarities between HumanEval and different data generation
methods

#### Categories

To study the categories of OSS-Instruct-generated data, we use INSTRUCTOR Su
et al. [2022], which is one of the SOTA embedding models and can generate
different text embeddings according to a task instruction. Inspired by
OctoPack Muennighoff et al. [2023] and the topic tags on GitHub, we manually
designed 10 categories specific to coding. As shown in Figure 4, we calculate
the cosine similarity between the embeddings of each sample in OSS-Instruct
and the 10 categories to obtain the category breakdown. Overall, OSS-Instruct
exhibits diversity and balance across different categories.

#### Length distribution

We depict the length distribution for both generated problems and solutions in
Figure 6. The x-axis represents the number of tokens in each problem/solution,
while the y-axis shows the correspondent number of samples.

#### Similarity with HumanEval

To study whether our data generation process produces more HumanEval-like
problems or solutions that contribute to high performance, we pair each sample
from our 75K dataset with each of the 164 HumanEval Chen et al. [2021] samples
and compute their cosine similarity using TF-IDF SPARCK JONES [1972]
embeddings. We then associate each OSS-Instruct sample with a HumanEval sample
with the highest similarity score. We also compare our dataset against Code
Alpaca, a 20K dataset applying Self-Instruct to code, and evol-codealpaca-v1
theblackcat102 [2023], an open-source reproduction of Evol-Instruct containing
110K coding instructions. We resort to the open-source implementation because
the official _Code Evol-Instruct_ Luo et al. [2023b] dataset is not released.
We decontaminate all the datasets beforehand using the same way discussed in
Section 2.2. Figure 6 shows that OSS-Instruct exhibits the lowest average
similarity among all the studied data generation techniques while Self-
Instruct shows the highest average similarity. This result indicates that the
improvements from OSS-Instruct are not merely due to including data from the
same distribution.

##  3 Implementation Details

#### Data generation

We use gpt-3.5-turbo-1106 as the foundation model to do OSS-Instruct due to
its high cost-effectiveness. We randomly extract 1–15 lines from each selected
code document from starcoderdata and let gpt-3.5-turbo-1106 imagine a self-
contained coding problem and a correct solution. Given the numerous seed code
snippets, we perform greedy decoding to maximize the consistency between the
generated problems and solutions.

#### Data decontamination

We apply data decontamination before training our Magicoder and
Magicoder$\mathcal{S}$ models. Following Li et al. [2023], we decontaminate
both our 75K OSS-Instruct dataset and the evol-codealpaca-v1 theblackcat102
[2023] dataset, an open-source reproduction of Evol-Instruct, by removing
exact matches from HumanEval Chen et al. [2021], MBPP Austin et al. [2021],
DS-1000 Lai et al. [2022], and GSM8K Cobbe et al. [2021]. Eventually, we
filtered out 9 problems for OSS-Instruct dataset and 89 for evol-
codealpaca-v1.

#### Training

We employ CodeLlama-Python-7B and DeepSeek-Coder-Base 6.7B as the base LLMs.
To obtain Magicoder series, we first finetune the base models on about 75K
synthetic data generated through OSS-Instruct using the transformers library
from Hugging Face Hugging Face [2023]. We finetune the base models for 2
epochs using two NVIDIA A100-80GB GPUs through the Distributed Data Parallel
(DDP) module from PyTorch. We set the initial learning rate at 5e-5 with 15
warmup steps and a linear scheduler. We use Adafactor Shazeer and Stern [2018]
as our optimizer and choose a batch size of 512 with a sequence truncation
length of 1216. To obtain Magicoder$\mathcal{S}$, we continue to finetune
Magicoder models with the evol-codealpaca-v1 dataset, an open-source Evol-
Instruct implementation containing about 110K samples. We use the same
hyperparameters except for 15 warmup steps and a 1024 max sequence length.

##  4 Evaluation

###  4.1 Python Text-to-Code Generation

HumanEval Chen et al. [2021] and MBPP Austin et al. [2021] are two of the most
widely used benchmarks for code generation. Each task in these benchmarks
includes a task description (_e.g.,_ docstring) as the prompt, where LLMs
generate corresponding code whose correctness is checked by a handful of test
cases. Because tests in these benchmarks can be insufficient Liu et al.
[2023b], for more rigorous evaluation, we use HumanEval+ and MBPP+, both
powered by the EvalPlus framework Liu et al. [2023b] to obtain
80$\times$/35$\times$ more tests. Following prior work Liu et al. [2023b],
Chen et al. [2023], for each task and LLM we use greedy decoding to generate
one sample and focus on comparing the pass@1 metric.

We consider a wide range of baseline models, including CodeLlama-Python
Rozière et al. [2023], WizardCoder Luo et al. [2023b], GPT-3.5 Turbo OpenAI
[2022], GPT-4 Turbo OpenAI [2023], StarCoder Li et al. [2023], CodeT5+ Wang et
al. [2023b], CodeGen-Mono Nijkamp et al. [2023], and Mistral Jiang et al.
[2023a]. All the results are consistently reported from the EvalPlus Liu et
al. [2023b] leaderboard.

Table 1: Pass@1 (%) results of different LLMs on HumanEval (+) and MBPP (+) computed with greedy decoding. The abbreviations “CL” and “SC” refer to the base models CodeLlama-Python and StarCoder, respectively. We report the results consistently from the EvalPlus Liu et al. [2023b] Leaderboard. Model | Release Date |  Size | Benchmark |  | Open-Source |
---|---|---|---|---|---|---
|  |  | HumanEval (+) | MBPP (+) | Weight | Data
GPT-3.5 Turbo | Nov 2023 |  - | 72.6 (65.9) | 81.7 (69.4) | ○ | ○
GPT-4 Turbo | Nov 2023 |  - |  85.4 (81.7) |  83.0 (70.7) | ○ | ○
CodeLlama-Python | Aug 2023 |  34B | 51.8 (42.7) | 67.2 (52.9) | ● | ○
WizardCoder-CL | Sep 2023 |  34B | 73.2 (64.6) | 73.2 (59.9) | ● | ○
CodeT5+ | May 2023 |  16B | 31.7 (26.2) | 54.6 (44.4) | ● | ●
CodeGen-Mono | Mar 2022 |  16B | 32.9 (27.4) | 52.6 (43.6) | ● | ●
StarCoder | May 2023 |  15B | 34.1 (29.3) | 55.1 (46.1) | ● | ●
CodeLlama-Python | Aug 2023 |  13B | 42.7 (36.6) | 61.2 (50.9) | ● | ○
WizardCoder-SC | Sep 2023 |  15B | 51.9 (45.1) | 61.9 (50.6) | ● | ○
StarCoder | May 2023 |  7B | 24.4 (20.7) | 33.1 (28.8) | ● | ●
Mistral | Oct 2023 |  7B | 28.7 (23.2) | 50.1 (40.9) | ● | ○
CodeT5+ | May 2023 |  6B | 29.3 (23.8) | 51.9 (40.9) | ● | ●
CodeGen-Mono | Mar 2022 |  6B | 29.3 (25.6) | 49.9 (42.1) | ● | ●
CodeLlama-Python | Aug 2023 |  7B | 37.8 (34.1) | 57.6 (45.4) | ● | ○
WizardCoder-CL | Sep 2023 |  7B | 48.2 (40.9) | 56.6 (47.1) | ● | ○
Magicoder-CL | Dec 2023 |  7B | 60.4 (55.5) | 64.2 (52.6) | ● | ●
Magicoder$\mathcal{S}$-CL | Dec 2023 |  7B |  70.7 (66.5) |  68.4 (56.6) | ● | ●

Table 1 shows the pass@1 results of different LLMs on these benchmarks. From
the results, we can first observe that Magicoder-CL has a clear improvement
over the base CodeLlama-Python-7B, and outperforms all studied open-source
models except CodeLlama-Python-34B and WizardCoder-CL-34B. Notably, Magicoder-
CL surpasses WizardCoder-SC-15B and has a substantial improvement on HumanEval
and HumanEval+ over CodeLlama-Python-34B.Magicoder$\mathcal{S}$-CL
demonstrates further improvements by being trained with the orthogonal Evol-
Instruct method. Magicoder$\mathcal{S}$-CL outperforms ChatGPT and all other
open-source models on HumanEval+. Moreover, although it scores slightly lower
than WizardCoder-CL-34B and ChatGPT on HumanEval, it surpasses both of them on
the more rigorous HumanEval+ dataset, indicating that
Magicoder$\mathcal{S}$-CL may produce more robust code.

###  4.2 Multilingual Code Generation

Table 2: Pass@1 results of different LLMs on MultiPL-E Cassano et al. [2022] following the same hyperparameter settings as the WizardCoder paper Luo et al. [2023b]: $\texttt{temperature}=0.2$, $\texttt{top\\_p}=0.95$, $\texttt{max\\_length}=512$, and $\texttt{num\\_samples}=50$. We evaluate all 7B models using bigcode-evaluation-harness Ben Allal et al. [2022] and report other results from the WizardCoder paper. Model | Size |  Programming Language |  |  |  |  |
---|---|---|---|---|---|---|---
|  |  Java | JavaScript | C++ | PHP | Swift | Rust
CodeLlama | 34B |  40.2 | 41.7 | 41.4 | 40.4 | 35.3 | 38.7
CodeLlama-Python | 34B |  39.5 | 44.7 | 39.1 | 39.8 | 34.3 | 39.7
CodeLlama-Instruct | 34B |  41.5 | 45.9 | 41.5 | 37.0 | 37.6 | 39.3
WizardCoder-CL | 34B |  44.9 | 55.3 | 47.2 | 47.2 | 44.3 | 46.2
StarCoderBase | 15B |  28.5 | 31.7 | 30.6 | 26.8 | 16.7 | 24.5
StarCoder | 15B |  30.2 | 30.8 | 31.6 | 26.1 | 22.7 | 21.8
WizardCoder-SC | 15B |  35.8 | 41.9 | 39.0 | 39.3 | 33.7 | 27.1
CodeLlama | 7B |  29.3 | 31.7 | 27.0 | 25.1 | 25.6 | 25.5
CodeLlama-Python | 7B |  29.1 | 35.7 | 30.2 | 29.0 | 27.1 | 27.0
Magicoder-CL | 7B |  36.4 | 45.9 | 36.5 | 39.5 | 33.4 | 30.6
Magicoder$\mathcal{S}$-CL | 7B |  42.9 | 57.5 | 44.4 | 47.6 | 44.1 | 40.3

In addition to Python, as shown in Table 2, we perform an extensive evaluation
on 6 widely used programming languages, _i.e.,_ Java, JavaScript, C++, PHP,
Swift, and Rust, using the MultiPL-E benchmark Cassano et al. [2022].We report
available results from the WizardCoder paper Luo et al. [2023b] and evaluate
our models consistently through bigcode-evaluation-harness Ben Allal et al.
[2022]. We skip proprietary models such as ChatGPT and GPT-4 as they are not
supported by the framework. Due to a significant inference latency when
running WizardCoder-CL-7B using the harness in our environment, we choose not
to include it in our analysis.

The results indicate that Magicoder-CL improves the base CodeLlama-Python-7B
by a large margin among all the studied programming languages. Moreover,
Magicoder-CL also achieves better results than the SOTA 15B WizardCoder-SC
among half of the programming languages.Additionally,
Magicoder$\mathcal{S}$-CL demonstrates further improvement over Magicoder-CL
on all programming languages, achieving comparable performance against
WizardCoder-CL-34B with only 7B parameters. It is worth noting that Magicoder-
CL is only trained with very limited multilingual data but still outperforms
other LLMs with similar or even larger sizes. Also, although the harness
evaluates models in _completion_ formats, Magicoders still show significant
improvements despite being only _instruction-tuned_. This implies that LLMs
can learn knowledge from the data beyond its format.

###  4.3 Code Generation for Data Science

Table 3: Pass@1 results on DS-1000 (completion format) with
$\texttt{temperature}=0.2$, $\texttt{top\\_p}=0.5$,
$\texttt{max\\_length}=1024$, and $\texttt{num\\_samples}=40$, following the
same hyperparameter setting used in WizardCoder Luo et al. [2023b]. We
evaluate all the 7B models with their preferred prompt formats and report
other results from WizardCoder.

Model | Size | \+ 155 |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---
Matplotlib | \+ 220 |  |  |  |  |  |  |  |
NumPy | \+ 291 |  |  |  |  |  |  |  |
Pandas | \+ 68 |  |  |  |  |  |  |  |
PyTorch | \+ 106 |  |  |  |  |  |  |  |
SciPy | \+ 115 |  |  |  |  |  |  |  |
Sklearn | \+ 45 |  |  |  |  |  |  |  |
TensorFlow | = 1000 |  |  |  |  |  |  |  |
Overall |  |  |  |  |  |  |  |  |
InCoder | 6.7B | 28.3 | 4.4 |  3.1 |  4.4 | 2.8 | 2.8 |  3.8 | 7.4
CodeGen-Mono | 16B | 31.7 | 10.9 |  3.4 |  7.0 | 9.0 | 10.8 |  15.2 | 11.7
Code-Cushman-001 | - | 40.7 | 21.8 |  7.9 |  12.4 | 11.3 | 18.0 |  12.2 | 18.1
StarCoder | 15B | 51.7 | 29.7 |  11.4 |  21.4 | 20.2 | 29.5 |  24.5 | 26.0
WizardCoder-SC | 15B | 55.2 | 33.6 |  16.7 |  26.2 | 24.2 | 24.9 |  26.7 | 29.2
CodeLlama-Python | 7B | 55.3 | 34.5 |  16.4 |  19.9 | 22.3 | 17.6 |  28.5 | 28.0
WizardCoder-CL | 7B | 53.5 | 34.4 |  15.2 |  25.7 | 21.0 | 24.5 |  28.9 | 28.4
Magicoder-CL | 7B | 54.6 | 34.8 |  19.0 |  24.7 | 25.0 | 22.6 |  28.9 | 29.9
Magicoder$\mathcal{S}$-CL | 7B | 55.9 | 40.6 |  28.4 |  40.4 | 28.8 | 35.8 |  37.6 | 37.5

The DS-1000 dataset Lai et al. [2022] contains 1K distinct data science coding
issues ranging from 7 popular data science libraries in Python. It evaluates
the realistic and practical use case of an LLM and offers unit tests for
validating each problem. DS-1000 has both _completion_ and _insertion_ modes,
but here we only evaluate _completion_ because the base CodeLlama-Python does
not support infilling. Table 3 shows the evaluation results where we include
the recent InCoder Fried et al. [2023], CodeGen Nijkamp et al. [2023], Code-
Cushman-001 Microsoft [2023a], StarCoder Li et al. [2023], CodeLlama-Python
Rozière et al. [2023], and WizardCoder Luo et al. [2023b]. We can see from the
table that Magicoder-CL-7B already outperforms all the baselines we evaluate,
including state-of-the-art WizardCoder-CL-7B and WizardCoder-SC-15B.
Magicoder$\mathcal{S}$-CL-7B further breaks the limit by introducing an 8.3
percentage point absolute improvement over WizardCoder-SC-15B.

###  4.4 Comparison with DeepSeek-Coder

Table 4: Pass@1 (greedy decoding) comparison between Magicoder and DeepSeek-Coder DeepSeek AI [2023] on HumanEval (+) and MBPP (+). DeepSeek-Coder results are reported from EvalPlus Liu et al. [2023b] Leaderboard. Model | Size |  Training Tokens | Benchmark |  | Open-Source |
---|---|---|---|---|---|---
|  |  | HumanEval (+) | MBPP (+) | Weight | Data
DeepSeek-Coder-Base | 1.3B |  2T | - | 55.4 (46.9) | ● | ○
| 6.7B |  2T | 47.6 (39.6) | 70.2 (56.6) | ● | ○
| 33B |  2T | 51.2 (43.3) | - | ● | ○
DeepSeek-Coder Instruct | 1.3B |  +2B | 64.6 (58.5) | 63.7 (53.1) | ● | ○
| 6.7B |  +2B |  73.8 (70.1) |  72.7 (63.4) | ● | ○
| 33B |  +2B |  78.7 (72.6) |  78.7 (66.7) | ● | ○
Magicoder-DS | 6.7B |  +90M | 66.5 (60.4) | 75.4 (61.9) | ● | ●
Magicoder$\mathcal{S}$-DS | 6.7B |  +240M |  76.8 (70.7) |  75.7 (64.4) | ● | ●

DeepSeek-Coder DeepSeek AI [2023] is a series of models released very recently
and they demonstrate superior coding performance. We only briefly discuss it
in this section because its technical details and instruction data are not
publicly available at the time of writing. We apply the same finetuning
strategy on DeepSeek-Coder-Base-6.7B as we performed on CodeLlama-Python-7B,
leading to Magicoder-DS and Magicoder$\mathcal{S}$-DS. Table 4 shows a similar
trend as Table 1 that the base model can be significantly improved after
applying OSS-Instruct. Remarkably, the Magicoder$\mathcal{S}$-DS variant
surpasses DeepSeek-Coder-Instruct-6.7B on all the benchmarks with $\times$8
fewer training tokens, and it also closely matches DeepSeek-Coder-Instruct-34B
on these datasets!

##  5 Ablations of Data Source

###  5.1 Impact of the Language Distribution

To understand the correlation between the programming languages appearing in
the training data and the downstream performance of different languages, we
conduct an additional ablation study about the training data. We classify the
75K training data into approximately 43K Python-only, and 32K non-Python data
according to whether `‘‘‘python` is a substring of the generated data. We do
not classify the data based on the seed code snippet because LLMs performing
OSS-Instruct may produce code in a different programming language than the
seed.

Table 5: Ablation study of using different programming languages as training data. We show the pass@1 results on HumanEval+ Liu et al. [2023b] for Python and the average pass@1 results on MultiPL-E Cassano et al. [2022] for the same set of programming languages used in Table 2 (_i.e.,_ Java, JavaScript, C++, PHP, Swift, and Rust). All the variants are finetuned with 2 epochs and evaluated through greedy-decoding. Model (7B) | Finetuning Data |  Python (HumanEval+) | Others (MultiPL-E)
---|---|---|---
CodeLlama-Python | - |  34.1 | 29.6
Magicoder-CL | Python (43K) |  47.6 | 32.7
Magicoder-CL | Others (32K) |  44.5 | 38.3
Magicoder-CL | Both (75K) |  55.5 | 37.8

Table 5 shows the evaluation results, where we consistently finetune the base
CodeLlama-Python-7B for 2 epochs on different data partitions using the same
training hyperparameters explained in Section 3. From the table, we can see
that, as can be imagined, training on Python or non-Python data can
substantially boost the performance of the base model in Python or non-Python
tasks, respectively. Interestingly, instruction tuning on different
programming languages can still boost the overall coding performance that
includes out-of-distribution languages. For example, when trained on only non-
Python data, Magicoder-CL still achieves a 10.4 percentage point improvement
over the base model in the Python-only evaluation. This implies LLMs can
establish correlations between different programming languages and perform
transfer learning of deeper code semantics.Finally, we observe a more
significant boost in Python evaluation when combining data from both sources,
with a slight decrease in multilingual performance compared with only
finetuning on multilingual data. We attribute this decrease to the dominant
amount of Python data (around 57%) during instruction tuning.

###  5.2 OSS-Instruct vs. Direct Finetuning

Table 6: Comparison between OSS-Instruct and directly finetuning on comment-function pairs with CodeLlama-Python-7B as the base model. Finetuning Data | HumanEval+ |  MultiPL-E
---|---|---
Base model w/o finetuning | 34.1 |  29.6
Comment-function pairs (75K) | 34.1 |  24.1
OSS-Instruct (75K) | 55.5 |  37.8

The fact that OSS-Instruct gets an LLM inspired from open-source code snippets
may lead to a natural question: why not directly finetuning on these open-
source code? To answer this question, we follow CodeSearchNet Husain et al.
[2020] to mine semantically relevant _comment-function_ pairs from the same
seed document corpus we use to construct the 75K OSS-Instruct dataset. We then
train the model to predict the function bodies from the function signatures
and comments. We prioritize comment-function pairs that overlap with our _75K
seed snippets_ , resulting in about 11K data points. To align with our 75K
samples, we collect the remaining 64K samples using the whole corpus of _75K
seed documents_. Eventually, we have the same number of comment-function pairs
with OSS-Instruct data.

We finetune the base CodeLlama-Python-7B for 2 epochs using the paired data,
following the same training setup discussed in Section 3. From Table 6, we
observe that finetuning on 75K paired comment-function data even worsens the
base model, while OSS-Instruct helps to introduce a substantial boost. We
conjecture that the degradation is owing to the substantial noise and
inconsistency that exists intrinsically in the data pairs, even though these
paired data exhibit _very similar_ format as HumanEval or MultiPL-E problems.
This further shows that data factuality, rather than the format, is essential
to code instruction tuning. It also indicates the superiority of OSS-Instruct
which can translate these loosely related code fragments into semantically-
consistent instruction-tuning data.

##  6 Related Work

#### Foundation models for code

Trained over billions of lines of code, LLMs have demonstrated outstanding
performance in a wide range of software engineering tasks, including code
generation Chen et al. [2021], Austin et al. [2021], program repair Wei et al.
[2023], Xia et al. [2023a, c], Xia and Zhang [2022], Jiang et al. [2023b], and
fuzzing Xia et al. [2023b], Deng et al. [2023b, a]. In particular, prominent
base models, such as CodeGen Nijkamp et al. [2023], CodeT5 Wang et al. [2021],
StarCoder Li et al. [2023], and CodeLlama Rozière et al. [2023], are pre-
trained over a huge number of codebase from scratch, establishing the
fundamental ability of general code generation and understanding. Furthermore,
these base models are also further finetuned Luo et al. [2023b] or prompted
Chen et al. [2023] to unlock the true potential of the model to specialize in
solving domain-specific coding tasks.

#### Instruction tuning

Instruction tuning aims at improving pretrained LLMs by finetuning them with a
mixture of instructions and corresponding responses Wei et al. [2022]. With
instruction tuning, LLMs can improve their generalization ability and directly
follow instructions without explicit user demonstration. However, obtaining
high-quality instructional data is oftentimes laborious. To this end, Wang et
al. [2023a] introduces _Self-Instruct_ , where a foundation LLM (GPT-3) is
used to generate synthetic instruction-response pairs with carefully crafted
prompts. The _same_ LLM is then instruction-tuned on the synthetic data to
distill such self-generated knowledge. Although Self-Instruct initially
suggests using the same LLM for both data generation and instruction tuning,
it has been further extended to create synthetic data with different LLMs. For
example, Alpaca Taori et al. [2023] and Code Alpaca Chaudhary [2023] apply
Self-Instruct to finetune Llama with ChatGPT-generated instructions.
CodeLlama-Instructs Rozière et al. [2023] are fintuned using Self-Instruct by
prompting Llama 2 Touvron et al. [2023] for coding problems and CodeLlama for
unit tests and solutions. To improve Self-Instruct, WizardLM Xu et al. [2023]
and WizardCoder Luo et al. [2023a] propose Evol-Instruct and Code Evol-
Instruct by guiding ChatGPT with heuristic prompts to make the synthetic data
more complex and diverse. In contrast, our proposed OSS-Instruct is orthogonal
to all existing methods by allowing LLMs to get inspired from real-world code
snippets for better controllability, quality, and creativity in coding tasks.

#### Evaluating LLMs for code

Most code benchmarks evaluate LLMs on generating _single-function_ programs
from natural language descriptions. Such benchmarks include HumanEval Chen et
al. [2021], MBPP Austin et al. [2021], APPS Hendrycks et al. [2021], and
CodeContests Li et al. [2022]. A handful of manual tests are used to assess
the functional correctness of LLM-generated solutions. However, insufficient
tests can lead to false negatives. Consequently, the EvalPlus framework Liu et
al. [2023b] produces HumanEval+ and MBPP+ by extending 80$\times$/35$\times$
more tests. Meanwhile, there are comprehensive benchmarks evaluating code
generation for data science (DS-1000 Lai et al. [2022]), addressing open-
source issues (SWE-bench Jimenez et al. [2023]), and repository-level code
generation (CrossCodeEval Ding et al. [2023] and RepoEval Zhang et al.
[2023]).

##  7 Conclusion and Future Work

We propose OSS-Instruct, a novel data generation method using Large Language
Models to generate low-bias and high-quality coding challenges from open-
source code snippets. This approach enables Magicoder, which significantly
improves the base LLM. Despite having less than 7B parameters, it can
outperform all evaluate LLMs with less than or equal to 16B parameters,
including the 15B WizardCoder. Combining OSS-Instruct with Evol-Instruct
allows us to build the enhanced Magicoder$\mathcal{S}$ models. They achieve
remarkable results by rivaling leading models like ChatGPT in HumanEval
benchmarks. We fully open source the model weights, training data, and source
code, to enable future research in LLM s for code.In the near future, we will
apply OSS-Instruct to larger base models. We will also continue advancing OSS-
Instruct by generating higher-quality data with a strategically designed
distribution of the seed code snippets and with more advanced teacher LLMs
such as GPT-4.

## References

  * Allal et al. [2023] Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, Logesh Kumar Umapathi, Carolyn Jane Anderson, Yangtian Zi, Joel Lamy Poirier, Hailey Schoelkopf, Sergey Troshin, Dmitry Abulkhanov, Manuel Romero, Michael Lappert, Francesco De Toni, Bernardo García del Río, Qian Liu, Shamik Bose, Urvashi Bhattacharyya, Terry Yue Zhuo, Ian Yu, Paulo Villegas, Marco Zocca, Sourab Mangrulkar, David Lansky, Huu Nguyen, Danish Contractor, Luis Villa, Jia Li, Dzmitry Bahdanau, Yacine Jernite, Sean Hughes, Daniel Fried, Arjun Guha, Harm de Vries, and Leandro von Werra.  Santacoder: don’t reach for the stars!, 2023.
  * Austin et al. [2021] Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton.  Program synthesis with large language models.  _CoRR_ , abs/2108.07732, 2021.  URL https://arxiv.org/abs/2108.07732.
  * Ben Allal et al. [2022] Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra.  A framework for the evaluation of code generation models.  https://github.com/bigcode-project/bigcode-evaluation-harness, 2022\.
  * Cambronero et al. [2023] José Cambronero, Sumit Gulwani, Vu Le, Daniel Perelman, Arjun Radhakrishna, Clint Simon, and Ashish Tiwari.  Flashfill++: Scaling programming by example by cutting to the chase.  _Proc. ACM Program. Lang._ , 7(POPL), jan 2023.  doi: 10.1145/3571226.  URL https://doi.org/10.1145/3571226.
  * Cassano et al. [2022] Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda.  Multipl-e: A scalable and extensible approach to benchmarking neural code generation, 2022.
  * Chaudhary [2023] Sahil Chaudhary.  Code alpaca: An instruction-following llama model for code generation.  https://github.com/sahil280114/codealpaca, 2023.
  * Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code, 2021.
  * Chen et al. [2023] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou.  Teaching large language models to self-debug, 2023.
  * Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems, 2021.
  * DeepSeek AI [2023] DeepSeek AI.  Deepseek coder: Let the code write itself.  https://github.com/deepseek-ai/DeepSeek-Coder, 2023.
  * Deng et al. [2023a] Yinlin Deng, Chunqiu Steven Xia, Haoran Peng, Chenyuan Yang, and Lingming Zhang.  Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models, 2023a.
  * Deng et al. [2023b] Yinlin Deng, Chunqiu Steven Xia, Chenyuan Yang, Shizhuo Dylan Zhang, Shujing Yang, and Lingming Zhang.  Large language models are edge-case fuzzers: Testing deep learning libraries via fuzzgpt, 2023b.
  * Ding et al. [2023] Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang.  Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023.  URL https://openreview.net/forum?id=wgDcbBMSfh.
  * Feng et al. [2018] Yu Feng, Ruben Martins, Osbert Bastani, and Isil Dillig.  Program synthesis using conflict-driven learning.  _SIGPLAN Not._ , 53(4):420–435, jun 2018.  ISSN 0362-1340.  doi: 10.1145/3296979.3192382.  URL https://doi.org/10.1145/3296979.3192382.
  * Fried et al. [2023] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis.  Incoder: A generative model for code infilling and synthesis.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=hQwb-lbM6EL.
  * Gulwani et al. [2017] Sumit Gulwani, Oleksandr Polozov, and Rishabh Singh.  Program synthesis.  _Foundations and Trends® in Programming Languages_ , 4(1-2):1–119, 2017.  ISSN 2325-1107.  doi: 10.1561/2500000010.  URL http://dx.doi.org/10.1561/2500000010.
  * Hendrycks et al. [2021] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt.  Measuring coding challenge competence with apps, 2021.
  * Honovich et al. [2023] Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick.  Unnatural instructions: Tuning language models with (almost) no human labor.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 14409–14428, Toronto, Canada, July 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.806.  URL https://aclanthology.org/2023.acl-long.806.
  * Hugging Face [2023] Hugging Face.  Hugging face: The ai community building the future.  https://huggingface.co/, 2023.  Accessed: 2023-12-01.
  * Husain et al. [2020] Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt.  Codesearchnet challenge: Evaluating the state of semantic code search, 2020.
  * Jiang et al. [2023a] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mistral 7b, 2023a.
  * Jiang et al. [2023b] Nan Jiang, Kevin Liu, Thibaud Lutellier, and Lin Tan.  Impact of code language models on automated program repair, 2023b.
  * Jimenez et al. [2023] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan.  Swe-bench: Can language models resolve real-world github issues?, 2023\.
  * Kocetkov et al. [2022] Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries.  The stack: 3 tb of permissively licensed source code, 2022.
  * Lai et al. [2022] Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu.  Ds-1000: A natural and reliable benchmark for data science code generation, 2022.
  * Li et al. [2023] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries.  Starcoder: may the source be with you!, 2023.
  * Li et al. [2022] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals.  Competition-level code generation with alphacode.  _Science_ , 378(6624):1092–1097, December 2022\.  ISSN 1095-9203.  doi: 10.1126/science.abq1158.  URL http://dx.doi.org/10.1126/science.abq1158.
  * Liu et al. [2023a] Jiawei Liu, Jinjun Peng, Yuyao Wang, and Lingming Zhang.  Neuri: Diversifying dnn generation via inductive rule inference, 2023a.
  * Liu et al. [2023b] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang.  Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023b.
  * Luo et al. [2023a] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang.  Wizardcoder: Empowering code large language models with evol-instruct.  _arXiv preprint arXiv:2306.08568_ , 2023a.
  * Luo et al. [2023b] Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang.  Wizardcoder: Empowering code large language models with evol-instruct, 2023b.
  * Microsoft [2023a] Microsoft.  Azure openai service models.  https://learn.microsoft.com/en-us/azure/cognitive-services/openai/concepts/models, 2023a.
  * Microsoft [2023b] Microsoft.  GitHub Copilot – Your AI pair programmer.  https://github.com/features/copilot, 2023b.
  * Muennighoff et al. [2023] Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre.  Octopack: Instruction tuning code large language models, 2023.
  * Nijkamp et al. [2023] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong.  Codegen: An open large language model for code with multi-turn program synthesis.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=iaYcJKpY2B_.
  * OpenAI [2022] OpenAI.  Chatgpt: Optimizing language models for dialogue.  https://openai.com/blog/chatgpt/, 2022.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Rozière et al. [2023] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve.  Code llama: Open foundation models for code, 2023.
  * Services [2023] Amazon Web Services.  AI Code Generator - Amazon CodeWhisperer - AWS.  https://aws.amazon.com/codewhisperer/, 2023.
  * Shazeer and Stern [2018] Noam Shazeer and Mitchell Stern.  Adafactor: Adaptive learning rates with sublinear memory cost, 2018.
  * SPARCK JONES [1972] KAREN SPARCK JONES.  A statistical interpretation of term specificity and its application in retrieval.  28(1):11–21, 2023/11/30 1972.  doi: 10.1108/eb026526.  URL https://doi.org/10.1108/eb026526.
  * Su et al. [2022] Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu.  One embedder, any task: Instruction-finetuned text embeddings.  2022\.  URL https://arxiv.org/abs/2212.09741.
  * Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * theblackcat102 [2023] theblackcat102.  The evolved code alpaca dataset.  https://huggingface.co/datasets/theblackcat102/evol-codealpaca-v1, 2023\.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * Wang et al. [2017] Xinyu Wang, Isil Dillig, and Rishabh Singh.  Program synthesis using abstraction refinement.  _Proc. ACM Program. Lang._ , 2(POPL), dec 2017.  doi: 10.1145/3158151.  URL https://doi.org/10.1145/3158151.
  * Wang et al. [2023a] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-instruct: Aligning language models with self-generated instructions.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13484–13508, Toronto, Canada, July 2023a. Association for Computational Linguistics.  doi: 10.18653/v1/2023.acl-long.754.  URL https://aclanthology.org/2023.acl-long.754.
  * Wang et al. [2021] Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi.  CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8696–8708, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.emnlp-main.685.  URL https://aclanthology.org/2021.emnlp-main.685.
  * Wang et al. [2023b] Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi D. Q. Bui, Junnan Li, and Steven C. H. Hoi.  Codet5+: Open code large language models for code understanding and generation, 2023b.
  * Wei et al. [2022] Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le.  Finetuned language models are zero-shot learners, 2022.
  * Wei et al. [2023] Yuxiang Wei, Chunqiu Steven Xia, and Lingming Zhang.  Copiloting the copilots: Fusing large language models with completion engines for automated program repair, 2023.
  * Xia and Zhang [2022] Chunqiu Steven Xia and Lingming Zhang.  Less training, more repairing please: Revisiting automated program repair via zero-shot learning, 2022.
  * Xia et al. [2023a] Chunqiu Steven Xia, Yifeng Ding, and Lingming Zhang.  Revisiting the plastic surgery hypothesis via large language models, 2023a.
  * Xia et al. [2023b] Chunqiu Steven Xia, Matteo Paltenghi, Jia Le Tian, Michael Pradel, and Lingming Zhang.  Universal fuzzing via large language models, 2023b.
  * Xia et al. [2023c] Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang.  Automated program repair in the era of large pre-trained language models.  In _2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)_ , pages 1482–1494, 2023c.  doi: 10.1109/ICSE48619.2023.00129.
  * Xu et al. [2023] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.  Wizardlm: Empowering large language models to follow complex instructions.  _arXiv preprint arXiv:2304.12244_ , 2023.
  * Yu et al. [2023] Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang.  Large language model as attributed training data generator: A tale of diversity and bias, 2023.
  * Zhang et al. [2023] Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen.  Repocoder: Repository-level code completion through iterative retrieval and generation, 2023.