#  Temperature and volume dependence of pion-pion scattering lengths 111
Supported by the Fostering Program in Disciplines Possessing Novel Features
for Natural Science of Sichuan University (Grant No. 2020SCUNL209).

Qing-Wu Wang 1 Hua-Zhong Guo 1 Email:
qw.wang<EMAIL_ADDRESS>1Department of Physics, Sichuan
University, Chengdu 610064, China

###### Abstract

$\mathbf{Abstract:}$ The s-wave pion-pion scattering lengths $a_{0}$ and
$a_{2}$ are studied at finite temperature and in finite spatial volume under
the framework of the Nambu–Jona-Lasinio model. With the proper time
regularization, the behavior beyond the pseudo transition temperature is
presented. The scattering length $a_{0}$ shows singularity near the Mott
temperature and $a_{2}$ is a continuous but non-monotonic function of
temperature. We present the finite volume effect on the scattering length and
have found that $a_{0}$ can be negative and its singularity disappears at
small volume size which may hint the existence of chiral phase transition as
volume decreases.

$\mathbf{Keywords:}$ scattering length, finite volume effect, chiral phase
transition

##  I introduction

Dynamical chiral symmetry breaking is an important feature of Quantum
Chromodynamics(QCD). With the rapid development in heavy ion experiment and
observation in astronomy, restoration of the chiral symmetry and the
deconfinement phase transition are expected to occur in ultra-relativistic
heavy-ion collisions or in the interior of neutron stars [1-4]. Many famous
research institutions, such as FAIR GSI, NICA JINR, and J-PARK, have carried
out experiments to study the properties of high-density matter which could
help to clarify the phase structure of quark matter.

As a gauge theory at short distances, perturbative Quantum Chromodynamics is a
remarkably successful and rich theory of the strong interactions. However,
many physical phenomena in a long distance have to be dealt with non-
perturbative method, especially in handling the problems related to low-energy
physics, the non-perturbation model is of particularly great use. Among many
effective models pion plays a crucial role with the fact that it occupies a
special place in nuclear and particle physics. In the standard picture, pion
is explained as the (pseudo-) Nambu-Goldstone boson. It arises as a
consequence of the dynamical breakdown of the chiral symmetry [5,6]. In the
chiral limit, where the masses of the two lightest quarks are turned off, the
pion mass is zero. When the chiral symmetry is broken spontaneously, the quark
condensate represents the leading order parameter. The Nambu-Goldstone bosons
can interact if they carry momentum. Weinberg’s low energy theorems state that
pion-pion scattering lengths are related to the pion mass with $a_{0}\sim
7M_{\pi}^{2}/(32\pi f_{\pi}^{2})$, $a_{2}\sim-M_{\pi}^{2}/(16\pi
f_{\pi}^{2})$. In the chiral limit, the pion-pion S-wave scattering lengths
vanish, and these quantities can be used as a sensitive probe of the chiral
symmetry breaking.

The pion-pion scattering, as one of the most fundamental hadronic processes of
QCD at the mesonic level, provides a direct link between the theoretical
formalisms of chiral symmetry and experiments. Most calculations so far have
been done in the infinite volume limit[7-13]. Works on the chiral phase
transition are stressed on the influence of temperature, baryon densities and
other outside parameters such as magnetic field [14-17]. But the quark-gluon
plasma(QGP) system produced by heavy-ion collisions experiments always has a
finite size. The volume of homogeneity before freeze out for Au–Au and Pb–Pb
collisions is between approximately $50\sim 250$ fm3. It is estimated that the
volume of the smallest QGP system could be as low as (2 fm)3 [18], motivated
by the estimated plasma size presumably formed in high-energy nucleus-nucleus
collision at RHIC [1 ]. Studies show that the chiral behavior depends on the
volume size of quark matter and the chiral symmetry breaking is closely
related to finite volume effects in QCD [19-22]. Besides, the finite size
effect on the dissociation and diffusion of chiral partners, on the phase
structure of the NJL model in D=3 Euclidean dimensions and on the viscosity,
bulk viscosity, electrical conductivity have been studied [23-25].

Since pion meson plays an important role in low-energy physics, it is
worthwhile to clarify its behaviors as extrapolation to high temperature and
small volume size.

The chiral phase transition of a finite system depends on the choice of the
boundary conditions [26,27]. For quark fields, the time direction is
constrained by anti-period requirement, but the choice of boundary condition
in spatial direction is at large. Typical boundary conditions are anti-period
boundary condition (APBC) and period boundary condition(PBC) which are used to
weaken the impact from physical boundary.

In this paper, the scheme of proper time regularization is adopted [28]. The
virtue of using proper time regularization here is that it enables us to
perform the sum over the thermal Matsubara frequencies analytically. Other
than most previous works which use cutoff scheme, we first use the proper time
regularization to study the pion-pion scattering length in a finite volume
size. We expect to extract the signals of chiral phase transition from pion-
pion interactions in the hot medium while considering the finite volume
effect.

The paper is organized as follows. In section II, the volume and temperature
dependence of effective quark mass is deduced from Nambu–Jona-Lasinio (NJL)
model under the proper time regularization. In section III, the equations for
the meson mass and the pion decay constant at finite volume are deduced. We
also give numerical results and analyze them in detail. The scattering length
formula and its corresponding numerical results are presented in section IV,
closely followed by a short conclusion in section V.

##  II NJL model at finite volume size

We use a two-flavor NJL Lagrangian model, which is motivated by the symmetries
of QCD, to describe the coupling between quarks and the chiral condensate in
the scalar-pseudoscalar sector. It reads as [29,30]

$\displaystyle\mathcal{L}_{NJL}$ $\displaystyle=$
$\displaystyle\bar{\psi}(i\gamma_{\mu}\partial^{\mu}-m_{q})\psi$ (1)
$\displaystyle+$ $\displaystyle
G[(\bar{\psi}\psi)^{2}+(\bar{\psi}i\gamma_{5}\tau\psi)^{2}],$

where $m_{q}$ is the current quark mass of flavor $q$ and $G$ is the four
quark effective coupling. In the limit of exact isospin symmetry,
$m_{u}=m_{d}=m$. The pion mass can be exploited through the effective
interaction for the exchange of a pion in the random-phase approximation.

The second term of four quark interaction in Eq. (1) is responsible for
exciting the pion as an isovector pseudoscalar. In the mean field
approximation, the effective quark mass is

$M=m+\sigma$ (2)

with

$\sigma=-2G\left\langle{\bar{\psi}\psi}\right\rangle$ (3)

and the two-quark condensate is defined as

$\left\langle{\bar{\psi}\psi}\right\rangle=-\int\frac{d^{4}p}{(2\pi)^{4}}Tr[S(p)],$
(4)

where $S(p)$ is the dressed quark propagator and the trace is taken in color,
flavor and Dirac spaces.

The integration of Eq.(4) is ultraviolet divergent. For simplicity a cutoff on
the momentum integration is usually taken which is valid when the cutoff is
lager than the relevant momenta. We will use the proper time regularization
[31-36] for two reasons. Firstly, we will work with a period or anti-period
boundary condition but the cutoff on the momentum breaks the symmetry in the
spatial direction. Secondly, a translation of the quark momentum is required
in deducing the equation of meson mass which requires the cutoff to tend to
infinity. The proper time method could overcome these difficulties by
introducing a new integration. Under this kind of regularization scheme the
trace term in Eq.(4) is replaced by an integral with a suitable choice of the
cutoff function. Here in the quark gap equation the key equation is a
replacement

$\frac{1}{A^{n}}\rightarrow\frac{1}{(n-1)!}\int_{\tau_{UV}}^{\tau_{IR}}d\tau\tau^{n-1}e^{-\tau
A}.$ (5)

Here, $\tau_{UV}$ is introduced to regularize the ultra-violet divergent and
an infrared cutoff $\tau_{IR}$ is adopted which appears also in the literature
[36-37].

By using a Wick rotation, the two quark condensate at infinity volume and zero
temperature can be written as

$\displaystyle\left\langle{\bar{\psi}\psi}\right\rangle$ $\displaystyle=$
$\displaystyle-N_{c}N_{f}\int\frac{d^{4}p}{(2\pi)^{4}}\frac{4M}{p^{2}+M^{2}}$
(6) $\displaystyle=$
$\displaystyle-24M\int^{\infty}_{-\infty}\frac{d^{4}p}{(2\pi)^{4}}\int^{\tau_{IR}}_{\tau_{UV}}d\tau
e^{-\tau(p^{2}+M^{2})}$ $\displaystyle=$
$\displaystyle-\frac{3M}{2\pi^{2}}\int^{\tau_{IR}}_{\tau_{UV}}d\tau\frac{e^{-\tau
M^{2}}}{\tau^{2}},$

with the number of color is $N_{c}=3$ and the number of flavor $N_{f}=2$. At
non-zero temperature, the quark four-momentum is replaced by
$p_{k}=(\vec{p},\omega_{k})$, with $\omega_{k}=(2k+1)\pi T$, $k\in\mathbb{Z}$.
The integration on fourth momentum in Eq. (6) is replaced by a sum of all the
fermion Matsubara frequencies $\omega_{k}$. The two-quark condensate is given
by

$\displaystyle\left\langle{\bar{\psi}\psi}\right\rangle$ $\displaystyle=$
$\displaystyle-24M\int^{\tau_{IR}}_{\tau_{UV}}d\tau e^{-\tau M^{2}}\times$ (7)
$\displaystyle[T\sum^{\infty}_{k=-\infty}\int^{\infty}_{0}\frac{dp}{2\pi^{2}}\vec{p}^{2}e^{-\tau(\vec{p}^{2}+\omega_{k}^{2})}]$
$\displaystyle=$
$\displaystyle-\frac{3MT}{\pi^{3/2}}\int^{\tau_{IR}}_{\tau_{UV}}d\tau\frac{e^{-\tau
M^{2}}}{\tau^{3/2}}\theta_{2}(0,e^{-4\pi^{2}\tau T^{2}}),$

where the Jacobi function is defined as
$\theta_{2}(0,q)=2\sqrt[4]{q}\sum\nolimits_{n=0}^{\infty}q^{n(n+1)}.$ Then the
quark mass is

$\displaystyle
M=m+\frac{6GMT}{\pi^{3/2}}\int^{\tau_{IR}}_{\tau_{UV}}d\tau\frac{e^{-\tau
M^{2}}}{\tau^{3/2}}\theta_{2}(0,e^{-4\pi^{2}\tau T^{2}}).$ (8)

For a specific boundary condition of finite volume, the quark momentum is
discretized and the integral over all spatial momenta is replaced by sum over
discrete momentum modes. Considering a cubic box with volume size $L$, the
discrete momenta that depend on the boundary conditions are

$\displaystyle\vec{p}_{PBC}^{2}$ $\displaystyle=$
$\displaystyle\frac{4\pi^{2}}{L^{2}}\sum\nolimits_{i=1}^{3}n_{i}^{2},\qquad
n_{i}=0,\pm 1,\pm 2\cdots,$ (9) $\displaystyle\vec{p}_{APBC}^{2}$
$\displaystyle=$
$\displaystyle\frac{4\pi^{2}}{L^{2}}\sum\nolimits_{i=1}^{3}\left(n_{i}+\frac{1}{2}\right)^{2},\quad
n_{i}=\pm 1,\pm 2\cdots~{}~{}~{}~{}~{}~{}$ (10)

for period and anti-period boundary conditions, respectively. The integration
measure is replaced by sum of discrete momentum with replacement of

$\int
dp\left(\cdots\right)\rightarrow\frac{2\pi}{L}\sum_{n_{i}}\left(\cdots\right).$
(11)

Then the quark mass is constrained by

$\displaystyle M$ $\displaystyle=$ $\displaystyle
m+48GM\int^{\tau_{IR}}_{\tau_{UV}}d\tau e^{-\tau M^{2}}[T\times$ (12)
$\displaystyle\sum^{\infty}_{k=-\infty}e^{-\tau\omega_{k}^{2}}\prod\limits_{i=1}^{3}\sum_{n_{i}}e^{-\tau
p_{i}^{2}}]$ $\displaystyle=$ $\displaystyle
m+48GMT\int^{\tau_{IR}}_{\tau_{UV}}d\tau e^{-\tau M^{2}}\times$
$\displaystyle\theta_{2}(0,e^{-4\pi^{2}\tau
T^{2}})\left[\frac{f(\theta)}{L}\right]^{3},$

with

$\displaystyle{f(\theta)}$ $\displaystyle=$
$\displaystyle\left\\{\begin{array}[]{l}{\theta_{2}(0,e^{-4\tau\pi^{2}/L^{2}})\qquad\qquad\text{for
APBC;}}\\\ {\theta_{3}(0,e^{-4\tau\pi^{2}/L^{2}})\qquad\qquad\text{for
PBC;}}\\\ \end{array}\right.$ (15)

Here $\theta_{3}(0,q)=1+2\sum\nolimits_{n=1}^{\infty}q^{n^{2}}$, while
$\theta_{2}$ is defined as before. Although the momentum integrals are given
by two different functions, it can be checked that these two functions
approach a same limit as $L$ increases to very large value. Thus, the quark
mass will not depend on the boundary condition in the infinite volume limit.

##  III Pion mass and decay constant

The pion, associated with the exact $SU_{L}(2)\times SU_{R}(2)$ symmetry,
occupies a special place in nuclear and particle physics. It is the most
relevant degree of freedom in the low energy regime of the strong interaction,
and is both (pseudo-)Nambu-Goldstone bosons and quark-antiquark bound-states.
The pion mass and decay constant are measures for the strength of chiral
symmetry breaking [13,38].

In the Lagrangian of Eq. (4), the four quark interaction term
$(\bar{\psi}\psi)^{2}$ is associated with the scalar $\sigma$ meson while term
$(\bar{\psi}i\gamma_{5}\tau\psi)^{2}$ the $\pi$ meson. By comparing with the
amplitude of $qq$ scattering for the exchange of a pion, the meson mass is
deduced from the proper polarization insertion of the four quark interaction.
In the random phase approximation, the polarization insertion is related to
quark propagator with

$\displaystyle\Pi_{ps}(k^{2})\delta_{\alpha\beta}=\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad$
$\displaystyle-i\int\frac{d^{4}p}{(2\pi)^{4}}Tr[i\gamma_{5}T_{\alpha}iS(p+k/2)i\gamma_{5}T_{\beta}iS(p-k/2)].~{}~{}~{}~{}~{}$
(16)

Here $T_{i}$ acts on the external quarks, $p$ is the quark momentum and $k$ is
the meson momentum. The trace is taken in Dirac, flavor and color space. Then
the meson mass is the solution of

$1-2G\Pi_{ps}(k^{2}=m^{2})=0,$ (17)

and the effective coupling strength between pion meson and quarks $g_{\pi qq}$
is defined as

$\displaystyle g_{\pi qq}^{2}=({\partial\Pi_{ps}}/{\partial
k^{2}})^{-1}|_{k^{2}=m^{2}}.$ (18)

Performing the trace in Eq. (III) the proper polarization is

$\displaystyle\Pi_{ps}(k^{2})$ $\displaystyle=$
$\displaystyle-4iN_{c}N_{f}\int\frac{d^{4}p}{(2\pi)^{4}}\times$ (19)
$\displaystyle\frac{M^{2}+p^{2}-k^{2}/4}{[(p+k/2)^{2}-M^{2}][(p-k/2)^{2}-M^{2}]}.~{}~{}$

After making appropriate shifts of quark momentum we can get

$\displaystyle\dfrac{1}{i}\Pi_{ps}(k^{2})$ $\displaystyle=$ $\displaystyle
4N_{c}N_{f}\int\frac{d^{4}p}{(2\pi)^{4}}\frac{1}{p^{2}-M^{2}}$ (20)
$\displaystyle-2N_{c}N_{f}k^{2}I(k)$

with

$\displaystyle
I(k)=\int\frac{d^{4}p}{(2\pi)^{4}}\frac{1}{[(p+k)^{2}-M^{2}](p^{2}-M^{2})}.$
(21)

Comparing Eq. (4) with Eq. (20), we have

$2G\Pi_{ps}=1-\frac{m}{M}+4iGN_{c}N_{f}m_{\pi}^{2}I(m_{\pi}^{2}).$ (22)

By inserting it into Eq. (17), the mass of pion meson is deduced with

$m_{\pi}^{2}=-\frac{m}{M}\frac{1}{4iGN_{c}N_{f}I(m_{\pi}^{2})}.$ (23)

The meson mass is proportional to the current quark mass and depends on the
effective quark mass. In the chiral limit with $m=0$, the meson mass is
$m_{\pi}^{2}=0$.

The main task now is to calculate the integration of $I(k)$. After Wick
rotation the calculation on $I(k)$ is available by introducing the Feynman
parameters which gives

$\displaystyle I(k)$ $\displaystyle=$
$\displaystyle\int_{0}^{1}dz\int\frac{d^{4}p}{(2\pi)^{4}}\times$ (24)
$\displaystyle\frac{1}{\\{[p-k(1-z)]^{2}-M^{2}+k^{2}(1-z)z\\}^{2}}.$

In the proper time scheme, we have

$\displaystyle I(m_{\pi}^{2})$ $\displaystyle=$
$\displaystyle\int_{0}^{1}dz\int_{\tau_{UV}}^{\tau_{IR}}d\tau\int\frac{d^{4}p}{(2\pi)^{4}}\times$
(25) $\displaystyle{\tau e^{-\tau[p^{2}+M^{2}-m_{\pi}^{2}(1-z)z]}}.$

At nonzero temperature and finite volume size, the discretizing in temporal
and spatial directions gives

$\displaystyle I(m_{\pi}^{2})$ $\displaystyle=$ $\displaystyle
T\int_{0}^{1}dz\int_{\tau_{UV}}^{\tau_{IR}}d\tau
e^{-\tau[M^{2}-m_{\pi}^{2}(1-z)z]}\times$ (26)
$\displaystyle\tau\theta_{2}(0,e^{-4\pi^{2}\tau
T^{2}})\left[\frac{f(\theta)}{L}\right]^{3},$

where $f(\theta)$ is defined as in Eq. (15).

Pion decay constant is an important quantity in low energy phenomena which can
be extracted from measurements of the decay $\pi^{-}\to\mu^{-}+\nu_{\mu}$.
From the vacuum to one pion and axial vector current matrix element
$\left<0\left|J^{i}_{5\mu}\right|\pi^{j}\right>$, the decay constant is given
by

$\displaystyle ik_{\mu}f_{\pi}\delta^{ij}$ $\displaystyle=$ $\displaystyle-
N_{c}g_{\pi qq}\int\dfrac{d^{4}p}{(2\pi)^{4}}tr[\gamma_{\mu}\gamma_{5}$ (27)
$\displaystyle\times S(p+\frac{1}{2}k)\gamma_{5}S(p-\frac{1}{2})k].$

Using the relation$tr\tau^{i}\tau^{j}=2\delta^{ij}$ and performing the traces
over spin labels yields

$ik_{\mu}f_{\mu}=N_{c}g_{\pi qq}4Mk_{\mu}I(k^{2}).$ (28)

The effective coupling $g_{\pi qq}$ is obtained by substituting Eq.(20) into
Eq.(18). Then the square of decay constant is

$f_{\pi}^{2}=-4iN_{c}N_{f}M^{2}\dfrac{I^{2}(m_{\pi})}{I(0)+I(m_{\pi})-m_{\pi}^{2}K(m_{\pi})}.$
(29)

The equation for the pion decay constant, independent of the regularization
scheme, is related to the quark mass. And we can get a relation

$f_{\pi}^{2}g_{\pi
qq}^{2}=4M^{2}\dfrac{I^{2}(m_{\pi})}{[I(0)+I(m_{\pi})-m_{\pi}^{2}K(m_{\pi})]^{2}}.$
(30)

The function $K$ is defined here as

$\displaystyle
K(k)=\int\frac{d^{4}p}{(2\pi)^{4}}\frac{1}{[(p+k)^{2}-M^{2}](p^{2}-M^{2})^{2}},$
(31)

and it can be calculated through $I(k)$ with

$\displaystyle
K(k)=\dfrac{1}{k^{2}-4M^{2}}\left[12M^{2}\dfrac{dI}{dk^{2}}(0)-I(k)+I(0)\right].~{}~{}$
(32)

However, to calculate the function $K(k)$ and latter the function $L(k)$
directly from Eq. (31), we can use the following Feynman parameter formula

$\displaystyle\dfrac{1}{A^{n}B^{m}}$ $\displaystyle=$
$\displaystyle\dfrac{(n+m-1)!}{(n-1)!(m-1)!}$ (33)
$\displaystyle\times\int_{0}^{1}dx\dfrac{x^{n-1}(1-x)^{m-1}}{[Ax+B(1-x)]^{n+m}}.~{}~{}~{}~{}~{}~{}~{}$

For an approximation, taking the function $I(k)$ as a smooth function of
$k^{2}$, we have $K(k)=0$. Then, the decay constant is simplified to

$f_{\pi}^{2}=-4iN_{c}N_{f}M^{2}I(0).$ (34)

As the meson mass is small, the combination of Eq.(23) for the pion mass with
Eq.(34) for the decay constant gives

$m_{\pi}^{2}f_{\pi}^{2}=mM(N_{f}G)^{-1}.$ (35)

Comparing with Eq.(2) and Eq.(3) for a small $m$, we have

$m_{\pi}^{2}f_{\pi}^{2}=-m\left<\bar{\psi}\psi\right>,$ (36)

which is the lowest order approximation to the current algebra result and
called as Gell-Mann–Oakes–Renner (GOR) relation [39].

The parameters we used in this paper are $m=4.8$ MeV, $G=3.19*10^{-6}$ MeV-2,
$\tau_{UV}=1/1080^{2}$ MeV -2, and $\tau_{IR}=1/190^{2}$ MeV-2. In this
parameter set, the effective quark mass is 202.3 MeV and the pion meson mass
is 135.1 MeV. The quark mass scaled to the mass at zero temperature is only
slightly dependent on different set of parameters [36]. And we will not have
the coupling $G$ depending on the volume size $L$ and the other properties.
The corresponding decay constant $f_{\pi}=93.0$ MeV and from the Weinberg’s
formula the scalar pion-pion scattering lengths in unit of $m_{\pi}^{-1}$ are
$a_{0}=0.147$ and $a_{2}=-0.042$, respectively.

Figure 1: Pion meson mass as a function of temperature with different boundary
condtion and volume size. The plot markers $\Diamond,\Delta,\square,\nabla$
and $\star$ stand for the volume size $L=1,1.5,2,3$ and 5 fm, respectively.

The pion mass and decay constant as functions of temperature with different
cubic volume size are presented in Fig. 1 and Fig. 2, respectively. The pion
mass increases and the decay constant decreases as temperature increases. The
data with volume size $L$ larger than 5 fm is close to the infinite volume
limit.

The GOR relation only keeps well in low temperature and in the infinite volume
limit. On the right hand side of the GOR relation Eq. (36), the current quark
mass comes from the Higgs mechanism which does not depend on the size effect
and is a constant parameter in NJL model here. The chiral symmetry is almost
restored at high temperature in finite size, so the quark condensate is close
to zero. Then the right hand side of GOR equation is almost temperature and
volume size independent. But on the left hand side of the equation the pion
mass times decay constant still depend on T, which makes the GOR no longer
hold.

Furthermore, with different choice of boundary conditions, the influence of
volume size, both on the mass and decay constant, are different. It will also
reflect on the scattering length.

Figure 2:  Decay constant as a function of temperature with different boundary
condition and volume size. The plot markers $\Diamond,\Delta,\square,\nabla$
and $\star$ stand for the volume size $L=1,1.5,2,3$ and 5 fm, respectively.

When the temperature increases, the effective quark mass decreases while the
pion mass increases. As $m_{q}(T)$ = $2M(T)$, which means that the pion can
dissociate into a constituent quark and an antiquark, it defines the Mott
temperature $T_{Mott}$ for pion meson. We can see from Fig. 3 that the Mott
temperature is about 155 MeV in the infinity volume limit. In Fig. 3, we
present the Mott temperature as a function of volume size for PBC and APBC.
The Mott temperature decreases (increases) with decreasing volume size for
APBC(ABC) which may hint that when the volume size effect can not be
neglected, APBC is favored, at relatively low temperature, to get the quark
gluon plasma.

Figure 3:  Mott temperature as function of volume size.

##  IV scattering length

From the Weinberg’s low energy theorems [40], the pion scattering lengths are
related to the pion mass and decay constant, which also represents a symmetry
breaking effect. The scattering of pion by pions involves only the lightest
pseudoscalar modes. It provides a direct link between the theoretical
formalism of chiral symmetry and experiment.

Three isospin channels are available for the pion-pion scattering process. The
invariant scattering amplitude $\left(\Pi^{a}\Pi^{b}\to\Pi^{c}\Pi^{d}\right)$
can be written as

$\displaystyle T_{ab,cd}(k,p\to
k^{\prime},p^{\prime})=A(s,t,u)\delta_{ab}\delta_{cd}$
$\displaystyle+B(s,t,u)\delta_{ac}\delta_{bd}+C(s,t,u)\delta_{ad}\delta_{bc},$
(37)

with incoming momenta $(k,p,k^{\prime},p^{\prime})$ and isospin indices
$(a,b,c,d)$. The Mandelstam variables $s$, $t$ and $u$ are defined as
$s=(k+p)^{2}$, $t=(k-k^{\prime})^{2}$ and $u=(k-p^{\prime})^{2}$. Six diagrams
(box and $\sigma$-propagation) contribute to pion-pion scattering in the tree
level which can be found in Refs. [7-10]. We present then in Fig. 4

Figure 4:  Box and $\sigma$-propagation diagrams for pion-pion scattering.

The scattering lengths at the kinematic threshold are

$\displaystyle a_{i}=\dfrac{1}{32\pi}A_{i}(s=4m_{\pi}^{2},t=0,u=0),$ (38)

with the three isospin amplitudes

$\displaystyle A_{0}=3A+B+C,\quad A_{1}=B-C,\quad A_{2}=B+C.~{}~{}~{}$ (39)

Here, A, B and C can be calculated from $T_{a}$, $T_{b}$, $T_{c}$, $T_{d}$,
$T_{e}$, and $T_{f}$ that correspond to the amplitudes of the six scattering
diagrams. From those diagrams, we have $T_{a}=T_{b}$ and $T_{e}=T_{f}$. Then,
A, B and C in Eq. (IV) and Eq. (39) are given by

$\displaystyle A=2T_{a}-T_{c}+T_{d},\quad B=C=T_{c}+T_{d}.$ (40)

Since $B=C$, we have $A_{1}=0$. So $a_{1}$ can not be calculated in this work.
The nonzero amplitudes are

$\displaystyle A_{0}=6T_{a}-T_{c}+3T_{d}+2T_{e}$ (41)

and

$A_{2}=2(T_{c}+T_{e}).$ (42)

Note that, in most works a superscript is use to indicate the different
isospin. Since here we only concern the s-wave scattering length and to avoid
confusion with power exponents, we use a subscript to distinguish the
different isospin scattering lengths as in Ref.[9].

The six $T_{i}$ values corresponding to the scattering diagrams are

$\displaystyle T_{a}$ $\displaystyle=$ $\displaystyle\dfrac{4N}{i}g_{\pi
qq}^{4}[k^{2}K(k)-I(0)-I(k)],$ (43) $\displaystyle T_{c}$ $\displaystyle=$
$\displaystyle\dfrac{8N}{i}g_{\pi
qq}^{4}[2k^{2}K(k)-I(0)-\dfrac{1}{2}k^{4}L(k)],$ (44) $\displaystyle T_{d}$
$\displaystyle=$ $\displaystyle\dfrac{8N}{i}g_{\pi
qq}^{4}\dfrac{I^{2}(k)}{(1-k^{2}/M^{2})I(2k)+k^{2}/(4M^{2})I(k)},~{}~{}~{}~{}$
(45) $\displaystyle T_{e}$ $\displaystyle=$ $\displaystyle\dfrac{8N}{i}g_{\pi
qq}^{4}\dfrac{[I(0)-k^{2}K(k)]^{2}}{I(0)+k^{2}/(4M^{2})I(k)},$ (46)

with $T_{b}=T_{a}$, $T_{f}=T_{e}$. Here $L(k)$ is given by

$\displaystyle
L(k)=\int\frac{d^{4}p}{(2\pi)^{4}}\frac{1}{[(p+k)^{2}-M^{2}]^{2}(p^{2}-M^{2})^{2}}.$
(47)

From Eq.(30), $g_{\pi qq}^{4}$ is given as

$g_{\pi qq}^{-4}=-N_{c}^{2}N_{f}^{2}[I(0)+I(k)-m_{\pi}^{2}K(k)]^{2}.$ (48)

with $k^{2}=m_{\pi}^{2}$. After performing the calculations of $I(k)$, $L(k)$
and $K(k)$, the scattering lengths in Eq.(38) can be obtained.

The numerical results are presented on Fig. 5 and Fig. 6 for period and
antiperiod boundary condition, respectively. The scattering lengths are
calculated as function of temperature at several volume sizes. As volume is
larger than 5 fm, the curve is close to the infinite volume limit. The results
is similar to results with cutoff regularization that the $a_{0}$ and $a_{2}$
vary only slowly firstly and then display steep singularity near the Mott
temperature [8,10]. The difference results of the two regularizations appear
beyond the Mott temperature. In the cutoff regularization, because the chiral
phase transition is first order, no results beyond the Mott temperature. In
the proper time regularization, the chiral phase transition is crossover. So
the effective quark mass and scattering lengths can have continuous values
beyond the Mott temperature or pseudo critical temperature of chiral phase
transition.

Figure 5: The scattering lengths $a_{0}$ for (anti-) period boundary
condition. The plot markers $\Diamond,\Delta,\square,\nabla$ and $\star$ stand
for the volume size $L=1,1.5,2,3$ and 5 fm, respectively.

The scattering lengths show apparently different behaviors as volume size
decreases for the two kinds of boundary condition. This can be expected from
the behaviors of quark mass, meson mass and decay constant as the volume size
is changing. Direct analysis of the Weinberg’s formula $a_{0}\sim
7M_{\pi}^{2}/(32\pi f_{\pi}^{2})$ with the results of $M_{\pi}$ and $f_{\pi}$
showed in Fig. 1 and Fig. 2 gives that the scattering length $a_{0}$ at
temperature lower than the pseudo critic temperature increases as the volume
size decrease for the antiperiod boundary condition, but decreases as the
volume size decrease for the period boundary condition. The same analysis can
be used for $a_{2}$ with $a_{2}\sim-M_{\pi}^{2}/(16\pi f_{\pi}^{2})$. However,
we see in Fig.5 that, $a_{0}$ increases and then decreases as volume size
decreases at low temperature.

The scattering length $a_{0}$ shows jump as the volume size is not small
enough. We define the jump as the pseudo critical temperature ($T_{c}$) which
is less than the Mott temperature. For the period boundary condition, the jump
position in the $a_{0}$ curve increases as volume size decreases and the jump
always exists. For the antiperiod boundary condition, $T_{c}$ in the $a_{0}$
curve decreases as volume size decreases. As the volume size in small enough
the jump disappears. When the volume size is large enough and temperature is
less than $T_{c}$, the scattering length $a_{0}$ is larger than zero and
increases with temperature; when temperature is larger than $T_{c}$, the
scattering length $a_{0}$ is less than zero and increases firstly and then
decreases as temperature increases.

The scattering length $a_{0}$ can be negative at small volume size, which can
be used as an examination of the existence of phase transition. Different from
the period boundary condition, for the antiperiod boundary condition, $a_{0}$
is a continuous function of temperature at $L=1$ fm, which may due to that
chiral symmetry is partly restored at small finite volume [27]. Since the
volume of the smallest QGP system could be as low as (2 fm)3 [18], a
continuous scattering length $a_{0}$ can serve as a criterion for testing
volume effects and different boundary conditions.

Figure 6: The scattering lengths $a_{2}$ for (anti-) period boundary
condition. The plot markers $\Diamond,\Delta,\square,\nabla$ and $\star$ stand
for the volume size 1, 1.5, 2, 3 and 5 fm, respectively.

The scattering length $a_{2}$ is always a connected function of temperature.
It is also different with the cut off results. At very small volume size, the
scattering length $a_{2}$ is monotonically increases. As the volume size
gradually increases, $a_{2}$ decreases with the temperature firstly and then
increases. A minimum exists, but which does not occurred at $T_{c}$.

For the cut off regularization scheme, $a_{0}$ increases within a narrow area
near the Mott temperature and both for $a_{0}$ and $a_{2}$ no data exist
beyond the Mott temperature [8,10].

##  V conclusion

. In this paper, we have studied the temperature and volume size dependence of
pion decay constant and pion-pion scatterings for different boundary
conditions. We show the chiral phase transition of the quark matter at finite
temperature and in finite spatial volume. Under the proper regularization, the
phase transition indicated from the effective quark mass is crossover
different from the results of cutoff regularization where the phase transition
is first order and the variation of pi-pi scattering length $a_{0}$ with
temperature beyond the Mott temperature between the two regularization scheme.

The calculated pi-pi scattering lengths of $a_{0}$ and $a_{2}$ in finite
spatial volume shows different behaviors for different boundary conditions.
Although we can not determine which boundary condition is the best one, but
the results deserve to arouse our attention.

The pion-pion scattering, as one of the most fundamental hadronic processes of
QCD at the mesonic level, may serve as a tools to check the different boundary
condition and regularization scheme and we hope the results obtained here may
be checked by other theoretical methods.

##  VI Acknowledgment

The authors thank the Chinese Institute of High Energy Physics and Prof. Huang
Mei for authorizing the download of the doctoral dissertation.

.

[1] J. Adams et al. (STAR), Nucl. Phys. A 757, 102 (2005).

[2] X. Luo and N. Xu, Nucl. Sci. Tech. 28, 112 (2017).

[3] K. Fukushima and T. Hatsuda, Rept. Prog. Phys. 74, 014001 (2011).

[4] E. Annala, T. Gorda, A. Kurkela, J. N¨attil¨a, and A. Vuorinen, Nature
Phys. 16, 907 (2020).

[5] T. Horn and C. D. Roberts, J. Phys. G 43, 073001 (2016).

[6] V. Pascalutsa and M. Vanderhaeghen, Phys. Rev. D 73, 034003 (2006).

[7] H. J. Schulze, J. Phys. G 21, 185 (1995).

[8] E. Quack, P. Zhuang, Y. Kalinovsky, S. P. Klevansky, and J. Hufner, Phys.
Lett. B 348, 1 (1995).

[9] M. Huang, P. Zhuang, and W. Chao, Phys. Lett. B 465, 55 (1999).

[10] W.-J. Fu and Y.-X. Liu, Phys. Rev. D 79, 074011 (2009).

[11] G. Colangelo, J. Gasser, and H. Leutwyler, Phys. Lett. B 488, 261 (2000).

[12] Z. T. Draper and S. R. Sharpe, Phys. Rev. D 105, 034508 (2022).

[13] J. Eser and J.-P. Blaizot, Phys. Rev. D 105, 074031 (2022).

[14] G. S. Bali, F. Bruckmann, G. Endrodi, Z. Fodor, S. D. Katz, and A.
Schafer, Phys. Rev. D86, 071502 (2012).

[15] B.-k. Sheng, X. Wang, and L. Yu, Phys. Rev. D 105, 034003 (2022).

[16] Q. W. Wang, Z. F. Cui, and H. S. Zong, Phys. Rev. D 94, 096003 (2016).

[17] Z. Fodor and S. D. Katz, JHEP 04, 050 (2004)

[18] L. F. Palhares, E. S. Fraga, and T. Kodama, J. Phys. G 38, 085101 (2011).

[19] J. Luecker, C. S. Fischer, and R. Williams, Phys. Rev. D 81, 094005
(2010).

[20] J. Gasser and H. Leutwyler, Phys. Lett. B 184, 83 (1987).

[21] J. Braun, B. Klein, and H. J. Pirner, Phys. Rev. D 72, 034017 (2005).

[22] Y. Xia, Q. Wang, H. Feng, and H. Zong, Chinese Physics C 43, 034101
(2019).

[23] L. M. Abreu, M. Gomes, and A. J. da Silva, Phys. Lett. B 642, 551 (2006),
hep-th/0610111.

[24] K. Saha, S. Ghosh, S. Upadhaya, and S. Maity, Phys. Rev. D 97, 116020
(2018), 1711.10169.

[25] P. Deb, S. Ghosh, J. Prakash, S. K. Das, and R. Varma, Chin. Phys. C 46,
044102 (2022), 2005.12037.

[26] B. Klein, Physics Reports 707-708, 1 (2017).

[27] Q. Wang, Y. Xia, and H. Zong, Mod. Phys. Lett. A 33, 1850232 (2018).

[28] J. S. Schwinger, Phys. Rev. 82, 664 (1951).

[29] M. Buballa, Phys. Rept. 407, 205 (2005).

[30] S. P. Klevansky, Rev. Mod. Phys. 64, 649 (1992).

[31] J. Braun, B. Klein, and H. Pirner (2005).

[32] J. Braun, B. Klein, H. J. Pirner, and A. H. Rezaeian, Phys. Rev. D 73,
074010 (2006).

[33] J. Braun, B. Klein, and H. J. Pirner, Phys. Rev. D 72, 034017 (2005).

[34] Z.-F. Cui, J.-L. Zhang, and H.-S. Zong, Sci. Rep. 7, 45937 (2017).

[35] S.-B. Liao, Phys. Rev. D 53, 2020 (1996).

[36] Z.-F. Cui, C. Shi, W.-M. Sun, Y.-L. Wang, and H.-S. Zong, Eur. Phys. J. C
74, 2782 (2014).

[37] Y. Ninomiya, W. Bentz, and I. C. Cloet, Physical Review C 91, 025202
(2015).

[38] T. Horn and C. D. Roberts, J. Phys. G 43, 073001 (2016)

[39] M. Gell-Mann, R. J. Oakes, and B. Renner, Phys. Rev. 175, 2195 (1968).

[40] S. Weinberg, Phys. Rev. Lett. 17, 616 (1966).


# R2C2-Coder: Enhancing and Benchmarking Real-world Repository-level Code
Completion
Abilities of Code Large Language Models

Ken Deng*, Jiaheng Liu*†, He Zhu*, Congnan Liu, Jingxin Li,
Jiakai Wang, Peng Zhao, Chenchen Zhang, Yanan Wu, Xueqiao Yin,
Yuanxing Zhang, Wenbo Su, Bangyu Xiang, Tiezheng Ge, Bo Zheng
Alibaba Group
{dengken.deng<EMAIL_ADDRESS>
###### Abstract

Code completion models have made significant progress in recent years.
Recently, repository-level code completion has drawn more attention in modern
software development, and several baseline methods and benchmarks have been
proposed. However, existing repository-level code completion methods often
fall short of fully using the extensive context of a project repository, such
as the intricacies of relevant files and class hierarchies. Besides, the
existing benchmarks usually focus on limited code completion scenarios, which
cannot reflect the repository-level code completion abilities well of existing
methods. To address these limitations, we propose the R2C2-Coder to enhance
and benchmark the real-world repository-level code completion abilities of
code Large Language Models, where the R2C2-Coder includes a code prompt
construction method R2C2-Enhance and a well-designed benchmark R2C2-Bench.
Specifically, first, in R2C2-Enhance, we first construct the candidate
retrieval pool and then assemble the completion prompt by retrieving from the
retrieval pool for each completion cursor position. Second, based on
R2C2-Enhance, we can construct a more challenging and diverse R2C2-Bench with
training, validation and test splits, where a context perturbation strategy is
proposed to simulate the real-world repository-level code completion well.
Extensive results on multiple benchmarks demonstrate the effectiveness of our
R2C2-Coder.

††footnotetext: * First three authors contributed equally.††footnotetext: †
Corresponding Author: Jiaheng Liu.

##  1 Introduction

Large Language Models for Code (Code LLMs), such as Codex [12], CodeGen [40,
41], and StarCoder [32], have shown their power to enhance developer
productivity as their promising results in code completion [4, 5, 12, 39].
Besides, to evaluate these models in real-world scenarios, multiple code
completion benchmarks are proposed. Recently, the challenging repository-level
code completion has drawn more attention, where the extensive cross-file
dependencies (i.e., contextual information from other code files within the
same repository) are used to enhance the completion performance. Therefore, a
direct but complex solution is to enhance the context length of LLMs [13, 14,
43]. Nonetheless, increasing the context length significantly raises costs and
is not affordable, especially concerning the large number of files in a given
repository. Hence, several recent baseline methods [16, 37] apply the widely-
used Retrieval-Augmented Generation (RAG) strategy by retrieving the most
relevant code snippets to improve the repository-level code completion.

However, this vanilla strategy exhibits significant limitations. First, it
usually preprocesses the source code files into code snippets under a
repository in Fig. 1, and performs retrieval on these code fragments for
retrieval query. However, these fragments only preserve local and discrete
information and fail to cover more diverse and fundamental contexts of program
semantics, such as the intricate network of dependencies, shared utility
functions, definition of the input and output parameters, etc. Besides, as
retrieved code snippets based on retrievers (e.g., BM25 [48]) usually contain
irrelevant or noisy contexts, which may affect the completion results, it is
important to evaluate and improve the robustness of the noisy contexts for
Code LLMs. Moreover, we observe that the existing repository-level code
completion benchmarks focus on limited completion scenes and cannot fully
simulate real-world usage well (See Section 4).

Figure 1: Examples of code snippets.

To address these limitations, we propose the R2C2-Coder to enhance and
benchmark the Real-world Repository-level Code Completion abilities of Code
LLMs, where a code prompt construction method (i.e., R2C2-Enhance) and a
challenging benchmark (i.e., R2C2-Bench) are proposed.

First, R2C2-Enhance includes two stages (i.e., candidate retrieval pool
construction, and completion prompt construction). In the retrieval pool
construction stage, we first build the retrieval pool using the abstraction
context based on the parser generator tool (e.g., Tree-sitter111https://tree-
sitter.github.io/tree-sitter/) and the snippet context, where the abstraction
context aims to represent the coarse-grained global information of each
programming file and the snippet context is built by extracted code fragments
to provide the fine-grained local information for completion. Then, in
completion prompt construction stage, for the current cursor position222The
cursor position denotes the location, where code completion is about to be
triggered., we first build the retrieval query and produce the retrieved
contexts by performing the retrieval between the retrieval query and the
retrieval pool, where the retrieved contexts are then combined with the
current code file to generate the completion prompt. Finally, the completion
prompt is sent to Code LLMs to generate the completion response.

Second, based on R2C2-Enhance, we can easily build a real-world repository-
level code completion benchmark called R2C2-Bench, which includes the
training, validation and test splits in 4 languages. Note that when producing
R2C2-Bench, we also apply a context perturbation strategy to simulate more
diverse and challenging code completion samples for generating the completion
prompt.

By performing experiments on the validation and test splits of R2C2-Bench, we
observe that our R2C2-Enhance achieves significant improvements when compared
with the existing methods without training. Besides, when we further fine-tune
the Code LLMs using the training split of R2C2-Bench, better results are
obtained on the validation and testing splits of R2C2-Bench. Moreover, we also
validate the generalization abilities of R2C2-Enhance on the widely-used
CrossCodeEval benchmark.

The contributions of our R2C2-Coder are as follows:

  * •

We investigate the limitations of existing repository-level code completion
(e.g., lack of sufficient context and benchmark with limited scenarios) and
propose the R2C2-Coder including R2C2-Enhance and R2C2-Bench to enhance and
benchmark real-world repository-level code completion abilities of Code LLMs.

  * •

For R2C2-Enhance, we propose to construct the candidate retrieval pool with
abstract and snippet contexts and generate the completion prompt using context
retrieval and prompt assemble. Based on R2C2-Enhance, we build a new
repository-level code completion benchmark called R2C2-Bench with training,
validation, and testing splits, where a context perturbation strategy is used
to simulate the real-world completion scenes better.

  * •

Comprehensive experimental results on multiple benchmark datasets demonstrate
the effectiveness and efficiency of our R2C2-Coder.

##  2 Related Works

Code Large Language Models. The successive achievements of generative language
modeling [9, 10, 11, 46, 55, 56, 6, 60, 35, 53, 23, 59, 22, 64, 18] have
inspired extensive studies on generative AI for software engineering. While
closed-source models [1, 12, 15] achieve significant dominance in the
benchmark leaderboards [12, 24, 36], their inaccessible model checkpoints and
source code are detrimental to subsequent innovation. In contrast, plenty of
open-source models, for instance, Code T5 [57, 58], CodeGen [40, 41],
StarCoder [3, 32, 38], Code Llama [49] and DeepSeekCoder [21], have been
proposed and have substantially driven the development of code intelligence.

Repository-level Code Completion. Existing methods are usually based on a
retrieval augmented generation (RAG), and we categorize related works into
ranking-based and fusion-based methods. Ranking-based approaches [34, 42, 44,
63] explicitly select those code snippets in the repository that are highly
similar to the incomplete code. In contrast, fusion-based methods [17, 50, 51]
focus on organizing the repository-level context to be jointly modeled with
the language model and allow the model to automatically select the most
relevant information.

Code Benchmark. Program synthesis is to prompt code models to solve
programming problems using the input description. The generated code snippet
should be functional to pass all test cases [4, 5, 12, 19, 31, 36, 62].
Besides, numerous benchmarks can be used to comprehensively evaluate code
models, such as code translation [29, 61, 65], code retrieval [25, 27, 33,
39], and vulnerability repair [26, 45, 47, 54]. Recently, several benchmarks
have been proposed on repository-level code completion [2, 3, 7, 17, 37, 42,
51, 50, 63]. For example, RepoBench [37] measures the retrieval and completion
performance of the system. CrossCodeEval [16] evaluates three cross-file
prompting levels: in-file, retrieval, and retrieval within references.
However, RepoBench only supports two programming languages and neglects the
FIM setting. CrossCodeEval only considers explicit file imports, resulting in
few cross-file dependencies and low multi-line prediction difficulty.

##  3 R2C2-Coder

In this section, we discuss the details of R2C2-Coder, which includes a code
prompt construction method R2C2-Enhance and a new benchmark R2C2-Bench. For
R2C2-Enhance, in Fig. 2, we first build the candidate retrieval pool based on
the abstract context and the snippet context to produce the coarse-grained
global information of each programming file and the fine-grained local
information of each code fragment, respectively. Then, for the current
triggered cursor location, we produce the retrieval query and obtain the
retrieved contexts based on the completion query and the candidate context
pool, where the retrieved contexts and the current in-file context are used to
construct the completion prompt. Based on R2C2-Enhance, we can easily produce
the R2C2-Bench, where a context perturbation strategy is used to generate more
diverse and challenging completion scenes.

Figure 2: Overview of our R2C2-Enhance. For the current completion cursor
position, we first generate the retrieval query using the prefix and suffix
contexts. Then, we perform context retrieval between the retrieval query and
the pre-constructed candidate retrieval pool to produce the retrieved
contexts. After that, we use the in-file context of the current code and the
retrieved contexts to assemble the completion prompt, which is then sent to
LLMs to generate the completion response.

###  3.1 R2C2-Enhance

####  3.1.1 Construction of the Candidate Retrieval Pool

Given a repository, we extract abstract and snippet contexts to construct the
candidate retrieval pool.

Figure 3: An example of the abstract syntax tree generated by the Tree-sitter
tool.

Abstract Context Extraction. In Fig. 2, we propose to use the parser generator
tool (i.e., Tree-sitter333https://tree-sitter.github.io/tree-sitter/) to
extract the abstract context for each file in the repository. Specifically, we
use Tree-sitter to generate the abstract syntax tree in Fig. 3, and then
preserve these nodes to represent the declarations (e.g., functions,
variables, classes etc.). After that, for each node, we remove the redundant
information (e.g., middle variables, empty lines, comments etc.) and only
preserve the information related to the definition for each declaration. Note
that the extraction process is easy and efficient (See Appendix B.1 for more
details.). Moreover, in Fig. 9 of Appendix B.1, we provide examples of the
extracted abstract contexts and observe that the abstract context mainly
provides the coarse-grained global information for each code file.

Snippet Context Extraction. For the snippet context, we iteratively scan the
files in the repository and extract contiguous $M$ lines (in all our
experiments, we set $M=10$ follow [16]) of overlapping code fragments, which
are the candidates for context retrieval.

####  3.1.2 Construction of the Completion Prompt

Context Retrieval. In Fig. 2, for the current cursor position, we first build
the retrieval query based on the prefix and suffix contexts. Specifically, for
the current cursor position, we first use the previous $P$ lines and
subsequent $S$ lines as the prefix and suffix contexts, respectively.
Therefore, the retrieval query usually has $P+S$ lines. Note that if the
prefix and suffix have one overlapped line, the retrieval query has $P+S-1$
lines. Then, we use the retriever (e.g., BM 25 [48]) to calculate the
similarity between the retrieval query and the candidate retrieval pool. After
that, we can select the retrieved contexts based on the similarities. Note
that in the retrieval process, we remove the abstract context and snippet
context of the current code file, where the cursor position is located.

Prompt Assemble. We assemble the completion prompt by using the current code
file, and retrieved contexts. Specifically, we first follow [16] to maintain
all context of the current code file as the in-file context, where the
completion cursor position is located. Then, we preserve the top-$K$ similar
abstract contexts as the coarse-grained global structure information, After
that, we append the most relevant snippet context based on the similarity
scores until the maximum number of tokens is $N$, where we set $N$ as 4,096 by
default. Finally, we can obtain the assembled prompt with $N$ tokens, which
will be sent to the Code LLMs for generating the completion results.

###  3.2 R2C2-Bench

In this section, we provide the details of our R2C2-Bench with training,
validation and test splits.

####  3.2.1 Dataset Generation

Initial Repository Collection. We collect permissively licensed repositories
from GitHub. Specifically, following [16], we contain four languages and keep
only repositories with the number of stars $>=3$. Then we filter out
repositories with fewer than 10 or more than 50 source code files. Finally, we
remove these repositories with at least one source code file that exactly
matches one of the code files in the Stack [30] dataset. As a result, we ended
up with 54972, 51796, 49790, and 35410 repositories for Python, Java,
TypeScript, and C#, respectively.

Completion Cursor Position Generation. Based on R2C2-Enhance, given completion
cursor position, we can easily generate the completion prompt. Therefore, it
is critical to generate informative and sufficient cursor positions for our
R2C2-Bench. Specifically, in Fig. 3, for each file in the repository, we reuse
the abstract syntax tree generated in the abstract context extraction stage,
and randomly choose one node of the tree as the cursor position to be
triggered for completion. Then, for the selected node, we can easily generate
one completion prompt sample for our R2C2-Bench.

Feature | Python | Java | TypeScript | C#
---|---|---|---|---
# Repositories | 260 | 397 | 451 | 245
# Files | 2500 | 2500 | 2500 | 2500
# Examples | 4394 | 6900 | 6506 | 5028
Avg. # cross files in prompt | 6.4 | 7.8 | 6.8 | 10.1
Avg. # lines in prompt | 320.5 | 333.7 | 365.7 | 373.8
Avg. # tokens in prompt | 3192.25 | 3262.3 | 3200.52 | 3313.23
Avg. # lines in reference | 1.73 | 1.75 | 2.02 | 1.78
Avg. # tokens in reference | 17.47 | 14.22 | 15.00 | 13.32

Table 1: The statistics of the testing split of R2C2-Bench.

Context Perturbation. In real-world repository-level code completion scenes,
we cannot always retrieve the relevant context well. For example, in certain
software development scenes, we create code files to implement different
usages under a repository. However, the relationships between these code files
are relatively low, which indicates that the constructed completion prompt
contains many irrelevant contexts. Therefore, to simulate the real-world
repository-level code completion scene and improve the semantic variety of the
completion prompt, we introduce a so-called context perturbation strategy.
Specifically, for $Q\%$ completion cursor position (we set $Q=10$ by default),
we first sort the contexts based on the similarities and follow the uniform
distribution to determine the proportion $R\%$. Then we randomly discard top
$R\%$ contexts with high similarities. After that, we perform the context
retrieval based on the retrieval query and the remaining contexts, and the
generated retrieved contexts are then combined with the current file to
produce the completion prompt as discussed in Sec. 3.1.2.

Based on the above meticulous procedure, we can produce sufficient completion
prompt samples using these repositories, and randomly select 400,000 as
training samples, where each language has 100,000 samples. For the validation
and testing samples, to ensure the quality of the dataset, we apply a series
of rule-based and model-based post-processing filter strategies discussed in
Appendix B.2.

####  3.2.2 Dataset Statistics

The statistics of R2C2-Bench are shown in Table 1444We use the StarCoder
tokenizer [32] to compute the number of tokens.. We have provided the
statistics of CrossCodeEval and validation split of R2C2-Bench in Appendix B.4

Model | Python | Java | TypeScript | C# | Average
---|---|---|---|---|---
EM | ES | EM | ES | EM | ES | EM | ES | EM | ES
StarCoder-7B | 17.0 | 49.5 | 21.6 | 55.9 | 18.1 | 53.6 | 17.6 | 51.0 | 18.6 | 52.5
\+ R2C2-Enhance | 21.4 | 46.2 | 27.6 | 51.9 | 24.8 | 52.8 | 25.2 | 45.8 | 24.8 | 49.2
\+ R2C2-Enhance w.o. Abs. | 21.5 | 46.2 | 26.1 | 51.4 | 24.6 | 52.8 | 24.9 | 45.7 | 24.3 | 49.0
\+ R2C2-Enhanced Tuning | 34.0 | 65.0 | 45.3 | 74.4 | 37.0 | 68.9 | 45.0 | 71.6 | 40.3 | 70.0
Code Llama-7B | 13.7 | 44.1 | 20.6 | 54.5 | 17.4 | 51.2 | 16.0 | 48.4 | 16.9 | 49.5
\+ R2C2-Enhance | 16.3 | 39.0 | 22.3 | 45.0 | 22.1 | 46.5 | 18.8 | 36.0 | 19.9 | 41.6
\+ R2C2-Enhance w.o. Abs. | 16.2 | 38.9 | 21.6 | 44.8 | 22.0 | 46.4 | 19.1 | 36.9 | 19.7 | 41.7
\+ R2C2-Enhanced Tuning | 34.5 | 66.5 | 47.4 | 77.2 | 40.2 | 71.7 | 48.0 | 74.7 | 42.5 | 72.5
DeepSeekCoder-6.7B | 19.4 | 52.5 | 24.2 | 59.3 | 22.0 | 58.8 | 20.7 | 54.1 | 21.6 | 56.2
\+ R2C2-Enhance | 25.8 | 51.3 | 31.8 | 56.5 | 29.7 | 57.8 | 33.1 | 53.6 | 30.1 | 54.8
\+ R2C2-Enhance w.o. Abs. | 25.8 | 51.3 | 30.7 | 55.6 | 29.6 | 58.0 | 31.9 | 52.5 | 29.5 | 54.3
\+ R2C2-Enhanced Tuning | 38.0 | 69.6 | 49.2 | 78.0 | 43.0 | 73.8 | 52.3 | 77.5 | 45.6 | 74.7

Table 2:  Exact match (%) and edit similarity (%) performance on R2C2-Bench.
The baseline results are produced under the “In-file Context Only” setting
while “+R2C2-Enhance” means the input is prepended with retrieval results
fetched by R2C2-Enhance at inference. In “+ R2C2-Enhance w.o. Abs.”, we report
results without using abstract contexts in the candidate retrieval pool.
“R2C2-Enhance w/ F.T.” means we fine-tune these LLMs on the training set of
R2C2-Bench with R2C2-Enhance.

##  4 Experiments

We conduct experiments on CrossCodeEval [16], CrossCodeEval+, and R2C2-Bench
with three popular Code LLMs (i.e., StarCoder-7B [32], DeepSeekCoder-7B [21]
and Code Llama-7B [49]) (See Appendix C.1 for more details). For
CrossCodeEval+, we collect the original repositories of CrossCodeEval [16]
then parse and extract the cross-file dependencies using R2C2-Enhance. During
retrieval, the abstracts and the snippets are placed together in the candidate
pool (Sec. 3.1.1). Note that we keep the cursor positions and the expected
outputs the same as those in the original CrossCodeEval (See Appendix C.2 for
more details).

###  4.1 Experimental Setup

In-file Context Only. We only provide the original code file, where the cursor
position is located. Without any explicit cross-file context as input, the
model should rely on its knowledge-based reasoning capabilities to accomplish
the code generation process.

Retrieval in CrossCodeEval. This setup is similar to the “+ Retrieval” setting
of CrossCodeEval [16], where the retrieval candidates are constructed by
exhaustively scanning files in the same repository and extracting
contiguous-$M$-line code fragments ($M=10$). Then, the top 5 similar
candidates and the test query assemble the prompt. Note that for better
baseline performance, we do not truncate cross-file contents into a total
length of 512 tokens as [16].

Retrieval with R2C2-Enhance. Based on R2C2-Enhance in Sec. 3.1, given a
repository, we first construct the candidate retrieval pool via abstract and
snippet context extraction. Then, we build the retrieval query including
prefix length $P=5$ and suffix length $S=5$. After that, candidates are
concatenated to the beginning of the in-file context based on their Jaccard
similarity [28] in descending order, until the cumulative length, including
the in-file context, reaches the maximum token limit of $N=4096$. Note that we
set the number of the abstract context $K$ as 3.

R2C2-Enhanced Tuning. To further boost the performance, we fine-tune code LLMs
on the training split of R2C2-Bench including 400,000 files mentioned in Sec.
3.2.1.

Evaluation Metrics. Following CrossCodeEval [16], we evaluate code LLMs with
code match metrics including exact match (EM) and edit similarity (ES). Exact
match expects that the prediction and the real code are extremely identical,
while edit similarity flexibly allows the generated samples to match the
original code through a certain number of editing operations.

###  4.2 Results

We present results on R2C2-Bench in Table 3.2.2, CrossCodeEval [16] in Table
4.2, and CrossCodeEval+ in Table 4.2. Based on Table 4.2 and Table 4.2, cross-
file context is undoubtedly efficacious, while retrieval with R2C2-Enhance
surely exhibits its superiority compared with retrieval method in
CrossCodeEval. Moreover, fine-tuning strategy significantly improves the
performance on both benchmarks. Besides, when abstract contexts are removed
from the candidate retrieval pool, these code LLMs suffer from degraded
performance. Notably, in Table 3.2.2, after applying R2C2-Enhanced tuning, the
Code Llama-7B, which is the worst in the in-file setting, outperforms the
StarCoder-7B. Overall, we could say that there exists a performance promotion
path, “In-file” $\rightarrow$ “+ R2C2-Enhance w.o. Abs.” $\rightarrow$ “+
R2C2-Enhance” $\rightarrow$ “+ R2C2-Enhanced Tuning”.

Model | Python | Java | TypeScript | C# | Average
---|---|---|---|---|---
EM | ES | EM | ES | EM | ES | EM | ES | EM | ES
StarCoder-7B | 24.3 | 51.3 | 27.7 | 61.4 | 30.8 | 67.1 | 44.7 | 69.8 | 30.9 | 62.1
\+ Retrieval | 27.0 | 51.0 | 33.6 | 62.9 | 33.1 | 66.7 | 47.1 | 67.7 | 34.1 | 61.9
\+ R2C2-Enhanced Tuning | 34.2 | 64.3 | 40.2 | 73.7 | 37.8 | 72.4 | 50.6 | 77.8 | 39.6 | 71.5
Code Llama-7B | 22.0 | 46.8 | 28.6 | 60.4 | 10.1 | 63.0 | 44.5 | 71.3 | 23.4 | 59.6
\+ Retrieval | 23.0 | 45.3 | 33.8 | 61.3 | 10.7 | 63.1 | 49.3 | 70.1 | 25.8 | 59.2
\+ R2C2-Enhanced Tuning | 34.7 | 65.4 | 41.0 | 72.7 | 11.4 | 69.5 | 53.2 | 77.8 | 31.5 | 70.6
DeepSeekCoder-6.7B | 25.9 | 51.8 | 31.3 | 62.5 | 33.8 | 67.5 | 42.9 | 67.6 | 32.8 | 62.2
\+ Retrieval | 28.0 | 52.6 | 36.2 | 62.3 | 35.6 | 66.7 | 48.4 | 67.7 | 35.9 | 62.1
\+ R2C2-Enhanced Tuning | 37.7 | 67.3 | 45.1 | 75.3 | 42.0 | 75.7 | 59.3 | 81.8 | 44.6 | 74.5

Table 3:  Exact match (%) and edit similarity (%) performance on
CrossCodeEval.

Model | Python | Java | TypeScript | C# | Average
---|---|---|---|---|---
EM | ES | EM | ES | EM | ES | EM | ES | EM | ES
StarCoder-7B | 24.3 | 51.3 | 27.7 | 61.4 | 30.8 | 67.1 | 44.7 | 69.8 | 30.9 | 62.1
\+ R2C2-Enhance | 30.9 | 51.9 | 38.1 | 63.6 | 35.5 | 67.7 | 68.4 | 78.2 | 40.7 | 64.5
\+ R2C2-Enhance w.o. Abs. | 30.9 | 51.8 | 35.3 | 62.6 | 34.5 | 67.6 | 65.2 | 77.1 | 39.2 | 64.0
\+ R2C2-Enhanced Tuning | 41.8 | 68.5 | 46.2 | 76.0 | 40.7 | 74.4 | 69.3 | 86.5 | 47.3 | 75.3
Code Llama-7B | 22.0 | 46.8 | 28.6 | 60.4 | 10.1 | 63.0 | 44.5 | 71.3 | 23.4 | 59.6
\+ R2C2-Enhance | 23.6 | 42.9 | 35.6 | 58.5 | 10.6 | 61.6 | 67.8 | 78.5 | 29.7 | 58.9
\+ R2C2-Enhance w.o. Abs. | 23.9 | 43.0 | 34.1 | 58.9 | 11.0 | 62.3 | 65.4 | 78.0 | 29.1 | 59.2
\+ R2C2-Enhanced Tuning | 43.0 | 71.8 | 48.8 | 76.8 | 13.0 | 73.2 | 70.3 | 87.0 | 39.0 | 76.1
DeepSeekCoder-6.7B | 25.9 | 51.8 | 31.3 | 62.5 | 33.8 | 67.5 | 42.9 | 67.6 | 32.8 | 62.2
\+ R2C2-Enhance | 32.7 | 54.0 | 41.6 | 64.6 | 38.8 | 67.9 | 69.0 | 78.8 | 43.1 | 65.4
\+ R2C2-Enhance w.o. Abs. | 32.5 | 53.7 | 38.8 | 63.5 | 37.8 | 67.5 | 65.7 | 77.5 | 41.5 | 64.7
\+ R2C2-Enhanced Tuning | 46.9 | 73.5 | 53.2 | 79.3 | 45.8 | 78.2 | 78.9 | 89.9 | 53.6 | 79.3

Table 4:  Exact match (%) and edit similarity (%) performance on
CrossCodeEval+.

Ablation Models | $P$ | $S$ | Python | Java | TypeScript | C# | Average
---|---|---|---|---|---|---|---
EM | ES | EM | ES | EM | ES | EM | ES | EM | ES
StarCoder-7B + R2C2-Enhanced Tuning | 5 | 5 | 40.8 | 68.5 | 42.0 | 74.1 | 35.6 | 68.1 | 50.0 | 76.6 | 42.1 | 71.8
Prefix & Suffix | 5 | 1 | 38.0 | 64.0 | 41.6 | 71.8 | 31.6 | 64.7 | 48.0 | 74.2 | 39.8 | 68.7
5 | 3 | 40.0 | 66.8 | 41.6 | 73.8 | 32.0 | 65.5 | 49.2 | 75.6 | 40.7 | 70.4
Prefix Only | 10 | - | 39.2 | 68.7 | 39.2 | 71.0 | 32.4 | 67.5 | 47.6 | 74.5 | 39.6 | 70.4
w.o. Snippet Context | 1 | 1 | 24.8 | 59.1 | 36.4 | 71.5 | 26.8 | 64.4 | 36.8 | 70.3 | 31.2 | 66.3
3 | 3 | 26.4 | 59.7 | 36.0 | 71.5 | 25.6 | 64.0 | 35.6 | 70.0 | 30.9 | 66.3
5 | 5 | 25.6 | 59.0 | 37.6 | 71.4 | 24.8 | 64.0 | 38.4 | 70.8 | 31.6 | 66.3
10 | 10 | 24.0 | 58.1 | 36.4 | 71.5 | 25.2 | 64.2 | 35.2 | 68.4 | 30.2 | 65.5
w.o. Abstract Context | 1 | 1 | 38.0 | 65.9 | 37.6 | 69.3 | 31.6 | 65.3 | 42.8 | 71.4 | 37.5 | 68.0
3 | 3 | 37.2 | 62.9 | 39.2 | 68.3 | 31.6 | 62.5 | 42.4 | 69.3 | 37.6 | 65.7
5 | 5 | 37.2 | 65.1 | 38.8 | 69.8 | 33.2 | 65.2 | 45.2 | 73.3 | 38.6 | 68.4
10 | 10 | 37.6 | 66.1 | 37.2 | 68.2 | 32.4 | 64.8 | 44.8 | 72.8 | 38.0 | 68.0

Table 5:  Exact match (%) and edit similarity (%) performance on R2C2-Bench
for StarCoder-7B. During retrieval, $P$ prefix lines and $S$ suffix lines are
concatenated as the query. In “Prefix Only”, no suffix lines are taken as the
query. “w.o.” stands for removing the snippet contexts or abstract contexts at
the fine-tuning and the inference stages simultaneously.

###  4.3 Analysis

Ablations. We conduct ablation studies on R2C2-Enhance and present the results
in Table 4.2. We fine-tune StarCoder-7B [32] with R2C2-Enhance and report the
performance on the validation set of R2C2-Bench. Meanwhile, we conduct
experiments on the retrieval manner, where we replace the FIM mode with
“prefix-only”. We could find that suffix contexts are necessary as the results
of $P=5,S=3$ even surpass those of $P=10$. Additionally, we remove either
snippet contexts (“w.o. Snippet Contexts”) or abstract contexts (“w.o.
Abstract Contexts”) simultaneously in fine-tuning and testing. Results on
various prefix lengths ($P$) and suffix lengths ($S$) are further reported for
the ablation model. Despite the larger impact without using snippet contexts,
abstract contexts still deliver 3.5% and 3.4% performance gains. Furthermore,
a case study in Fig. 6 shows how the code LLMs benefit from abstract context
extraction.

Context Perturbation Helps Cope with Confusing Contexts. In real-world
scenarios, retrievers would produce low-quality results as shown in Fig. 7. To
enhance the model robustness on irrelevant contexts, we select $Q\%$ of the
cursors and randomly remove the top similar contexts during R2C2-Enhanced
tuning. Fig. 4 visualizes how the performance of StarCoder-7B on EM and ES
scales when using different perturbation rates for R2C2-Enhanced tuning. The
perturbation rate $Q\%$ for fine-tuning varies in $0\%,10\%,20\%,50\%,$ and
$80\%$, while the perturbation rate for the validation set is always kept as
10%. The best results are achieved under perturbation rate $Q\%=10\%$, which
is also the default setting of R2C2-Enhanced tuning for Table 3.2.2 , 4.2 and
4.2.

Similarity Metric | Python | Java | TypeScript | C# | Average |  Execution Time (ms)
---|---|---|---|---|---|---
EM | ES | EM | ES | EM | ES | EM | ES | EM | ES
Jaccard | 40.8 | 68.5 | 42.0 | 74.1 | 35.6 | 68.1 | 50.0 | 76.6 | 42.1 | 71.8 | 2.941
BM25 | 38.4 | 64.6 | 40.8 | 71.5 | 31.6 | 65.6 | 50.0 | 75.4 | 40.2 | 69.3 | 3.142
UniXCoder | 39.2 | 66.0 | 40.8 | 70.9 | 33.6 | 66.1 | 49.2 | 75.9 | 40.7 | 69.7 | 250.9

Table 6: Performance of StarCoder-7B with different similarity metrics on the
validation set of R2C2-Bench. The retrieval time consumption (ms) per sample
is further reported.

Figure 4: Performance of StarCoder-7B + R2C2-Enhanced Tuning at various
perturbation rates on the validation set of R2C2-Bench.

Lexical Retrievers are Cost-effective. For ranking-based code retrieval,
several approaches could be adopted as the similarity metric. We categorize
the metrics into lexical ones, including Jaccard similarity [28] and BM25
[48], and neural ones such as UniXcoder [20]. In real-world scenarios, we
often need to make a trade-off, as lexical retrievers could save more response
time (RT) while neural retrievers would theoretically achieve higher
performance. To find the most suitable retriever for R2C2-Enhance, we evaluate
the code match performance and retrieval duration with Jaccard similarity,
BM25, and UniXCoder. Table 4.3 presents the code match performance and
retrieval duration of each retriever. Although UniXCoder outperforms BM25 on
code match, its time consumption is extremely worse than lexical retrievers.
We find that Jaccard similarity is the best retriever regarding both
effectiveness and efficiency. Thus, we choose Jaccard similarity by default.
See Appendix C.3 for details of execution time benchmarking.

Scalability of Model Performance. We present how the performance of StarCoder
scales w.r.t model parameters in Table 4.3. We find that almost no performance
gain is obtained for StarCoder-3B when using R2C2-Enhance with abstract and
snippet contexts. After R2C2-Enhanced tuning, StarCoder-3B can outperform the
inference-only results of StarCoder-7B. In Table 4.3 we further evaluate the
fine-tuned StarCoder-7B when using different sizes of tuning datasets based on
our R2C2-Enhanced tuning. We observe that when increasing from 10K samples to
100K samples per language, better results are obtained. When continually
increasing, the results are relatively stable. Thus, we choose 10K samples per
language as the default training set size of R2C2-Bench.

Scalability on Multi-line Predictions. In Fig. 5, we provide a detailed
analysis of the scalability on multi-line predictions. Specifically, in Fig.
5(a), we provide the counts of completion references with different lines and
observe that our R2C2-Bench has more multi-line completion scenes. Then, in
Fig. 5(b) and Fig. 5(c), we provide the results of StarCoder-7B with
R2C2-Enhanced Tuning on CrossCodeEval+ and R2C2-Bench, and observe that the
scores on R2C2-Bench are lower than CrossCodeEval+ a lot, specifically in
multi-line scenes, which means that our R2C2-Bench is more challenging than
the CrossCodeEval+.

Limitations of CrossCodeEval. We summarize three major drawbacks of
CrossCodeEval [16] as follows: 1) The retrieval results provided in
CrossCodeEval are produced by sliding-window-based searching, which could
potentially miss long-term dependencies that exceed the length of $M$. In
contrast, abstract extraction expands the reception field of code LLMs with
coarse-grained global information. 2) CrossCodeEval only considers explicit
dependencies (i.e., file imports), but there could be implicit references
during software development. For example, in Java, usages in the same package
do not require the import statement. As shown in Fig. 8, such implicit
dependencies could not be detected by Ding et al. [16], but are successfully
covered by our R2C2-Enhance during constructing CrossCodeEval+. 3) Compared
with CrossCodeEval, our R2C2-Bench provides more complex cross-file
dependencies (see “Avg. # cross files in prompt” in Table 1 and Table B.4) and
more difficult multi-line test samples in Fig. 5).

(a) Multi-line Reference Counts (b) Exact Match (c) Edit Similarity

Figure 5: (a) Statistics of references from 1 to 5 lines in R2C2-Bench and
CrossCodeEval+. (b) Exact Match of StarCoder-7B w/ R2C2-Enhanced tuning on
R2C2-Bench and CrossCodeEval+ when expected output varies from 1 to 5 lines.
(c) Edit Similarity of StarCoder-7B w/ R2C2-Enhanced tuning on R2C2-Bench and
CrossCodeEval+ when expected output varies from 1 to 5 lines.

Model | Python | Java | TypeScript | C# | Average
---|---|---|---|---|---
EM | ES | EM | ES | EM | ES | EM | ES | EM | ES
StarCoder-3B | 10.8 | 38.6 | 18.0 | 51.1 | 12.4 | 49.0 | 14.8 | 50.0 | 14.0 | 47.2
\+ R2C2-Enhance | 16.8 | 34.1 | 18.0 | 39.7 | 16.0 | 39.0 | 12.8 | 31.2 | 15.9 | 36.0
\+ R2C2-Enhanced Tuning | 40.0 | 64.4 | 37.2 | 68.0 | 28.0 | 61.8 | 45.6 | 71.0 | 37.7 | 66.3
StarCoder-7B | 15.2 | 45.2 | 26.0 | 59.1 | 14.4 | 52.4 | 16.0 | 52.6 | 17.9 | 52.3
\+ R2C2-Enhance | 23.2 | 46.4 | 30.8 | 56.8 | 25.2 | 52.6 | 26.4 | 48.9 | 26.4 | 51.2
\+ R2C2-Enhanced Tuning | 40.8 | 68.5 | 42.0 | 74.1 | 35.6 | 68.1 | 50.0 | 76.6 | 42.1 | 71.8

Table 7: Performance on the validation set of R2C2-Bench.

Training samples ($\times 4$) | Python | Java | TypeScript | C# | Average
---|---|---|---|---|---
EM | ES | EM | ES | EM | ES | EM | ES | EM | ES
10,000 | 38.0 | 63.7 | 40.4 | 71.3 | 31.2 | 65.6 | 44.0 | 74.4 | 38.4 | 68.7
50,000 | 40.0 | 67.2 | 39.6 | 71.1 | 34.0 | 66.3 | 49.2 | 75.6 | 40.7 | 70.0
100,000 | 40.8 | 68.5 | 42.0 | 74.1 | 35.6 | 68.1 | 50.0 | 76.6 | 42.1 | 71.8
200,000 | 39.6 | 66.9 | 43.6 | 73.9 | 35.4 | 66.0 | 49.6 | 75.6 | 42.1 | 70.6

Table 8: Performance of StarCoder-7B with R2C2-Enhanced Tuning on the
validation set of R2C2-Bench. Experiments are conducted on R2C2-Bench using
different sizes of training data.

##  5 Conclusion

In this paper, we propose the R2C2-Coder including R2C2-Enhance and R2C2-Bench
to enhance and benchmark the real-world repository-level code completion
abilities of code LLMs. Specifically, first, R2C2-Enhance is a code prompt
construction method, which includes candidate retrieval pool construction and
completion prompt construction. Second, the R2C2-Bench is a more challenging
repository-level code completion dataset with R2C2-Bench with training,
validation, and test splits. Comprehensive results on multiple benchmarks
demonstrate the effectiveness of our R2C2-Coder.

##  6 Acknowledgement

This research project is supported by the Alibaba Aone Copilot Group.

## References

  * Achiam et al. [2023] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F.L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al., 2023.  Gpt-4 technical report .
  * Agrawal et al. [2023] Agrawal, L.A., Kanade, A., Goyal, N., Lahiri, S.K., Rajamani, S.K., 2023.  Guiding language models of code with global context using monitors .
  * Allal et al. [2023] Allal, L.B., Li, R., Kocetkov, D., Mou, C., Akiki, C., Ferrandis, C.M., Muennighoff, N., Mishra, M., Gu, A., Dey, M., et al., 2023.  Santacoder: don’t reach for the stars!  arXiv preprint arXiv:2301.03988 URL: https://arxiv.org/abs/2301.03988.
  * Athiwaratkun et al. [2023] Athiwaratkun, B., Gouda, S.K., Wang, Z., Li, X., Tian, Y., Tan, M., Ahmad, W.U., Wang, S., Sun, Q., Shang, M., Gonugondla, S.K., Ding, H., Kumar, V., Fulton, N., Farahani, A., Jain, S., Giaquinto, R., Qian, H., Ramanathan, M.K., Nallapati, R., Ray, B., Bhatia, P., Sengupta, S., Roth, D., Xiang, B., 2023.  Multi-lingual evaluation of code generation models, in: The Eleventh International Conference on Learning Representations.  URL: https://openreview.net/forum?id=Bo7eeXm6An8.
  * Austin et al. [2021] Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al., 2021.  Program synthesis with large language models.  ArXiv preprint abs/2108.07732.  URL: https://arxiv.org/abs/2108.07732.
  * Bai et al. [2024] Bai, G., Liu, J., Bu, X., He, Y., Liu, J., Zhou, Z., Lin, Z., Su, W., Ge, T., Zheng, B., Ouyang, W., 2024.  Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues.  arXiv .
  * Bairi et al. [2023] Bairi, R., Sonwane, A., Kanade, A., Iyer, A., Parthasarathy, S., Rajamani, S., Ashok, B., Shet, S., et al., 2023.  Codeplan: Repository-level coding using llms and planning.  arXiv preprint arXiv:2309.12499 .
  * Bavarian et al. [2022] Bavarian, M., Jun, H., Tezak, N., Schulman, J., McLeavey, C., Tworek, J., Chen, M., 2022.  Efficient training of language models to fill in the middle.  arXiv preprint arXiv:2207.14255 URL: https://arxiv.org/abs/2207.14255.
  * Black et al. [2022] Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., Pieler, M., Prashanth, U.S., Purohit, S., Reynolds, L., Tow, J., Wang, B., Weinbach, S., 2022.  GPT-NeoX-20B: An open-source autoregressive language model, in: Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models, Association for Computational Linguistics, virtual+Dublin. pp. 95–136.  URL: https://aclanthology.org/2022.bigscience-1.9, doi:10.18653/v1/2022.bigscience-1.9.
  * Black et al. [2021] Black, S., Gao, L., Wang, P., Leahy, C., Biderman, S., 2021.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow.  URL: https://doi.org/10.5281/zenodo.5297715, doi:10.5281/zenodo.5297715.
  * Brown et al. [2020] Brown, T.B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D.M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D., 2020.  Language models are few-shot learners, in: Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., Lin, H. (Eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.  URL: https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html.
  * Chen et al. [2021] Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H.P.d.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al., 2021.  Evaluating large language models trained on code.  ArXiv preprint abs/2107.03374.  URL: https://arxiv.org/abs/2107.03374.
  * Chen et al. [2023a] Chen, S., Wong, S., Chen, L., Tian, Y., 2023a.  Extending context window of large language models via positional interpolation.  CoRR abs/2306.15595.
  * Chen et al. [2023b] Chen, Y., Qian, S., Tang, H., Lai, X., Liu, Z., Han, S., Jia, J., 2023b.  Longlora: Efficient fine-tuning of long-context large language models.  arXiv:2309.12307 .
  * Chowdhery et al. [2023] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H.W., Sutton, C., Gehrmann, S., et al., 2023.  Palm: Scaling language modeling with pathways 24, 1–113.
  * Ding et al. [2023] Ding, Y., Wang, Z., Ahmad, W.U., Ding, H., Tan, M., Jain, N., Ramanathan, M.K., Nallapati, R., Bhatia, P., Roth, D., Xiang, B., 2023.  Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion, in: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (Eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.  URL: http://papers.nips.cc/paper_files/paper/2023/hash/920f2dced7d32ab2ba2f1970bc306af6-Abstract-Datasets_and_Benchmarks.html.
  * Ding et al. [2022] Ding, Y., Wang, Z., Ahmad, W.U., Ramanathan, M.K., Nallapati, R., Bhatia, P., Roth, D., Xiang, B., 2022.  Cocomic: Code completion by jointly modeling in-file and cross-file context.  arXiv preprint arXiv:2212.10007 URL: https://arxiv.org/abs/2212.10007.
  * Du et al. [2024] Du, X., Yu, Z., Gao, S., Pan, D., Cheng, Y., Ma, Z., Yuan, R., Qu, X., Liu, J., Zheng, T., Luo, X., Zhou, G., Yuan, B., Chen, W., Fu, J., Zhang, G., 2024.  Chinese tiny llm: Pretraining a chinese-centric large language model.  arXiv:2404.04167.
  * Gu et al. [2024] Gu, A., Rozière, B., Leather, H., Solar-Lezama, A., Synnaeve, G., Wang, S.I., 2024.  Cruxeval: A benchmark for code reasoning, understanding and execution .
  * Guo et al. [2022] Guo, D., Lu, S., Duan, N., Wang, Y., Zhou, M., Yin, J., 2022.  Unixcoder: Unified cross-modal pre-training for code representation, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7212–7225.
  * Guo et al. [2024a] Guo, D., Zhu, Q., Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al., 2024a.  Deepseek-coder: When the large language model meets programming–the rise of code intelligence.  arXiv preprint arXiv:2401.14196 .
  * Guo et al. [2023] Guo, H., Yang, J., Liu, J., Yang, L., Chai, L., Bai, J., Peng, J., Hu, X., Chen, C., Zhang, D., et al., 2023.  Owl: A large language model for it operations.  arXiv preprint arXiv:2309.09298 .
  * Guo et al. [2024b] Guo, J., Wu, J., Wang, Z., Liu, J., Yang, G., Ding, Y., Gong, R., Qin, H., Liu, X., 2024b.  Compressing large language models by joint sparsification and quantization.  ICML .
  * Hendrycks et al. [2020] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., Steinhardt, J., 2020.  Measuring massive multitask language understanding .
  * Huang et al. [2021] Huang, J., Tang, D., Shou, L., Gong, M., Xu, K., Jiang, D., Zhou, M., Duan, N., 2021.  Cosqa: 20,000+ web queries for code search and question answering .
  * Huq et al. [2022] Huq, F., Hasan, M., Haque, M.M.A., Mahbub, S., Iqbal, A., Ahmed, T., 2022.  Review4repair: Code review aided automatic program repairing 143, 106765.
  * Husain et al. [2019] Husain, H., Wu, H.H., Gazit, T., Allamanis, M., Brockschmidt, M., 2019.  Codesearchnet challenge: Evaluating the state of semantic code search.  arXiv preprint arXiv:1909.09436 URL: https://arxiv.org/abs/1909.09436.
  * Jaccard [1912] Jaccard, P., 1912.  The distribution of the flora in the alpine zone. 1.  New phytologist 11, 37–50.
  * Jiao et al. [2023] Jiao, M., Yu, T., Li, X., Qiu, G., Gu, X., Shen, B., 2023.  On the evaluation of neural code translation: Taxonomy and benchmark, in: 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE), IEEE. pp. 1529–1541.
  * Kocetkov et al. [2022] Kocetkov, D., Li, R., Allal, L.B., Li, J., Mou, C., Ferrandis, C.M., Jernite, Y., Mitchell, M., Hughes, S., Wolf, T., et al., 2022.  The stack: 3 tb of permissively licensed source code.  arXiv preprint arXiv:2211.15533 URL: https://arxiv.org/abs/2211.15533.
  * Lai et al. [2023] Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, W., Fried, D., Wang, S.I., Yu, T., 2023.  DS-1000: A natural and reliable benchmark for data science code generation, in: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J. (Eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, PMLR. pp. 18319–18345.  URL: https://proceedings.mlr.press/v202/lai23b.html.
  * Li et al. [2023] Li, R., Allal, L.B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C., Li, J., Chim, J., et al., 2023.  Starcoder: may the source be with you!  arXiv preprint arXiv:2305.06161 URL: https://arxiv.org/abs/2305.06161.
  * Li et al. [2024] Li, Z., Zhang, J., Yin, C., Ouyang, Y., Rong, W., 2024.  Procqa: A large-scale community-based programming question answering dataset for code search .
  * Liao et al. [2023] Liao, D., Pan, S., Huang, Q., Ren, X., Xing, Z., Jin, H., Li, Q., 2023.  Context-aware code generation framework for code repositories: Local, global, and third-party library awareness .
  * Liu et al. [2024] Liu, J., Bai, Z., Zhang, Y., Zhang, C., Zhang, Y., Zhang, G., Wang, J., Que, H., Chen, Y., Su, W., et al., 2024.  E2-llm: Efficient and extreme length extension of large language models.  arXiv preprint arXiv:2401.06951 .
  * Liu et al. [2023a] Liu, J., Xia, C.S., Wang, Y., Zhang, L., 2023a.  Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, in: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (Eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.  URL: http://papers.nips.cc/paper_files/paper/2023/hash/43e9d647ccd3e4b7b5baab53f0368686-Abstract-Conference.html.
  * Liu et al. [2023b] Liu, T., Xu, C., McAuley, J.J., 2023b.  Repobench: Benchmarking repository-level code auto-completion systems abs/2306.03091.  URL: https://doi.org/10.48550/arXiv.2306.03091, doi:10.48550/ARXIV.2306.03091, arXiv:2306.03091.
  * Lozhkov et al. [2024] Lozhkov, A., Li, R., Allal, L.B., Cassano, F., Lamy-Poirier, J., Tazi, N., Tang, A., Pykhtar, D., Liu, J., Wei, Y., et al., 2024.  Starcoder 2 and the stack v2: The next generation .
  * Lu et al. [2021] Lu, S., Guo, D., Ren, S., Huang, J., Svyatkovskiy, A., Blanco, A., Clement, C., Drain, D., Jiang, D., Tang, D., Li, G., Zhou, L., Shou, L., Zhou, L., Tufano, M., GONG, M., Zhou, M., Duan, N., Sundaresan, N., Deng, S.K., Fu, S., LIU, S., 2021.  CodeXGLUE: A machine learning benchmark dataset for code understanding and generation, in: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).  URL: https://openreview.net/forum?id=6lE4dQXaUcb.
  * Nijkamp et al. [2023a] Nijkamp, E., Hayashi, H., Xiong, C., Savarese, S., Zhou, Y., 2023a.  Codegen2: Lessons for training llms on programming and natural languages.  arXiv preprint arXiv:2305.02309 URL: https://arxiv.org/abs/2305.02309.
  * Nijkamp et al. [2023b] Nijkamp, E., Pang, B., Hayashi, H., Tu, L., Wang, H., Zhou, Y., Savarese, S., Xiong, C., 2023b.  Codegen: An open large language model for code with multi-turn program synthesis, in: International Conference on Learning Representations.  URL: https://openreview.net/forum?id=iaYcJKpY2B_.
  * Pei et al. [2023] Pei, H., Zhao, J., Lausen, L., Zha, S., Karypis, G., 2023.  Better context makes better code language models: A case study on function call argument completion, in: Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAI Press.  URL: https://doi.org/10.1609/aaai.v37i4.25653, doi:10.1609/aaai.v37i4.25653.
  * Peng et al. [2023] Peng, B., Quesnelle, J., Fan, H., Shippole, E., 2023.  Yarn: Efficient context window extension of large language models.  CoRR abs/2309.00071.
  * Phan et al. [2024] Phan, H.N., Phan, H.N., Nguyen, T.N., Bui, N.D., 2024.  Repohyper: Better context retrieval is all you need for repository-level code completion.  arXiv preprint arXiv:2403.06095 .
  * Prenner and Robbes [2023] Prenner, J.A., Robbes, R., 2023.  Runbugrun–an executable dataset for automated program repair .
  * Radford et al. [2019] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., 2019.  Language models are unsupervised multitask learners.  OpenAI preprint URL: https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.
  * Richter and Wehrheim [2022] Richter, C., Wehrheim, H., 2022.  Tssb-3m: Mining single statement bugs at massive scale, in: Proceedings of the 19th International Conference on Mining Software Repositories, pp. 418–422.
  * Robertson et al. [2009] Robertson, S., Zaragoza, H., et al., 2009.  The probabilistic relevance framework: Bm25 and beyond.  Foundations and Trends® in Information Retrieval 3, 333–389.
  * Roziere et al. [2023] Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X.E., Adi, Y., Liu, J., Remez, T., Rapin, J., et al., 2023.  Code llama: Open foundation models for code .
  * Shrivastava et al. [2023a] Shrivastava, D., Kocetkov, D., de Vries, H., Bahdanau, D., Scholak, T., 2023a.  Repofusion: Training code models to understand your repository.  arXiv preprint arXiv:2306.10998 .
  * Shrivastava et al. [2023b] Shrivastava, D., Larochelle, H., Tarlow, D., 2023b.  Repository-level prompt generation for large language models of code, in: Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., Scarlett, J. (Eds.), Proceedings of the 40th International Conference on Machine Learning, PMLR. pp. 31693–31715.  URL: https://proceedings.mlr.press/v202/shrivastava23a.html.
  * Su et al. [2024] Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., Liu, Y., 2024.  Roformer: Enhanced transformer with rotary position embedding 568, 127063.
  * Sun et al. [2024] Sun, T., Chai, L., Jian Yang, Y.Y., Guo, H., Liu, J., Wang, B., Yang, L., Li, Z., 2024.  Unicoder: Scaling code large language model via universal code.  ACL .
  * Tian et al. [2024] Tian, R., Ye, Y., Qin, Y., Cong, X., Lin, Y., Liu, Z., Sun, M., 2024.  Debugbench: Evaluating debugging capability of large language models .
  * Touvron et al. [2023] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al., 2023.  Llama: Open and efficient foundation language models .
  * Wang and Komatsuzaki [2021] Wang, B., Komatsuzaki, A., 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wang et al. [2023a] Wang, Y., Le, H., Gotmare, A.D., Bui, N.D., Li, J., Hoi, S.C., 2023a.  Codet5+: Open code large language models for code understanding and generation .
  * Wang et al. [2021] Wang, Y., Wang, W., Joty, S., Hoi, S.C., 2021.  CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation, in: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Online and Punta Cana, Dominican Republic. pp. 8696–8708.  URL: https://aclanthology.org/2021.emnlp-main.685, doi:10.18653/v1/2021.emnlp-main.685.
  * Wang et al. [2023b] Wang, Z.M., Peng, Z., Que, H., Liu, J., Zhou, W., Wu, Y., Guo, H., Gan, R., Ni, Z., Zhang, M., Zhang, Z., Ouyang, W., Xu, K., Chen, W., Fu, J., Peng, J., 2023b.  Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models.  arXiv preprint arXiv: 2310.00746 .
  * Wu et al. [2024] Wu, Y., Liu, J., Bu, X., Liu, J., Zhou, Z., Zhang, Y., Zhang, C., Bai, Z., Chen, H., Ge, T., Ouyang, W., Su, W., Zheng, B., 2024.  Conceptmath: A bilingual concept-wise benchmark for measuring mathematical reasoning of large language models.  arXiv .
  * Yan et al. [2023] Yan, W., Tian, Y., Li, Y., Chen, Q., Wang, W., 2023.  Codetransocean: A comprehensive multilingual benchmark for code translation .
  * Yu et al. [2024] Yu, H., Shen, B., Ran, D., Zhang, J., Zhang, Q., Ma, Y., Liang, G., Li, Y., Wang, Q., Xie, T., 2024.  Codereval: A benchmark of pragmatic code generation with generative pre-trained models, in: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, pp. 1–12.
  * Zhang et al. [2023] Zhang, F., Chen, B., Zhang, Y., Liu, J., Zan, D., Mao, Y., Lou, J.G., Chen, W., 2023.  Repocoder: Repository-level code completion through iterative retrieval and generation.  arXiv preprint arXiv:2303.12570 URL: https://arxiv.org/abs/2303.12570.
  * Zhang et al. [2024] Zhang, G., Qu, S., Liu, J., Zhang, C., Lin, C., Yu, C.L., Pan, D., Cheng, E., Liu, J., Lin, Q., Yuan, R., Zheng, T., Pang, W., Du, X., Liang, Y., Ma, Y., Li, Y., Ma, Z., Lin, B., Benetos, E., Yang, H., Zhou, J., Ma, K., Liu, M., Niu, M., Wang, N., Que, Q., Liu, R., Liu, S., Guo, S., Gao, S., Zhou, W., Zhang, X., Zhou, Y., Wang, Y., Bai, Y., Zhang, Y., Zhang, Y., Wang, Z., Yang, Z., Zhao, Z., Zhang, J., Ouyang, W., Huang, W., Chen, W., 2024.  Map-neo: Highly capable and transparent bilingual large language model series.  arXiv preprint arXiv: 2405.19327 .
  * Zhu et al. [2022] Zhu, M., Jain, A., Suresh, K., Ravindran, R., Tipirneni, S., Reddy, C.K., 2022.  Xlcost: A benchmark dataset for cross-lingual code intelligence .

Figure 6: A test case from CrossCodeEval+. With the help of abstract
extraction, the code model could understand the entire class YAMX at a glance
and then generate the correct method invocation. Figure 7: A test case from
R2C2-Bench. The statement delete_child occurs multiple times in the prefix and
suffix, resulting in retrieval contexts that are highly relevant to
delete_child. The vanilla code model tends to predict the term that frequently
appears in retrieval results. On the contrary, the model could focus on the
true reference after R2C2-Enhanced tuning.

##  Appendix A Broader Impacts and Limitations

Broader Impacts. In this paper, we propose a methodology with prominent
efficiency and ingenious utilization of cross-file information for real-world
code completion. We anticipate that this methodology, alongside our fine-
tuning strategy and the corresponding benchmarks, will assist developers in
enhancing coding efficiency and liberating productivity. Therefore, we hope
our work can enable programmers to focus on scenarios with greater creativity.

Limitations. Firstly, to attain enhanced performance, it is essential to fine-
tune several hyperparameters. Secondly, despite the need for only 400,000
training samples to achieve outstanding performance, the process demands a
certain amount of GPU resources for fine-tuning. Thirdly, four programming
languages are covered in our setting, and a large number of languages are not
considered in our paper. In our future work, we will continue to investigate
to support more languages for facilitating research in this field.

##  Appendix B More Details

###  B.1 Details on Abstract Context Extraction

In Fig. 11, we have provided a relatively complex abstract syntax tree for the
code in Fig. 10. Besides, we also provide two examples of the extracted
abstract context for Python and Java in Fig. 9.

Figure 8: A case found in CrossCodeEval [16]. As Java supports the package
statement, developers could call cross-file APIs in the same package. Such
kinds of implicit dependencies are ignored by Ding et al. [16]. Figure 9:
Examples of the generated abstract contexts for Python and Java. Figure 10: A
code example. Figure 11: An example of the abstract syntax tree generated by
the Tree-sitter tool.

###  B.2 Quality Control Procedure

We first define the ground-truth completion result as the “reference”, and
design a series of rule-based and model-based post-processing filters to
ensure the quality of the dataset as follows. First, we filter examples if (1)
too short ($<10$ tokens) reference. (2). too long ($5$ lines) reference.
Second, we keep that the percentage of the multiple lines for the reference is
larger than 30%. Moreover, to ensure that the reference isn’t predictably
inferred solely from the current file (possibly owing to strong clues in
function names and comments), we feed the examples (input prompts) to
DeepSeekCoder-1.3B model [21] to complete the statement and remove the exact
matches. This step removes 57.1%, 39.8%, 48.0%, and 47.2% of the samples for
Java, Python, TypeScript, and C#, respectively. Finally, we perform human
annotations on a subsample of R2C2-Bench and found that the dataset has a
satisfactory quality to serve the goal of cross-file code completion.

###  B.3 Dataset Statistics of the validation split of R2C2-Bench

In Table 9, we provide the statistics of the validation split of R2C2-Bench.

Feature | Python | Java | TypeScript | C#
---|---|---|---|---
# Repositories | 23 | 43 | 42 | 21
# Files | 250 | 250 | 250 | 250
# Examples | 425 | 717 | 687 | 521
Avg. # cross files in prompt | 6.03 | 7.76 | 6.84 | 11.28
Avg. # lines in prompt | 326.04 | 333.98 | 367.46 | 374.81
Avg. # tokens in prompt | 3527.72 | 3467.47 | 3451.46 | 3460.65
Avg. # lines in reference | 1.65 | 1.80 | 2.12 | 1.77
Avg. # tokens in reference | 16.19 | 14.64 | 15.06 | 13.52
Table 9: The statistics of the validation split of R2C2-Bench.

###  B.4 Dataset Statistics of CrossCodeEval

As shown in Table 10, we have provided the dataset statistics of the
CrossCodeEval benchmark.

Feature | Python | Java | TypeScript | C#
---|---|---|---|---
# Repositories | 471 | 239 | 193 | 99
# Files | 1368 | 745 | 779 | 642
# Examples | 2665 | 2139 | 3356 | 1768
Avg. # cross files in prompt | 3.19 | 3.15 | 3.16 | 3.98
Avg. # lines in prompt | 90.6 | 106.7 | 116.5 | 71.1
Avg. # tokens in prompt | 938.9 | 995.3 | 944.9 | 584.1
Avg. # lines in reference | 1.0 | 1.1 | 1.7 | 1.7
Avg. # tokens in reference | 13.2 | 14.5 | 17.4 | 12.5
Table 10: The statistics of the CrossCodeEval.

##  Appendix C More Experiments

###  C.1 Details of the baseline models

StarCoder [32] is a generative decoder-only LM series whose size varies from
1B to 15.5B. Trained on the Stack [30] dataset, StarCoder supports a maximum
context length of 8K.

DeepSeekCoder [21] is a suite of code models with sizes ranging from 1.3B to
33B trained on a 2-trillion-token manual collected corpus. Benefit from FIM
strategy [8] and Rotary Position Embedding (RoPE) [52], these models are
capable of efficient code generation and infilling under a 16K context window.

Code Llama [49] is a family of large language models based on Llama 2 [55]
with 7B, 13B, 34B, and 70B parameters. These models are trained on sequences
of 16K tokens and capable of inference on 100K. We conduct experiments with
the foundational Code Llama models, not the Python-specialized versions or
instruction-tuned ones.

###  C.2 Details of CrossCodeEval and CrossCodeEval+

CrossCodeEval [16] is an arising benchmark for cross-file code completion that
is built on a diverse set of real-world, open-sourced, permissively-licensed
Github repositories in four popular programming languages: Python, Java,
TypeScript, and C#. To create accurate and high-quality test samples, the
authors adopt static analysis and filter out those samples solved by
starcoderbase-1B within in-file only context.

CrossCodeEval+. We collect the original code bases of CrossCodeEval [16] then
parse and extract the cross-file dependencies with our R2C2-Enhance. During
retrieval, the abstracts and the snippets are placed together in the candidate
pool (Sec. 3.1.1). Note that we keep the cursor positions and the expected
outputs the same as those in CrossCodeEval. Here, we name the reconstructed
CrossCodeEval as CrossCodeEval+.

###  C.3 Execution Time Benchmarking

Execution time measurements are conducted on an Intel Xeon CPU E5-2682 v4 with
a base frequency of 2.50 GHz and a turbo boost of up to 4.00 GHz.