# Image reconstruction from tissue scattered events for ${\beta}^{+}\gamma$
coincidences in Compton-PET

Satyajit Ghosh  Department of Physics, Indian Institute of Technology Bombay,
Mumbai, INDIA  Pragya Das  Department of Physics, Indian Institute of
Technology Bombay, Mumbai, INDIA

Abstract For long time non-pure beta emitters are avoided from PET imaging due
to extra dose and increase in background from Compton scattering. But advent
of high-resolution Compton camera system opens up new domain of imaging.
Various non-pure beta emitters are formed as beam irradiation byproduct in
therapy which can be used in online beam range verification. In this case, the
number of usable counts for imaging is generally 1-3 order lesser than normal
PET scan. On the other hand, we know that in human PET scanner, 30-60% can be
tissue scattered coincidences in 3D case containing 80% single scattered
events. In this work, we have investigated feasibility of imaging using only
single scattered coincidences for non-pure beta emitters in a Compton-PET
system. The locus of tissue scatter point can be reduced to in generally two
points after using Compton cone from both ends of 511 keV detections. Finally,
annihilation point is estimated using Compton cone of 1157 keV gamma and time-
of-flight information for the 511 keV. We believe independent assessment of
underlying activity from single scattered data sets will increase confidence
in image interpretation.
---

##  1 Introduction

For long time non-pure beta emitter radioisotopes (e.g.,
44mSc,94Tc,14O,68Ga,124I,10C) are not used in PET imaging. This is because of
extra dose and Compton scattering background that the quasi-simultaneously
emitted extra gamma ray produces. But with the development of excellent
resolution Compton camera systems this situation had changed. New concept of
imaging using triple coincidence data was proposed [1]. In this new imaging,
Compton cone drawn using extra gamma interaction points was used to estimate
original annihilation point on the LOR, similar to TOF-PET imaging [2].
Application of these type of radioisotopes is in generally of two types. It is
used as conventional radiopharmaceutical, e.g., [44Sc]Sc-PSMA-617 [3] in
prostate cancer imaging. Besides, various non-pure beta emitters are formed as
beam irradiation byproduct in ion therapy [4]. Hence, it is online or offline
beam range monitoring agent. But in this case, generally the emitted count is
1-3 orders magnitude smaller than conventional PET scan [5]. On the other
hand, it is known that tissue scattering can contribute to 30-60% of
coincidences in human 3D-PET [6] in which 80% are single scattered [7]. So, in
this work, we have investigated the feasibility of image reconstruction from
those tissue scattered events in Compton-PET system. Our aim is to produce a
physically meaningful image from the single scattered data which is
independent from unscattered data. We believe having two independent image of
same underlying activity distribution will assists us in better diagnosis. In
this context, it is worth to mention that the motivation of WGI imaging
concept is indeed to use all types data independently [8].

We performed GATE [9] simulation with finite resolution parameters for a
Compton-PET system with silicon as scatterer ring and LaBr3:Ce as absorber
ring. Geometrical arrangement and parameters were chosen keeping in mind the
sensitivity and resolution. Line sources of 44Sc was used. And a cylindrical
water phantom of diameter 10 cm was placed axially. At first, we had shown
that the locus of tissue scattering point of a single scattered coincident
(single scatter surface) is prolate spheroid (for scattering angle,
${\theta}_{s}<{90}^{0}$) and spindle toroid (for ${\theta}_{s}>{90}^{0}$)
where to acquire these single scattered coincidences photo-peak and off-peak
energy windows positioned in accordance with detector resolution were used.
Data acquisition was performed using an appropriately defined trigger logic.
Compton cones from both end of 511 keV detection were projected on the single
scatter surface to obtain two 3D curves which cut each other in generally at
two points forming two possible broken LORs. Finally, annihilation point was
estimated by projecting Compton cone of 1157 keV and TOF information was used
to choose between the two. The image we obtained is physically meaningful and
proves the feasibility of single scattered imaging in non-pure beta emitter
cases.

##  2 Materials and Methods

At first, we have discussed the locus of single scattering point in case of
Compton-PET system. For proving the feasibility a GATE simulation was
performed. Trigger logic was developed for data extraction. Finally the image
reconstruction algorithm was proposed for single scatter imaging.

###  2.1 Locus of scattering point

We have drawn a typical Compton-PET set in figure (1). From here on, we named
the locus as single scatter surface to avoid any confusion. Now, we assume a
single scattered coincident event where annihilation happened at point O and
tissue scattering at C. To find the locus of the scattering point C, at first,
we write down the equation of locus depending only on scattering angle in
tissue (${\theta}_{s}$),

$\overrightarrow{AC}.\overrightarrow{CB}=\left|\overrightarrow{AC}\right|\left|\overrightarrow{CB}\right|\
cos\ {\theta}_{s}$
$\Rightarrow\left(\overrightarrow{r}-\overrightarrow{r_{A}}\right).\left(\overrightarrow{r_{B}}-\overrightarrow{r}\right)=\left|\overrightarrow{r}-\overrightarrow{r_{A}}\right|\left|\overrightarrow{r_{B}}-\overrightarrow{r}\right|\
cos\ {\theta}_{s}$ (1)

where tissue scattering angle is calculated using this equation

${\theta}_{s}=\arccos\left(2-\frac{511}{{E_{1}}+{E_{2}}}\right)$

where $E_{1}$ and $E_{2}$ are energies deposited in scatterer and absorber by
the tissue scattered photon and here we have assumed a full energy deposition.

Figure 1: Compton-PET set up with a single scatter event get detected at
points A, B (scatterer ring detection points) and points D, E (absorber ring
detection points) where annihilation happended at O and tissue scattering
happended at point C; tissue scattering angle is ${\theta}_{s}$ and scattering
angles in scatterer ring are ${\theta}_{a}$, ${\theta}_{b}$; locus of
scattering point shown in blue curve and Compton cones from both ends of 511
keV detection are shown in green colored cones.

Now applying Compton cone constraint from both side of 511 keV detection the
tissue scattering point can be further localised. The equations of the Compton
cones are

$\left(\overrightarrow{r}-\overrightarrow{r_{A}}\right).\
\widehat{n_{A}}=\left|\overrightarrow{r}-\overrightarrow{r_{A}}\right|\ cos\
{\theta}_{a}$ (2)

and,

$\left(\overrightarrow{r}-\overrightarrow{r_{B}}\right).\
\widehat{n_{B}}=\left|\overrightarrow{r}-\overrightarrow{r_{B}}\right|\ cos\
{\theta}_{b}$ (3)

where ${\theta}_{a}$ and ${\theta}_{b}$ are scattering angles from scatterer
ring and $\widehat{n_{a}}$ and $\widehat{n_{b}}$ are unit vectors along line
joining from absorption to scattering point respectively.

It is known that eq. (1), which represents single scatter surface (blue curve
in figure 1), is a surface equation of a prolate spheroid for
${\theta}_{s}<{90}^{0}$ and of spindle toroid for ${\theta}_{s}>{90}^{0}$. For
further constraining the locus of scattering point, we have solved eq. (1)
with eq. (2) and (3) which means solution between single scatter surface and
cones. We found that the solution to be closed contour 3D curves on the single
scatter surface. And two curves from both end of 511 keV detection cut each
other in generally at two points. Further discussion about this is given in
section 3.

###  2.2 GATE simulation

We performed a GATE [9] simulation of a Compton-PET system. Silicon scatterer
ring of thickness 2.5 cm and radius 20 cm was chosen. Radius was chosen larger
since we are working with human scanner. And LaBr3:Ce absorber of ring radius
28 cm and thickness 3 cm was used. Axial width of each ring was 28 cm. Energy
resolutions of scatterer and absorber were 2.5% and 5% @511 keV and time
resolutions were 1 ns and 200 ps respectively. Finally the spatical resolution
was chosen to be 2 mm and 5 mm respectively. For image resolution study a 44Sc
line source, situated at the centre of the scanner, of activity 1 MBq was
used. Activity was chosen low to have a smaller number of random events. A
cylindrical water phantom of diameter 10 cm and height 28 cm was defined
axially. The decision of working with a human scale Compton-PET set up was due
to the fact that the scatter fraction in human PET scan is significant enough
to interest us in the proposed idea whereas in small animal imaging scatter
fraction is not so high.

###  2.3 Trigger logic

After generating the data from GATE simulation, we had defined a trigger logic
to select out usable valid triple gamma single scattered events. A coincidence
time window of 10 ns was used for data selection. At first, two different
energy windows, for 511 keV, the energy window was from 10-255 keV and for
1157 keV gamma it is 255-818 keV were used to select out scatterer detector
interactions. If three hits in the above specified energy windows (two for 511
keV and one for 1157 keV) for the scatterer were obtained then we collected
all the events in absorber ring falling in that coincidence time window and
sorted out events with only three hits in absorber ring. In next stage,
correspondence between individual scatter hit and absorber interactions were
made. At first, the absorber hit corresponding to 1157 keV is identified
depending on closeness of summed energy to 1157 keV. Then remaining two
absorber hits were allocated depending on closeness from scatter hits.
Finally, single scattered coincidences were acquired using photo-peak and off-
peak energy windows of 495-525 keV and 250-495 keV respectively.

###  2.4 Image reconstruction

Image reconstruction was performed without applying any typical algorithm
(e.g., MLEM, OSEM [7]). Rather the annihilation points were estimated
independently for each event. At first, we had calculated two possible
scattering points on the single scatter surface as explained in section 2.1.
Then Compton cone of 1157 keV was projected on the two separate broken LORs to
obtain at most four possible annihilation points (figure 2). One point among
those was selected depending on FOV constraint and TOF information.

Figure 2: A single scattered event is detected at points A, B in scatterer
ring and at point D, E in absorber ring; the intersection between single
scatter surface, two Compton cones for 511 keV lefts us with two possible
scattering points K, J; the intersection between Compton cone of 1157 keV and
broken LORs gives us four possible annihilation points; one among those are
chosen depending on FOV constraint and TOF information.

##  3 Results

As described in the section 2.1, using Compton cones from both sides of 511
keV detection, the locus can be further constrained. The cross section between
single scatter surface and the cone in these cases is a type of 3D curves (red
and green curve in figure 3) such that two such 3D curves cut each other in
generally at two points. It is worth to mention here that the generation of
two cross points is not due to finite resolution of detectors and hence those
can be quite a distant apart (figure 3).

Figure 3: The solution between single scatter surface (yellow envelope) and
Compton cones from each end of 511 keV detection is shown as green and red
curve, in generally these two curves cut each other at two points, it is to be
noted that two point is not due to finite resolution of detector.

We had performed the GATE simulation of Compton-PET system with parameters
described in section 2.2. Then root output data was processed using the
trigger logic described in section 2.3. The trigger logic was implemented
through MATLAB scripts. Figure 4-6 shows the 2D energy histogram plot between
scatter energy deposition vs. absorber energy deposition for 511 and 1157
keVs. For unscattered photons we can find x+y=511 keV line where x and y are
absorber and scatter deposition respectively which shows that the proposed
trigger logic is able to collect 511 keV data (figure 4). The line is
discontinued at scatter energy 10 keV and 255 keV, because of energy window on
scatter deposition (see section 2.3). Besides the width of the x+y=511 line is
decided by photo-peak width chosen in trigger logic. On the other hand, for
1157 keV detection similar x+y=1157 keV line can be seen (figure 5). Here we
have discontinuity on scatterer energy at 255 keV and 818 keV due to energy
window applied in trigger logic. Point to be noted, here unlike 511 keV we
have count below the x+y=1157 line. This is because there is no window applied
on total energy like photo-peak energy window for 511 keV. We have assumed a
full deposition of energy of 1157 keV gamma in scatterer and absorber.
Finally, for single scattered events, rather than having a line we have area
bounded by x+y=250 keV, x+y=495 keV, x=10 keV, and x=255 keV (figure 6). First
two bounds are due to off-peak window and last two are applied in initial
stage of trigger logic.

Figure 4: The count histogram color plot between scatterer and absorber energy
deposition for 511 keV unscattered photon detection. Figure 5: The count
histogram color plot between scatterer and absorber energy deposition for 1157
keV photon detection. Figure 6: The count histogram color plot between
scatterer and absorber energy deposition for 511 keV single scattered photon
detection. Figure 7: Single scattered image (cross sectional) of line source,
pixel size was chosen to be $8\times 8$ $mm^{2}$.  Figure 8: Intensity line
profile (horizontal) of the single scatter image through (0,0) point, pixel
size was chosen to be $8\times 8\ mm^{2}$.

Finally, we had produced image from single scattered data set (figure 7). For
image resolution study, we had calculated FWHM of intensity line profile
(horizontal) of the image. The histograms were shown in figure (8) with FWHM
calculated to be 35.864 mm. To sum up, we were able to produce physically
meaningful single scatter images. This proves the feasibility of single
scatter imaging for Compton-PET system with triple gamma source.

##  4 Conclusion

We have proposed the idea of feasibility of imaging from single scattered
(inside tissue) data in triple gamma imaging. Tissue scattered data in human
PET scan can go up to 40-60%. On the other hand, triple gamma imaging suffers
from low count specially in online ion range verification in ion therapy and
hence in that context the idea of imaging from scattered data is relevant.
Although a better resolution image than unscattered image can’t be expected
from scattered data due to inherent resolution effects, we believe that
producing image from two independent data sets – unscattered and single
scattered – will improve our diagnosis ability. We have shown the feasibility
of the proposed concept. Analysing GATE simulation data, we are able to
produce physically meaningful images. The trigger logic used here is not
claimed to be perfect. Rather simplicity is invoked to make the task
computationally simple as this work is related to only feasibility. We believe
that the idea proposed can be beneficial in triple gamma imaging based beam
range monitoring and late point imaging in case of Scandium DOTA-TOC imaging.

##  5 Acknowledgement

Authors wish to gratefully acknowledge the Center for Development of Advanced
Computing (C-DAC), Pune, India, for providing the supercomputing facilities
[10] for data analysis.

## References

  * [1] C Grignon et al.  “Nuclear medical imaging using $\beta$\+ $\gamma$ coincidences from 44Sc radio-nuclide with liquid xenon as detection medium”  In _Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment_ 571.1-2 Elsevier, 2007, pp. 142–145
  * [2] D Giovagnoli et al.  “A Pseudo-TOF Image Reconstruction Approach for Three-Gamma Small Animal Imaging”  In _IEEE Transactions on Radiation and Plasma Medical Sciences_ IEEE, 2020
  * [3] Elisabeth Eppard  “Pre-Therapeutic Dosimetry Employing Scandium-44 for Radiolabeling PSMA-617”  In _Prostatectomy_ IntechOpen, 2018
  * [4] K Parodi, A Ferrari, F Sommerer and H Paganetti  “Clinical CT-based calculations of dose and positron emitter distributions in proton therapy using the FLUKA Monte Carlo code”  In _Physics in Medicine & Biology_ 52.12 IOP Publishing, 2007, pp. 3369
  * [5] Christopher Kurz et al.  “Investigating the limits of PET/CT imaging at very low true count rates and high random fractions in ion-beam therapy monitoring”  In _Medical Physics_ 42.7 Wiley Online Library, 2015, pp. 3979–3991
  * [6] Habib Zaidi and Marie-Louise Montandon  “Scatter compensation techniques in PET”  In _PET clinics_ 2.2 Elsevier, 2007, pp. 219–234
  * [7] Dale L Bailey, Michael N Maisey, David W Townsend and Peter E Valk  “Positron emission tomography”  Springer, 2005
  * [8] Eiji Yoshida et al.  “Whole gamma imaging: a new concept of PET combined with Compton imaging”  In _Physics in Medicine & Biology_ 65.12 IOP Publishing, 2020, pp. 125013
  * [9] Sébastien Jan et al.  “GATE: a simulation toolkit for PET and SPECT”  In _Physics in Medicine & Biology_ 49.19 IOP Publishing, 2004, pp. 4543
  * [10] Param Yuva-II  “India’s fastest supercomputer, 2013”  In _URL http://cdac. in/index. aspx_ , 2013


#  Astraios: Parameter-Efficient Instruction Tuning Code Large Language Models

Terry Yue Zhuo 1,2 Armel Zebaze 3 Nitchakarn Suppattarachai 1
Leandro von Werra 3 Harm de Vries 4 Qian Liu 5 Niklas Muennighoff 6
1 Monash University 2 CSIRO’s Data61 3 Hugging Face
4 ServiceNow Research 5 Sea AI Lab 6 Contextual AI
<EMAIL_ADDRESS>
https://github.com/bigcode-project/astraios The work was partially done at
Hugging Face.

###### Abstract

The high cost of full-parameter fine-tuning (FFT) of Large Language Models
(LLMs) has led to a series of parameter-efficient fine-tuning (PEFT) methods.
However, it remains unclear which methods provide the best cost-performance
trade-off at different model scales. We introduce Astraios, a suite of 28
instruction-tuned OctoCoder models using 7 tuning methods and 4 model sizes up
to 16 billion parameters. Through investigations across 5 tasks and 8
different datasets encompassing both code comprehension and code generation
tasks, we find that FFT generally leads to the best downstream performance
across all scales, and PEFT methods differ significantly in their efficacy
based on the model scale. LoRA usually offers the most favorable trade-off
between cost and performance. Further investigation into the effects of these
methods on both model robustness and code security reveals that larger models
tend to demonstrate reduced robustness and less security. At last, we explore
the relationships among updated parameters, cross-entropy loss, and task
performance. We find that the tuning effectiveness observed in small models
generalizes well to larger models, and the validation loss in instruction
tuning can be a reliable indicator of overall downstream performance.

Figure 1: Mean task performance of Astraios models across 5 representative
tasks and 8 datasets. We indicate the average percentage of total parameters
updated for each PEFT method.

##  1 Introduction

Large language models (LLMs) (Zhao et al., 2023) trained on Code (Code LLMs)
have shown strong performance on various software engineering tasks (Hou et
al., 2023). There are three main model paradigms: (A) Code LLMs for code
completion (Nijkamp et al., 2022; Fried et al., 2022; Li et al., 2023); (B)
Task-specific fine-tuned Code LLMs for a single task (Hou et al., 2023); and
(C) Instruction-tuned (Ouyang et al., 2022) Code LLMs that excel at following
human instructions and generalizing well on unseen tasks (Wang et al., 2023b;
Muennighoff et al., 2023c). Recent instruction-tuned Code LLMs, including
WizardCoder (Luo et al., 2023) and OctoCoder (Muennighoff et al., 2023a), have
achieved state-of-the-art performance on various tasks without task-specific
fine-tuning. However, with the increasing parameters of Code LLMs, it becomes
more expensive to perform full-parameter fine-tuning (FFT) to obtain
instruction-tuned models. In practise, to save computational cost, parameter-
efficient fine-tuning (PEFT) have been applied. This training strategy aims to
achieve comparable performance to FFT by updating fewer parameters. While
there are many PEFT methods (Ding et al., 2022), the predominant PEFT method
is still LoRA, which is proposed in 2021 (Hu et al., 2021). However, there is
no empirical evidence showing LoRA remains the best for instruction-tuned code
LLMs. In this paper, we investigate instruction-tuned code LLMs with the
following research question: what are the best PEFT methods for Code LLMs?

Existing analysis on PEFT methods presents several opportunities for further
exploration: (1) Beyond Task-Specific LLMs. Most prior works (Zhou et al.,
2022; Ding et al., 2023) only focus on the model paradigm (B), where the
selected base models are fine-tuned on specific downstream tasks. While these
studies provide insights into PEFT methods on task-specific LLMs, the
transferability of their findings to the instruction tuning paradigm is
unclear. (2) Diverse Domains. Studies on PEFT methods tend to evaluate in the
predominant domains like vision (Sung et al., 2022; He et al., 2023) and text
(Houlsby et al., 2019; He et al., 2021), leaving other domains like code
underexplored. (3) Inclusive PEFT Methods. Prior investigations on PEFT mainly
consider a limited number of methods, such as adapter-based tuning (Houlsby et
al., 2019) or reparametric tuning (Aghajanyan et al., 2021), which does not
capture the full breadth of available methods. (4) Multidimensional
Evaluation. Previous works only consider limited evaluation on representative
downstream tasks (Chen et al., 2022; Fu et al., 2023; Ding et al., 2023). We
argue that other evaluation dimensions like model robustness (Han et al.,
2021) and output code safety (Weidinger et al., 2021; Zhuo et al., 2023b;
Pearce et al., 2022; Dakhel et al., 2023) are also important, especially in
the era of LLM agents (Ouyang et al., 2022; Xie et al., 2023). (5)
Scalability. Most prior PEFT work has only explored LLMs with insufficient
scales of model sizes and training time, which makes their scalability
questionable (Lester et al., 2021; Chen et al., 2022; Hu et al., 2023).

To explore these identified opportunities further, we introduce Astraios, a
suite of 28 instruction-tuned Code LLMs, which are fine-tuned with 7 tuning
methods based on the StarCoder (Li et al., 2023) base models (1B, 3B, 7B,
16B). We instruction-tune the models based on the open-source dataset,
CommitPackFT from OctoPack (Muennighoff et al., 2023a), to balance their
downstream capabilities. We utilize PEFT configurations with Hugging Face’s
best practices (Mangrulkar et al., 2022) and integrate a few PEFT methods from
recent frameworks (Hu et al., 2023). We first inspect the scalability of
different tuning methods through the lens of cross-entropy loss during
instruction tuning. Specifically, we assess the scales of model size and
training time. Our main evaluation focuses on 5 representative code tasks,
including clone detection (Svajlenko and Roy, 2021), defect detection (Zhou et
al., 2019), code synthesis (Muennighoff et al., 2023a), code repair
Muennighoff et al. (2023a), and code explain (Muennighoff et al., 2023a). We
further study the tuning methods from two aspects: model robustness (Wang et
al., 2023a) and code security (Pearce et al., 2022). We assess how well models
can generate code based on the perturbed examples and how vulnerable the
generated code can be.

The main experimental results can be found in Figure 1, where we observe that
FFT generally leads to the best downstream performance across all scales. In
addition, we find that PEFT methods differ significantly in their efficacy
depending on the model scale. At 16B parameters, Parallel Adapter (He et al.,
2021) and LoRA (Hu et al., 2021) are the most competitive methods with FFT.
Meanwhile, at 1B parameters, they are both slightly outperformed by P-Tuning
and (IA)3. Thus, the choice of the PEFT method should be considered along with
the model scale at hand. Nevertheless, LoRA usually offers the most favourable
trade-off between cost and performance.

Meanwhile, we also observe that larger PEFT Code LLMs perform better on code
generation tasks while they do not show such patterns on code comprehension
tasks like clone detection and defect detection. In addition, increasing model
size improves generation task performance but exhibits vulnerabilities to
adversarial examples and biases towards insecure code. Additionally, we
investigate the relationships among updated parameters, cross-entropy loss,
and task performance. We find that the final loss of small PEFT models can be
extrapolated to the larger ones. We also observe strong correlations between
final loss and overall downstream task performance. Although the instruction
dataset we choose is general and is not directly correlated with the benchmark
downstream tasks, we suggest that the performance on such general data can
serve as a proxy for the downstream performance.

##  2 The Astraios Suite and Benchmark

In this section, we document our model choices, training configurations, and
evaluations in detail for easy reproducing our experimental results in this
paper.

Figure 2: Percentage (%) of total parameters updated for each PEFT method in
Astraios models. Table 1: Summary of tuning methods and the trainable
parameters of different model scales.

Type | Name | 1B | 3B | 7B | 16B
---|---|---|---|---|---
Low-Rank | LoRA (Hu et al., 2021) | 3,588,096 | 7,372,800 | 12,472,320 | 17,776,640
Prompt | P-Tuning (Liu et al., 2023) | 12,650,496 | 23,882,496 | 50,466,816 | 113,448,960
Adapter | (IA)3 (Liu et al., 2022) | 251,904 | 516,096 | 870,912 | 1,239,040
AdapterH (Houlsby et al., 2019) | 50,331,648 | 103,809,024 | 176,160,768 | 251,658,240
AdapterP (Pfeiffer et al., 2020) | 25,165,824 | 51,904,512 | 88,080,384 | 125,829,120
Parallel (He et al., 2021) | 26,738,688 | 54,263,808 | 90,832,896 | 128,450,560
FFT | FFT | 1,137,207,296 | 3,043,311,104 | 7,327,263,232 | 15,517,456,384

###  2.1 Model

#### Base Model

There are many Code LLMs available that could be a suitable base model.
However, some of them are not fully open such as Code-Llama (Roziere et al.,
2023), where the training data is not discussed. To maximize transparency, we
select the StarCoder series as our base models. Concretely, four model scales
including 1B, 3B, 7B and 16B parameters are selected.

#### PEFT Model

We focus on three kinds of PEFT methods (Ding et al., 2022): (1) Adapter-based
Tuning (Houlsby et al., 2019): An early approach, which injects small-scale
neural modules as adapters to LLMs and only tune these adapters for model
adaptation. (2) Prompt-based Tuning (Li and Liang, 2021): It wraps the
original input with additional context introducing virtual task-specific
tokens without adding layers of modules like adapters. (3) Intrinsic-rank-
based Tuning (Aghajanyan et al., 2021): A representative method is LoRA, which
assumes that the change of weights during model tuning has a low rank and thus
low-rank changes to the matrices suffice. For all methods, we utilize the
implementations in the open-source PEFT
library111https://github.com/huggingface/peft (Mangrulkar et al., 2022) and
the LLM-Adapters work (Hu et al., 2023) built on top of it. We benchmark 6
PEFT methods, including 4 adapter-based, 1 prompt-based, and 1 intrinsic-rank-
based tuning methods as shown in Table 1. We also include FFT for each model
size. The ratio of updated parameters of each PEFT method is presented in
Figure 2.

###  2.2 Instruction Tuning

#### Dataset

Following previous work, we select the dataset CommitPackFT+OASST from
OctoPack (Muennighoff et al., 2023a) as the instruction tuning dataset, which
helps StarCoder to achieve superior performance. We note that there could be
other choices by utilizing other datasets (e.g., the publicly available
dataset CodeAlpaca (Chaudhary, 2023)) . However, they usually focus on a
certain aspect of code-related tasks and lack generality to different tasks.

#### Configuration

We train all models with a sequence length of 2048 tokens, with the batch size
as 1, the warm-up step as 12, and the global steps as 200. We set the learning
rate as $1\times 10^{-4}$ for PEFT models and $1\times 10^{-6}$ FFT models
with a cosine scheduler in both cases. For PEFT methods, we use 8-bit-
quantized models during training (Dettmers et al., 2022).

###  2.3 Evaluation

#### Code Comprehension

To evaluate code comprehension, we select two representative tasks: clone
detection and defect detection. Clone detection aims to identify segments of
code that are either exact duplicates or structurally similar with variations
in identifiers, literals, types, layout, and comments, or even more broadly
similar in terms of functionality. Defect detection targets for identifying
bugs, vulnerabilities, or antipatterns in code. We select two widely-used
datasets from CodeXGLUE benchmark Lu et al. (2021): BigCloneBench (Svajlenko
and Roy, 2021) and Devign (Zhou et al., 2019). As the original BigCloneBench
and Devign are designed to evaluate classification models, we prepend
additional instructions to prompt the instruction-tuned models to complete
such tasks. We follow the evaluation settings of CodeXGLUE and use F1 and
Accuracy for BigClone and Devign, respectively. Due to the non-trivial number
of test examples in these two datasets, we sample 2,000 from each to save
costs. As BigCloneBench and Devign are in the binary classification tasks, we
use temperature 0 for model inference to get deterministic outputs.

#### Code Generation

We use HumanEvalPack (Muennighoff et al., 2023a), a benchmark recently
proposed that enables easy evaluation of instruction-tuned Code LLMs. The
benchmark is structured around three core tasks in code generation, each
designed to test different capabilities of the model. The first task, Code
Synthesis, involves the model in synthesizing functional code given a function
with a docstring detailing the desired code behavior. The second task, Code
Repair, challenges the model to identify and fix a subtle bug in an otherwise
correct code function, using provided unit tests as a guide. The third and
final task, Code Explanation, requires the model to generate a clear and
concise explanation for a correctly written code function. For the evaluation
on HumanEvalPack, we use its Python and Java splits and compute Pass@1 for
each task. We use temperature 0.2 and sample 20 outputs per test example.

#### Model Robustness

Evaluating the robustness of code generation models is crucial in
understanding their real-world applicability and reliability. Models that can
maintain high-performance levels despite variations and perturbations in input
data are more likely to be effective in diverse and dynamic coding
environments (Bielik and Vechev, 2020; Henkel et al., 2022; Wang et al.,
2023a). Motivated by such model behaviors, we utilize ReCode (Wang et al.,
2023a), a benchmark framework designed to assess the robustness of Code LLMs.
We use HumanEval (Chen et al., 2021) as the base dataset and curated it to
mimic natural variations while preserving the semantic integrity of the
original inputs. The perturbations cover a range of transformations (Zhuo et
al., 2023c) on code format, function, variable names, code syntax, and
docstrings. These transformations are not arbitrary but represent changes
occurring naturally in coding practices. The quality of the perturbed data in
ReCode is verified through human evaluation and objective similarity scores,
ensuring the relevance and reliability of the dataset for robustness
assessment. We use temperature 0.2 and 20 samples per test example for the
generation. To compute the level of model robustness, we adopt Robust Pass@k
(RP@k) from ReCode and also compute Robust Change@k (RC@k) as follows:

$RP@k:=\mathbb{E}_{x}\left[1-\frac{{n-r_{c}s(x)}}{{\binom{n}{k}}}\right]$ (1)
$RC@k:=\left|Pass@k-Robust\ Pass@k\right|$ (2)

#### Code Security

One limitation of Code LLMs is their tendency to generate code with potential
security vulnerabilities, as various studies have highlighted (Dakhel et al.,
2023; Asare et al., 2023). In our work, we aim to empirically examine how PEFT
methods can influence the security aspects of Code LLM outputs. We utilize the
“Asleep at the Keyboard” (AATK) benchmark (Pearce et al., 2022), which
includes 89 security-centric scenarios, to provide a comprehensive evaluation
across three distinct dimensions: Diversity of Weakness (DoW), encompassing 18
unique vulnerability classes from the MITRE Common Weakness Enumeration (CWE)
taxonomy, sourced from the 2021 CWE Top 25 Most Dangerous Software Weaknesses;
Diversity of Prompt (DoP), assessing responses to different prompts within the
SQL injection vulnerability class; and Diversity of Domain (DoD), involving
scenarios in Verilog, a hardware description language. Our analysis
predominantly focuses on the DoW axis, comprising 54 scenarios–25 in C and 29
in Python–covering 18 CWEs. This focus is due to the automatic evaluation
challenges associated with the other two dimensions. After filtering out
scenarios that lack an automated test, we thoroughly examine 40 scenarios,
including 23 in C and 17 in Python. We use temperature 0.2 and 20 samples per
test example for the generation.

##  3 Preliminary Study: Cross-Entropy Loss

Cross-entropy loss has been used as the principal performance metric in
training LLMs for NLP tasks (Brown et al., 2020; Hernandez et al., 2021; Zhang
et al., 2022b). Most studies on modeling loss focus on either pre-training
(Kaplan et al., 2020) or FFT (Chung et al., 2022). Previous studies have
consistent findings on loss (Kaplan et al., 2020; Hoffmann et al., 2022;
Aghajanyan et al., 2023): The final loss tends to decrease when the training
computation (e.g., model sizes, training data and training time) increases.
These observations indicate that more training time and more trainable model
parameters can lead to better alignment with the tuning data. However, there
is no systematic investigation for PEFT, especially for Code LLMs. Based on
the updated parameters for each tuning method in Table 1, we hypothesize that
each PEFT method has a similar trend to previous findings of loss. Inspired by
Kaplan et al. (2020), we study the loss change for instruction tuning Code
LLMs, varying two factors: (1) Model Size (1B - 16B); and (2) Training Time
(measured in global step, maximum 200 steps). Due to the limited budget, We do
not study how the amount of training data may affect the loss.

Figure 3: Final loss across model sizes.

#### Model Size Scaling

We present the results of final loss in Figure 3 when varying the model size
from 1B to 16B. Our first observation is that train and test loss are well
aligned, indicating that the models trained on the selected tuning methods are
not overfitted. The second observation is that both train and test loss also
strictly decrease when the model size increases. Although these observations
are aligned with the aforementioned observations (Kaplan et al., 2020;
Hoffmann et al., 2022), they show the different scales of loss change,
suggesting different tuning methods may require different levels of power.
Compared to other tuning methods, FFT demonstrates a slightly better loss
performance than PEFT methods like LoRA and Parallel Adapter. As we notice
that heavier PEFT methods (which update more parameters) tend to have a better
final loss, we hypothesize that more trainable parameters in the model may
result in a smaller loss, regardless of how the parameters are updated during
training.

1B Astraios models.

3B Astraios models.

7B Astraios models.

16B Astraios models.

Figure 4: Test loss of Astraios models across training time measured by Global
Step.

#### Training Time Scaling

We show the changes in test loss on the Astraios when varying the training
time in Figure 4. We notice that the loss continues decreasing when the model
is trained longer. Although the loss changes of (IA)3 are consistently
insignificant. Notably, the loss of P-Tuning decreases drastically to 50 steps
but behaves similarly to other prompt-based methods. In terms of tuning
stability, we observe that P-tuning is more unstable than other methods, where
the loss change appears to be a non-monotonic pattern. When comparing FFT
against PEFT methods, we find that FFT tends to decrease even after 200 steps,
while PEFT methods do not show a decreasing trend clearly. We hypothesize that
it may be due to the number of updated parameters, where FFT updates the full
parameters in the model.

Table 2: Results of Astraios models on Defect Detection and Clone Detection.
The best performance is highlighted in bold. The second best performance is
underlined.

Method | Defect Detection | Clone Detection
---|---|---
1B | 3B | 7B | 16B | 1B | 3B | 7B | 16B
LoRA | 44.15 | 44.90 | 49.05 | 31.95 | 9.30 | 12.05 | 14.10 | 8.80
P-Tuning | 53.70 | 27.75 | 40.55 | 11.00 | 19.27 | 23.52 | 13.35 | 3.24
AdapterH | 45.75 | 45.80 | 46.25 | 41.75 | 8.59 | 8.17 | 12.05 | 8.18
AdapterP | 45.55 | 46.05 | 46.85 | 27.35 | 8.88 | 8.63 | 12.05 | 9.00
Parallel | 34.50 | 33.50 | 52.55 | 42.30 | 9.55 | 8.94 | 10.16 | 17.21
(IA)3 | 53.90 | 33.55 | 37.20 | 23.70 | 8.28 | 11.76 | 23.19 | 8.13
FFT | 50.80 | 44.20 | 48.30 | 43.65 | 8.34 | 12.68 | 8.04 | 12.62

##  4 Main Results: Task Performance

Figure 5: Accuracy results of Astraios models on Defect Detection.

Figure 6: F1 results of Astraios models on Clone Detection.

As cross-entropy loss only indicates how well Code LLMs can be aligned with
the training data, it greatly depends on the specific training content and may
not serve as a reliable proxy of performance on various tasks of source code.
Therefore, We seek to examine how well selective PEFT methods contribute to
task performance in this section. To benchmark the performance, we leverage
the representative code downstream tasks: (1) Defect Detection, (2) Code
Clone, (3) Code Synthesis, (4) Code Repair and (5) Code Explanation. For the
first two code comprehension tasks, there is no existing study stating that
the larger code LLMs result in a better understanding of code. We are the
first to study this aspect when varying the model sizes. Regarding the latter
three code generation tasks, previous power-law studies (Kaplan et al., 2020;
Hoffmann et al., 2022) have shown that increasing model sizes can also lead to
better task performance on generation tasks. We further validate this finding
on the PEFT settings.

#### Code Comprehension

Table 2 shows the results of the two code comprehension tasks when varying the
model sizes. Surprisingly, as shown in Figures 6 and 6, the results of both
tasks are not well aligned with the patterns we observe on code generation
tasks. All tuning methods consistently behave like the inverse scaling, which
has been discussed in McKenzie et al. (2023). We hypothesize that Code LLMs
have not seen enough task-specific training data and cannot generalize to
those unseen tasks (Yadlowsky et al., 2023). As Astraios models are pre-
trained on various source code from GitHub repositories for next token
prediction and fine-tuned on GitHub commits for code refinement, they may not
have a profound understanding of defects and cloned code.

Table 3: Pass@1 results of Astraios models on HumanEvalPack Python and Java
splits. The best performance is highlighted in bold. The second best
performance is underlined.

| Method | Code Synthesis | Code Repair | Code Explanation
---|---|---|---|---
| 1B | 3B | 7B | 16B | 1B | 3B | 7B | 16B | 1B | 3B | 7B | 16B
Python | LoRA | 17.26 | 25.37 | 32.01 | 38.08 | 3.29 | 11.16 | 21.74 | 27.50 | 20.49 | 22.53 | 25.34 | 30.52
P-Tuning | 15.79 | 24.33 | 29.39 | 35.58 | 1.86 | 13.69 | 20.34 | 18.72 | 9.48 | 11.92 | 14.60 | 15.43
AdapterH | 15.70 | 23.87 | 28.26 | 33.29 | 3.14 | 15.55 | 22.50 | 22.28 | 17.77 | 22.35 | 24.24 | 26.07
AdapterP | 17.04 | 24.76 | 30.67 | 34.97 | 3.69 | 12.87 | 19.54 | 26.46 | 16.07 | 24.05 | 22.87 | 30.67
Parallel | 15.98 | 26.65 | 28.81 | 35.88 | 4.91 | 8.11 | 16.13 | 26.43 | 19.70 | 23.14 | 23.93 | 31.10
(IA)3 | 16.13 | 25.34 | 30.52 | 36.80 | 2.01 | 14.05 | 17.07 | 23.60 | 9.51 | 11.86 | 14.30 | 16.19
FFT | 16.95 | 25.21 | 32.38 | 38.47 | 3.26 | 14.45 | 21.40 | 29.88 | 15.37 | 23.45 | 26.13 | 30.85
Java | LoRA | 2.84 | 16.52 | 24.27 | 40.33 | 3.72 | 5.06 | 13.60 | 30.35 | 7.07 | 14.33 | 14.70 | 16.86
P-Tuning | 10.67 | 14.73 | 20.73 | 37.19 | 0.00 | 7.53 | 11.74 | 22.25 | 6.07 | 9.79 | 17.32 | 13.02
AdapterH | 8.99 | 13.45 | 17.53 | 33.41 | 0.12 | 6.89 | 14.70 | 24.91 | 6.74 | 9.57 | 13.99 | 14.85
AdapterP | 10.46 | 16.77 | 21.28 | 33.68 | 3.66 | 6.52 | 15.40 | 32.07 | 6.65 | 11.62 | 14.15 | 16.28
Parallel | 9.60 | 15.91 | 21.59 | 38.56 | 0.49 | 5.09 | 8.87 | 29.39 | 7.62 | 12.16 | 14.51 | 17.93
(IA)3 | 10.34 | 16.46 | 21.95 | 39.91 | 2.87 | 4.54 | 13.02 | 25.30 | 6.13 | 13.99 | 17.04 | 15.85
FFT | 10.18 | 17.04 | 23.87 | 41.16 | 0.00 | 5.61 | 16.10 | 32.47 | 7.16 | 13.60 | 15.12 | 16.62

#### Code Generation

Table 3 demonstrates the performance on three different code generation tasks
on the Python and Java splits in HumanEvalPack. Over the six benchmarks, we
first observe that FFT results in consistent gains when the model parameters
increase. When examining the PEFT methods, We find they can also provide
reasonable performance scalability similar to FFT. Therefore, the lower test
loss may lead to better performance across various downstream generation tasks
for Code LLMs. However, we notice that the benefit of base model sizes may
also differ from tasks and languages. For instance, 1B and 3B models typically
underperform in code repair compared to code synthesis. When the model
parameters expand to 7B and 16B, their performance across these tasks becomes
more comparable.

#### Overall Performance

To compare the overall task performance of different tuning methods, we
compute the mean cumulative scores for each tuning method per model size. We
present the rankings in Figure 1. We show that FFT remains the best regarding
overall task performance, while LoRA and Parallel Adapter are comparable to
FFT. However, there is still a huge performance gap between most PEFT methods
and FFT, suggesting that they cannot guarantee optimal performance. Regarding
the tuning efficiency, we use updated parameters as the metric to summarize
two more findings. Firstly, (IA)3 is efficient enough to perform reasonably by
updating much fewer parameters than the other PEFT methods. Secondly, we
notice that AdapterP always performs better than AdapterH, even though
AdapterH updates more model parameters. The counter-intuitive observation
indicates that AdapterH may not be worth deploying in real-world practice.

##  5 Further Analysis

In this section, we further study two aspects of Code LLMs beyond task
performance. Specifically, we highlight the importance of model robustness and
generated code security, which indicate real-world practicality. We tend to
understand the trend of model behavior across tuning methods and model sizes.

###  5.1 Model Robustness

Table 4: RP@1 and RC@1 results of Astraios models on ReCode. The best
performance is highlighted in bold. The second best performance is underlined.

| Method | Format | Function | Syntax | Docstring
---|---|---|---|---|---
| 1B | 3B | 7B | 16B | 1B | 3B | 7B | 16B | 1B | 3B | 7B | 16B | 1B | 3B | 7B | 16B
Robust Pass | LoRA | 28.05 | 35.98 | 43.29 | 51.22 | 12.80 | 15.24 | 23.78 | 29.27 | 8.54 | 13.41 | 15.85 | 18.29 | 10.98 | 15.24 | 17.68 | 20.73
P-Tuning | 18.29 | 29.88 | 39.63 | 48.78 | 7.32 | 15.85 | 21.34 | 23.78 | 6.71 | 11.59 | 14.02 | 17.68 | 6.71 | 14.63 | 18.29 | 21.34
AdapterH | 10.98 | 34.15 | 40.24 | 46.95 | 4.88 | 14.02 | 17.07 | 23.78 | 7.32 | 11.59 | 12.20 | 15.85 | 6.10 | 12.80 | 14.63 | 17.68
AdapterP | 9.76 | 35.37 | 43.90 | 50.00 | 1.22 | 15.85 | 21.34 | 26.22 | 4.88 | 12.20 | 14.63 | 18.29 | 3.05 | 15.24 | 19.51 | 20.12
Parallel | 26.22 | 32.32 | 42.68 | 50.00 | 10.37 | 11.59 | 21.95 | 26.83 | 7.93 | 12.80 | 14.63 | 17.07 | 8.54 | 15.24 | 17.68 | 21.95
(IA)3 | 26.83 | 33.54 | 42.07 | 50.61 | 12.80 | 17.07 | 21.34 | 26.83 | 7.93 | 12.20 | 14.63 | 17.07 | 10.37 | 15.85 | 18.90 | 22.56
FFT | 20.12 | 35.37 | 45.73 | 53.05 | 5.49 | 15.85 | 21.34 | 30.49 | 7.32 | 14.63 | 15.85 | 19.51 | 6.10 | 14.02 | 18.90 | 22.56
Robust Change | LoRA | 10.98 | 14.63 | 15.24 | 15.85 | 4.27 | 6.10 | 4.27 | 6.10 | 8.54 | 7.93 | 12.20 | 17.07 | 6.10 | 6.10 | 10.37 | 14.63
P-Tuning | 6.10 | 9.76 | 12.80 | 17.68 | 4.88 | 4.27 | 5.49 | 7.32 | 5.49 | 8.54 | 12.80 | 13.41 | 5.49 | 5.49 | 8.54 | 9.76
AdapterH | 0.61 | 15.85 | 15.85 | 15.85 | 5.49 | 4.27 | 7.32 | 7.32 | 3.05 | 6.71 | 12.20 | 15.24 | 4.27 | 5.49 | 9.76 | 13.41
AdapterP | 3.66 | 14.63 | 17.68 | 15.85 | 4.88 | 4.88 | 4.88 | 7.93 | 1.22 | 8.54 | 11.59 | 15.85 | 3.05 | 5.49 | 6.71 | 14.02
Parallel | 12.20 | 11.59 | 15.85 | 15.24 | 3.66 | 9.15 | 4.88 | 7.93 | 6.10 | 7.93 | 12.20 | 17.68 | 5.49 | 5.49 | 9.15 | 12.80
(IA)3 | 10.98 | 12.80 | 14.02 | 14.63 | 3.05 | 3.66 | 6.71 | 9.15 | 7.93 | 8.54 | 13.41 | 18.90 | 5.49 | 4.88 | 9.15 | 13.41
FFT | 7.32 | 14.02 | 17.68 | 15.24 | 7.32 | 5.49 | 6.71 | 7.32 | 5.49 | 6.71 | 12.20 | 18.29 | 6.71 | 7.32 | 9.15 | 15.24

While the performance on downstream tasks is essential, we argue that the
evaluation of model robustness is also necessary to characterize different
tuning methods systematically. We therefore consider benchmarking the
robustness of code synthesis, one of the most representative downstream tasks
of source code.

Table 4 reports each tuning method’s worst-case RP@1 and RC@1 of each
perturbation category. Among the four types of perturbation, all models
perform the worst on syntax transformation, confirming the findings in Wang et
al. (2023a). Furthermore, RP@1 per tuning method increases when the model size
is scaled up, indicating the generation capability is consistently improved.
We noticed that FFT may not perform better than other PEFT methods on smaller
models, such as 1B and 3B. However, it results in the best RP@1 on larger
models like 16B. By comparing different model sizes, we observe that RC@1
consistently increases when the model gets bigger, indicating that larger
models will be less robust.

Figure 7: Mean RC@1 of Astraios on ReCode. Lower RC@1 indicates better
robustness. We indicate the percentage of total parameters updated for each
PEFT method.

To rank among the tuning methods through the lens of robustness, we compute
the mean RC@1 similar to Section 4 and illustrate in Figure 7. We observe that
FFT and LoRA do not show strong robustness. Instead, adapter-based tuning
seems more robust while having comparable performance to FFT, which is similar
to what Han et al. (2021) have found in NLP tasks.

###  5.2 Code Security

Table 5: Valid and Insecure rates of Astraios models on AATK benchmark. We
note that the insecure rate is calculated based on the valid programs. The
best performance is highlighted in bold. The second best performance is
underlined.

Method | Valid% ($\uparrow$) | Insecure% ($\downarrow$)
---|---|---
1B | 3B | 7B | 16B | 1B | 3B | 7B | 16B
LoRA | 85.9 | 89.1 | 75.9 | 87.1 | 23.1 | 26.2 | 20.9 | 35.0
P-Tuning | 70.1 | 68.6 | 86.8 | 82.0 | 32.8 | 25.9 | 28.1 | 34.5
AdapterH | 84.5 | 90.9 | 87.5 | 86.8 | 29.0 | 26.0 | 31.9 | 34.1
AdapterP | 83.9 | 92.1 | 82.8 | 86.3 | 31.7 | 25.2 | 26.6 | 37.8
Parallel | 88.9 | 94.1 | 70.0 | 86.0 | 30.2 | 19.3 | 22.3 | 32.6
(IA)3 | 78.0 | 62.1 | 77.4 | 86.6 | 34.8 | 25.2 | 23.1 | 30.4
FFT | 82.9 | 93.6 | 80.1 | 84.1 | 22.6 | 27.4 | 21.2 | 38.3

Previous studies (Dakhel et al., 2023; Asare et al., 2023). have shown that
Code LLMs can generate code with security vulnerabilities, which can be
exploited by malicious users. However, few studies have studied different
tuning methods from the output security perspective. In this experiment, we
intend to understand how tuning methods affect the capability to generate
secure code on AATK benchmark.

We follow the original setting in Pearce et al. (2022) and compute the valid
and insecure rates, which are illustrated in Table 5. When comparing the valid
rate of PEFT methods, it does not show better performance when the model size
increases, indicating that current models may not learn the program validity
intrinsically. However, we observe that the changes in the insecure rate show
that larger models are more likely to generate insecure code. This observation
suggests that the growth of learning capability can result in learning more
data, including insecure programs. The study on the insecure rate among tuning
methods further shows that FFT and LoRA are still better than the other tuning
methods regarding the security level. While the other methods have a similar
insecure rate, P-Tuning may have more chances to generate less secure
programs, which may not be suitable for deploying in security-sensitive
scenarios.

##  6 Discussion

In this section, we seek to conduct a preliminary analysis of the performance
of Code LLMs through the lens of updated parameters. Specifically, we ask two
questions: (1) What is the relationship between the updated parameters and
cross-entropy loss?; and (2) Can we utilize the performance of loss to predict
the task performance of Code LLMs?.

Figure 8: Relationships between cross-entropy loss and the number of updated
parameters.

#### Loss of small models can be projected to larger ones.

The relationship between the updated parameters of Astraios models and their
final loss is analyzed in Figure 8. Our analysis does not reveal a consistent
pattern across different model sizes when it comes to the correlation between
model loss and updated parameters. However, an interesting finding is the
consistency in relative loss performance across different model sizes when
comparing various tuning methods. This consistency suggests that the
improvements achieved by each tuning method are likely to be similar
regardless of the model’s size. Therefore, the loss observed in smaller
models, when tuned with different methods, can be a useful predictor for the
performance of the larger models.

Figure 9: Relationships between cross-entropy loss and overall task
performance.

#### Instruct-tuning loss is a strong predictor of downstream performance.

Assuming that the model has been instruction-tuned already but not yet done
for the evaluation, we seek to understand if we can utilize such loss to
predict its performance on downstream tasks. Despite our instruction data
being derived from general sources like GitHub commits and broad NLP domains,
which are not directly aligned with the downstream tasks discussed in Section
4, we find some strong correlations. Motivated by the aforementioned scenario,
we aggregate all the data points of mean task performance and their
corresponding final loss in Figure 9. We observe that the models with lower
loss generally have better overall performance on downstream tasks.
Specifically, the pattern is stronger on test loss than on train loss. We
explain by the fact that the models do not learn to fit the test split and can
present a more accurate determination of their actual performance. Our
observation suggests that general instruction data can work as a good proxy of
downstream tasks in Code LLMs, similar to the prior findings in NLP (Anil et
al., 2023; Wei et al., 2023).

##  7 Related Work

#### Code Large Language Models

Many base Code LLMs have been proposed recently (Chen et al., 2021; Nijkamp et
al., 2022; Fried et al., 2022; Allal et al., 2023; Zheng et al., 2023; Li et
al., 2023; Roziere et al., 2023) mostly targeting code completion. With the
help of these base Code LLMs, there have been extensive studies fine-tuning
task-specific Code LLMs to perform software engineering tasks like automatic
program repair (Xia and Zhang, 2023; Xia et al., 2023a), code translation (Pan
et al., 2023) and code summarization (Wang et al., 2023b, 2022a). Later, a
series of works has been proposed for instruction-tuning the base Code LLMs
(Luo et al., 2023; Shen et al., 2023; Muennighoff et al., 2023a; Bai et al.,
2023), aiming to enhance the generalization capabilities of these models on
diverse tasks. As fine-tuning Code LLMs with full parameters is costly, most
models have been tuned with LoRA (Hu et al., 2021), a parameter-efficient
tuning method. In this work, we seek to answer how good LoRA is and if there
are other comparable tuning methods.

#### Model Analysis Across Scales

Understanding why and how neural models behave is crucial for developing more
advanced ones. Existing studies have investigated predictable patterns in the
behavior of trained language models across scales (Kaplan et al., 2020;
Henighan et al., 2020; Hernandez et al., 2021; Hoffmann et al., 2022; Wei et
al., 2022; Muennighoff et al., 2023b; Xia et al., 2023b) and their learning
dynamics (McGrath et al., 2022; Tirumala et al., 2022; Biderman et al., 2023).
However, they either focus on pre-training or task-specific full-parameter
fine-tuning. There is no attempt to understand the mechanism of parameter-
efficient instruction tuning. In this paper, we work on this perspective and
analyze Code LLMs (Wan et al., 2022; Troshin and Chirkova, 2022; Zhuo et al.,
2023a).

##  8 Conclusion

This work studies the parameter-efficient instruction-tuning of Code LLMs. We
introduce a model suite consisting of 28 instruction-tuned OctoCoder across
scales and PEFT methods. We characterize the tuning methods on representative
downstream tasks, model robustness, and output security, highlighting the
importance of understanding these models via comprehensive evaluation. We also
discuss the relationships among updated parameters, cross-entropy loss, and
task performance. We hope these analyses will inspire further follow-up work
on understanding the mechanism of tuning methods and developing new
approaches.

## Acknowledgements

We thank Monash University and Hugging Face for providing compute instances.
We are extremely grateful to Cristian Rojas for help on the initial
exploration, Zhensu Sun for the discussion, Dmitry Abulkhanov for the paper
review, Brendan Dolan-Gavitt for providing the evaluation script of “Asleep At
The Keyboard” benchmark, the BigCode community for providing the base models
(Li et al., 2023) and instruction tuning data (Muennighoff et al., 2023a) from
GitHub commits, and Mangrulkar et al. (2022); Hu et al. (2023) for
implementing PEFT methods.

## References

  * Aghajanyan et al. (2021) Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021.  Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 7319–7328.
  * Aghajanyan et al. (2023) Armen Aghajanyan, Lili Yu, Alexis Conneau, Wei-Ning Hsu, Karen Hambardzumyan, Susan Zhang, Stephen Roller, Naman Goyal, Omer Levy, and Luke Zettlemoyer. 2023.  Scaling laws for generative mixed-modal language models.  _arXiv preprint arXiv:2301.03728_.
  * Allal et al. (2023) Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al. 2023.  SantaCoder: don’t reach for the stars!  _arXiv preprint arXiv:2301.03988_.
  * Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023.  Palm 2 technical report.  _arXiv preprint arXiv:2305.10403_.
  * Asare et al. (2023) Owura Asare, Meiyappan Nagappan, and N Asokan. 2023.  Is github’s copilot as bad as humans at introducing vulnerabilities in code?  _Empirical Software Engineering_ , 28(6):1–24.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Ben Allal et al. (2022) Loubna Ben Allal, Niklas Muennighoff, Logesh Kumar Umapathi, Ben Lipkin, and Leandro von Werra. 2022.  A framework for the evaluation of code generation models.  https://github.com/bigcode-project/bigcode-evaluation-harness.
  * Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023.  Pythia: A suite for analyzing large language models across training and scaling.  In _International Conference on Machine Learning_ , pages 2397–2430. PMLR.
  * Bielik and Vechev (2020) Pavol Bielik and Martin Vechev. 2020.  Adversarial robustness for code.  In _International Conference on Machine Learning_ , pages 896–907. PMLR.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chaudhary (2023) Sahil Chaudhary. 2023.  Code Alpaca: An Instruction-following LLaMA model for code generation.  https://github.com/sahil280114/codealpaca.
  * Chen et al. (2022) Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. 2022.  Revisiting Parameter-Efficient Tuning: Are We Really There Yet?  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 2612–2626.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Dakhel et al. (2023) Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Zhen Ming Jack Jiang. 2023.  Github copilot ai pair programmer: Asset or liability?  _Journal of Systems and Software_ , 203:111734.
  * Dettmers et al. (2022) Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022.  Gpt3. int8 (): 8-bit matrix multiplication for transformers at scale.  _Advances in Neural Information Processing Systems_ , 35:30318–30332.
  * Ding et al. (2022) Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2022.  Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models.  _arXiv preprint arXiv:2203.06904_.
  * Ding et al. (2023) Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023.  Parameter-efficient fine-tuning of large-scale pre-trained language models.  _Nature Machine Intelligence_ , 5(3):220–235.
  * Edalati et al. (2022) Ali Edalati, Marzieh Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. 2022.  Krona: Parameter efficient tuning with kronecker adapter.  _arXiv preprint arXiv:2212.10650_.
  * Fried et al. (2022) Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Scott Yih, Luke Zettlemoyer, and Mike Lewis. 2022.  InCoder: A Generative Model for Code Infilling and Synthesis.  In _The Eleventh International Conference on Learning Representations_.
  * Fu et al. (2023) Zihao Fu, Haoran Yang, Anthony Man-Cho So, Wai Lam, Lidong Bing, and Nigel Collier. 2023.  On the effectiveness of parameter-efficient fine-tuning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 12799–12807.
  * Han et al. (2021) Wenjuan Han, Bo Pang, and Ying Nian Wu. 2021.  Robust Transfer Learning with Pretrained Language Models through Adapters.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 854–861.
  * He et al. (2021) Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021.  Towards a Unified View of Parameter-Efficient Transfer Learning.  In _International Conference on Learning Representations_.
  * He et al. (2023) Xuehai He, Chunyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. 2023.  Parameter-efficient model adaptation for vision transformers.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 37, pages 817–825.
  * Henighan et al. (2020) Tom Henighan, Jared Kaplan, Mor Katz, Mark Chen, Christopher Hesse, Jacob Jackson, Heewoo Jun, Tom B Brown, Prafulla Dhariwal, Scott Gray, et al. 2020.  Scaling laws for autoregressive generative modeling.  _arXiv preprint arXiv:2010.14701_.
  * Henkel et al. (2022) Jordan Henkel, Goutham Ramakrishnan, Zi Wang, Aws Albarghouthi, Somesh Jha, and Thomas Reps. 2022.  Semantic robustness of models of source code.  In _2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER)_ , pages 526–537. IEEE.
  * Hernandez et al. (2021) Danny Hernandez, Jared Kaplan, Tom Henighan, and Sam McCandlish. 2021.  Scaling laws for transfer.  _arXiv preprint arXiv:2102.01293_.
  * Hoffmann et al. (2022) Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_.
  * Hou et al. (2023) Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2023.  Large language models for software engineering: A systematic literature review.  _arXiv preprint arXiv:2308.10620_.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP.  In _International Conference on Machine Learning_ , pages 2790–2799. PMLR.
  * Hu et al. (2021) Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2021.  LoRA: Low-Rank Adaptation of Large Language Models.  In _International Conference on Learning Representations_.
  * Hu et al. (2023) Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria. 2023.  LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models.  _arXiv preprint arXiv:2304.01933_.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_.
  * Karimi Mahabadi et al. (2021) Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021.  Compacter: Efficient low-rank hypercomplex adapter layers.  _Advances in Neural Information Processing Systems_ , 34:1022–1035.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The Power of Scale for Parameter-Efficient Prompt Tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059.
  * Li et al. (2023) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023.  StarCoder: may the source be with you!  _arXiv preprint arXiv:2305.06161_.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-Tuning: Optimizing Continuous Prompts for Generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4582–4597.
  * Liu et al. (2022) Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  _Advances in Neural Information Processing Systems_ , 35:1950–1965.
  * Liu et al. (2023) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2023.  GPT understands, too.  _AI Open_.
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021.  CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_.
  * Luo et al. (2023) Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. 2023.  WizardCoder: Empowering Code Large Language Models with Evol-Instruct.  _arXiv preprint arXiv:2306.08568_.
  * Mangrulkar et al. (2022) Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022.  PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods.  https://github.com/huggingface/peft.
  * McGrath et al. (2022) Thomas McGrath, Andrei Kapishnikov, Nenad Tomašev, Adam Pearce, Martin Wattenberg, Demis Hassabis, Been Kim, Ulrich Paquet, and Vladimir Kramnik. 2022.  Acquisition of chess knowledge in alphazero.  _Proceedings of the National Academy of Sciences_ , 119(47):e2206625119.
  * McKenzie et al. (2023) Ian R McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, et al. 2023.  Inverse Scaling: When Bigger Isn’t Better.  _arXiv preprint arXiv:2306.09479_.
  * Muennighoff et al. (2023a) Niklas Muennighoff, Qian Liu, Armel Zebaze, Qinkai Zheng, Binyuan Hui, Terry Yue Zhuo, Swayam Singh, Xiangru Tang, Leandro von Werra, and Shayne Longpre. 2023a.  Octopack: Instruction tuning code large language models.  _arXiv preprint arXiv:2308.07124_.
  * Muennighoff et al. (2023b) Niklas Muennighoff, Alexander M Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023b.  Scaling Data-Constrained Language Models.  _arXiv preprint arXiv:2305.16264_.
  * Muennighoff et al. (2023c) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023c.  Crosslingual Generalization through Multitask Finetuning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.
  * Nijkamp et al. (2022) Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022.  CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis.  In _The Eleventh International Conference on Learning Representations_.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Pan et al. (2023) Rangeet Pan, Ali Reza Ibrahimzada, Rahul Krishna, Divya Sankar, Lambert Pouguem Wassi, Michele Merler, Boris Sobolev, Raju Pavuluri, Saurabh Sinha, and Reyhaneh Jabbarvand. 2023.  Understanding the Effectiveness of Large Language Models in Code Translation.  _arXiv preprint arXiv:2308.03109_.
  * Pearce et al. (2022) Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022.  Asleep at the keyboard? assessing the security of github copilot’s code contributions.  In _2022 IEEE Symposium on Security and Privacy (SP)_ , pages 754–768. IEEE.
  * Perez et al. (2021) Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021.  True few-shot learning with language models.  _Advances in Neural Information Processing Systems_ , 34:11054–11070.
  * Pfeiffer et al. (2020) Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020.  MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7654–7673.
  * (54) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.
  * Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023.  Code llama: Open foundation models for code.  _arXiv preprint arXiv:2308.12950_.
  * Shen et al. (2023) Bo Shen, Jiaxin Zhang, Taihong Chen, Daoguang Zan, Bing Geng, An Fu, Muhan Zeng, Ailun Yu, Jichuan Ji, Jingyang Zhao, et al. 2023.  Pangu-coder2: Boosting large language models for code with ranking feedback.  _arXiv preprint arXiv:2307.14936_.
  * Sung et al. (2022) Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022.  Vl-adapter: Parameter-efficient transfer learning for vision-and-language tasks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5227–5237.
  * Svajlenko and Roy (2021) Jeffrey Svajlenko and Chanchal K Roy. 2021.  Bigclonebench.  _Code Clone Analysis: Research, Tools, and Practices_ , pages 93–105.
  * Tirumala et al. (2022) Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. 2022.  Memorization without overfitting: Analyzing the training dynamics of large language models.  _Advances in Neural Information Processing Systems_ , 35:38274–38290.
  * Troshin and Chirkova (2022) Sergey Troshin and Nadezhda Chirkova. 2022.  Probing Pretrained Models of Source Codes.  In _Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pages 371–383.
  * Wan et al. (2022) Yao Wan, Wei Zhao, Hongyu Zhang, Yulei Sui, Guandong Xu, and Hai Jin. 2022.  What do they capture? a structural analysis of pre-trained language models for source code.  In _Proceedings of the 44th International Conference on Software Engineering_ , pages 2377–2388.
  * Wang et al. (2022a) Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael R Lyu. 2022a.  No more fine-tuning? an experimental evaluation of prompt tuning in code intelligence.  In _Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ , pages 382–394.
  * Wang et al. (2023a) Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nallapati, Murali Krishna Ramanathan, Dan Roth, and Bing Xiang. 2023a.  ReCode: Robustness Evaluation of Code Generation Models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13818–13843, Toronto, Canada. Association for Computational Linguistics.
  * Wang et al. (2023b) Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. 2023b.  Codet5+: Open code large language models for code understanding and generation.  _arXiv preprint arXiv:2305.07922_.
  * Wang et al. (2021) Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021.  CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 8696–8708.
  * Wang et al. (2022b) Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. 2022b.  Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning.  In _The Eleventh International Conference on Learning Representations_.
  * Wei et al. (2022) Jason Wei, Najoung Kim, Yi Tay, and Quoc V Le. 2022.  Inverse scaling can become u-shaped.  _arXiv preprint arXiv:2211.02011_.
  * Wei et al. (2023) Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lü, Rui Hu, et al. 2023.  Skywork: A more open bilingual foundation model.  _arXiv preprint arXiv:2310.19341_.
  * Weidinger et al. (2021) Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021.  Ethical and social risks of harm from language models.  _arXiv preprint arXiv:2112.04359_.
  * Xia et al. (2023a) Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023a.  Automated program repair in the era of large pre-trained language models.  In _Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery_.
  * Xia and Zhang (2023) Chunqiu Steven Xia and Lingming Zhang. 2023.  Conversational automated program repair.  _arXiv preprint arXiv:2301.13246_.
  * Xia et al. (2023b) Mengzhou Xia, Mikel Artetxe, Chunting Zhou, Xi Victoria Lin, Ramakanth Pasunuru, Danqi Chen, Luke Zettlemoyer, and Veselin Stoyanov. 2023b.  Training Trajectories of Language Models Across Scales.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13711–13738, Toronto, Canada. Association for Computational Linguistics.
  * Xie et al. (2023) Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, Leo Z. Liu, Yiheng Xu, Hongjin Su, Dongchan Shin, Caiming Xiong, and Tao Yu. 2023.  OpenAgents: An Open Platform for Language Agents in the Wild.  _CoRR_ , abs/2310.10634.
  * Yadlowsky et al. (2023) Steve Yadlowsky, Lyric Doshi, and Nilesh Tripuraneni. 2023.  Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models.  _arXiv preprint arXiv:2311.00871_.
  * Zaken et al. (2022) Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022.  BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 1–9.
  * Zhang et al. (2022a) Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2022a.  Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning.  In _The Eleventh International Conference on Learning Representations_.
  * Zhang et al. (2022b) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022b.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_.
  * Zheng et al. (2023) Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Lei Shen, Zihan Wang, Andi Wang, Yang Li, et al. 2023.  Codegeex: A pre-trained model for code generation with multilingual benchmarking on humaneval-x.  In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , pages 5673–5684.
  * Zhou et al. (2022) Xin Zhou, Ruotian Ma, Yicheng Zou, Xuanting Chen, Tao Gui, Qi Zhang, Xuan-Jing Huang, Rui Xie, and Wei Wu. 2022.  Making parameter-efficient tuning more efficient: A unified framework for classification tasks.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 7053–7064.
  * Zhou et al. (2019) Yaqin Zhou, Shangqing Liu, Jingkai Siow, Xiaoning Du, and Yang Liu. 2019.  Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks.  _Advances in neural information processing systems_ , 32.
  * Zhuo et al. (2023a) Terry Yue Zhuo, Xiaoning Du, Zhenchang Xing, Jiamou Sun, Haowei Quan, Li Li, and Liming Zhu. 2023a.  Pop Quiz! Do Pre-trained Code Models Possess Knowledge of Correct API Names?  _arXiv preprint arXiv:2309.07804_.
  * Zhuo et al. (2023b) Terry Yue Zhuo, Yujin Huang, Chunyang Chen, and Zhenchang Xing. 2023b.  Red teaming chatgpt via jailbreaking: Bias, robustness, reliability and toxicity.  _arXiv preprint arXiv:2301.12867_ , pages 12–2.
  * Zhuo et al. (2023c) Terry Yue Zhuo, Zhou Yang, Zhensu Sun, Yufei Wang, Li Li, Xiaoning Du, Zhenchang Xing, and David Lo. 2023c.  Data Augmentation Approaches for Source Code Models: A Survey.  _arXiv preprint arXiv:2305.19915_.

##  Appendix A What is Astraios?

Astraios is a suite of 28 instruction-tuned StarCoder models, employing 7
different PEFT methods across 4 model sizes, with up to 16B parameters. Named
after the Greek Titan god of the stars, Astraios, this model collection
represents a vast array of “stars”, each model illuminating a path to
understanding the cost-performance trade-offs in Code LLMs. Through extensive
testing across various tasks and datasets, Astraios evaluates the efficacy of
fine-tuning methods with an emphasis on understanding their performance
implications at different model scales, robustness, and security aspects. The
suite serves as a celestial guide in the Code LLM universe, helping to chart
the most efficient and effective methods for model fine-tuning.

##  Appendix B Artifacts

Name | Public Link
---|---
Base Models
StarCoderBase 1B | https://huggingface.co/bigcode/starcoderbase-1b
StarCoderBase 3B | https://huggingface.co/bigcode/starcoderbase-3b
StarCoderBase 7B | https://huggingface.co/bigcode/starcoderbase-7b
StarCoderBase | https://huggingface.co/bigcode/starcoderbase
Instruction Tuning Data
CommitPackFT + OASST | https://huggingface.co/datasets/bigcode/guanaco-commits
Original PEFT Implementation
LoRA | https://github.com/huggingface/peft
P-Tuning | https://github.com/huggingface/peft
AdapterH | https://github.com/AGI-Edgerunners/LLM-Adapters
AdapterP | https://github.com/AGI-Edgerunners/LLM-Adapters
Parallel | https://github.com/AGI-Edgerunners/LLM-Adapters
(IA)3 | https://github.com/huggingface/peft
Prompt | https://github.com/huggingface/peft
AdaLoRA | https://github.com/huggingface/peft
Evaluation Framework
Code Generation LM Evaluation Harness | https://github.com/bigcode-project/bigcode-evaluation-harness
Astraios Models
Astraios LoRA 1B | https://huggingface.co/bigcode/astraios-1b-lora
Astraios P-Tuning 1B | https://huggingface.co/bigcode/astraios-1b-ptuning
Astraios AdapterH 1B | https://huggingface.co/bigcode/astraios-1b-adapterh
Astraios AdapterP 1B | https://huggingface.co/bigcode/astraios-1b-adapterp
Astraios Parallel 1B | https://huggingface.co/bigcode/astraios-1b-parallel
Astraios (IA)3 1B | https://huggingface.co/bigcode/astraios-1b-ia3
Astraios LoRA 3B | https://huggingface.co/bigcode/astraios-3b-lora
Astraios P-Tuning 3B | https://huggingface.co/bigcode/astraios-3b-ptuning
Astraios AdapterH 3B | https://huggingface.co/bigcode/astraios-3b-adapterh
Astraios AdapterP 3B | https://huggingface.co/bigcode/astraios-3b-adapterp
Astraios Parallel 3B | https://huggingface.co/bigcode/astraios-3b-parallel
Astraios (IA)3 3B | https://huggingface.co/bigcode/astraios-3b-ia3
Astraios LoRA 7B | https://huggingface.co/bigcode/astraios-7b-lora
Astraios P-Tuning 7B | https://huggingface.co/bigcode/astraios-7b-ptuning
Astraios AdapterH 7B | https://huggingface.co/bigcode/astraios-7b-adapterh
Astraios AdapterP 7B | https://huggingface.co/bigcode/astraios-7b-adapterp
Astraios Parallel 7B | https://huggingface.co/bigcode/astraios-7b-parallel
Astraios (IA)3 7B | https://huggingface.co/bigcode/astraios-7b-ia3
Astraios LoRA 16B | https://huggingface.co/bigcode/astraios-lora
Astraios P-Tuning 16B | https://huggingface.co/bigcode/astraios-ptuning
Astraios AdapterH 16B | https://huggingface.co/bigcode/astraios-adapterh
Astraios AdapterP 16B | https://huggingface.co/bigcode/astraios-adapterp
Astraios Parallel 16B | https://huggingface.co/bigcode/astraios-parallel
Astraios (IA)3 16B | https://huggingface.co/bigcode/astraios-ia3

Table 6:  Used and produced artifacts.

##  Appendix C Contributions

Terry Yue Zhuo trained 1B and 3B models, conducted most evaluations, analyzed
the results, wrote the paper and led the project. Armel Zebaze trained 7B and
15B models, evaluated 15B models on Code Synthesis and Code Repair, analyzed
the results and helped edit the paper. Nitchakarn Suppattarachai evaluated two
comprehension tasks. Niklas Muennighoff advised on the experiments and helped
with plotting. Niklas Muennighoff, Qian Liu, Harm de Vries and Leandro von
Werra provided suggestions and helped edit the paper.

##  Appendix D Instruction Tuning

All the instruction tuning experiments have been conducted on A100 80G GPUs.
For all PEFT strategies, we use the 8-bit quantized base models for training.
For FFT, we use the original base models without quantization.

#### LoRA

We use the attention dimension of 8, the alpha parameter of 16, dropout
probability of 0.05, and target modules of ”[c_proj, c_attn, q_attn]”. We keep
the other hyperparameters as default.

#### P-Tuning

We use the 30 virtual tokens and remain the other hyperparameters as default.

#### AdapterH

We use target modules of ”[c_fc, mlp.c_proj]”. We keep the other
hyperparameters as default.

#### AdapterP

We use target modules of ” [mlp.c_proj]”. We keep the other hyperparameters as
default.

#### Parallel

We use target modules of ”[c_fc, mlp.c_proj]”. We keep the other
hyperparameters as default.

#### (IA)3

We target modules of ”c_attn, mlp.c_proj]” and feedforward modules of ”
[mlp.c_proj]”.

#### Prompt (Lester et al., 2021)

We use the 30 virtual tokens and keep the other hyperparameters as default.

#### AdaLoRA (Zhang et al., 2022a)

We use the target average rank of the incremental matrix of 8, the initial
rank for each incremental matrix of 12, 200 steps of initial fine-tuning
warmup, 1000 step of final fine-tuning, the alpha parameter of 16, dropout
probability of 0.05, the time interval between two budget allocations of 10,
EMA for sensitivity smoothing of 0.85, EMA for uncertainty quantification of
0.85, and target modules of ”[c_proj, c_attn, q_attn]”. We keep the other
hyperparameters as default.

##  Appendix E Evaluation Setup

#### Devign

We generate the outputs with a max length of 512 tokens in the style of greedy
decoding. All other parameters are defaulted in Ben Allal et al. (2022). For
the one-shot example, we randomly sample from the train set.

#### BigCloneBench

We generate the outputs with a max length of 512 tokens in the style of greedy
decoding. All other parameters are defaulted in Ben Allal et al. (2022). For
the one-shot example, we randomly sample from the train set.

#### HumanEvalPack

We generate 20 outputs per example with a max length of 2048 tokens and a
temperature of 0.2. All other parameters are defaulted in Ben Allal et al.
(2022).

#### ReCode

We generate the outputs with a max length of 1024 tokens in the style of
greedy decoding. All other parameters are defaulted in Ben Allal et al.
(2022).

#### Asleep At The Keyboard

We generate 20 outputs per example with a max length of 1024 tokens and a
temperature of 0.2. All other parameters are defaulted in Ben Allal et al.
(2022).

##  Appendix F Failure of Scaling

1B model.

3B models.

Figure 10: Test loss of selected models across training time measured by
Global Step.

Figure 11: Final loss across model sizes.

During the initial experiment, we also train the models with Prompt Tuning
(Lester et al., 2021) and AdaLoRA (Zhang et al., 2022a). Although the loss
continues decreasing when the training time increases, we observe the
phenomenon of model size scales in contrast to Section 2.2. As shown in Figure
11, the final loss of these two tuning strategies consistently increases as
the model size increases, which is contrary to what we observe for other PEFT
methods. In the new version of LLM-Adapter (Hu et al., 2023), we notice that
the learning rate has been specifically mentioned. For Prompt Tuning, the
authors use $3\times 10^{-2}$ instead of $3\times 10^{-4}$, which is used in
their other selected PEFT strategies. Therefore, we hypothesize that some
tuning strategies may require a much higher learning rate to achieve optimal
performance. We further try a few learning rates on training 1B and 3B
StarCoderBase models and find that $3\times 10^{-2}$ works well for Prompt
Tuning. In addition, $3\times 10^{-2}$ and $1\times 10^{-3}$ also work much
better for AdaLoRA. With the new set of learning rates, we find that these
tuning strategies are aligned with our findings in Section 3. Different from
the conclusion of Kaplan et al. (2020) that the choice of learning rate
schedule is mostly irrelevant in language model pre-training, we suggest that
hyperparameters of learning rate schedule may matter a lot for scaling
parameter-efficient language model on fine-tuning.

##  Appendix G Visualization on HumanEvalPack

Python Code Synthesize

Java Code Synthesize

Python Code Repair

Java Code Repair

Python Code Explain

Java Code Explain

Figure 12: Pass@1 results of Astraios models on HumanEvalPack.

##  Appendix H Mitigating Inverse Scaling

Figure 13: Results on Defect Detection with 1-shot demonstration.

Figure 14: Results on Clone Detection with 1-shot demonstration.

We have attempted to see if the inverse-scaling-like patterns in code
comprehension tasks can be mitigated and more aligned with scaling laws. As
Wei et al. (2022) have shown that 1-shot demonstrations can make all inverse
scaling tasks U-shaped or flat, we try to see if 1-shot examples can help with
defection detection and clone detection. To select the 1-shot examples, we
randomly sample a fixed sample from the train set of each benchmark. We re-
evaluate all Astraios models on the two tasks and present the results in
Figures 14 and 14. For defect detection, all PEFT strategies become flatter
than the previous patterns, which is similar to what Wei et al. (2022)
observe. However, for clone detection, the patterns of some tuning strategies
like LoRA and FFT do not turn flat. Although the performances of LoRA and FFT
have been scaling up to 7B, they decrease at 15B. We hypothesize that our size
scaling is still not significant enough to represent an increasing pattern
after 15B for LoRA and FFT with 1-shot demonstrations.

##  Appendix I Further Discussion

We further measure the correlations among final loss in Section 3, overall
task performance in Section 4, and numbers of updated parameters via three
metrics, Kendall ($\tau$), Pearson ($r_{p}$), and Spearman ($r_{s}$)
coefficients. Kendall coefficient measures the ordinal association and is
robust against outliers, making it useful for non-normal data distributions.
Pearson’s coefficient assesses linear correlation, which is ideal for normal
data distributions with expected linear relationships. Spearman’s coefficient,
like Kendall coefficient, is a non-parametric measure that assesses rank
correlation, useful for identifying monotonic but non-linear relationships.

Table 7: Correlations between trainable parameters and final loss. $p$-values
are provided in gray.

Model Size | Train Loss | Test Loss
---|---|---
$\tau$ | $r_{p}$ | $r_{s}$ | $\tau$ | $r_{p}$ | $r_{s}$
1B | .4286 | .3113 | .6071 | .3333 | .3358 | .4643
3B | .5238 | .3433 | .7143 | .2381 | .3835 | .4286
7B | .5238 | .3555 | .7143 | .2381 | .4091 | .4286
16B | .5238 | .3524 | .7143 | .2381 | .3986 | .4286
Overall | .4339 (.00) | .3328 (.08) | .5616 (.00) | .3598 (.01) | .3308 (.09) | .4953 (.01)

We compute the correlations between updated parameters of Astraios models and
the final loss of corresponding models in Table 7. From the table, we first
observe that the updated parameters are more correlated to the final train
loss than the test loss. However, they all imply that there is a moderated
correlation, which can be used for cross-entropy loss in model training. We
also observe that when we aggregate all statistics across model sizes, the
correlations may slightly decrease.

Table 8: Correlations between final loss and overall task performance.
$p$-values are provided in gray.

Model Size | Train Loss | Test Loss
---|---|---
$\tau$ | $r_{p}$ | $r_{s}$ | $\tau$ | $r_{p}$ | $r_{s}$
1B | -.2381 | -.4319 | -.285 | .04 | -.4328 | -.0357
3B | .5238 | .7819 | .7143 | .8095 | .7859 | .9286
7B | .5238 | .7165 | .6786 | .8095 | .8230 | .9286
16B | .3333 | .8096 | .5000 | .8095 | .9211 | .8929
Overall | .7302 (.00) | .9027 (.00) | .9201 (.00) | .8466 (.00) | .9277 (.00) | .9579 (.00)

We compute the correlations between the model loss and their mean downstream
scores calculated in Section 4. We show the results in Table 8, where we
compute correlations for each model size and the final aggregated statistics.
Our observation on the size-level correlations indicates that the task
performance of 1B models is hard to align with the final loss, while bigger
models tend to be much more correlated to both train and test loss. We explain
the hypothesis that 1B models do not have enough capability to learn
instructions. When aggregating the data points, we find that correlations are
much stronger than the size-level prediction. The strong correlations imply
that model loss on the general instruction data can work as a good proxy of
downstream tasks in Code LLMs. When comparing the correlations on train loss
to the test loss, we observe the correlations are stronger on the latter one.
This can be explained by the fact that models tend to FFT on the training
data, where the loss on the train split can not generalize well on the unseen
tasks and data. Moreover, we also ask: What is the relationship between the
downstream task performance and the updated parameters? Therefore, We
investigate the correlation between tuned parameters and cumulative scores.
The correlations are 0.3016 (.02), 0.4128 (.03) and 0.4138 (.03) for Kendall,
Pearson and Spearman correlations, respectively. We draw the conclusion –
Possible.

##  Appendix J Limitations and Future Work

#### Experiment Noise

We observe that our empirical results are based solely on a single run of each
task, due to budget constraints that prevent us from tuning and evaluating the
same Code LLMs multiple times. Although the single evaluation approach limits
the breadth of our results and may introduce unexpected experiment noise, it
provides a preliminary insight into the performance and potential of PEFT in
different scenarios. Future investigations with multiple runs are necessary to
establish more robust conclusions and understand the variance and reliability
of our results.

#### Fair Evaluation

To compare different PEFT strategies fairly, we have used the same training
configurations described in Section 2.2. However, as we find that some PEFT
strategies like Prompt Tuning may be sensitive to the training hyperparameters
in Section 3, the consistent configurations can be unfair. On the other hand,
finding the optimal hyperparameters for each PEFT strategy is impractical and
can cost more than training with FFT. A more efficient approach is to reuse
the hyperparameters in previous work, which motivates us to adopt the default
settings in the PEFT library and LLM-Adapter framework. Meanwhile, we believe
there may be other practical approaches to benchmark PEFT strategies,
encouraging the community to investigate further.

#### PEFT Strategy

We notice that there are many more PEFT strategies (Karimi Mahabadi et al.,
2021; Zaken et al., 2022; Wang et al., 2022b; Edalati et al., 2022) have been
proposed recently. Due to the limited computation budget, we do not include
them all in our Astraios model suite. However, we have publicly made all our
source code, data, and models available. We encourage future development in
analyzing PEFT strategies on Code LLMs, which helps design more efficient PEFT
strategies.

#### Data Scaling

One limitation of our work is that we do not verify the validity of data
scaling on PEFT strategies. However, this factor has been well-studied in
various works (Kaplan et al., 2020; Hoffmann et al., 2022; Muennighoff et al.,
2023b) for model pre-training and fine-tuning. As we find that the performance
of PEFT on Code LLMs monotonically increases when scaling up the model size
and training time, these selected PEFT strategies are likely aligned with the
previous findings of data scaling. We recommend further verification on this
aspect.

#### Model Architecture

Another limitation of our study is that we do not vary the model architecture
of Code LLMs. It is possible that some findings may not generalize to other
encoder-decoder Code LLMs like CodeT5 (Wang et al., 2021) and CodeT5+ (Wang et
al., 2023b). However, as StarCoder is built upon the enhanced GPT-2 (Radford
et al., ) architecture, we believe that our observations can be transferred to
other GPT-based LLMs.

#### Scaling Parameter-Constrained Language Models

Although we demonstrate the possibility of predicting the final loss based on
the updated parameters and vice versa, we note that a scaling law generally
needs more than 100 models and their final loss. Ideally, the training
experiments should be consistent with different PEFT strategies, meaning that
training hundreds of models is needed. Furthermore, task performance is hard
to predict, as there is much more noise in the downstream tasks than the final
loss. We foresee that predicting such overall performance is very challenging.

##  Appendix K Prompts

The prompting format can significantly impact performance. In the spirit of
true few-shot learning (Perez et al., 2021), we do not optimize prompts and go
with the format provided by the respective model authors or the most intuitive
format if none is provided. For each task not designed for evaluating
instruction-tuned Code LLMs, we define an instruction. The instruction is to
ensure that models behave correctly and that their outputs can be parsed
effortlessly.

Question: {context}
Is there a defect in the Code, and respond to YES or NO.

Answer:

Figure 15: Prompt for Devign.

Question: Code 1: {context_1}
.
Code 2: {context_2}
Is there a clone relation between the Code1 and Code2, and respond to YES or
NO.

Answer:

Figure 16: Prompt for BigCloneBench.

Question: {instruction}
{context}

Answer:
{function_start}

Figure 17: Prompt for HumanEvalPack.

Question: Create a Python script for this problem.

Answer: {function_start}

Figure 18: Prompt for Code Completion on ReCode.

Question: Create a script for this problem.

Answer: {function_start}

Figure 19: Prompt for Asleep At The Keyboard.

##  Appendix L Timeline

#### Sep/2023

Experiment Design; Model Training; Model Evaluation.

#### Oct/2023

Model Training; Evaluation Discussion; Model Evaluation.

#### Nov/2023

Model Evaluation; Result Discussion; Paper Writing.

#### Dec/2023

Paper Finalization; Codebase Construction.