#  BioCoder: A Benchmark for Bioinformatics Code Generation with Contextual
Pragmatic Knowledge

Xiangru Tang† , Bill Qian†,, Rick Gao†,, Jiakang Chen†,∗, Xinyun Chen‡, Mark
Gerstein†
†Yale University
‡Google Deepmind
{xiangru.tang<EMAIL_ADDRESS><EMAIL_ADDRESS>
https://BioCoder-benchmark.github.io/ Contributed equally.

###### Abstract

Pre-trained language models like ChatGPT have significantly improved code
generation. As these models scale up, there is an increasing need for the
output to handle more intricate tasks. Moreover, in bioinformatics, generating
functional programs poses additional notable challenges due to the amount of
domain knowledge, the need for complicated data operations, and intricate
functional dependencies between the operations. Here, we present BioCoder, a
benchmark developed to evaluate existing pre-trained models in generating
bioinformatics code. In relation to function-code generation, BioCoder covers
potential package dependencies, class declarations, and global variables. It
incorporates 1026 functions and 1243 methods in Python and Java from GitHub
and 253 examples from the Rosalind Project. BioCoder incorporates a fuzz-
testing framework for evaluation, and we have applied it to evaluate many
models including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder,
StarCoder+, InstructCodeT5+, and ChatGPT. Our detailed analysis of these
models emphasizes the importance of domain knowledge, pragmatic code
generation, and contextual understanding. Our dataset, benchmark, Docker
images, and scripts required for testing are all available at
https://github.com/gersteinlab/biocoder.

##  1 Introduction

Large language models (LLMs) like OpenAI’s ChatGPT and Microsoft’s Copilot are
triggering profound societal impacts (1; 2; 3; 4; 5) especially in the field
of code generation (6; 7; 8; 9; 10), and thus the demand for high-quality and
challenging benchmarks is growing.

The landscape of existing datasets for language models is largely populated
with simple functions, often limited to a handful of lines. Combined with a
significant lack of closed-domain datasets across diverse fields, this
landscape highlights the need for a more robust benchmarking system. Although
domain-specific datasets, such as DS1000 for data science, have emerged, they
fall short of adequately addressing specific tasks in fields like
bioinformatics. Open-domain alternatives, including HumanEval (7), MBPP (11),
and APPS (12), offer entry-level programming tasks, but their utility is
limited as they lack the ability to test more niche, domain-specific code
blocks. This shortfall is largely due to a lack of appropriate fine-tuning and
context; therefore, a more comprehensive, encompassing approach to
benchmarking is clearly needed.

Figure 1: An overview: BioCoder is a code generation benchmark designed for
challenging, practical bioinformatics scenarios, offering an extensible
testing framework for evaluating the performance of LLMs.

To bridge these gaps, we introduce BioCoder (see Figure. 1), a benchmark for
code generation incorporating 2269 bioinformatics-specific coding problems.
BioCoder caters to the wide spectrum of programming skills and queries
prevalent in bioinformatics, encapsulating the frequent usage of popular
bioinformatics packages like BioPython and specific file formats such as FASTQ
(see Appendix. N for more details). Further expanding the scope of BioCoder,
we included an additional 253 questions from the Rosalind project. This
project specializes in generating Python functions addressing key
bioinformatics topics such as genetic sequencing and DNA/RNA analysis.
BioCoder assures the inclusion of all potential external packages and code
that could be utilized by the generated program. This consideration extends to
the recognition that real-world functions often necessitate managing multiple
external function calls and global variable usage; hence, we included all
potentially required class declarations in the input. Lastly, we performed
ablation studies to determine whether the models are strictly memorizing the
solutions rather than being proficient at generating code (see Appendix. M).

Benchmark | Num. | S. PL | Data Statistics | Scenario
---|---|---|---|---
T.N. | P.C. | P.L. | S.C. | S.L.
HumanEval (2021) | $164$ | Python | $7.8$ | $450.6$ | $13.7$ | $180.9$ | $6.8$ | Code Exercise
MBPP (2021b) | $974$ | Python | $3.1$ | $78.6$ | $1.0$ | $181.1$ | $6.7$ | Code Exercise
APPS (2021) | $5,000$ | Python | $21.0$ | $1743.4$ | $41.6$ | $473.8$ | $21.4$ | Competitions
CodeContests (2022) | $165$ | Multi. | $203.7$ | $1989.2$ | $66.4$ | $2239.3$ | $92.1$ | Competitions
DS-1000 (2022) | $1,000$ | Python | $1.6$ | $879.1$ | $31.6$ | $137.4$ | $5.0$ | Data Science
DSP (2023) | $1,119$ | Python | $2.1$ | $756.9$ | $17.8$ | $226.3$ | $7.6$ | Data Science
MBXP (2023) | $974^{*}$ | Multi. | $3.1$ | $419.9$ | $14.8$ | $-$ | $-$ | Multilingual
MBXP-HumanEval (2023) | $164^{*}$ | Multi. | $7.8$ | $825.6$ | $30.0$ | $-$ | $-$ | Multilingual
HumanEval-X (2023a) | $164^{*}$ | Multi. | $7.8$ | $468.4$ | $15.5$ | $264.6$ | $12.1$ | Multilingual
MultiPL-HumanEval (2023) | $164^{*}$ | Multi. | $7.8$ | $453.9$ | $13.0$ | $-$ | $-$ | Multilingual
MultiPL-MBPP (2023) | $974^{*}$ | Multi. | $3.1$ | $181.2$ | $5.4$ | $-$ | $-$ | Multilingual
PandasEval (2022) | $101$ | Python | $6.5$ | $244.5$ | $7.2$ | $46.2$ | $1.3$ | Public Library
NumpyEval (2022) | $101$ | Python | $3.5$ | $222.9$ | $7.0$ | $29.9$ | $1.1$ | Public Library
TorchDataEval (2022) | $50$ | Python | $1.1$ | $329.0$ | $8.6$ | $50.7$ | $1.3$ | Private Library
MTPB (2023a) | $115$ | Python | $-$ | $72.7$ | $1.0$ | $-$ | $-$ | Multi-Turn
ODEX (2022a) | $945$ | Python | $1.8$ | $26.6$ | $2.0$ | $50.4$ | $1.9$ | Open-Domain
BIG-Bench (2022) | $32$ | Python | $4.7$ | $341.8$ | $3.0$ | $-$ | $-$ | Code Exercise
Ours (public set) | 460 | Multi. | 1000 | 10465.6 | 243.5 | 706.8 | 26.2 | Bioinformatics
Ours (hidden set) | 2,269 | Multi. | 1000 | 12296.7 | 298.8 | 919.5 | 26.2 | Bioinformatics
Ours (similar set) | 460 | Multi. | 1000 | 9885.6 | 240.8 | 767.5 | 26.8 | Bioinformatics

Table 1: Summary of benchmarks for NL2Code. Num. denotes the number of
instances in the benchmark. PL denotes the Programming Language of the
project. T.N. denotes the average Number of Test cases. P.C. and P.L. (S.C.
and S.L.) stand for the average number of Characters and Lines in Problem
description (code Solution). ∗ denotes the number of instances per programming
language.

The main contributions of this paper are summarized as follows:

  * •

We present a new high-quality dataset for code generation, extracted and
curated from 1,720 bioinformatics repositories referenced in peer-reviewed
bioinformatics articles, aiding in the practical and realistic evaluation of
code generation tasks.

  * •

We meticulously processed the data, rephrasing more detailed text
descriptions, as well as associated comments and specifications, including
considerations needed in coding.

  * •

We provide an extendable parsing tool that can extract all pertinent
information associated with the target function in expansive projects. This
includes import dependencies, potential usage of global variables, possible
invocation of other functions along with their signatures, as well as class
definitions replete with their attributes and methods. By integrating this
contextual and pragmatic knowledge into the input, we can more thoroughly test
the abilities of LLMs to use tools and comprehend the entirety of a project.

  * •

We offer a Python transformer-based library for code LLMs, furnishing a
seamless interface for both training and inferencing in code generation tasks.

  * •

We provide a fuzzer testing tool capable of scaling to handle substantial
datasets. Our benchmark results, derived from 1000 iterations, are
particularly reliable, indicating the Pass@K rate.

  * •

We actively encourage new contributors, by providing comprehensive
instructions and software tools to work with BioCoder

##  2 Related Work

BioCoder is a code generation benchmark designed for challenging, practical
bioinformatics scenarios, offering an extensible testing framework for
evaluating the performance of LLMs. We provide a brief overview the related
work in both code generation models and benchmarks.

###  2.1 Code Generation with LLMs

In recent years, Code LLMs, trained on vast unlabeled code data, have shown
promising results in generating code, demonstrating impressive zero-shot
performance on natural language to code benchmarks (24; 25; 26; 27). One
successful approach to enhancing model performance has been to scale up both
the model parameters and training data (28; 6). Several top-performing large-
scale LLMs have been developed, such as Google’s PaLM (8), LaMDA (29),
DeepMind’s Chinchilla (30), and Gopher (31). These models have proven their
code generation prowess, exemplified by technical reports from OpenAI (6; 7;
10). The AI community has also seen the release of several open-source LLMs,
such as Tsinghua University’s bilingual GLM-130B (32), coding-application
CodeGeeX-13B (33), and StarCoder (24). Salesforce released CodeGen-Mono-16B
(21), while Meta developed LLaMA (34) and the InCoder model (35). Smaller-
scale LLMs, such as PolyCoder (36) and SantaCoder (27), also contribute to the
field.

Figure 2: A diagram of the BioCoder construction process involving custom
GitHub repository cleaning, parsing, and function selection, as well as
context and test case creation and a massively dockerized testing framework.

###  2.2 Code Generation Datasets and Benchmarks

Early work on code generation benchmarks used lexical exact match, data flow,
and abstract syntax tree (AST) methods. However, these measures proved to be
unreliable due to their sensitivity to inconsequential differences in the
generated code. In response, execution-based evaluation approaches have become
more prevalent (7; 17; 14; 22; 15). These approaches execute tests on the
generated code to verify its functional correctness, ensuring unbiased
evaluations irrespective of implementation method or style variations.

As a result, the field of code generation has seen a burgeoning number of
execution-based benchmarks (Table. 1), each presenting unique properties in
terms of size, language coverage, complexity, and practical applicability. For
instance, HumanEval (7) and MBPP (11) are benchmarks frequently employed for
code generation tasks and consist of only 164 and 974 simple Python functions
respectively, representing a small sample size. These benchmarks also overlook
multi-language coding scenarios—a gap partially bridged by benchmarks like
HumanEval-X (18) and MCoNaLa (37), covering nine and three languages
respectively.

However, all the benchmarks discussed above share the same shortcoming of only
benchmarking generic functions, rather than domain-specific ones. DS-1000 (15)
represents a more domain-specific dataset, featuring 1,000 data science
workflows extracted from Python functions. A study reported (15) that a
model’s performance on benchmarks such as HumanEval and MBPP does not
necessarily correlate with its performance on DS-1000. This discrepancy
underscores the need for benchmarks that more accurately emulate real-world,
domain-specific code generation.

Furthermore, the performance of current LLMs is significantly impacted by the
provided context (38). While DS-1000 includes eight packages, it fails to
fully reflect a typical coding environment. This gap is partially bridged
through benchmarks such as CoderEval (39), which incorporate some dependencies
and function calls; however, these benchmarks are rudimentary in nature, and
once again consist primarily of domain-agnostic functions.

Our work shares common ground with CoderEval. Both our approach and CoderEval
can evaluate models beyond the simple generation of standalone functions.
Given the necessity to handle context-dependent code, both methodologies
employ Docker-based testing. However, our approach contrasts with that of
CoderEval by placing a specific emphasis on bioinformatics. We ensure each
function demands a certain level of domain expertise in bioinformatics.
Moreover, our dataset surpasses the scale of CoderEval, which only consists of
230 functions from 43 Python projects and 230 methods from 10 Java projects.
In contrast, we source 2,522 functions from over two thousand repositories,
offering a broader and more challenging context for code generation tasks. We
further compare our benchmark to CoderEval in Appendix G.

| Public | Hidden | Similar
---|---|---|---
| Py | Java | Overall | Py | Java | Overall | Py | Java | Overall
Avg. Comment Lines | 4.96 | 2.66 | 4.40 | 8.77 | 4.90 | 6.65 | 5.75 | 3.14 | 5.12
Avg. Tokens of G.T. | 189.25 | 106.54 | 169.28 | 353.67 | 107.88 | 219.02 | 216.62 | 100.92 | 188.68
Avg. Lines of G.T. | 24.30 | 11.10 | 21.11 | 43.28 | 12.19 | 26.25 | 26.50 | 10.32 | 22.59
Avg. Parameters of G.T. | 2.39 | 1.70 | 2.23 | 2.92 | 1.25 | 2.00 | 2.48 | 1.10 | 2.15
Avg. Classes/Function Decl. | 20.25 | 2.52 | 15.97 | 19.45 | 32.96 | 26.85 | 20.20 | 1.16 | 15.60
Avg. Global Variables | 1.90 | - | - | 2.26 | - | - | 1.87 | - | -
Avg. Imports | 11.91 | 1.52 | 9.40 | 10.37 | 5.00 | 7.43 | 11.63 | 1.16 | 9.10

Table 2: Summary statistics for the BioCoder dataset. Statistics are
represented as the averages per ground truth function. The overalls are
calculated utilizing weighted averages of the number of functions in both the
Python and Java subsets. G.T. stands for the ground truth function. The
“similar data” are a subset of the hidden data, curated to mimic the overall
distribution of the public data. This set includes an additional 157 Python
problems and 50 Java problems, maintaining the same 253 Rosalind problem
tally, thus reflecting the composition of the public data.

##  3 The BioCoder Benchmark

In this section, we introduce four parts of BioCoder: the cleaning and
inspection of the dataset, benchmark collection, metrics used, and testing
construction.

###  3.1 Dataset Filtering

Our dataset begins with an initial web scrape of 1,743 bioinformatics-adjacent
GitHub repositories (see Figure. 2). These repositories were curated from
peer-reviewed bioinformatics articles (40), resulting in a comprehensive list
of 1720 bioinformatics repositories on GitHub with open-source licenses, in
addition to 23 notable repositories. The collection includes code in languages
such as C, C++, PHP, Python, R, Ruby, SQL, Perl, Java, Matlab, and C#,
although for now, we only explore Python and Java, with plans to scale up to
other languages in the future.

The 1,743 repositories were then filtered based on popularity and community
ratings, as well as a manual round of review, resulting in 28 high-quality,
highly domain-specific repositories that are commonly used in the field of
bioinformatics. After determining the set of 28 high-quality, highly domain-
specific repositories, we then wrote separate custom Python and Java parsers
to automatically parse all the selected GitHub repositories. These parsers
generated an AST of each code file in each repository and then scraped all the
relevant data, including function content, function signature, important
imports, and cross-file dependencies for each function in each code file.
After parsing all repositories, we were left with a large set of over 20,000
Python functions and over 50,000 Java functions.

Given the large baseline of functions, we initiated two rounds of automatic
filtering. In the first round, every function and its associated comments
underwent keyword filtering. The keywords were scraped from bioinformatics
Wikipedia articles, with each function requiring at least 10 keyword matches
to be considered relevant to bioinformatics. This round was critical to avoid
any unnecessary OpenAI API calls, which are expensive. The remaining functions
advanced to the second round, where we inputted each function into GPT-3.5,
which assessed bioinformatics relevance. With the workload significantly
reduced, we manually sorted through the remaining functions, leaving us with a
final count of 1,026 Python functions and 1,243 Java functions (see Table. 2).
Note that in Java, there are no global variables, because all variables are
encapsulated in classes.

###  3.2 Benchmark Construction

#####  BioCoder-Py and BioCoder-Java

BIOCODER-PY and BIOCODER-JAVA. For each of the functions that passed all
rounds of filtering in Section. 3.1, we manually wrote custom code context,
inclusive of necessary imports, cross-file dependencies, and pertinent fuzzer
test cases (explained in more detail in Section. 3.4). We then crafted custom
prompts based on the parsed function data and function summaries, ensuring the
inclusion of any necessary imports and cross-file dependencies (see Figure 4).
More prompt statistics can be found in Table. 8. Finally, we presented the
model with a prompt to generate the function, offering the function signature
as a starting point. Examples of the different prompt types can be found in
Appendix B. Prompts were generated partly using GPT-3.5, as GPT-3.5 was used
to generate function summaries for all the functions in the public dataset.
These function summaries were used as part of the prompt in order to
efficiently describe the functions. More details about this method can be
found in the Appendix E.

Figure 3: Sample prompts for LLM code generation. Our prompts follow the same
general outline. First, imports are declared at the top of the prompt, then
global variables (if there are any), then function declarations, then class
dependencies, and finally, our actual instruction regarding the function to be
generated. We end the prompt with the function signature for the LLM to
generate to streamline the testing process.

##### BioCoder-Rosalind.

To compile the Rosalind portion of the benchmark, we began by scraping the
problem descriptions from the Rosalind website, identifying problems with
available solutions, and gathering all possible solutions. Subsequently, we
developed a custom scraper to assemble ten test cases for each Rosalind
problem. Using these test cases, we crafted a script to automatically assess
whether the available solutions successfully ran against the collected test
cases.

Solutions that were successfully executed against all test cases formed the
‘golden code’ section of the Rosalind benchmark. Each Rosalind benchmark
context is custom-made, incorporating the scraped test cases and injecting
them into the generated code. The prompts for the Rosalind problems are
constructed using the scraped problem descriptions, supplemented with a brief
section outlining the context into which the generated code would be
integrated. This rigorous filtering process resulted in 253 functions meeting
all our criteria. Selected examples for the Rosalind dataset are shown in
Appendix C. Statistics of token counts, comment lines per function, and
parameters per function can be found in Appendix A.

###  3.3 Metric

We use the Pass@K metric to measure functional accuracy in this benchmark (7).
The metric Pass@K evaluates the efficiency of a model’s code generation
ability. Specifically, for a given model, this metric quantifies the
probability that the model can solve a particular problem. A problem is deemed
“solve” if, among the k-function samples produced by the model, at least one
sample passes all the provided test cases. The mathematical estimation of
Pass@K for a particular problem is articulated as follows:
$\text{Pass@K}:=\underset{\text{Problems}}{\mathbb{E}}\left[1-\frac{\binom{n-c}{k}}{\binom{n}{k}}\right]$.
where $n$ is the number of samples generated by the model, and $c$ is the
number of samples that pass all test cases.

###  3.4 Testing Framework

Our testing framework is a comprehensive process starting with a manual review
of selected functions, leading to the creation of a context file and a golden
code file for each problem (see Figure 4), as discussed in 3.2.

For Python and Java functions, in the context file, we employ a custom syntax
to indicate the insertion points for custom randomly generated test cases.
Through this syntax, we cater to four types of random generation: integers,
floats, strings, and Boolean values. During the runtime, each of these
insertion points are replaced with language-specific code to insert a
dynamically generated test case. The tester can be run for any number of
iterations, depending on the number of fuzzer tests desired.

For Rosalind functions, the process is simpler and more efficient as the
functions are less complex. The golden code’s output is generated and cached
ahead of time. During testing, the tester executes the generated code within
the corresponding context, and the output of this execution is compared with
the cached golden code output.

For every fuzzer test case and Rosalind test case, we ran the golden output
against itself, to ensure that the golden output passes each test with one
hundred percent reliability. Furthermore, to ensure system security and test
reliability, we ran our tests in Docker environments. We constructed a system
using Amazon Web Services, coordinating tasks across multiple compute nodes to
accelerate the process without compromising the validity of the results. After
creating a generalized Docker image, equipped with all necessary Python
requirements, we summarized our testing framework in Appendix K.

Figure 4: Test case for UntangleWorms example. The context file includes
various import dependencies and a class definition with a method placeholder
for the solution. The UntangleWorms class comes from a GitHub repository file
(https://github.com/CellProfiler/CellProfiler/blob/master/cellprofiler/modules/untangleworms.py)
that was scraped in our study. UntangleWorms is an image analysis tool that
was initially part of the paper “An image analysis toolbox for high-throughput
C. elegans assays.”

##  4 Models and Results

To test BioCoder, we opted to benchmark StarCoder-15B (24), StarCoder+-15B
(24), InCoder (25), SantaCoder (27), CodeGen (6B-mono and 16B-mono) (21),
CodeGen2-7B (41), InstructCodeT5+ (42), and GPT3.5-Turbo (10) through Azure
OpenAI Service. Full details of the model context lengths and model sizes can
be found in Table. 4 and Table. 4, respectively. Note that we have permission
to use all these models under open-source licenses. The selection of these
models was informed by two primary factors: their demonstrated proficiency in
open-domain code generation and our limited project budget and timeline. Our
prompts were fed directly into the LLMs, and we preserved the output for
subsequent analyses. For all models, we used the following hyperparameter
settings:

  * •

top_k=50

  * •

top_p=0.95

  * •

temperature=0.7

  * •

early_stopping=True

  * •

num_return_sequences=1

  * •

do_sample=True

We utilized similar parameters to make model testing consistent across all the
tested models. This approach allowed for a more unbiased comparison of how
each model performs on our benchmark.

Model | Context limit
---|---
InCoder (25) | 2048
SantaCoder (27) | 2048
StarCoder (43) | 8192
StarCoderPlus (43) | 8192
InstructCodeT5+ (42) | 2048
CodeGen (21) | 2048
CodeGen2 (41) | 2048
GPT-3.5-Turbo | 8192
Table 3: Model Context Lengths

Model | Parameters
---|---
InCoder (25) | 6B
SantaCoder (27) | 1.1B
StarCoder (43) | 15.5B
StarCoderPlus (43) | 15.5B
InstructCodeT5+ (42) | 16B
CodeGen-6B (21) | 6B
CodeGen-16B (21) | 16B
CodeGen2 (41) | 7B*
GPT-3.5-Turbo | Unknown
Table 4: Model Sizes

Aiming to accurately represent the performance of the LLM outputs, we
implemented basic correction mechanisms to rectify minor syntax and style
errors that did not impact functionality. For instance, all StarCoder outputs
were appended with a post-script. Consequently, each LLM output was passed
through these correction mechanisms before being sent to the testing framework
for evaluation (see Table. 5 and 6). Furthermore, to empirically evaluate the
hypothesis regarding the efficacy of smaller, specialized LLMs in closed-
domain code generation, as opposed to large open-domain models like ChatGPT,
we also fine-tuned StarCoder and documented the resulting performance. We ran
inference on HPC clusters, using 8x A100 GPU hardware.

Model | Prompt | Java | Python |
---|---|---|---|---
Pass@1 | Pass@5 | Pass@10 | Pass@20 | Pass@1 | Pass@5 | Pass@10 | Pass@20 |
InCoder-6B | Summary at Top | 0 | 0 | 0 | 0 | 0.828 | 2.016 | 3.006 | 4.459 |
Uncommented | 0 | 0 | 0 | 0 | 0.032 | 0.159 | 0.318 | 0.637 |
Summary Only | 0 | 0 | 0 | 0 | 1.688 | 5.320 | 8.332 | 12.006 |
Summary at Bottom | \ | \ | \ | \ | 0.610 | 2.587 | 4.303 | 6.274 |
Necessary Only | 0 | 0 | 0 | 0 | 0.032 | 0.159 | 0.318 | 0.637 |
SantaCoder-1.1B | Summary at Top | 0 | 0 | 0 | 0 | 0.637 | 1.338 | 1.844 | 2.548 |
Uncommented | 0 | 0 | 0 | 0 | 0.287 | 0.764 | 0.955 | 1.274 |
Summary Only | 0 | 0 | 0 | 0 | 2.965 | 9.848 | 14.227 | 18.181 |
Summary at Bottom | \ | \ | \ | \ | 0.510 | 1.949 | 3.013 | 4.459 |
Necessary Only | 0 | 0 | 0 | 0 | 0.032 | 0.159 | 0.318 | 0.637 |
StarCoder-15.5B | Summary at Top | 0 | 0 | 0 | 0 | 3.694 | 13.197 | 19.359 | 24.554 |
Uncommented | 0 | 0 | 0 | 0 | 0.318 | 1.062 | 1.591 | 2.548 |
Summary Only | 0 | 0 | 0 | 0 | 4.682 | 15.225 | 21.200 | 27.166 |
Summary at Bottom | \ | \ | \ | \ | 6.465 | 13.824 | 16.746 | 19.076 |
Necessary Only | 0 | 0 | 0 | 0 | 0.127 | 0.603 | 1.123 | 1.911 |
StarCoder+ | Summary at Top | 0 | 0 | 0 | 0 | 2.675 | 9.133 | 14.019 | 19.650 |
Uncommented | 0 | 0 | 0 | 0 | 0.510 | 0.955 | 1.274 | 1.911 |
Summary Only | 1.300 | 5.031 | 8.042 | 12.000 | 2.548 | 8.279 | 12.864 | 18.057 |
Summary at Bottom | \ | \ | \ | \ | 4.172 | 11.772 | 14.933 | 17.197 |
Necessary Only | 0 | 0 | 0 | 0 | 0.127 | 0.457 | 0.609 | 0.637 |
InstructCodeT5+ | All prompt types | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
CodeGen-6B-mono | Summary at Top | 0 | 0 | 0 | 0 | 0.637 | 0.637 | 0.637 | 0.637 |
Uncommented | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Summary Only | 0 | 0 | 0 | 0 | 0.637 | 0.637 | 0.637 | 0.637 |
Summary at Bottom | \ | \ | \ | \ | 2.070 | 4.535 | 5.896 | 7.006 |
Necessary Only | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
CodeGen-16B-mono | Summary at Top | 0 | 0 | 0 | 0 | 0.637 | 0.637 | 0.637 | 0.637 |
Uncommented | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
Summary Only | 0 | 0 | 0 | 0 | 0.637 | 0.637 | 0.637 | 0.637 |
Summary at Bottom | \ | \ | \ | \ | 2.166 | 5.137 | 6.022 | 6.369 |
Necessary Only | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
CodeGen2-7B | Summary at Top | 0 | 0 | 0 | 0 | 0.637 | 0.637 | 0.637 | 0.637 |
Uncommented | 0 | 0 | 0 | 0 | 0.510 | 0.637 | 0.637 | 0.637 |
Summary Only | 0 | 0 | 0 | 0 | 0.860 | 2.494 | 3.962 | 6.242 |
Summary at Bottom | \ | \ | \ | \ | 0.510 | 1.019 | 1.207 | 1.274 |
Necessary Only | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
GPT-3.5-Turbo | Summary at Top | 4.100 | 7.235 | 8.989 | 11.600 | 22.771 | 33.461 | 36.551 | 39.490 |
Uncommented | 6.300 | 11.563 | 14.436 | 18.000 | 11.019 | 19.075 | 21.680 | 24.204 |
Summary Only | 17.400 | 33.199 | 37.878 | 42.000 | 24.682 | 33.997 | 37.132 | 40.127 |
Summary at Bottom | \ | \ | \ | \ | 13.439 | 20.040 | 22.460 | 25.478 |
Necessary Only | 43.500 | 52.582 | 53.995 | 55.400 | 28.758 | 39.529 | 44.029 | 47.771 |
Table 5: Zero-shot performance with five prompt versions of BioCoder. For all settings, we performed trials twice for Pass@K. Results are in %. Model | Prompt | Rosalind |
---|---|---|---
Pass@1 | Pass@5 | Pass@10 | Pass@20 |
InCoder | Description | 0.020% | 0.099% | 0.198% | 0.395% |
SantaCoder | Description | 0.158% | 0.658% | 1.075% | 1.581% |
StarCoder | Description | 0.534% | 2.042% | 3.228% | 4.743% |
StarCoderPlus | Description | 0.356% | 1.313% | 1.978% | 2.767% |
StarCoder (fine-tuned) | Description | 1.623 | 3.109 | 5.328 | 7.036 |
InstructCodeT5+ | Description | 0.059% | 0.296% | 0.593% | 1.186% |
CodeGen | Description | 0.692% | 2.088% | 3.055% | 3.953% |
CodeGen2 | Description | 0.059% | 0.296% | 0.593% | 1.186% |
ChatGPT | Description | 23.671% | 31.953% | 36.702% | 40.725% |
Table 6: Performance of Rosalind. In this table, we have omitted the
percentage symbol (%), although these figures represent the Pass@K in the form
of percentages. For all settings, n=20.

The results in Table. 5 and Table. 6 align with our initial hypothesis, which
proposed that larger models would likely outperform their smaller
counterparts. However, the significant performance gap between ChatGPT and all
other code-generation models was surprising. This stark contrast underscores
the crucial role of both the dataset size and parameter size of the base
models in accomplishing closed-domain code generation prompts. Interestingly,
despite the rudimentary nature of our fine-tuning on StarCoder, the results
still highlighted a significant improvement compared with the non-fine-tuned
model. This stark contrast in performance bolsters our original assertion:
achieving success in closed-domain tasks can be realized either through large
open-domain LLMs, such as ChatGPT, or via fine-tuning smaller models. These
smaller models could potentially achieve comparable performance but with
significantly reduced computational and memory requirements. Furthermore, in
Table. 5 demonstrates that the performance of models improves with the
inclusion of dependencies in prompts, indicating that including dependencies
is an important part of promoting.

##  5 Analysis and Discussion

Looking more closely at the results in Table. 5, it is clear that the larger
models with more parameters generally perform better than the smaller models.
It is well known that the GPT-3.5 model includes hundreds of billions of
parameters and thus dwarfs the other models in this study in both size and
performance. For instance, GPT-3.5 achieved an impressive 47.771% accuracy on
the Necessary Only prompt style, while the best non-GPT result was StarCoder
achieving roughly 27.166% accuracy on the Summary Only prompt style.

Looking at the other models, it is interesting to note that while
InstructCodeT5+, CodeGen, and CodeGen2 are all larger than InCoder and
SantaCoder, they perform far worse. This is likely due to InstructCodeT5+,
CodeGen, and CodeGen2 being trained for single-line completions rather than
function completion. Furthermore, InstructCodeT5+, CodeGen, and CodeGen2 have
relatively small context limits, which likely hurts their performance. As for
the remaining model, SantaCoder notably performs impressively well for being
only a roughly 1B parameter model, which is an indication of aggressive fine-
tuning on Python code.

We also note that context limits have a very large impact on how different
models perform on different prompts. Across the board (with GPT-3.5 being the
sole exception), the models performed the best on the Summary Only prompt
style. This is likely because the Summary Only prompt style is the most likely
to fit within a model’s context limit. This is especially pronounced for the
InCoder and SantaCoder models, as they both have small context limits of 2,048
tokens. Their Pass@K performance for Python decreases dramatically when
switching from short Summary Only prompts to longer Summary at Top/Bottom
especially when compared with StarCoder and StarCoder+, which both have larger
context limits.

Focusing on Java’s performance, it is clear that most of the publicly
available LLMs have not been fine-tuned for Java, resulting in the near 0%
Pass@K values that can be seen in Table. 5. The one exception to this is
StarCoder+, which performed reasonably well on the Summary Only prompts,
achieving 12.000% Pass@20 performance. Finally, Finally, Rosalind’s
performance results in Table. 6are roughly in line with Python’s performance
in Table. 5, indicating that our benchmark is at roughly the same difficulty
as the problems found on the Rosalind website.

Statistics
---
Failure Reason: different output | 8661
Failure Reason: invalid syntax | 117665
Failure Reason: runtime error | 55351
Failure Reason: function timed out | 4
Passed Tests | 7982
Table 7: Error Distribution.

Prompt | Mean | Median | STDev
---|---|---|---
Java | 2278.82 | 2599.00 | 1331.81
Python | 2790.75 | 2194.00 | 2539.79
Rosalind | 564.49 | 509.00 | 286.47
Overall | 1510.66 | 812.50 | 1882.80
Table 8: Statistics of Prompt Tokens

Table. 8 provides an overview of the error statistics collected from our test
runs. The errors include: ‘different output’ where the generated code’s output
did not match the golden code’s output; ‘invalid syntax’ where syntax errors
in the generated code prevented code execution; ‘function timed out’ where
code execution exceeded time limits; and ‘runtime error’ where the generated
code compiled successfully but failed to run. The vast majority of the
generated code samples tested encountered a sort syntax or runtime error
without resulting in an output.

Many models, including SantaCoder, InCoder, and CodeGen, struggled to generate
the functions in our dataset effectively. Table. 8 sheds some light as to why:
these models could not process the extensive context necessary for typical
bioinformatics code due to their default context length limitation of 2,048
tokens, while the average Python prompt consists of 2,790 tokens, and the
average Java prompt consists of 2,278 tokens.

As shown by the scatterplots in Appendix J, it seems that on models with an
average Pass@K score of at least 2%, there is an inverse relationship between
the number of tokens in the prompt and the Pass@K score. Furthermore, for
models such as SantaCoder and GPT, the performance fell sharply after around
500 tokens. This could be due to the massive amount of context “confusing” the
models. When we look at the Necessary Only prompts, a similar trend occurs for
longer contexts, but the phenomenon is not as pronounced, as on average these
prompts are significantly shorter. Further discussion on the results of each
model can be found in Appendix I.

Despite these challenges, we firmly believe that this dataset holds pivotal
importance for benchmarking future models, especially ones with larger context
limits. For instance, models like GPT4, which offer a 32k context limit, fall
comfortably within our token range. Furthermore, other non-code language
models, such as certain storytelling models, have unlocked context lengths of
up to 100k tokens or more.

##  6 Conclusions and Future Work

Our study underscores the challenges in code generation, emphasizing the
shortcomings of current models in the face of complex tasks. We present a
highly challenging natural language to code tasks, providing input rich with
dependencies and imports. Existing models struggle to comprehend the
application of these imported toolkits or functions contained in other files.
Our tasks are marked by extensive input and a high level of specialization.
These programs are closer to real-world scenarios, requiring professional-
level code-writing skills, rather than merely catering to beginners. This
suggests that the code in question can typically be produced only by
professional programmers.

##  7 Limitation

As the novel benchmark within the field of bioinformatics, there remains a
multitude of areas for future exploration. Currently, we have covered most of
the existing models (at the time, August 2023). New models can be included in
further work, including but not limited to LLaMa2. Additionally, we will move
beyond function-level code generation as current models do not have the
capacity of the token to generate file-sized code. We included only a few
well-established bioinformatics repositories; in future work, without a
manpower constraint, we could include additional repositories that span more
niche sub-studies of bioinformatics and span across other languages, such as
R.

Finally, as with all novel computer science research, there are many societal
impacts to consider. There is always a risk that the benchmark may encourage
incorrect code generation outputs, resulting in researchers deploying
incorrect code, which could lead to inaccurate conclusions, initiating a
snowball effect of misinformation. Beyond the immediate implication for
research outcomes, our benchmark and dataset potentially could be misused
maliciously. For example, users might use our data to train models that
generate harmful biomedical experiments, such as designing dangerous
molecules. While our intention is to advance knowledge and use it in a
beneficial manner, there must be a level of caution and responsibility in
employing the benchmark and dataset we provide.

## References

  * Milmo [2023] Dan Milmo.  Chatgpt reaches 100 million users two months after launch.  _The Guardian_ , 2023.
  * Asare et al. [2022] Owura Asare, Meiyappan Nagappan, and N Asokan.  Is github’s copilot as bad as humans at introducing vulnerabilities in code?  _arXiv preprint arXiv:2204.04741_ , 2022.
  * Kalliamvakou [2022] Eirini Kalliamvakou.  Research: quantifying github copilot’s impact on developer productivity and happiness.  _The GitHub Blog, Sep_ , 7, 2022.
  * Eloundou et al. [2023] Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock.  GPTs are GPTs: An early look at the labor market impact potential of large language models.  _arXiv preprint arXiv:2303.10130_ , 2023.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and et al.  On the opportunities and risks of foundation models.  _CoRR_ , abs/2108.07258, 2021.  URL https://arxiv.org/abs/2108.07258.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ , 2020.
  * Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating large language models trained on code, 2021.
  * Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ , 2022.
  * Zhang et al. [2022] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ , 2022.
  * OpenAI [2023] OpenAI.  Gpt-4 technical report, 2023.
  * Austin et al. [2021a] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al.  Program synthesis with large language models.  _arXiv preprint arXiv:2108.07732_ , 2021a.
  * Hendrycks et al. [2021] Dan Hendrycks, Steven Basart, Saurav Kadavath, and Mantas Mazeika.  Measuring coding challenge competence with apps.  _NeurIPS_ , 2021.
  * Austin et al. [2021b] Jacob Austin, Augustus Odena, Maxwell Nye, and Maarten Bosma.  Program synthesis with large language models.  _arXiv preprint_ , 2021b.
  * Li et al. [2022] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals.  Competition-level code generation with alphacode.  _arXiv preprint arXiv:2203.07814_ , 2022.
  * Lai et al. [2022] Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu.  DS-1000: a natural and reliable benchmark for data science code generation.  _ArXiv_ , abs/2211.11501, 2022.
  * Khattab et al. [2023] Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia.  Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp, 2023.
  * Athiwaratkun et al. [2023] Ben Athiwaratkun, Sanjay Krishna Gouda, Zijian Wang, Xiaopeng Li, Yuchen Tian, Ming Tan, Wasi Uddin Ahmad, Shiqi Wang, Qing Sun, Mingyue Shang, Sujan Kumar Gonugondla, Hantian Ding, Varun Kumar, Nathan Fulton, Arash Farahani, Siddhartha Jain, Robert Giaquinto, Haifeng Qian, Murali Krishna Ramanathan, Ramesh Nallapati, Baishakhi Ray, Parminder Bhatia, Sudipta Sengupta, Dan Roth, and Bing Xiang.  Multi-lingual evaluation of code generation models, 2023.
  * Zheng et al. [2023a] Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang.  Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x, 2023a.
  * Cassano et al. [2023] Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda.  MultiPL-E: a scalable and polyglot approach to benchmarking neural code generation.  _IEEE Transactions on Software Engineering_ , pages 1–17, 2023.  doi: 10.1109/TSE.2023.3267446.  URL https://arxiv.org/abs/2208.08227.
  * Zan et al. [2022] Daoguang Zan, Bei Chen, Dejian Yang, Zeqi Lin, Minsu Kim, Bei Guan, Yongji Wang, Weizhu Chen, and Jian-Guang Lou.  Cert: Continual pre-training on sketches for library-oriented code generation, 2022.
  * Nijkamp et al. [2023a] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong.  CodeGen: an open large language model for code with multi-turn program synthesis.  In _The Eleventh International Conference on Learning Representations_ , 2023a.  URL https://openreview.net/forum?id=iaYcJKpY2B_.
  * Wang et al. [2022a] Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig.  Execution-based evaluation for open-domain code generation.  _arXiv preprint arXiv:2212.10481_ , 2022a.
  * Srivastava et al. [2022] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _arXiv preprint arXiv:2206.04615_ , 2022.
  * Li et al. [2023a] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries.  Starcoder: may the source be with you!, 2023a.
  * Fried et al. [2023] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen tau Yih, Luke Zettlemoyer, and Mike Lewis.  Incoder: A generative model for code infilling and synthesis, 2023.
  * Wang et al. [2021] Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi.  Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation, 2021.
  * Allal et al. [2023] Loubna Ben Allal, Raymond Li, Denis Kocetkov, Chenghao Mou, Christopher Akiki, Carlos Munoz Ferrandis, Niklas Muennighoff, Mayank Mishra, Alex Gu, Manan Dey, et al.  Santacoder: don’t reach for the stars!  _arXiv preprint arXiv:2301.03988_ , 2023.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Thoppilan et al. [2022] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_ , 2022.
  * Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_ , 2022.
  * Rae et al. [2021] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al.  Scaling language models: Methods, analysis & insights from training gopher.  _arXiv preprint arXiv:2112.11446_ , 2021.
  * Zeng et al. [2022] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang.  GLM-130B: an open bilingual pre-trained model.  _arXiv preprint arXiv:2210.02414_ , 2022.
  * Zheng et al. [2023b] Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang.  CodeGeeX: A pre-trained model for code generation with multilingual evaluations on HumanEval-X.  _arXiv preprint arXiv:2303.17568_ , 2023b.  doi: 10.48550/arXiv.2303.17568.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  LLaMA: open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Fried et al. [2022] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis.  InCoder: a generative model for code infilling and synthesis.  _arXiv preprint arXiv:2204.05999_ , 2022.  doi: 10.48550/ARXIV.2204.05999.  URL https://arxiv.org/abs/2204.05999.
  * Xu et al. [2022] Frank F. Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn.  A systematic evaluation of large language models of code.  In _Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming_ , MAPS 2022, page 1–10, New York, NY, USA, 2022. Association for Computing Machinery.  ISBN 9781450392730.  doi: 10.1145/3520312.3534862.  URL https://doi.org/10.1145/3520312.3534862.
  * Wang et al. [2023a] Zhiruo Wang, Grace Cuenca, Shuyan Zhou, Frank F. Xu, and Graham Neubig.  Mconala: A benchmark for code generation from multiple natural languages, 2023a.
  * Wang et al. [2022b] Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, et al.  Recode: Robustness evaluation of code generation models.  _arXiv preprint arXiv:2212.10264_ , 2022b.
  * Yu et al. [2023] Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Tao Xie, and Qianxiang Wang.  Codereval: A benchmark of pragmatic code generation with generative pre-trained models.  _arXiv preprint_ , 2023.
  * Russell et al. [2018] Pamela H. Russell, Rachel L. Johnson, Shreyas Ananthan, Benjamin Harnke, and Nichole E. Carlson.  A large-scale analysis of bioinformatics code on GitHub.  _PLOS ONE_ , 13(10):e0205898, October 2018\.  doi: 10.1371/journal.pone.0205898.  URL https://doi.org/10.1371/journal.pone.0205898.
  * Nijkamp et al. [2023b] Erik Nijkamp, Hiroaki Hayashi, Caiming Xiong, Silvio Savarese, and Yingbo Zhou.  Codegen2: Lessons for training llms on programming and natural languages.  _arXiv preprint_ , 2023b.
  * Wang et al. [2023b] Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi.  Codet5+: Open code large language models for code understanding and generation.  _arXiv preprint arXiv:2305.07922_ , 2023b.
  * Li et al. [2023b] Raymond Li, Loubna Ben Allal, Yangtian Zi, and Niklas Meunnighoff.  Starcoder: May the source be with you!  _arXiv preprint_ , 2023b.
  * Lachaux et al. [2020] Marie-Anne Lachaux, Baptiste Roziere, Lowik Chanussot, and Guillaume Lample.  Unsupervised translation of programming languages, 2020.

##  Appendix A Function Statistics

Below are our distributions of comment lines, token lengths, and the number of
parameters per function across our entire dataset. Generally, our number of
parameters and comments are left-skewed, while our prompt lengths, while left-
skewed, had a much more even distribution.

Figure 5: Comment lines per Function Distribution Figure 6: Prompt Token
Length Distribution Figure 7: Parameters per Function Distribution

Below is the distribution of the number of output tokens, averaged across
either Python or Java. The vast concentration of the generations is
concentrated at 0-250, as those are the minimum reserved tokens for
generations if the prompts are too large. Across each language, we utilized
nine models to generate outputs.

Figure 8: The distribution of the number of models’ output tokens.

##  Appendix B Python Examples

###  B.1 Python Example 1

####  B.1.1 Full Prompt

⬇

1#This is in python

2#Write a function called "unpipe_name" that takes in a string parameter
called "name". The function takes a string containing multiple gene names
separated by pipes, and returns a single gene name devoid of duplicates and
pipe characters. The function checks if there are any duplicates, and removes
any meaningless target names. If there are multiple gene names present, the
function takes the longest name as the final name. If there are any ambiguous
gene names, the function logs a warning and selects the longest name as the
final name. The function should also import "cnvlib.params".

3#

4#def unpipe_name(name):

5#

6#Here are the imports:

7#import numpy as np

8#import logging

9#from . import params

10#from skgenome import tabio

11#Here are the global variables:

12#MIN_REF_COVERAGE = -5.0

13#MAX_REF_SPREAD = 1.0

14#NULL_LOG2_COVERAGE = -20.0

15#GC_MIN_FRACTION = 0.3

16#GC_MAX_FRACTION = 0.7

17#INSERT_SIZE = 250

18#IGNORE_GENE_NAMES = ’-’, ’.’, ’CGH’

19#ANTITARGET_NAME = ’Antitarget’

20#ANTITARGET_ALIASES = ANTITARGET_NAME, ’Background’

21#Here are the class declarations:

22#Here are the additional function declarations:

23#def do_import_picard(fname, too_many_no_coverage):

24# summary: Reads a file in ’picardhs’ format, processes the data, and
returns a modified dataframe.

25# param: fname (string) - the file name/path to be read in ’picardhs’
format.

26# param: too_many_no_coverage (int) - if the number of bins with no coverage
is greater than this value, a warning message is logged. Default is 100.

27# return: garr (pandas dataframe) - a modified dataframe with added columns
’gene’ and ’log2’ based on the original dataframe read from the input file.

28#def unpipe_name(name):

29# summary: Remove duplicate gene names and pipe characters from a given
string.

30# param: name (str) - the string containing the gene names.

31# return: new_name (str) - the cleaned gene name string.

32#def do_import_theta(segarr, theta_results_fname, ploidy):

33# summary: A function for importing theta results and estimating copy number
and log2 ratios of segments.

34# param: segarr (numpy array) - array of segments

35# param: theta_results_fname (str) - name of theta results file

36# param: ploidy (int) - ploidy of genome (default is 2)

37# return: generator of numpy arrays - array of segments with estimated copy
number and log2 ratios.

38#def parse_theta_results(fname):

39# summary: Parses THetA results into a data structure with NLL, mu, C, and
p* columns.

40# param: fname (str) - name of the file to parse the results from

41# return: (dict) - a dictionary containing the NLL, mu_normal, mu_tumors, C,
and p* values

42#Here are the comments and the specs:

43#Write a function called "unpipe_name" that takes in a string parameter
called "name". The function takes a string containing multiple gene names
separated by pipes, and returns a single gene name devoid of duplicates and
pipe characters. The function checks if there are any duplicates, and removes
any meaningless target names. If there are multiple gene names present, the
function takes the longest name as the final name. If there are any ambiguous
gene names, the function logs a warning and selects the longest name as the
final name. The function should also import "cnvlib.params".

44#def unpipe_name(name):

####  B.1.2 Small Prompt

⬇

1Write a function with the following specs:

2\--specs begin here\--

3#Write a function called "unpipe_name" that takes in a string parameter
called "name". The function takes a string containing multiple gene names
separated by pipes, and returns a single gene name devoid of duplicates and
pipe characters. The function checks if there are any duplicates, and removes
any meaningless target names. If there are multiple gene names present, the
function takes the longest name as the final name. If there are any ambiguous
gene names, the function logs a warning and selects the longest name as the
final name. The function should also import "cnvlib.params".

4param: name (str) - the string containing the gene names.

5return: new_name (str) - the cleaned gene name string.

6\--specs end here\--

7Note the function will be embedded in the following context

8\--context begins here\--

9import random

10import hashlib

11import numpy as np

12import skimage

13import skimage.measure

14import scipy.ndimage

15import os

16import logging

17from functools import wraps

18from scipy import stats

19import sys

20import math

21IGNORE_GENE_NAMES = ("-", ".", "CGH")

22<<insert solution here>>

23def main():

24 string1 = <|string|>

25 string2 = ’CGH’

26 name=f’{string1}|{string2}’

27 print(unpipe_name(name))

28// context continues

29\--context ends here\--

30Make sure to only generate the function and not any of the context. Make
sure you are generating valid, runnable code. Begin your solution with:

31def unpipe_name(name):

32MAKE SURE TO INDENT THE BODY OF YOUR FUNCTION BY A TAB

####  B.1.3 No Comment Prompt

⬇

1This is in python

2Write a function called "unpipe_name" that takes in a string parameter called
"name". The function takes a string containing multiple gene names separated
by pipes, and returns a single gene name devoid of duplicates and pipe
characters. The function checks if there are any duplicates, and removes any
meaningless target names. If there are multiple gene names present, the
function takes the longest name as the final name. If there are any ambiguous
gene names, the function logs a warning and selects the longest name as the
final name. The function should also import "cnvlib.params".

3

4def unpipe_name(name):

5

6Here are the imports:

7import numpy as np

8import logging

9from . import params

10from skgenome import tabio

11Here are the global variables:

12MIN_REF_COVERAGE = -5.0

13MAX_REF_SPREAD = 1.0

14NULL_LOG2_COVERAGE = -20.0

15GC_MIN_FRACTION = 0.3

16GC_MAX_FRACTION = 0.7

17INSERT_SIZE = 250

18IGNORE_GENE_NAMES = ’-’, ’.’, ’CGH’

19ANTITARGET_NAME = ’Antitarget’

20ANTITARGET_ALIASES = ANTITARGET_NAME, ’Background’

21Here are the class declarations:

22Here are the additional function declarations:

23def do_import_picard(fname, too_many_no_coverage):

24 summary: Reads a file in ’picardhs’ format, processes the data, and returns
a modified dataframe.

25 param: fname (string) - the file name/path to be read in ’picardhs’ format.

26 param: too_many_no_coverage (int) - if the number of bins with no coverage
is greater than this value, a warning message is logged. Default is 100.

27 return: garr (pandas dataframe) - a modified dataframe with added columns
’gene’ and ’log2’ based on the original dataframe read from the input file.

28def unpipe_name(name):

29 summary: Remove duplicate gene names and pipe characters from a given
string.

30 param: name (str) - the string containing the gene names.

31 return: new_name (str) - the cleaned gene name string.

32def do_import_theta(segarr, theta_results_fname, ploidy):

33 summary: A function for importing theta results and estimating copy number
and log2 ratios of segments.

34 param: segarr (numpy array) - array of segments

35 param: theta_results_fname (str) - name of theta results file

36 param: ploidy (int) - ploidy of genome (default is 2)

37 return: generator of numpy arrays - array of segments with estimated copy
number and log2 ratios.

38def parse_theta_results(fname):

39 summary: Parses THetA results into a data structure with NLL, mu, C, and p*
columns.

40 param: fname (str) - name of the file to parse the results from

41 return: (dict) - a dictionary containing the NLL, mu_normal, mu_tumors, C,
and p* values

42Here are the comments and the specs:

43Write a function called "unpipe_name" that takes in a string parameter
called "name". The function takes a string containing multiple gene names
separated by pipes, and returns a single gene name devoid of duplicates and
pipe characters. The function checks if there are any duplicates, and removes
any meaningless target names. If there are multiple gene names present, the
function takes the longest name as the final name. If there are any ambiguous
gene names, the function logs a warning and selects the longest name as the
final name. The function should also import "cnvlib.params".

44def unpipe_name(name):

####  B.1.4 Reference Code

⬇

1def unpipe_name(name):

2 """Fix the duplicated gene names Picard spits out.

3

4 Return a string containing the single gene name, sans duplications and pipe

5 characters.

6

7 Picard CalculateHsMetrics combines the labels of overlapping intervals

8 by joining all labels with ’|’, e.g. ’BRAF|BRAF’ \-- no two distinct

9 targeted genes actually overlap, though, so these dupes are redundant.

10 Meaningless target names are dropped, e.g. ’CGH|FOO|-’ resolves as ’FOO’.

11 In case of ambiguity, the longest name is taken, e.g. "TERT|TERT Promoter"

12 resolves as "TERT Promoter".

13 """

14 if ’|’ not in name:

15 return name

16 gene_names = set(name.split(’|’))

17 if len(gene_names) == 1:

18 return gene_names.pop()

19 cleaned_names = gene_names.difference(IGNORE_GENE_NAMES)

20 if cleaned_names:

21 gene_names = cleaned_names

22 new_name = sorted(gene_names, key=len, reverse=True)[0]

23 if len(gene_names) > 1:

24 logging.warning(’WARNING: Ambiguous gene name %r; using %r’, name,

25 new_name)

26 return new_name

####  B.1.5 Annotation Prompts

⬇

1#This is in python

2#Write a function called "unpipe_name" that takes in a string parameter
called "name". The function takes a string containing multiple gene names
separated by pipes, and returns a single gene name devoid of duplicates and
pipe characters. The function checks if there are any duplicates, and removes
any meaningless target names. If there are multiple gene names present, the
function takes the longest name as the final name. If there are any ambiguous
gene names, the function logs a warning and selects the longest name as the
final name. The function should also import "cnvlib.params".

3#

4#def unpipe_name(name):

####  B.1.6 Context

⬇

1import random

2import hashlib

3import numpy as np

4import skimage

5import skimage.measure

6import scipy.ndimage

7import os

8import logging

9from functools import wraps

10from scipy import stats

11import sys

12import math

13IGNORE_GENE_NAMES = ("-", ".", "CGH")

14<<insert solution here>>

15def main():

16 string1 = <|string|>

17 string2 = ’CGH’

18 name=f’{string1}|{string2}’

19 print(unpipe_name(name))

20if __name__ == "__main__":

21 main()

###  B.2 Python Example 2

####  B.2.1 Full Prompt

⬇

1#This is in python

2#write a function called "UnifyLevels" that takes three parameters: baseLevel
(a constant array of integers representing the base levels), addonLevel (a
constant array of integers representing the additional levels), and windowSize
(an integer representing the window size). The function should merge the two
lists of breakpoints, but drop addonLevel values that are within windowSize of
baseLevel values. The function should return an array of integers called
joinedLevel. If addonLevel is an empty array, the function should return
baseLevel. The output should be sorted and of dtype=np.int_.

3#

4#def UnifyLevels(baseLevel, addonLevel, windowSize):

5#

6#Here are the imports:

7#from scipy import stats

8#import math

9#import numpy as np

10#import logging

11#import pandas as pd

12#Here are the global variables:

13#Here are the class declarations:

14#Here are the additional function declarations:

15#def segment_haar(cnarr, fdr_q):

16# summary: Segment CNVkit data using HaarSeg algorithm

17# param: cnarr (CopyNumArray) - binned, normalized copy ratios

18# param: fdr_q (float) - false discovery rate q-value

19# return: segarr (CopyNumArray) - CBS data table as a CNVkit object

20#def one_chrom(cnarr, fdr_q, chrom):

21# summary: This function segments copy number data for a single chromosome
using the HaarSeg algorithm.

22# param: cnarr (pandas.DataFrame) - a dataframe with columns ’chromosome’,
’start’, ’end’, ’log2’, and ’probes’.

23# param: fdr_q (float) - the false discovery rate threshold for segmenting
the copy number data.

24# param: chrom (str) - the chromosome to segment.

25# return: table (pandas.DataFrame) - a dataframe with columns ’chromosome’,
’start’, ’end’, ’log2’, ’gene’, and ’probes’, representing the segmented copy
number data.

26#def variants_in_segment(varr, segment, fdr_q):

27# summary: Generates a table of variant segments based on allele frequencies

28# param: varr (object) - variant data

29# param: segment (object) - genomic segment data

30# param: fdr_q (float) - false discovery rate threshold

31# return: table (object) - pandas DataFrame with segmented data

32#def haarSeg(I, breaksFdrQ, W, rawI, haarStartLevel, haarEndLevel):

33# summary: Perform segmentation on a 1D array of log-ratio values according
to the HaarSeg algorithm.

34# param: I (array) - A 1D array of log-ratio values, sorted according to
their genomic location.

35# param: W (array) - Weight matrix, corresponding to quality of measurement,
with values :math:‘1/(\\\sigma^2)‘. Must have the same size as I.

36# param: rawI (array) - The minimum between the raw test-sample and control-
sample coverages (before applying log ratio, but after any background
reduction and/or normalization). These raw red / green measurments are used to
detect low-value probes, which are more sensitive to noise. Used for the non-
stationary variance compensation. Must have the same size as I.

37# param: breaksFdrQ (float) - The FDR q parameter. This value should lie
between 0 and 0.5.

38# param: haarStartLevel (int) - The detail subband from which we start to
detect peaks.

39# param: haarEndLevel (int) - The detail subband until which we use to
detect peaks.

40# return: dict - Returns a dictionary containing the start and end points of
each segment and the mean value of each segment.

41#def FDRThres(x, q, stdev):

42# summary: Calculates the False Discovery Rate (FDR) threshold.

43# param: x (unknown type) - array of values.

44# param: q (unknown type) - a floating-point number.

45# param: stdev (unknown type) - a floating-point number representing the
standard deviation.

46# return: T (unknown type) - a floating-point number representing the FDR
threshold.

47#def SegmentByPeaks(data, peaks, weights):

48# summary: Average the values of the probes within each segment.

49# param: data (array) - the probe array values

50# param: peaks (array) - Positions of copy number breakpoints in the
original array

51# param: weights (None or array) - optional array of weights of same length
as the data array

52# return: segs (array) - array of segment values obtained by averaging the
values of the probes within each segment.

53#def HaarConv(signal, weight, stepHalfSize):

54# summary: Convolve haar wavelet function with a signal, applying circular
padding.

55# param: signal (array of floats) - signal to be convolved.

56# param: weight (array of floats) - optional weights for the steps of the
convolution.

57# param: stepHalfSize (int) - half size of the step to be used in the
convolution.

58# return: array (array of floats) - of floats, representing the convolved
signal.

59#def FindLocalPeaks(signal):

60# summary: Finds local maxima on positive values, local minima on negative
values.

61# param: signal (const array of floats): an array of floating point numbers

62# return: peakLoc (array of ints): Locations of extrema in ‘signal‘

63#def UnifyLevels(baseLevel, addonLevel, windowSize):

64# summary: Merge two lists of breakpoints and drop addonLevel values that
are too close to baseLevel values.

65# param: baseLevel (const array of ints) - a list of integers representing
the base level.

66# param: addonLevel (const array of ints) - a list of integers representing
the addon level.

67# param: windowSize (int) - an integer representing the maximum distance
between addonLevel and baseLevel values.

68# return: joinedLevel (array of ints) - a sorted array of integers
representing the merged list of breakpoints.

69#def PulseConv(signal, pulseSize):

70# summary: Convolve a pulse function with a signal applying circular padding
to the signal for non-stationary variance compensation.

71# param: signal (const array of floats) - the signal to be convolved.

72# param: pulseSize (int) - the size of the pulse function.

73# return: array of floats - the convolved signal.

74#def AdjustBreaks(signal, peakLoc):

75# summary: Improve localization of breaks in a signal by adjusting peak
locations.

76# param: signal (const array of floats) - the signal to adjust the peak
locations in.

77# param: peakLoc (const array of ints) - the current peak locations in the
signal.

78# return: newPeakLoc (array of ints) - the adjusted peak locations in the
signal.

79#def table2coords(seg_table):

80# summary: Returns x and y arrays for plotting with the help of the input
segment table.

81# param: seg_table (list) - a list of tuples containing start, size, and
value.

82# return: x (list) - a list of x-coordinates for plotting.

83# return: y (list) - a list of y-coordinates for plotting.

84#Here are the comments and the specs:

85#write a function called "UnifyLevels" that takes three parameters:
baseLevel (a constant array of integers representing the base levels),
addonLevel (a constant array of integers representing the additional levels),
and windowSize (an integer representing the window size). The function should
merge the two lists of breakpoints, but drop addonLevel values that are within
windowSize of baseLevel values. The function should return an array of
integers called joinedLevel. If addonLevel is an empty array, the function
should return baseLevel. The output should be sorted and of dtype=np.int_.

86#def UnifyLevels(baseLevel, addonLevel, windowSize):

####  B.2.2 Small Prompt

⬇

1Write a function with the following specs:

2\--specs begin here\--

3#write a function called "UnifyLevels" that takes three parameters: baseLevel
(a constant array of integers representing the base levels), addonLevel (a
constant array of integers representing the additional levels), and windowSize
(an integer representing the window size). The function should merge the two
lists of breakpoints, but drop addonLevel values that are within windowSize of
baseLevel values. The function should return an array of integers called
joinedLevel. If addonLevel is an empty array, the function should return
baseLevel. The output should be sorted and of dtype=np.int_.

4param: baseLevel (const array of ints) - a list of integers representing the
base level.

5param: addonLevel (const array of ints) - a list of integers representing the
addon level.

6param: windowSize (int) - an integer representing the maximum distance
between addonLevel and baseLevel values.

7return: joinedLevel (array of ints) - a sorted array of integers representing
the merged list of breakpoints.

8\--specs end here\--

9Note the function will be embedded in the following context

10\--context begins here\--

11import random

12import hashlib

13import numpy as np

14import skimage

15import skimage.measure

16import scipy.ndimage

17import os

18import logging

19from functools import wraps

20from scipy import stats

21import sys

22import math

23<<insert solution here>>

24def main():

25 np.random.seed(<|int;range=0,1000|>)

26 baseLevel = np.random.randint(20, size=(10))

27 np.random.seed(<|int;range=0,1000|>)

28 addonLevel = np.random.randint(20, size=(10))

29 print(UnifyLevels(baseLevel, addonLevel, 3))

30if __name__ == "__main__":

31// context continues

32\--context ends here\--

33Make sure to only generate the function and not any of the context. Make
sure you are generating valid, runnable code. Begin your solution with:

34def UnifyLevels(baseLevel, addonLevel, windowSize):

35MAKE SURE TO INDENT THE BODY OF YOUR FUNCTION BY A TAB

####  B.2.3 No Comment Prompt

⬇

1This is in python

2write a function called "UnifyLevels" that takes three parameters: baseLevel
(a constant array of integers representing the base levels), addonLevel (a
constant array of integers representing the additional levels), and windowSize
(an integer representing the window size). The function should merge the two
lists of breakpoints, but drop addonLevel values that are within windowSize of
baseLevel values. The function should return an array of integers called
joinedLevel. If addonLevel is an empty array, the function should return
baseLevel. The output should be sorted and of dtype=np.int_.

3

4def UnifyLevels(baseLevel, addonLevel, windowSize):

5

6Here are the imports:

7from scipy import stats

8import math

9import numpy as np

10import logging

11import pandas as pd

12Here are the global variables:

13Here are the class declarations:

14Here are the additional function declarations:

15def segment_haar(cnarr, fdr_q):

16 summary: Segment CNVkit data using HaarSeg algorithm

17 param: cnarr (CopyNumArray) - binned, normalized copy ratios

18 param: fdr_q (float) - false discovery rate q-value

19 return: segarr (CopyNumArray) - CBS data table as a CNVkit object

20def one_chrom(cnarr, fdr_q, chrom):

21 summary: This function segments copy number data for a single chromosome
using the HaarSeg algorithm.

22 param: cnarr (pandas.DataFrame) - a dataframe with columns ’chromosome’,
’start’, ’end’, ’log2’, and ’probes’.

23 param: fdr_q (float) - the false discovery rate threshold for segmenting
the copy number data.

24 param: chrom (str) - the chromosome to segment.

25 return: table (pandas.DataFrame) - a dataframe with columns ’chromosome’,
’start’, ’end’, ’log2’, ’gene’, and ’probes’, representing the segmented copy
number data.

26def variants_in_segment(varr, segment, fdr_q):

27 summary: Generates a table of variant segments based on allele frequencies

28 param: varr (object) - variant data

29 param: segment (object) - genomic segment data

30 param: fdr_q (float) - false discovery rate threshold

31 return: table (object) - pandas DataFrame with segmented data

32def haarSeg(I, breaksFdrQ, W, rawI, haarStartLevel, haarEndLevel):

33 summary: Perform segmentation on a 1D array of log-ratio values according
to the HaarSeg algorithm.

34 param: I (array) - A 1D array of log-ratio values, sorted according to
their genomic location.

35 param: W (array) - Weight matrix, corresponding to quality of measurement,
with values :math:‘1/(\\\sigma^2)‘. Must have the same size as I.

36 param: rawI (array) - The minimum between the raw test-sample and control-
sample coverages (before applying log ratio, but after any background
reduction and/or normalization). These raw red / green measurments are used to
detect low-value probes, which are more sensitive to noise. Used for the non-
stationary variance compensation. Must have the same size as I.

37 param: breaksFdrQ (float) - The FDR q parameter. This value should lie
between 0 and 0.5.

38 param: haarStartLevel (int) - The detail subband from which we start to
detect peaks.

39 param: haarEndLevel (int) - The detail subband until which we use to detect
peaks.

40 return: dict - Returns a dictionary containing the start and end points of
each segment and the mean value of each segment.

41def FDRThres(x, q, stdev):

42 summary: Calculates the False Discovery Rate (FDR) threshold.

43 param: x (unknown type) - array of values.

44 param: q (unknown type) - a floating-point number.

45 param: stdev (unknown type) - a floating-point number representing the
standard deviation.

46 return: T (unknown type) - a floating-point number representing the FDR
threshold.

47def SegmentByPeaks(data, peaks, weights):

48 summary: Average the values of the probes within each segment.

49 param: data (array) - the probe array values

50 param: peaks (array) - Positions of copy number breakpoints in the original
array

51 param: weights (None or array) - optional array of weights of same length
as the data array

52 return: segs (array) - array of segment values obtained by averaging the
values of the probes within each segment.

53def HaarConv(signal, weight, stepHalfSize):

54 summary: Convolve haar wavelet function with a signal, applying circular
padding.

55 param: signal (array of floats) - signal to be convolved.

56 param: weight (array of floats) - optional weights for the steps of the
convolution.

57 param: stepHalfSize (int) - half size of the step to be used in the
convolution.

58 return: array (array of floats) - of floats, representing the convolved
signal.

59def FindLocalPeaks(signal):

60 summary: Finds local maxima on positive values, local minima on negative
values.

61 param: signal (const array of floats): an array of floating point numbers

62 return: peakLoc (array of ints): Locations of extrema in ‘signal‘

63def UnifyLevels(baseLevel, addonLevel, windowSize):

64 summary: Merge two lists of breakpoints and drop addonLevel values that are
too close to baseLevel values.

65 param: baseLevel (const array of ints) - a list of integers representing
the base level.

66 param: addonLevel (const array of ints) - a list of integers representing
the addon level.

67 param: windowSize (int) - an integer representing the maximum distance
between addonLevel and baseLevel values.

68 return: joinedLevel (array of ints) - a sorted array of integers
representing the merged list of breakpoints.

69def PulseConv(signal, pulseSize):

70 summary: Convolve a pulse function with a signal applying circular padding
to the signal for non-stationary variance compensation.

71 param: signal (const array of floats) - the signal to be convolved.

72 param: pulseSize (int) - the size of the pulse function.

73 return: array of floats - the convolved signal.

74def AdjustBreaks(signal, peakLoc):

75 summary: Improve localization of breaks in a signal by adjusting peak
locations.

76 param: signal (const array of floats) - the signal to adjust the peak
locations in.

77 param: peakLoc (const array of ints) - the current peak locations in the
signal.

78 return: newPeakLoc (array of ints) - the adjusted peak locations in the
signal.

79def table2coords(seg_table):

80 summary: Returns x and y arrays for plotting with the help of the input
segment table.

81 param: seg_table (list) - a list of tuples containing start, size, and
value.

82 return: x (list) - a list of x-coordinates for plotting.

83 return: y (list) - a list of y-coordinates for plotting.

84Here are the comments and the specs:

85write a function called "UnifyLevels" that takes three parameters: baseLevel
(a constant array of integers representing the base levels), addonLevel (a
constant array of integers representing the additional levels), and windowSize
(an integer representing the window size). The function should merge the two
lists of breakpoints, but drop addonLevel values that are within windowSize of
baseLevel values. The function should return an array of integers called
joinedLevel. If addonLevel is an empty array, the function should return
baseLevel. The output should be sorted and of dtype=np.int_.

86def UnifyLevels(baseLevel, addonLevel, windowSize):

####  B.2.4 Reference Code

⬇

1def UnifyLevels(baseLevel, addonLevel, windowSize):

2 """Unify several decomposition levels.

3

4 Merge the two lists of breakpoints, but drop addonLevel values that are too

5 close to baseLevel values.

6

7 Parameters

8 \----------

9 baseLevel : const array of ints

10 addonLevel : const array of ints

11 windowSize : int

12

13 Returns

14 \-------

15 joinedLevel : array of ints

16

17 Source: HaarSeg.c

18 """

19 if not len(addonLevel):

20 return baseLevel

21 joinedLevel = []

22 addon_idx = 0

23 for base_elem in baseLevel:

24 while addon_idx < len(addonLevel):

25 addon_elem = addonLevel[addon_idx]

26 if addon_elem < base_elem - windowSize:

27 joinedLevel.append(addon_elem)

28 addon_idx += 1

29 elif base_elem - windowSize <= addon_elem <= base_elem + windowSize:

30 addon_idx += 1

31 else:

32 assert base_elem + windowSize < addon_elem

33 break

34 joinedLevel.append(base_elem)

35 last_pos = baseLevel[-1] + windowSize if len(baseLevel) else -1

36 while addon_idx < len(addonLevel) and addonLevel[addon_idx] <= last_pos:

37 addon_idx += 1

38 if addon_idx < len(addonLevel):

39 joinedLevel.extend(addonLevel[addon_idx:])

40 return np.array(sorted(joinedLevel), dtype=np.int_)

####  B.2.5 Context

⬇

1import random

2import hashlib

3import numpy as np

4import skimage

5import skimage.measure

6import scipy.ndimage

7import os

8import logging

9from functools import wraps

10from scipy import stats

11import sys

12import math

13<<insert solution here>>

14def main():

15 np.random.seed(<|int;range=0,1000|>)

16 baseLevel = np.random.randint(20, size=(10))

17 np.random.seed(<|int;range=0,1000|>)

18 addonLevel = np.random.randint(20, size=(10))

19 print(UnifyLevels(baseLevel, addonLevel, 3))

20if __name__ == "__main__":

21 main()

##  Appendix C Rosalind Examples

The Rosalind dataset consists of 253 problems from the bioinformatics training
website
rosalind.info. For each of the 253 problems that make up the Rosalind portion
of BioCoder, we created both a prompt that we fed into the LLMs as well as a
golden code solution that we either wrote ourselves or found on public GitHub
repositories.

For each problem, to create the prompt, we first scraped the problem’s page on
the Rosalind website to obtain a problem description. This problem description
would then make up the first section of the prompt. For instance, for the
Rosalind problem ba1a, here is the scraped problem description:

⬇

1# This is the first problem in a collection of "code challenges" to accompany
Bioinformatics Algorithms: An Active-Learning Approach by Phillip Compeau &
Pavel Pevzner.

2# A k-mer is a string of length k.

3# We define Count(Text, Pattern) as the number of times that a k-mer Pattern
appears as a substring of Text.

4# For example,

5#
$\textit{Count}(\text{ACA}\color{green}\textbf{ACTAT}\color{black}\text{GCAT}\color{green}\textbf{ACTAT}\color{black}\text{CGGGA}\color{green}\textbf{ACTAT}\color{black}\text{CCT},
{\color{green}\textbf{ACTAT}}) = 3$.

6# We note that
Count($\text{CG}\color{green}\textbf{ATATA}\color{black}\text{TCC}\color{green}\textbf{ATA}\color{black}\text{G}$,
$\color{green}\textbf{ATA}$) is equal to 3 (not 2) since we should account for
overlapping occurrences of Pattern in Text.

7# To compute Count(Text, Pattern), our plan is to "slide a window" down Text,
checking whether each k-mer substring of Text matches Pattern. We will
therefore refer to the k-mer starting at position i of Text as Text(i, k).
Throughout this book, we will often use 0-based indexing, meaning that we
count starting at 0 instead of 1. In this case, Text begins at position 0 and
ends at position |Text| - 1 (|Text| denotes the number of symbols in Text).
For example, if Text = GACCATACTG,

8# then Text(4, 3) = ATA. Note that the last k-mer of Text begins at position
|Text| - k, e.g., the last 3-mer of GACCATACTG starts at position 10 - 3 = 7.
This discussion results in the following pseudocode for computing Count(Text,
Pattern).

9# PatternCount(Text, Pattern)

10# count <- 0

11# for i <- 0 to |Text| - |Pattern|

12# if Text(i, |Pattern|) = Pattern

13# count <-s count + 1

14# return count

15# Implement PatternCount

16# Given: {DNA strings}} Text and Pattern.

17# Return: Count(Text, Pattern).

18# Sample Dataset

19# GCGCG

20# GCG

21# Sample Output

22# 2

From the sample problem description above you can see that a problem
description generally consists of an explanation of the problem at hand, as
well as a sample test case. The second section of the prompt consists of a
custom block of text that prompts the LLM to generate its code to fit the
context that generated Rosalind code will be run in. For instance, for the
same Rosalind problem ba1a, here is the custom block of text that makes up the
second section of the prompt:

⬇

1 #write the solve() function to solve the above problem

2#Do NOT print anything

3#Do NOT make comments

4#Do NOT call the main() function.

5#Use 4 spaces for tabs.

6#input_str: a string

7#return output: another string

8

9

10def main():

11 with open("input.txt", "r") as f:

12 output = solve(f.read())

13 print(output)

14

15# Write your solution here

16# Begin with: def solve(input_str):

From the example above, you can see that the custom block of text consists of
a prompt telling the LLM to generate a solve function as well as the context
that the generated code will be run in, namely the main function that is
included in the custom text. Each prompt consists of these two sections, i.e.,
the first section of the prompt will include a problem description consisting
of an explanation of the problem as well as a sample test case, and the second
section of the prompt will consist of custom prompting text that tells the LLM
to generate a solve function and provides the context in which the Rosalind
code will be run.

For each of the 253 problems that make up the Rosalind portion of BIOCODER, we
also either found online or wrote custom golden code solutions. Each golden
code solution is meant to be run in the custom context, following the main and
solve structure that is described in every Rosalind prompt. For instance, here
is the golden code solution for the Rosalind problem ba1a:

⬇

1# Compute the Number of Times a Pattern Appears in a Text

2

3def substrings(text, size):

4 for i in range(len(text) - size + 1):

5 yield text[i : i + size]

6

7def pattern_count(text, pattern):

8 return sum(pattern == x for x in substrings(text, len(pattern)))

9

10def solve(input_str):

11 text, pattern = input_str.splitlines()

12

13 return pattern_count(text, pattern)

As you can see, a golden code solution consists of the solve function
requested by the corresponding prompt, as well as any additional helper
functions that may be called (possibly recursively) by the solve function. For
more information on the Rosalind portion of the BIOCODER dataset, three
detailed examples of Rosalind prompts and golden code are provided below to
further illustrate the differences between the code and the prompt.

###  C.1 Example 1

The given input is a problem description in the field of bioinformatics,
namely the Reverse Complement Problem. The problem is related to the DNA
sequences, which are strings containing the characters ‘A’, ‘T’, ‘G’, and ‘C’.
Each of these characters represents a nucleotide in the DNA.

The problem requires model to find the reverse complement of a given DNA
string. The reverse complement of a DNA string is obtained by replacing each
nucleotide in the string with its complement (‘A’ with ‘T’, ‘T’ with ‘A’, ‘C’
with ‘G’, and ‘G’ with ‘C’), and then reversing the entire string. For
example, the reverse complement of “GTCA” is “TGAC.”

The problem description includes a sample dataset (DNA string) and its
corresponding output, which is its reverse complement. The problem also
specifies the format for the solution: a Python function named ‘solve’, which
takes a DNA string as an input and returns its reverse complement as output.
The function should not print anything or include any comments.

⬇

1

2Reverse Complement Problem:

3In DNA strings, symbols ’A’ and ’T’ are complements of each other, as are ’C’
and ’G’.

4Given a nucleotide p, we denote its complementary nucleotide as p. The
reverse complement of a DNA string Pattern = p1…pn is the string Pattern = pn
… p1 formed by taking the complement of each nucleotide in Pattern, then
reversing the resulting string.

5

6For example, the reverse complement of Pattern = "GTCA" is Pattern = "TGAC".

7

8

9Problem:

10Find the reverse complement of a DNA string.

11Given: A DNA string Pattern.

12Return: Pattern, the reverse complement of Pattern.

13

14Sample Dataset:

15AAAACCCGGT

16

17Sample Output:

18ACCGGGTTTT

19

20write the solve() function to solve the above problem.

21

22Do NOT print anything

23Do NOT make comments

24Do NOT call the main() function.

25Use 4 spaces for tabs.

26

27input_str: a string

28return output: another string

29

30def main():

31 with open("input.txt", "r") as f:

32 output = solve(f.read())

33 print(output)

34

35

36Write your solution here:

37Begin with: def solve(input_str):

The given sample solution to the problem is written in Python. It defines two
functions: ‘revcomp’ and ‘main’. The ‘revcomp’ function computes the reverse
complement of a DNA string by first reversing the string and then replacing
each nucleotide with its complement using Python’s ‘str.maketrans’ method. The
‘main’ function reads a DNA string from a file, computes its reverse
complement using the ‘revcomp’ function, and then prints the result.

The ground-truth code is a solution to the problem presented in the ‘Rosalind’
platform, a platform that provides a collection of bioinformatics problems.
This platform is known for helping students and researchers learn
bioinformatics by solving the computational problems it presents.

⬇

1Find the Reverse Complement of a String

2

3

4def revcomp(seq):

5 return seq[::-1].translate(str.maketrans("ACGT", "TGCA"))

6

7

8def main(file):

9 seq = open(file).read().splitlines()[0]

10 print(revcomp(seq))

###  C.2 Example 2

The input here consists of several parts.

The initial portion discusses the Hardy–Weinberg Principle and Mendel’s laws,
which are used in genetics to predict how genes will distribute throughout a
population over time. The Hardy–Weinberg Principle assumes that a population
is large and remains in genetic equilibrium, meaning the frequency of each
allele (a version of a gene) remains constant across generations, under
certain conditions.

The problem statement is presented: if we know the proportion of homozygous
recessive individuals (those with two copies of the recessive allele) for each
of several genetic factors in a population, can we predict the probability
that a randomly selected individual carries at least one copy of the recessive
allele for each factor? The task is to write a Python function named ’solve’
to solve this problem.

A sample dataset is given, represented as an array of floating-point numbers,
each of which indicates the proportion of homozygous recessive individuals for
a specific genetic factor in the population. The ’solve’ function will receive
this array as a string and should return another string representing the
solution array.

The final portion of the input is a sample Python code that reads the dataset
from a file, calls the ‘solve’ function to compute the solution, and then
prints the solution. However, the user is instructed not to print anything,
not to write comments, and not to call this ’main’ function in your solution.
The user is also instructed to use four spaces for indentation.

⬇

1Genetic Drift and the Hardy-Weinberg Principle Mendel’s laws of segregation
and independent assortment are excellent for the study of individual organisms
and their progeny, but they say nothing about how alleles move through a
population over time.

2

3Our first question is: when can we assume that the ratio of an allele in a
population, called the allele frequency, is stable? G. H. Hardy and Wilhelm
Weinberg independently considered this question at the turn of the 20th
Century, shortly after Mendel’s ideas had been rediscovered. They concluded
that the percentage of an allele in a population of individuals is in genetic
equilibrium when five conditions are satisfied: The population is so large
that random changes in the allele frequency are negligible. No new mutations
are affecting the gene of interest; The gene does not influence survival or
reproduction, so natural selection is not occurring; Gene flow, or the change
in allele frequency due to migration into and out of the population, is
negligible. Mating occurs randomly with respect to the gene of interest. The
Hardy-Weinberg principle states that if a population is in genetic equilibrium
for a given allele, then its frequency will remain constant and evenly
distributed through the population. Unless the gene in question is important
to survival or reproduction, Hardy-Weinberg usually offers a reasonable enough
model of population genetics. One of the many benefits of the Mendelian theory
of inheritance and simplifying models like Hardy-Weinberg is that they help us
predict the probability with which genetic diseases will be inherited, so as
to take appropriate preventative measures. Genetic diseases are usually caused
by mutations to chromosomes, which are passed on to subsequent generations.
The simplest and most widespread case of a genetic disease is a single gene
disorder, which is caused by a single mutated gene. Over 4,000 such human
diseases have been identified, including cystic fibrosis and sickle-cell
anemia. In both of these cases, the individual must possess two recessive
alleles for a gene in order to contract the disease. Thus, carriers can live
their entire lives without knowing that they can pass the disease on to their
children. The above introduction to genetic equilibrium leaves us with a basic
and yet very practical question regarding gene disorders: if we know the
number of people who have a disease encoded by a recessive allele, can we
predict the number of carriers in the population?

4

5Problem:

6

7To model the Hardy-Weinberg principle, assume that we have a population of
$N$ diploid individuals. If an allele is in genetic equilibrium, then because
mating is random, we may view the $2N$ chromosomes as receiving their alleles
uniformly. In other words, if there are $m$ dominant alleles, then the
probability of a selected chromosome exhibiting the dominant allele is simply
$p = \frac{m}{2N}$. Because the first assumption of genetic equilibrium states
that the population is so large as to be ignored, we will assume that $N$ is
infinite, so that we only need to concern ourselves with the value of $p$.

8

9Given: An array $A$ for which $A[k]$ represents the proportion of homozygous
recessive individuals for the $k$-th Mendelian factor in a diploid population.
Assume that the population is in genetic equilibrium for all factors.

10

11Return: An array $B$ having the same length as $A$ in which $B[k]$
represents the probability that a randomly selected individual carries at
least one copy of the recessive allele for the $k$-th factor.

12

13Sample Dataset

140.1 0.25 0.5

15

16write the solve() function to solve the above problem

17

18Do NOT print anything

19Do NOT make comments

20Do NOT call the main() function.

21

22Use 4 spaces for tabs.

23

24input_str: a string

25return output: another string

26

27def main():

28 with open("input.txt", "r") as f:

29 output = solve(f.read())

30 print(output)

31

32Write your solution here

33Begin with: def solve(input_str):

In the output portion, a Python function afrq is presented, which takes an
array of proportions of homozygous recessive individuals for each genetic
factor in a population and returns an array of probabilities that a randomly
selected individual carries at least one copy of the recessive allele for each
factor. The main function uses this afrq function to solve the problem based
on the input data. It takes the input data from a file, applies the afrq
function to compute the solution, and then prints the solution in the form of
a space-separated string of floating-point numbers, each rounded to three
decimal places.

⬇

1# Counting Disease Carriers

2

3from math import sqrt

4from .helpers import Parser

5

6def afrq(a):

7 """Counting Disease Carriers"""

8 return [2 * sqrt(x) - x for x in a]

9

10def main(file):

11 b = afrq(Parser(file).floats())

12 print(*[round(x, 3) for x in b])

###  C.3 Example 3

This Python script is designed to find the most frequent k-mers (substrings of
length k) in a given DNA sequence (a string), while allowing for a specified
number of mismatches. This problem is referred to as the "Frequent Words with
Mismatches Problem."

The input to this problem is a string that contains two distinct parts
separated by a newline:

The DNA sequence is represented as a string of characters ‘A’, ‘C’, ‘G’, and
‘T’. Two integers, k and d, are separated by a space. The integer k refers to
the length of the substrings (k-mers) of interest, and d specifies the maximum
number of mismatches that are allowed in a k-mer.

⬇

1

2We defined a mismatch in ‘‘Compute the Hamming Distance Between Two Strings”.
We now generalize “Find the Most Frequent Words in a String” to incorporate
mismatches as well. Given strings Text and Pattern as well as an integer d, we
define Countd(Text, Pattern) as the total number of occurrences of Pattern in
Text with at most d mismatches. For example, Count1(AACAAGCTGATAAACATTTAAAGAG,
AAAAA) = 4 because AAAAA appears four times in this string with at most one
mismatch: AACAA, ATAAA, AAACA, and AAAGA. Note that two of these occurrences
overlap.

3

4A most frequent k-mer with up to d mismatches in Text is simply a string
Pattern maximizing Countd(Text, Pattern) among all k-mers. Note that Pattern
does not need to actually appear as a substring of Text; for example, AAAAA is
the most frequent 5-mer with 1 mismatch in AACAAGCTGATAAACATTTAAAGAG, even
though AAAAA does not appear exactly in this string. Keep this in mind while
solving the following problem.

5

6Frequent Words with Mismatches Problem

7

8Find the most frequent k-mers with mismatches in a string. Given: A string
Text as well as integers k and d. Return: All most frequent k-mers with up to
d mismatches in Text.

9

10Sample Dataset

11ACGTTGCATGTCGCATGATGCATGAGAGCT

124 1

13

14Sample Output

15GATG ATGC ATGT

16

17write the solve() function to solve the above problem

18Do NOT print anything

19Do NOT make comments

20Do NOT call the main() function.

21Use 4 spaces for tabs.

22input_str: a string

23return output: another string

24

25def main():

26 with open("input.txt", "r") as f:

27 output = solve(f.read())

28 print(output)

29

30Write your solution here

31Begin with: def solve(input_str):

Here is the corresponding reference code:

⬇

1# Find the Most Frequent Words with Mismatches in a String

2

3from .ba1g import hamming

4from .ba1b import count_kmers, most_frequent

5from itertools import product

6

7# Note, the best kmer might not be observed in our sequence. The simplistic

8# method here simply checks all possible kmers (which is ~17M for k = 12)

9

10def generate_kmers(k):

11 return ("".join(x) for x in product(["A", "C", "G", "T"], repeat=k))

12

13def count_hamming_kmers(kmers, d, k):

14 for x in generate_kmers(k):

15 count = sum(kmers[y] for y in kmers if hamming(x, y) <= d)

16 if count > 0:

17 yield [x, count]

18

19def main(file):

20 seq, ints = open(file).read().splitlines()

21 k, d = list(map(int, ints.split()))

22 kmers = count_kmers(seq, k)

23 hkmers = dict(count_hamming_kmers(kmers, d, k))

24 print(*most_frequent(hkmers))

##  Appendix D Additional Model Information

Model | Parameters
---|---
InCoder Fried et al. [2023] | 6B
SantaCoder Allal et al. [2023] | 1.1B
StarCoder Li et al. [2023b] | 15.5B
StarCoderPlus Li et al. [2023b] | 15.5B
InstructCodeT5+ Wang et al. [2023b] | 16B
CodeGen Nijkamp et al. [2023a] | 6B-mono*
CodeGen Nijkamp et al. [2023a] | 16B-mono*
CodeGen2 Nijkamp et al. [2023b] | 7B*
GPT-3.5-Turbo | Unknown
Table 9: Size of models.

We were not able to run the 16B models due to issues with scaling and sharding
on the A100s.

We found that the following parameters produced the best results across all
models.

  * •

top_k=50

  * •

top_p=0.95

  * •

temperature=0.7

  * •

early_stopping=True

  * •

num_return_sequences=1

  * •

do_sample=True

We utilized similar parameters to make model testing consistent across all
tested models. This approach allowed for a more unbiased comparison of how
each model performed on our benchmark. We determined these weights by
averaging the recommended parameters for the above models, and then used
Bayesian optimization to determine the most optimal parameters.

Furthermore, note that we used the version of GPT-3.5-Turbo hosted on Azure
OpenAI Service (gpt-35-turbo-0301). There may be some minor differences
compared with the OpenAI Platform version.

##  Appendix E Utilizing ChatGPT for Prompt Generation

ChatGPT served as a pivotal component in our prompt generation, enabling us to
assemble comprehensive annotations for the ground-truth functions and their
dependencies. This eliminates the need for extensive manual labor that would
otherwise be spent on annotation. Utilizing GPT3.5 Turbo calls further ensures
consistency across annotations, mitigating the variability that might arise
from human annotators.

The prompt was constructed as follows:

⬇

1Generate a prompt for the following function, such that a programmer could
reproduce it based solely on the description. Emphasize crucial components
like the return statement, complex function calls, and ambiguous variables.
However, avoid line-by-line descriptions; aim to provide a high-level
overview. The response should be in plain text, free from any formatting such
as Markdown. Keep the length under 200 words or 15 lines, whichever is
shorter.

2Here is the function for you to describe:

3<function>

4Here are the additional dependencies that may or may not be used within the
function:

5<dependencies>

6Again, limit your responses to no more than 200 words or 15 lines, whichever
comes first.

7Start your prompt with "write a function".

##  Appendix F Model Context Limits

Model | Context limit
---|---
InCoder Fried et al. [2023] | 2048
SantaCoder Allal et al. [2023] | 2048
StarCoder Li et al. [2023b] | 8192
StarCoderPlus Li et al. [2023b] | 8192
InstructCodeT5+ Wang et al. [2023b] | 2048
CodeGen Nijkamp et al. [2023a] | 2048
CodeGen2 Nijkamp et al. [2023b] | 2048
GPT-3.5-Turbo | 8192
Table 10: Length limits of different models.

Since the limit includes both input and output tokens, we reserved 256 tokens
for the output, therefore reducing the input length limit by 256. In our best
effort to continue testing with these context limits, instead of treating a
failing test case immediately, we attempted to truncate the prompts such that
they fit barely below the context limit of each respective model. For all
models, we also tested a version with the function summary at the top. Note
that for all models, the context limit was identical regardless of the
parameter count.

##  Appendix G CoderEval Comparison

To validate the performance of code LLMs, multiple existing benchmarks (e.g.,
AiXBench and the widely used HumanEval benchmark from OpenAI) are proposed,
including only cases of generating a standalone function, i.e., a function
that invokes or accesses only built-in functions and standard libraries.
However, standalone functions constitute only about 30% of the functions from
real open-source projects. To assess a model’s performance for pragmatic code
generation (i.e., code generation for real settings of open source or
proprietary code), CoderEval Yu et al. [2023] proposes a benchmark named
CoderEval of pragmatic code generation with generative pre-trained models.

In comparison to CoderEval, our approach is more focused on bioinformatics, as
we ensure each function requires a certain level of domain knowledge in
bioinformatics, as opposed to entry-level data science or software engineering
tasks.

Moreover, we employ a more advanced parser, implementing a comprehensive
parsing of the respective GitHub repositories. This includes the imported
third-party packages and the classes that have been imported from another
actual file.

We also conduct function testing at the file level, rather than the repository
level, manually extracting the context at the end and running it through a
framework that catches more errors than CoderEval.

Additionally, our dataset is larger than CoderEval’s, which only consists of
230 functions from 43 Python projects and 230 methods from 10 Java projects,
while we have data from more than two thousand sources.

CoderEval classifies tasks into six levels according to the source of
dependency outside the function, while we directly use the most complex
dependencies.

CoderEval employs a human-labeled version description from 13 experienced
engineers, while we leverage ChatGPT to generate function descriptions.
Similarly, for test cases, our use of fuzz testing allows us to scale to large
data volumes without the need for intensive manual annotation.

Yet, we share some similarities with CoderEval. Both BioCoder and CoderEval
can be used to assess the performance of models beyond merely generating
standalone functions. Given the need to address dependencies with context, we
both establish testing based on Docker, cloning GitHub repositories and their
associated import dependencies. With ’k’ candidate outputs generated by
different models (e.g., 20), we simulate scenarios in actual IDEs.

##  Appendix H Prompt Explanations

Below is an explanation of the prompt types:

  1. 1.

Summary Only We begin the prompt from these prompts, where only the summary
and the function signature are contained in the prompt. The summary is added
as a comment directly above the signature, which is uncommented. The summary
includes nearly complete details about the task; however, it intentionally
does not thoroughly explain what the context is. Therefore, this result is
best treated as a baseline when compared with other prompt types. An example
is shown below:

⬇

1#This is in python

2#Write a function named "planewise_morphology_closing" that accepts two
parameters, "x_data" and "structuring_element". Inside the function, create a
new variable named "y_data" that will hold the result of the morphological
closing operation. Loop through each item in "x_data" and perform the same
operation on each. Do this by using the "enumerate" function to get both the
index and plane for each item. Use the "closing" function from the
"skimage.morphology" module to perform the morphological closing operation on
each plane using the "structuring_element" parameter. Finally, return the
newly created "y_data" array that holds the result of the operation. Ensure
that the function uses the "numpy" and "skimage.morphology" modules, as they
are dependencies for the function.

3#

4#def planewise_morphology_closing(x_data, structuring_element):

  2. 2.

Uncommented We add the summary back into all of the parsed context. Therefore,
the entire prompt contains all imports, global variables, classes, internal
class functions, etc., that were imported into the file originally. For
functions over ten lines, we summarize the parameters, return type, and logic
instead of including the actual function code in order to reduce the number of
tokens used and eliminate data that the model would not need. Note that, by
design, if these extraneous functions were used, the test would have counted
as failed, because it would be extremely inefficient code. An example is shown
below:

⬇

1This is in python

2Here are the imports:

3from cellprofiler.library.modules import closing

4from ._help import HELP_FOR_STREL

5from cellprofiler_core.module import ImageProcessing

6from cellprofiler_core.setting import StructuringElement

7Here are the global variables:

8Here are the class declarations:

9class Closing(ImageProcessing):

10 attributes: self.structuring_element,self.function

11 methods:

12 def create_settings(self):

13 summary: creates settings for the Closing function

14 param: self (unknown) - instance of the Closing function

15 return: None

16 def settings(self):

17 summary: Returns settings from superclass with self.structuring_element
added.

18 param: None

19 return: list - __settings__ with self.structuring_element appended.

20 def visible_settings(self):

21 summary: This function returns the visible settings of a Closing object.

22 param: self (Closing) - the Closing object whose settings are to be
returned.

23 return: list - the visible settings of the Closing object.

24 def run(self, workspace):

25 summary: Applies morphological closing to an image in a workspace.

26 param: workspace (object) - the workspace containing the image.

27 return: None

28Here are the additional function declarations:

29def planewise_morphology_closing(x_data, structuring_element):

30 summary: Applies morphology closing operation to each plane in x_data using
structuring_element and returns the resulting y_data.

31 param: x_data (numpy array) - 3D array containing the input data.

32 param: structuring_element (numpy array) - Structuring element used to
perform the closing operation.

33 return: y_data (numpy array) - 3D array containing the result of closure
operation on each plane in x_data.

34Here are the comments and the specs:

35Write a function named "planewise_morphology_closing" that accepts two
parameters, "x_data" and "structuring_element". Inside the function, create a
new variable named "y_data" that will hold the result of the morphological
closing operation. Loop through each item in "x_data" and perform the same
operation on each. Do this by using the "enumerate" function to get both the
index and plane for each item. Use the "closing" function from the
"skimage.morphology" module to perform the morphological closing operation on
each plane using the "structuring_element" parameter. Finally, return the
newly created "y_data" array that holds the result of the operation. Ensure
that the function uses the "numpy" and "skimage.morphology" modules, as they
are dependencies for the function.

36def planewise_morphology_closing(x_data, structuring_element):

Note that it is nearly certain that each prompt will contain extraneous data
that does not need to be used for the function. The goal is for the LLM to
ensure it uses the correct context for the function. Note that for prompts
that exceed the model’s context limit, this prompt likely fails.

  3. 3.

Summary at Bottom We generate these using the previous prompt (uncommented),
and comment out all the context. Note that for prompts that exceed the model’s
context limit, this prompt likely fails. In addition, note that there are no
results for "summary at bottom" for Java due to an incompatibility with Java
syntax. We could not generate this type of prompt for Java similar to how we
generated it for Python.

⬇

1#Here are the imports:

2#from cellprofiler.library.modules import closing

3#from ._help import HELP_FOR_STREL

4#from cellprofiler_core.module import ImageProcessing

5#from cellprofiler_core.setting import StructuringElement

6#Here are the global variables:

7#Here are the class declarations:

8#class Closing(ImageProcessing):

9# attributes: self.structuring_element,self.function

10# methods:

11# def create_settings(self):

12# summary: creates settings for the Closing function

13# param: self (unknown) - instance of the Closing function

14# return: None

15# def settings(self):

16# summary: Returns settings from superclass with self.structuring_element
added.

17# param: None

18# return: list - __settings__ with self.structuring_element appended.

19# def visible_settings(self):

20# summary: This function returns the visible settings of a Closing object.

21# param: self (Closing) - the Closing object whose settings are to be
returned.

22# return: list - the visible settings of the Closing object.

23# def run(self, workspace):

24# summary: Applies morphological closing to an image in a workspace.

25# param: workspace (object) - the workspace containing the image.

26# return: None

27#Here are the additional function declarations:

28#def planewise_morphology_closing(x_data, structuring_element):

29# summary: Applies morphology closing operation to each plane in x_data
using structuring_element and returns the resulting y_data.

30# param: x_data (numpy array) - 3D array containing the input data.

31# param: structuring_element (numpy array) - Structuring element used to
perform the closing operation.

32# return: y_data (numpy array) - 3D array containing the result of closure
operation on each plane in x_data.

33#Here are the comments and the specs:

34#Write a function named "planewise_morphology_closing" that accepts two
parameters, "x_data" and "structuring_element". Inside the function, create a
new variable named "y_data" that will hold the result of the morphological
closing operation. Loop through each item in "x_data" and perform the same
operation on each. Do this by using the "enumerate" function to get both the
index and plane for each item. Use the "closing" function from the
"skimage.morphology" module to perform the morphological closing operation on
each plane using the "structuring_element" parameter. Finally, return the
newly created "y_data" array that holds the result of the operation. Ensure
that the function uses the "numpy" and "skimage.morphology" modules, as they
are dependencies for the function.

35def planewise_morphology_closing(x_data, structuring_element):

  4. 4.

Summary at Top These prompts are generated from the previous prompts (summary
at bottom); however, the summary is copied at the top (moved for Java). This
is intended for models with shorter context lengths, as when we truncated the
prompt, the summary would still be intact, along with a portion of the
context.

⬇

1#This is in python

2#Write a function named "planewise_morphology_closing" that accepts two
parameters, "x_data" and "structuring_element". Inside the function, create a
new variable named "y_data" that will hold the result of the morphological
closing operation. Loop through each item in "x_data" and perform the same
operation on each. Do this by using the "enumerate" function to get both the
index and plane for each item. Use the "closing" function from the
"skimage.morphology" module to perform the morphological closing operation on
each plane using the "structuring_element" parameter. Finally, return the
newly created "y_data" array that holds the result of the operation. Ensure
that the function uses the "numpy" and "skimage.morphology" modules, as they
are dependencies for the function.

3#

4#def planewise_morphology_closing(x_data, structuring_element):

5#

6#Here are the imports:

7#from cellprofiler.library.modules import closing

8#from ._help import HELP_FOR_STREL

9#from cellprofiler_core.module import ImageProcessing

10#from cellprofiler_core.setting import StructuringElement

11#Here are the global variables:

12#Here are the class declarations:

13#class Closing(ImageProcessing):

14# attributes: self.structuring_element,self.function

15# methods:

16# def create_settings(self):

17# summary: creates settings for the Closing function

18# param: self (unknown) - instance of the Closing function

19# return: None

20# def settings(self):

21# summary: Returns settings from superclass with self.structuring_element
added.

22# param: None

23# return: list - __settings__ with self.structuring_element appended.

24# def visible_settings(self):

25# summary: This function returns the visible settings of a Closing object.

26# param: self (Closing) - the Closing object whose settings are to be
returned.

27# return: list - the visible settings of the Closing object.

28# def run(self, workspace):

29# summary: Applies morphological closing to an image in a workspace.

30# param: workspace (object) - the workspace containing the image.

31# return: None

32#Here are the additional function declarations:

33#def planewise_morphology_closing(x_data, structuring_element):

34# summary: Applies morphology closing operation to each plane in x_data
using structuring_element and returns the resulting y_data.

35# param: x_data (numpy array) - 3D array containing the input data.

36# param: structuring_element (numpy array) - Structuring element used to
perform the closing operation.

37# return: y_data (numpy array) - 3D array containing the result of closure
operation on each plane in x_data.

38#Here are the comments and the specs:

39#Write a function named "planewise_morphology_closing" that accepts two
parameters, "x_data" and "structuring_element". Inside the function, create a
new variable named "y_data" that will hold the result of the morphological
closing operation. Loop through each item in "x_data" and perform the same
operation on each. Do this by using the "enumerate" function to get both the
index and plane for each item. Use the "closing" function from the
"skimage.morphology" module to perform the morphological closing operation on
each plane using the "structuring_element" parameter. Finally, return the
newly created "y_data" array that holds the result of the operation. Ensure
that the function uses the "numpy" and "skimage.morphology" modules, as they
are dependencies for the function.

40#def planewise_morphology_closing(x_data, structuring_element):

  5. 5.

Necessary Only We use a mixture of our syntax solving algorithm and hand
annotation to determine exactly which objects of the code are necessary for
the function to execute, and then use them in place of the original context.
Note that this is very similar to the environment used for testing the
functions.

⬇

1Write a function with the following specs:

2\--specs begin here\--

3#Write a function named "planewise_morphology_closing" that accepts two
parameters, "x_data" and "structuring_element". Inside the function, create a
new variable named "y_data" that will hold the result of the morphological
closing operation. Loop through each item in "x_data" and perform the same
operation on each. Do this by using the "enumerate" function to get both the
index and plane for each item. Use the "closing" function from the
"skimage.morphology" module to perform the morphological closing operation on
each plane using the "structuring_element" parameter. Finally, return the
newly created "y_data" array that holds the result of the operation. Ensure
that the function uses the "numpy" and "skimage.morphology" modules, as they
are dependencies for the function.

4param: x_data (numpy array) - 3D array containing the input data.

5param: structuring_element (numpy array) - Structuring element used to
perform the closing operation.

6return: y_data (numpy array) - 3D array containing the result of closure
operation on each plane in x_data.

7\--specs end here\--

8Note the function will be embedded in the following context

9\--context begins here\--

10import numpy

11import skimage.morphology

12import os

13<<insert solution here>>

14def main():

15 numpy.random.seed(<|int;range=0,100|>)

16 pixels = numpy.random.randint(2, size=(10, 10, 10))

17 structuring_element = skimage.morphology.square(3)

18 print(planewise_morphology_closing(pixels, structuring_element))

19if __name__ == "__main__":

20 main()

21\--context ends here\--

22Make sure to only generate the function and not any of the context. Make
sure you are generating valid, runnable code. Begin your solution with:

23def planewise_morphology_closing(x_data, structuring_element):

##  Appendix I Model Ablation Study

For the ablation study, we considered two representative functions, one for
Java and one for Python, and determined how the various models performed in
generating the following functions. The Java function we selected is the
following:

⬇

1public static String getReverseComplementedSequence(SAMRecord record, int
startIndex, int length) {

2 if (startIndex < 0) {

3 startIndex = record.getReadLength() + startIndex;

4 }

5 byte[] rangeBytes = Arrays.copyOfRange(record.getReadBases(), startIndex,
startIndex + length);

6 SequenceUtil.reverseComplement(rangeBytes);

7 return new String(rangeBytes);

8}

and the (abridged) context that the function getReverseComplementedSequence
had access to is the following:

⬇

1import org.apache.commons.math3.distribution.HypergeometricDistribution;

2import java.text.DecimalFormat;

3import java.util.*;

4import java.util.function.Function;

5import htsjdk.samtools.util.SequenceUtil;

6import java.util.Objects;

7

8class SAMRecord {

9 public byte[] arr2;

10

11 public SAMRecord(byte[] arr) {

12 arr2 = arr;

13 }

14

15 public byte[] getReadBases() {

16 return arr2;

17 }

18

19 public int getReadLength() {

20 return arr2.length;

21 }

22}

23

24final class Utils {

25 /**

26 * Method returns reverse complemented sequence for the part of the record.
Can work with 3’ and 5’ ends

27 * (if start index < 0, then it will found the index in the end of sequence
by adding the length of record).

28 * @param record read from SAM file to process

29 * @param startIndex index where start the sequence

30 * @param length length of pert of sequence

31 * @return reverse complemented part of record

32 */

33 <<insert solution here>>

34

35 public static String reverse(String string) {

36 return new StringBuffer(string).reverse().toString();

37 }

38

39 public static String complement(String string) {

40 final byte[] bases = htsjdk.samtools.util.StringUtil.stringToBytes(string);

41 complement(bases);

42 return htsjdk.samtools.util.StringUtil.bytesToString(bases);

43 }

44}

We chose this as the representative Java function because it is highly
bioinformatics related and the solution will require an understanding of the
context around it, namely the custom SAMRecord class. Furthermore, the model
will have to understand that there are many helpful utility functions
available in the Utils class that can be used to help complete the function,
such as the reverse and complement functions.

The Python function that we selected for this ablation study is the following:

⬇

1def absolute_clonal(cnarr, ploidy, purity, is_reference_male,
is_sample_female

2 ):

3 """Calculate absolute copy number values from segment or bin log2 ratios."""

4 absolutes = np.zeros(len(cnarr), dtype=np.float_)

5 for i, row in enumerate(cnarr):

6 ref_copies, expect_copies = _reference_expect_copies(row.chromosome,

7 ploidy, is_sample_female, is_reference_male)

8 absolutes[i] = _log2_ratio_to_absolute(row.log2, ref_copies,

9 expect_copies, purity)

10 return absolutes

and the (abridged) context that the function absolute_clonal had access to is
the following:

⬇

1import random

2import hashlib

3import numpy as np

4import os

5

6def _reference_expect_copies(chrom, ploidy, is_sample_female,
is_reference_male

7 ):

8 """Determine the number copies of a chromosome expected and in reference.

9 For sex chromosomes, these values may not be the same ploidy as the

10 autosomes. The "reference" number is the chromosome’s ploidy in the

11 CNVkit reference, while "expect" is the chromosome’s neutral ploidy in the

12 given sample, based on the specified sex of each. E.g., given a female

13 sample and a male reference, on chromosome X the "reference" value is 1 but

14 "expect" is 2.

15 Returns

16 \-------

17 tuple

18 A pair of integers: number of copies in the reference, and expected in

19 the sample.

20 """

21 chrom = chrom.lower()

22 if chrom in [’chrx’, ’x’]:

23 ref_copies = ploidy // 2 if is_reference_male else ploidy

24 exp_copies = ploidy if is_sample_female else ploidy // 2

25 elif chrom in [’chry’, ’y’]:

26 ref_copies = ploidy // 2

27 exp_copies = 0 if is_sample_female else ploidy // 2

28 else:

29 ref_copies = exp_copies = ploidy

30 return ref_copies, exp_copies

31

32def _log2_ratio_to_absolute_pure(log2_ratio, ref_copies):

33 """Transform a log2 ratio to absolute linear scale (for a pure sample).

34 Purity adjustment is skipped. This is appropriate if the sample is germline

35 or if scaling for tumor heterogeneity was done beforehand.

36 .. math :: n = r*2^v

37 """

38 ncopies = ref_copies * 2 ** log2_ratio

39 return ncopies

40

41def _log2_ratio_to_absolute(log2_ratio, ref_copies, expect_copies,
purity=None

42 ):

43 """Transform a log2 ratio to absolute linear scale (for an impure sample).

44 Does not round to an integer absolute value here.

45 Math::

46 log2_ratio = log2(ncopies / ploidy)

47 2^log2_ratio = ncopies / ploidy

48 ncopies = ploidy * 2^log2_ratio

49 With rescaling for purity::

50 let v = log2 ratio value, p = tumor purity,

51 r = reference ploidy, x = expected ploidy,

52 n = tumor ploidy ("ncopies" above);

53 v = log_2(p*n/r + (1-p)*x/r)

54 2^v = p*n/r + (1-p)*x/r

55 n*p/r = 2^v - (1-p)*x/r

56 n = (r*2^v - x*(1-p)) / p

57 If purity adjustment is skipped (p=1; e.g. if germline or if scaling for

58 heterogeneity was done beforehand)::

59 n = r*2^v

60 """

61 if purity and purity < 1.0:

62 ncopies = (ref_copies * 2 ** log2_ratio - expect_copies * (1 - purity)

63 ) / purity

64 else:

65 ncopies = _log2_ratio_to_absolute_pure(log2_ratio, ref_copies)

66 return ncopies

67

68<<insert solution here>>

We chose this as the representative Python function because, like the Java
function that we chose, it is both highly bioinformatics related and the
solution will require an understanding of the context around it, namely the
custom _reference_expect_copies and _log2_ratio_to_absolute functions, as well
as their problem, will require the use of some of the general imports such as
the numpy package.

  1. 1.

InCoder: Similar to many other models, InCoder has very poor Java performance,
due to the fact that it was trained and fine-tuned on a Python corpus.
Therefore, InCoder achieves 0% Pass@K for the Java prompts. The main issue
that InCoder runs into is that for Java, it seems to struggle to understand
the objective in the prompt and fails to generate the actual function;
instead, it seems to regurgitate the prompt, which will obviously result in a
fail. For instance here is a sample InCoder output for the function
getReverseComplementedSequence:

⬇

1 # Do not write any comments in your code.

2#This is in java

3#Write a function called "getReverseComplementedSequence" that takes in a
SAMRecord object, an integer called "startIndex", and an integer called
"length". The function should check if startIndex is less than 0, and if it
is, set startIndex to the SAMRecord object’s read length plus startIndex. The
function should then create a byte array called "rangeBytes" that is a copy of
a portion of the SAMRecord object’s read bases starting from the startIndex
and ending at startIndex + length. The function should then call
SequenceUtil.reverseComplement on rangeBytes, and finally return a new String
created from the rangeBytes byte array.

4#Here are the imports:

5#import org.apache.commons.math3.distribution.HypergeometricDistribution;

6#import java.text.DecimalFormat;

7#import java.util.*;

8#import java.util.function.Function;

9#import htsjdk.samtools.util.SequenceUtil;

10#import java.util.Objects;

11#Here are the class declarations:

12#class SAMRecord {

13# public byte[] arr2;

14# public SAMRecord(byte[] arr) {

15# arr2 = arr;

16# }

17# public byte[] getReadBases() {

18# return arr2;

19# }

20# public int getReadLength() {

21# return arr2.length;

22# }

23#}

24#class Region {

25# /**

26# * Chromosome name

27# */

28# public final String chr;

29# /**

30# * Region start position

31# */

32# public final int start;

33# /**

34# * Region end position

35# */