[1,3]Zijun Zhang

1]Division of Artificial Intelligence in Medicine, Cedars-Sinai Medical
Center, Los Angeles, 90048, CA, USA

2]Department of Statistics, University of California, Los Angeles, Los
Angeles, 90095, CA, USA

3]Department of Computational Biomedicine, Cedars-Sinai Medical Center, Los
Angeles, 90048, CA, USA

# Efficient and Scalable Fine-Tune of Language Models for Genome Understanding

Huixin Zhan<EMAIL_ADDRESS>Ying Nian Wu<EMAIL_ADDRESS><EMAIL_ADDRESS>[  [  [

###### Abstract

Although DNA foundation models have advanced the understanding of genomes,
they still face significant challenges in the limited scale and diversity of
genomic data. This limitation starkly contrasts with the success of natural
language foundation models, which thrive on substantially larger scales.
Furthermore, genome understanding involves numerous downstream genome
annotation tasks with inherent data heterogeneity, thereby necessitating more
efficient and robust fine-tuning methods tailored for genomics. Here, we
present Lingo: Language prefix fIne-tuning for GenOmes. Unlike DNA foundation
models, Lingo strategically leverages natural language foundation models’
contextual cues, recalibrating their linguistic knowledge to genomic
sequences. Lingo further accommodates numerous, heterogeneous downstream fine-
tune tasks by an adaptive rank sampling method that prunes and stochastically
reintroduces pruned singular vectors within small computational budgets.
Adaptive rank sampling outperformed existing fine-tuning methods on all
benchmarked 14 genome understanding tasks, while requiring fewer than 2% of
trainable parameters as genomic-specific adapters. Impressively, applying
these adapters on natural language foundation models matched or even exceeded
the performance of DNA foundation models. Lingo presents a new paradigm of
efficient and scalable genome understanding via genomic-specific adapters on
language models.

###### keywords:

Pre-trained foundation models, Genome, Parameter-efficient fine-tuning,
Adaptive rank sampling

##  1 Main

DNA foundation models, such as DNABERT [1], DNABERT-$2$ [2], and Nucleotide
Transformer (NT) [3], have made significant progress in decoding the
linguistic intricacies of the genome. An important paradigm of utilizing such
DNA foundation models is “pre-training+finetuning”, i.e., pre-training on
unlabeled genomic sequences, and then adaptation to a particular genome
understanding task. A critical aspect of genome annotation and downstream
tasks is their considerable number and diversity. For example, state-of-the-
art deep learning models in epigenetics alone can encompass nearly 22,000
individual tasks [4]. This multiplicity of tasks, considered alongside the
large parameter size of these models, poses a significant challenge. As models
grow in size and complexity, the practice of full-model fine-tuning (FMFT) —
entailing the retraining of every model parameter for each task — becomes
increasingly impractical for genomic studies [5]. Using one of the largest NT
models with $2.5$B parameters as an example — deploying independent instances
of fine-tuned models, each with $2.5$B parameters, is prohibitively resource
intensive. Moreover, as the proportion of model parameters relative to
training data grows, the risk of overfitting during fine-tuning rises [6].

Computationally, there are two lines of solutions to efficiently fine-tune DNA
foundation models on a large scale: first, model compression to reduce model
size; second, parameter-efficient fine-tuning (PEFT) to add task-specific
adapters in the small-parameter regime. While model compression approaches
have been well-established in recent years, implementing them on large
language models can be very expensive, as these techniques typically
necessitate FMFT [7]. As a countermeasure, PEFTs focus on fine-tuning the
model on only a small number of additional parameters, significantly
decreasing the computational costs. In PEFTs, low-rank adaptation, e.g., low-
rank adapters (LoRA) [8] and adaptive low-rank adaptation (AdaLoRA) [9] are
increasingly prominent. While LoRA introduces fine-tuning through fixed-rank
LoRA blocks, AdaLoRA adaptively decreases the total rank of all LoRA blocks,
maintaining salient singular values based on their importance scores. Building
on the foundations of low-rank adaptation methods, KronA [10] and FedPara [11]
introduce advanced techniques: KronA replaces LoRA projections with Kronecker
factors to enhance representability, while FedPara leverages a novel re-
parameterization with low-rank weights and a Hadamard product, achieving full-
rank matrix and tensor expressivity with reduced parameters. However, all
aforementioned methods operate deterministically based on pre-pruning states,
leading to potential sub-optimal outcomes in genome understanding tasks,
particularly when these pre-pruning states are not ideal. This deterministic
nature of PEFTs becomes a significant limitation due to the considerable
heterogeneity of genomic sequences, culminating in challenges posed by
inherent data heterogeneity. For example, as illustrated in Figure 1a, the
genetic elements encompass both coding sequences and non-coding regulatory
sequences, such as histone modifications and transcription factor binding
sites. This diverse array of elements highlights the complexity and
multifaceted nature of genomic composition and regulation. Therefore, it is
essential to introduce randomness to the inherently unstable pre-pruning
states.

Echoing the substantial parameters in DNA foundation models that hamper their
downstream fine-tuning capability, their development has been limited by data
scalability issues [3, 12, 13]. GPT-3 [12], trained on approximately $45$
terabytes of text data, vastly surpasses the NT with $173.9$ billion
nucleotides in its multispecies dataset from NCBI
111https://www.ncbi.nlm.nih.gov/, thereby demonstrating a significant
difference in the scale of training data between a broad-scope pre-trained
natural language foundation model (PLM) and a specialized DNA foundation
model. Moreover, the relative constraint in genomic data availability and
diversity hinders the ability of these models to reach the levels of efficacy
and robustness that are evident in their natural language processing (NLP) and
computer vision (CV) counterparts, underscoring a critical area for future
development of DNA foundation models. For instance, whole-genome sequencing
data from 20,314 individuals in the gnomAD consortium revealed 261.9 million
distinct genetic variants, indicating a variant-to-base pair ratio of
approximately 4.4e-6 in this cohort [14]. As a consequence, the
generalizability and therefore reasoning abilities of DNA foundation models
has been questioned [15]. In contrast, PLMs have achieved remarkable progress
in NLP [16, 17] and CV [18]. More importantly, recent advances have shown that
PLMs possess surprising abilities of in-context reasoning and cross-modality
learning [19, 20, 21]. PLMs that are only trained on natural languages exhibit
compression ability of images that surpass established compression algorithms
[22]. The idea of large language models as universal compute machines has been
demonstrated effective for NLPs and images, but not yet for DNAs. Therefore,
empirically, our objective is to explore and substantially expand the domain-
shift ability of PLMs on genome understanding. Nevertheless, the venture of
PLMs into the field of genomic sequences introduces unique challenges. PLMs
are initially trained on vast amounts of general language data and thus
develop a strong understanding of natural language. However, genomic
sequences, despite being sequential and preserving complex patterns, do not
conform to the rules of human language. Moreover, the effective tokenization
and context length in genomics are still debatable [23].

Here, we introduce a new paradigm of efficient and scalable genome
understanding via genomic-specific adapters on PLMs. To adapt the PLMs to
genome domain, we develop a novel Lingo: Language prefix fIne-tuning for
GenOmes approach to prime PLMs, i.e., open pre-trained transformers (OPTs),
for genome understanding tasks. Unlike the direct input of DNA sequences,
Lingo leverages the inherent contextual learning capabilities of PLMs to guide
their transition from processing natural language to interpreting genomic
sequences, thereby recalibrates their extensive linguistic knowledge to the
intricacies of genomic sequences. Methodologically, Lingo’s adaptive rank
sampling prunes and stochastically reintroduces pruned singular vectors,
adhering to a cubic budget schedule. This technique is widely applicable
across various foundation models, particularly useful in addressing the
unstable pre-pruning frequently observed in genomics. In addition, we have
selected byte-level byte-pair encoding (BBPE) tokenization for genomic
sequences, incorporating the use of frequently occurring token IDs to
effectively train on DNA sequences.

We applied Lingo on a comprehensive set of genome understanding tasks. Across
all PLMs we tested, Lingo’s adaptive rank sampling achieves superior
performance among PEFT methods. On Lingo-trained OPT with 350M parameters,
i.e. OPT-350M, its performances matched or surpassed FMFT on all 14 genomic
sequence datasets we benchmarked, while utilizing under $2\%$ of the trainable
parameters as genomic-specific adapters. More impressively, compared to state-
of-the-art DNA foundation models trained on DNA sequences alone, Lingo also
demonstrated superior performance by applying these genomic-specific adapters
to PLMs. In the specific context of the ten datasets in histone marker
prediction task in yeast, Lingo-trained OPT-350M is among the Top-2 performed
models in 9/10 tasks, consistently surpassing either or both of DNABERT-2 and
Nucleotide Transformer. Our Lingo framework provides a powerful new strategy
for genome understanding, and a significant step for extending artificial
general intelligence in genomics.

a

b

c

d

e

Figure 1: a Heterogeneity inherent in genomic sequence data. Genome sequences
consist of 2% coding sequences and 98% non-coding sequences, including
regulatory elements, histone marks, and transcription factors, illustrating
the complexity of genomic structure. b Lingo framework for domain-shift genome
understanding. Instead of applying a full-model fine-tuning to the Pretrained
Language Model (PLM) modules (illustrated in blue), our method uses low-rank
approximation with adaptive rank sampling on Low-rank Adapters (LoRA) blocks
(depicted in yellow). These LoRA blocks are integrated with the classification
head, regulated by a scaling hyperparameter. c The adaptive rank sampling
method. In this technique, we generate masks for every singular value based on
a Bernoulli distribution. These masks are then applied to the singular values,
enabling the pruning and stochastic reintroduction of pruned singular vectors.
d Illustration of natural language prompting. These promptings encompass a
prefix and a suffix, strategically positioned around the genomic sequence. The
prefix is designed to transmit domain-specific information, while the suffix
provides annotation details. e Different methodologies for tokenizing genomic
sequences. Displayed for each row are the original genomic sequence and its
respective transformations into “words” tokenization, 6-mer tokenization, and
byte-level byte-pair encoding (BBPE) tokenization.

##  2 Results

###  2.1 Language prefix fine-tuning framework for genome understanding

We developed a language prefix fine-tuning framework, Lingo, for scalable and
effective genome understanding tasks. Unlike conventional DNA foundation
models that are only pre-trained on DNA sequence, Lingo leverages the natural
language PLMs, incorporating a genomic-specific adapter through a novel and
robust approach designed to accommodate the heterogeneity of genomic data. As
shown in Figure 1b and 1c, we introduce our adaptive rank sampling in Lingo.
Adaptive rank sampling, rather than fine-tuning the entire PLM modules $W_{0}$
(in blue), effectively reduces the rank count (Figure 1b). This approach
projects high-dimensional weight matrices into smaller subspaces, utilizing
singular value decomposition (SVD) [24] for a low-rank approximation of
gradient updates, represented as $W_{0}+\Delta W=W_{0}+P\Lambda Q$. Here,
$P\in\mathbb{R}^{d_{p}\times r}$ and $Q\in\mathbb{R}^{r\times d_{q}}$, where
the rank $r\ll\min(d_{p},d_{q})$. Fine-tuning is then exclusively conducted on
the genomic-specific adapter, specifically the matrices $P$ and $Q$, along
with the classification heads (illustrated in olive). The genomic-specific
adapter optimizes computational efficiency and model performance in high-
dimensional data contexts. Adaptive rank sampling, as depicted in Figure 1c,
involves the definition of masks $R_{k,ii}^{t}$, which are employed both for
pruning each singular value and subsequently reintroducing it. These masks are
conceptualized as random variables, each following a Bernoulli distribution
characterized by the parameter $p$, denoted as
$R_{k,ii}^{t}\sim\text{Bernoulli}(p)$. This stochastic approach introduces an
element of randomness essential for our analysis and model robustness.
Adaptive rank sampling adeptly retains crucial singular values, taking into
account both their importance and sensitivity during current batch training.

Lingo further leverages text prototypes, comprising a prefix and a suffix
flanking the genomic sequence (Figure 1d). The prefix conveys domain
information, while the suffix conveys annotation details. For the processing
of these prefixes and suffixes, we employ the BBPE tokenizer of GPT-2 [12].
Furthermore, as illustrated in Figure 1e, although various typical methods
exist for tokenizing genomic sequences, such as one-hot or k-mer tokenization,
we opt for BBPE tokenization for genomic sequences as well. This tokenizer is
well-suited for DNA sequences, adeptly capturing the frequent patterns of
nucleotides. To achieve this, the BBPE tokenizer begins with an initialization
of a dictionary containing all individual bytes in UTF-8 encoding. It then
iteratively merges the most frequently occurring token pairs. Each newly
formed pair is subsequently incorporated into the dictionary as a novel token,
a process illustrated in steps $①$ through $⑤$.

We evaluated Lingo against FMFT, LoRA, AdaLoRA, adaptive rank sampling, on
foundation models using three genome understanding tasks, i.e., histone marker
prediction in yeast, i.e., Histone (Yeast), promoter detection in human, i.e.,
Promoter (Human), and histone mark prediction in human, i.e., Histone (Human).
The statistics for all three datasets and the setup are shown in Supplementary
Table S0 and Supplementary Table S1. All Lingo in the results section reflects
language prefix fine-tuning with adaptive rank sampling.

###  2.2 Full model fine-tune of OPTs matches DNA foundation models

To demonstrate the potential of applying natural language PLMs towards genome
understanding tasks, we first performed FMFT on OPTs on previously established
benchmark genome sequence datasets [2]. The results for FMFT on different
models are shown in Figure 2a and 2b. As baseline comparisons, we report the
previously published evaluation metrics of DNA foundation models, including
DNABERT-2, and four DNA foundation models from NT, with sizes spanning from
$500$M to $2.5$B parameters. Consistent with previous evaluation metrics, the
Matthews Correlation Coefficient (MCC) [25] and Area Under the Curve (AUC)
were computed for the OPT-125M and OPT-350M models, which were fully fine-
tuned on Histone (Yeast) and Promoter (Human) tasks. The NT models have been
pre-trained on three distinct datasets: the human reference genome (HR), the
$1000$G dataset [13], and genomes from multiple species (MS); therefore they
are referred to as HR-$500$M, NT-$500$M (with $1000$G data), NT-$2.5$B (with
$1000$G data), and MS-$2.5$B.

Full-model fine-tuned OPTs achieved on par performance with DNA foundation
models. In Figure 2a, on FMFT of H3 Histone (Yeast) task, we find that the OPT
with $350$M parameters (OPT-$350$M) lies on the Pareto front (shown in black
dotted line) with two DNA foundation models, i.e., DNABERT-$2$ and MS-$2.5$B.
Specifically, the OPT-$350$M achieves a $2.5\%$ decrease in MCC while
utilizing a parameter size of $14\%$ compared to MS-$2.5$B. The OPT model with
$125$M parameters (OPT-$125$M) outperforms the DNA foundation model HR-$500$M
with $25\%$ of HR-$500$M parameters. Similarly, in Figure 2b, in the context
of FMFT for the Prom_tata Promoter (Human) task, both OPT-$125$M and
OPT-$350$M models are positioned on the Pareto front, as indicated by the
black dotted line. This performance is comparable with three additional DNA
foundation models: DNABERT-$2$, HR-$500$M, and MS-$2.5$B.

In Figures 2c and 2d, we present the average MCC and AUC for the ten Histone
(Yeast) datasets and three Promoter (Human) datasets, respectively.
Specifically, Figure 2c reveals that the average MCC for OPT-125M and OPT-350M
surpasses that of NT with 500M parameters, i.e., NT-500M, by 0.85% and 5.39%,
respectively. Similarly, Figure 2d shows that the average AUC for OPT-125M and
OPT-350M exceeds DNABERT-2 by 1.57% and 0.7%, respectively. Despite the pre-
training of OPT models on natural language datasets, their FMFT performance
proves highly competitive against DNA foundation models. This competitive edge
is likely attributed to their robust reasoning abilities, as discussed in [19,
20, 21]. In light of these findings, it is evident that the transferability of
skills learned from NLP to genomic sequence understanding holds substantial
promise.

a

b

c

d

Figure 2: a Full-model fine-tuning performance of various models on the H3
dataset. The x-axis represents the parameter size, with a lower value being
preferable, while the y-axis indicates the MCC, where higher values are
desirable. The Pareto front is marked by a dotted line, with the upper left
corner signifying models that achieve a balance of lower parameter size and
higher MCC. Notably, OPT-350M aligns with the Pareto front alongside two DNA
foundation models, DNABERT-2 and MS-2.5B. b Full-model fine-tuning performance
on the Prom_tata dataset. Here, the x-axis measures parameter size, and the
y-axis shows the AUC, with the Pareto front again indicated by a dotted line.
The upper left corner, indicating optimal performance, features both OPT-125M
and OPT-350M models. c Illustration of the average MCC on the Histone (Yeast)
dataset. OPT-125M and OPT-350M outperform the NT-500M model. d Average AUC on
the Promoter (Human) dataset. OPT-125M and OPT-350M surpassing DNABERT-2 in
performance.

###  2.3 Adaptive rank sampling enhances genomic adapters across foundation
models

We then asked if OPTs could remain competitive to DNA foundation models in the
small-parameter adapter regime using PEFT methods. Towards this end, we
benchmarked the performance of three PEFT methods across both DNA foundation
models and OPTs, while also evaluating the benefit of adaptive rank sampling
across various types of DNA foundation models and PLMs (Figure 3a \- 3g). The
Pareto front is depicted with a dotted line. Across all tests for OPTs,
adaptive rank sampling consistently outperforms the other two PEFT methods.
For instance, Figure 3c shows OPT-350M surpassing LoRA with a 2.5% increase in
AUC and 1e5 fewer trainable parameters for the Prom_tata Promoter (Human)
task. Similarly, Figure 3g illustrates OPT-350M outperforming LoRA by 1.7% in
AUC, again with 1e5 fewer trainable parameters on the same task. The superior
performance of adaptive rank sampling on OPTs is consistent across ten Histone
(Yeast) datasets and three Promoter (Human) datasets (Supplementary Figure
S2aa - Figure S2az).

Across all DNA foundation models and PLMs we tested, our adaptive rank
sampling consistently outperforms other PEFT methods for both Histone (Yeast)
and Promoter (Human) tasks. To provide a clearer illustration of the AUC and
MCC in relation to the parameter size for LoRA, AdaLoRA and adaptive rank
sampling, we have detailed these metrics of Histone (Yeast) in Supplementary
Table S2 and Promoter (Human) in Supplementary Table S3.

When applied to DNA foundation models, the performance of adaptive rank
sampling aligns with the Pareto front observed in FMFT outcomes and is
comparable to that of FMFT. As an example, in Supplementary Table S3, adaptive
rank sampling posts a 0.926 AUC with merely 6.9M trainable parameters (only
1.3% of FMFT’s parameters), whereas FMFT achieves a 0.95 AUC using 500M
trainable parameters for NT-500M model. Supplementary Table S2 shows the MCCs
for various models and methods on the Histone (Yeast) task, where adaptive
rank sampling’s performance lies on a Pareto front with that of FMFT. This
demonstrates that adaptive rank sampling, by accommodating the unstable pre-
pruning states in genomic data, learns effective genomic-specific adapters
across all foundation models.

a

b

c

d

e

f

g

h

Figure 3: a, b, c, and d: Relationship between the AUC and parameter size for
the Prom_tata dataset within the Promoter (Human) task, focusing on DNABERT-2,
OPT-125M, OPT-350M, and NT-500M models. Panels a, b, c, and d are arranged in
ascending order of parameter size, ranging from 117M to 500M, to facilitate a
comparative analysis based on model complexity. In each panel, models
positioned in the upper-left indicate superior performance, characterized by
higher AUC and lower parameter size. e, f, g, and h: Illustration of the MCC
versus parameter size for the H4 dataset in the Histone (Yeast) task, again
examining the same set of models: DNABERT-2, OPT-125M, OPT-350M, and NT-500M.

a

b

c

d

e

f

Figure 4: a MCC comparisons between adaptive rank sampling and Lingo across
ten datasets on Histone (Yeast) for the OPT-125M model. A one-sided paired
t-test was performed between Lingo and adaptive rank sampling. b MCC between
adaptive rank sampling and Lingo is showed for the OPT-350M model on Histone
(Yeast). A one-sided paired t-test was performed. c Average AUC on the
Promoter (Human) task for five different methods – full-model fine-tune, LoRA,
AdaLoRA, adaptive rank sampling, and Lingo – applied to a variety of models
including DNABERT-2, NT-500M, OPT-125M, and OPT-350M. d Comparison of the
average MCC on Histone (Yeast) for the same five methods across various
models. e Frequency of OPT-125M model registers Top-2 performance across four
PEFT methods: LoRA, AdaLoRA, adaptive rank sampling, and Lingo. The solid bars
in the graph represent the count of datasets where a method achieved top-2
performance, while the hatched bars indicate the total number of datasets
considered. f Performance of the OPT-350M model with the same four PEFT
methods.

###  2.4 Leveraging prompts for in-context learning in pre-trained natural
language models

To further leverage the in-context learning ability and address the NLP
pretraining of OPTs, we applied and evaluated Lingo on OPTs. Lingo adeptly
utilizes contextual signals inherent in PLMs, effectively reorienting their
capabilities from standard natural language processing tasks to the intricate
and specialized field of genomic sequence analysis. This redirection is
achieved by fine-tuning PLMs to recognize and interpret the unique patterns
and structures present in genomic data, thereby extending their applicative
scope beyond conventional linguistic tasks to the nuanced domain of genomics.
We conducted further tests to determine whether Lingo demonstrates superior
performance over DNA foundation models and other PEFT methods, including
adaptive rank sampling alone. Notably, we define Lingo as applying adaptive
rank sampling with language prefix exclusively on natural language PLMs, while
adaptive rank sampling alone can be applied to both PLMs and DNA foundation
models.

On both OPT-125M and OPT-350M, Lingo consistently outperforms adaptive rank
sampling alone. In Figures 4a and 4b, both radar figures present the MCC
across ten datasets for the Histone (Yeast) task. The performances of adaptive
rank sampling alone and Lingo are represented by orange and blue areas,
respectively. A one-sided paired t-test was performed between these two
methods, revealing that Lingo significantly outperforms adaptive rank
sampling, with P-values of 0.013 and 0.005 for both tests, respectively.
Additionally, these tests were extended to three datasets in Promoter (Human),
with corresponding results presented in Supplementary Figures S2ba -
Supplementary Figures S2bb. Details of these metrics can be found in
Supplementary Table S2 and Supplementary Table S3. For example, in
Supplementary Table S2, Lingo-trained OPT-125M achieves an MCC of 0.735 for
the H4 dataset. In comparison, AdaLoRA and LoRA attain AUCs of 0.734 and
0.725, respectively, on the Histone (Yeast) task. In Supplementary Table S3,
Lingo-trained OPT-125M achieves an AUC of 0.98 for the prom_notata dataset. In
comparison, FMFT, AdaLoRA, and LoRA attain AUCs of 0.947, 0.931, and 0.907,
respectively, on the Promoter (Human) task.

Lingo also outperforms other PEFT methods in terms of average performances for
both tasks. Specifically, Figure 4c shows that in the OPT-350M model, Lingo
surpasses adaptive rank sampling by 0.9%. Moreover, as illustrated in Figure
4d, Lingo demonstrates superior performance over adaptive rank sampling in
both OPT-125M and OPT-350M models, with improvements of 2.2% and 2.6%,
respectively.

We highlight the Lingo performances of OPTs in comparison to DNA foundation
models (Figure 4e and Figure 4f). Compared to the two DNA foundation models,
i.e. DNABERT-2 and NT-500M, Lingo-trained OPTs are consistently among the
Top-2 for Promoter (Human) and Histone (Yeast) tasks. This is significant,
because PEFT is computationally efficient than training DNA foundation models
from scratch, while a small-parameter genomic adapter on OPTs consistently
outperforms at least one of such DNA foundation models. For the three datasets
in Promoter (Human) task, Lingo-trained OPT-125M is among the Top-2 performed
models in 3/3 tasks in Figure 4e. Similarly, in Figure 4f, for the ten
datasets in Histone (Yeast) task, Lingo-trained OPT-350M is among the Top-2
performed models in 9/10 tasks.

###  2.5 One-hot encoding addresses the challenge of semantic disambiguation

We sought to investigate the effect of semantic ambiguity in our framework,
and whether a DNA-specific tokenization method could improve our framework’s
performance via mitigating semantic ambiguity. So far for the previous PLMs,
BBPE tokenization is utilized for encoding genomic sequences. Following the
aggregation of the most frequent pairs, a notable observation is the potential
overlap of token identifiers between genomic sub-sequences and conventional
English lexicon. However, it is important to acknowledge that the semantic
nature of these genomic sub-sequences, compared to their linguistic lexicon,
is fundamentally different. Thus, we hypothesized that the coexistence of
multiple semantic interpretations within the same token set could adversely
impact downstream task efficacy. To further investigate this hypothesis, we
apply an additional experimental approach that utilizes simple one-hot
encoding for genomic sequence representations.

Our experiments demonstrate that semantic disambiguation does not decrease
Lingo’s performance; in contrast, one-hot encoding achieved subpar performance
compared to BBPE, likely due to its lack of ability to capture contextual
information. In Table 1, we show the performances for Lingo for BBPE
tokenization, i.e., Lingo+BBPE, and one-hot encoding,i.e. Lingo+One-hot,
respectively, for Promoter (Human) task. For example, in the prom_notata
dataset, the OPT-125M model with Lingo+BBPE attains an AUC of 0.98. In
comparison, the same model utilizing Lingo+One-hot achieves a commendable AUC
of 0.971. The findings indicate that while one-hot encoding does not surpass
BBPE tokenization in AUC, it nonetheless demonstrates sufficient efficacy.
This can be attributed to its proficiency in mitigating the ambiguities
arising from semantic multiplicity in PLMs. Therefore, our study concludes
that despite the relative simplicity of one-hot encoding, which may not
comprehensively capture contextual nuances and dependencies inherent in
genomic sequences, it effectively addresses the challenge of semantic
disambiguation, thereby necessitating future investigations of effective
tokenization methods in our framework.

Table 1: AUC comparisons between two tokenization methods used with Lingo on the OPT-125M model: one-hot tokenization (Lingo \+ One-hot) and BBPE tokenization (Lingo \+ BBPE).  Model |  Method |  Prom_all |  Prom_notata |  Prom_tata
---|---|---|---|---
OPT-$125$M |  Lingo \+ BBPE |  $0.954$ |  $0.980$ |  $0.895$
|  Lingo \+ One-hot |  $0.921$ |  $0.971$ |  $0.850$
OPT-$350$M |  Lingo \+ BBPE |  $0.957$ |  $0.983$ |  $0.890$
|  Lingo \+ One-hot |  $0.947$ |  $0.960$ |  $0.807$

###  2.6 Efficient and accurate genome-scale prediction by language prefix
fine-tuning

a

b

c

d

e

Figure 5:  a Scaling law of the number of attention computations with
increasing sequence length across three tasks: Promoter (Human), Histone
(Yeast), and Histone (Human). Notably, the increase in attention computations
is quadratic in relation to the sequence length. b Macro AUC comparison among
three models – NT-500M, OPT-125M, and OPT-350M – under FLOPs-adjusted training
steps, demonstrating that OPT-350M outperforms OPT-125M, which in turn
surpasses NT-500M, within the same training steps. c Comparison of AUC between
OPT-125M and NT-500M for each label in the Histone (Human) task. d Comparison
of AUC between OPT-125M and OPT-350M for each label on Histone (Human). e
Pearson correlation of multitask learning performance for three models:
OPT-125M, OPT-350M, and NT-500M.

Lastly, we demonstrate Lingo can efficiently scale up to whole-genome scale
multitask learning, achieving superior prediction performance within a fixed
computational budget compared to Nucleotide Transformer. For each 1000-bp DNA
sequence in this task, over 100 cell-line specific histone modification
markers, annotated based on the ENCODE datasets [26], were previously compiled
by DeepSEA [27]. As genome-scale high-throughput sequencing assays become a
routine to probe biological systems [28], this represents a realistic scenario
towards deploying DNA foundation models in academic labs, where specific
biological questions are of interest, yet huge computational resources to
fine-tune DNA foundation models may be restricted. In Figure 5a, we initially
illustrate that the Histone (Human) task presents significantly greater
complexity compared to both the Histone (Yeast) and Promoter (Human) tasks.
This figure delves into the computational complexity inherent in the self-
attention mechanism of transformer models, a topic frequently noted for its
quadratic nature. The complexity is a direct consequence in which self-
attention processes the input sequence. As a notable observation from this
figure, even a modest increase in the input sequence length from 500 to 1000
leads to a substantial escalation in the number of attention computations,
rising to as much as 1e9.

Compared to the NT-500M model, both Lingo-trained OPT-125M and OPT-350M
demonstrate superior performance and greater computational efficiency. We
applied Lingo to OPTs and adaptive rank sampling (as language prefix is not
applicable to DNA foundation models) to the NT-500M on the Histone (Human)
task, while all models were trained under a fixed budget of 6 ranks per layer
on average. To ensure a fair comparison, we adjusted the computational costs
of all three foundation models based on their training FLOPs. This approach
allowed for a direct comparison of the models’ performance efficacy under a
uniform computational constraint, providing an equitable assessment of each
model’s capabilities when adjusted for FLOPs efficiency. The FLOPs efficiency
of OPT-125M, denoted as FLOPs${}_{\text{OPT}-125\text{M}}$, was designated as
the foundational benchmark for computational efficiency. Figure 5b reveals
that although the OPT-125M model’s final performance is lower than that of the
350M model, it demonstrates greater computational efficiency. This is
evidenced by its higher Macro AUC at FLOPs-adjusted training steps. Similarly,
when comparing with the NT-500M model, both OPT-125M and OPT-350M exhibit
superior performance, indicating heightened computational efficiency, e.g.,
OPT-125M achieves $4.443\times$ efficiency (via
$\frac{\text{FLOPs}_{\text{OPT-125M}}}{\text{FLOPs}_{500\text{M}-1000\text{G}}}$)
compared to NT-500M. Consequently, our analysis leads to the conclusion that
PLMs trained by Lingo, in comparison to the NT-500M model, are more
computationally efficient.

Using our Lingo framework, OPT-350M achieved the highest accuracy on this
genome-scale understanding task. As shown in Figure 5c, the Lingo-trained
OPT-125M model achieves an AUC of 0.757, already surpassing the 0.744 AUC
achieved by the NT-500M model. Furthermore, Figure 5d demonstrates that Lingo-
trained OPT-350M, with an AUC of 0.774, outperforms OPT-125M. These results
indicate that Lingo-trained PLMs exhibit more accurate performance over
NT-500M model in multi-label classification tasks within the Histone (Human),
with the improvement of OPT-350M concentrated in the poorly-performed tasks in
OPT-125M (bottomleft corner, Figure 5d). Finally, we analyzed the Pearson
correlation coefficients [29] of the AUC across 104 labels in the Histone
(Human) for the three different models (Figure 5e). This figure illustrates
that the Pearson correlation coefficients for the OPT models (OPT-125M and
OPT-350M) are more closely aligned with each other, indicating a higher degree
of similarity in their performance patterns. In contrast, the correlation
between OPT models and NT-500M is less pronounced. This disparity suggests
that the Lingo-trained OPT models, despite variations in their sizes, tend to
follow a more consistent performance trend compared to the divergence observed
when compared with the NT-500M model.

##  3 Discussion

Recently, there have been significant advancements in the field of genomic
domain, attributed to the development of DNA foundation models such as
DNABERT-2 [2] and Nucleotide Transformer (NT) [3]. However, the development of
these models in the domain of genomics has been hindered by challenges in data
scalability. This contrasts with the significant advancements made by pre-
trained foundation models in fields such as NLP and CV, where they have
demonstrated remarkable progress. To effectively adapt PLMs for applications
in the genome domain, we introduce a novel Lingo method designed to prime
PLMs, specifically OPT models, for genome understanding tasks. This approach
diverges from the straightforward input of DNA sequences. Instead, Lingo
harnesses the inherent contextual learning capabilities of PLMs. It guides
their transition from processing natural language to interpreting genomic
sequences. By inserting text prefix and suffix, this strategy enables PLMs to
recalibrate their extensive linguistic knowledge to the unique complexities of
genomic sequences, leveraging their existing expertise in a novel context.

In the realm of PEFT, additive fine-tuning, such as prefix tuning [30, 31] and
prompt tuning methods [32], in PLMs involves adding new parameters to the
existing model, enabling it to learn additional information while retaining
its original knowledge. Partial fine-tuning, such as SAM [33], in contrast,
adjusts only a subset of the model’s existing parameters, aiming for quicker
adaptation to specific tasks with less computational cost.

Reparameterization-based fine-tuning offers a distinct advantage over these
methods. By reparameterizing the model’s weights, it allows for a more
flexible and efficient adaptation to new tasks without the need for extensive
additional parameters or the risk of overwriting valuable pre-learned
representations. Within this reparameterization scheme, low-rank adaptation
methods, have garnered considerable attention. While each approach presents
unique advantages, a comparative analysis reveals certain limitations inherent
to prefix and prompt tuning methods when contrasted with low-rank adaptation
strategies. Predominantly, prefix and prompt tuning methods exhibit
constraints in their capacity for extensive model reconfiguration. This
limitation stems from their focus on input-level modifications, which may not
suffice for tasks necessitating deeper model transformations. Additionally,
these methods display a pronounced dependency on the nature of pre-training
tasks, potentially limiting their effectiveness in scenarios vastly divergent
from initial training contexts. While low-rank adaptation is a well-
established method in PLMs, there is a notable absence of mature applications
of PLMs in DNA foundation models. DNABERT-2 employs low-rank adapters [8] with
a fixed number of ranks to curtail the quantity of trainable parameters.
However, deterministic PEFTs can yield sub-optimal performance when the
initial states are less than ideal. To solve this issue, we propose an
adaptive rank sampling to prune and stochastically reintroduce pruned singular
vectors. Lingo demonstrates superior potential. Our empirical observations on
three genome understanding tasks demonstrate that OPT-350M, when combined with
adaptive rank sampling, positions itself on the Pareto front in comparison to
its full-model fine-tuning baseline, utilizing merely $0.94\%$ of the
trainable parameters. Interestingly, we also observe that PLMs exhibit
superior performance and are more computationally efficient in multi-label
classification on Histone (Human). For instance, OPT-125M achieves an
efficiency that is $4.443\times$ greater than that of NT-500M.

##  4 Methods

###  4.1 Datasets

The sequences for Histone (Yeast) and Promoter (Human) are obtained from the
GUE framework [2] and the sequences for Histone (Human) are extracted from the
DeepSEA framework [27]. The detailed statistics for the three tasks are shown
in Supplementary Table S0.

###  4.2 BBPE tokenization for DNA sequences

In Figure 1e, we present three popular tokenizers for DNA sequences, labeled
as ($1$), ($2$), and ($3$). The “words” tokenizer employs a dictionary derived
from the four nucleotides. The tokenized sequence length of an input DNA
equates to the number of nucleotides. However, this method lacks contextual
information. In contrast, the $6$-mer tokenizer is gaining popularity in DNA
sequencing [23]. The concept of $k$-mer revolves around extracting continuous
subsequences of $k$ nucleotides from a DNA sequence. However, one drawback of
$k$-mer tokenization is the increased computational complexity, especially
with larger $k$ values. To address this, the BBPE tokenizer initializes a
dictionary consisting of all individual bytes in UTF-8 encoding. It
progressively selects the most frequent pairs of tokens to merge. Each
combined pair is then added to the dictionary as a new token (shown in $①-⑤$).
OPTs are tailored with GPT-2’s BBPE tokenizer [34]. This tokenizer is well-
suited for DNA sequences because it efficiently captures the recurring
patterns of nucleotides. By focusing on the frequency of specific sequences,
it offers a nuanced encoding that can illuminate biological motifs. This
dictionary only consists of three tokens: “AAC”, “TC”, and “GA”. In our
experiments, we conducted Lingo combined with one-hot/BBPE tokenization. This
was done to investigate whether one-hot encoding effectively resolves the
challenge of semantic disambiguation, particularly concerning potential
overlaps between genomic sub-sequence token identifiers and those found in the
conventional English lexicon in BBPE.

###  4.3 Domain-shift genome understanding

In this subsection, we introduce Lingo, importance scores computation, and the
cubic budget schedule with adaptive rank sampling

####  4.3.1 Language prefix fine-tuning

Next, we introduce a novel Lingo approach. This approach primes PLMs,
particularly OPTs, to tackle genome understanding tasks. Through Lingo, the
model is adeptly fine-tuned. Central to our methodology is the hypothesis that
the integration of domain-specific prompts, such as "Domain: DNA Promoter",
markedly enhances the model’s ability to discern the unique rules and patterns
inherent in genomic sequences, distinct from those in natural language. To
implement this, we append a prefix "Domain: DNA Promoter\nSequence: " and a
suffix "\nAnnotation:" to the DNA sequence inputs, formatted in Markdown to
ensure clarity and structure.

In this example, text highlighted in pink remains constant, while the text in
teal, sequences, and labels are variables, tailored to specific genomic
instances. This representation not only facilitates a clear demarcation of the
input components but also reflects the flexible nature of our Lingo in
accommodating diverse genome understanding tasks.

####  4.3.2 Importance score

PLMs contain many weight matrices to perform matrix multiplication. These
weight matrices typically have full-rank. However, performing FMFT is not
efficient. Thus, our goal is to reduce the number of ranks to project the high
dimensional weights matrices to smaller subspaces. Mathematically, for a pre-
trained weight matrix $W_{0}\in\mathbb{R}^{d_{p}\times d_{q}}$, we approximate
the gradient updates using SVD for a low-rank representation, i.e.,
$W_{0}+\Delta W=W_{0}+P\Lambda Q$, where $P\in\mathbb{R}^{d_{p}\times r}$,
$Q\in\mathbb{R}^{r\times d_{q}}$, and the rank $r\ll\min(d_{p},d_{q})$. Thus,
for $n$ matrices in the PLM, we need to perform SVD: $\Delta
W_{k}=P_{k}\Lambda_{k}Q_{k}$ for $k=1,...,n$. Our importance score computation
considers the importance from both singular values, which capture the
magnitude of changes and indicate dominant variation directions, and singular
vectors, which denote their orientations. Thus, each triplet
$\Sigma_{i}=\\{\lambda_{k,i},P_{k,*i},Q_{k,i*}\\}$ is constructed with the
$i$-th singular value $\lambda_{k,i}$ and the corresponding singular vectors
$P_{k,*i},Q_{k,i*}$. The importance score for each singular value is then
computed as [35]:

$S_{k_{i}}=s(\lambda_{k,i})+\frac{1}{d_{p}}\sum_{j=1}^{d_{p}}s(P_{k,ji})+\frac{1}{d_{q}}\sum_{j=1}^{d_{q}}s(Q_{k,ij}).$
(1)

At each time step $t$, each entry in the matrix is associated with an
importance score, computed as the product of its sensitivity and uncertainty,
i.e., $s^{t}(w_{ij})=\bar{I}^{t}(w_{ij})\bar{U}^{t}(w_{ij}),$ where
$\bar{I}^{t}(w_{ij})$ denotes the stabilized sensitivity and
$\bar{U}^{t}(w_{ij})$ represents the stabilized uncertainty. These stabilized
scores refine the original scores through a weighted adjustment. The updating
rules for $\bar{I}^{t}(w_{ij})$ and $\bar{U}^{t}(w_{ij})$ are:

$\bar{I}^{t}(w_{ij})=\beta_{1}\bar{I}^{t-1}(w_{ij})+(1-\beta_{1})I^{t}(w_{ij}),$
(2)

and

$\bar{U}^{t}(w_{ij})=\beta_{2}\bar{U}^{t-1}(w_{ij})+(1-\beta_{2})|I^{t}(w_{ij})-\bar{I}^{t-1}(w_{ij})|,$
(3)

where $I^{t}(w_{ij})=|w_{ij}\bigtriangledown_{w_{ij}}\mathcal{L}^{t}|$ and
$\mathcal{L}^{t}$ denotes the binary cross-entropy for a batch of data
$\mathcal{D}$. Therefore, the sensitivity captures how much the loss responds
to changes in a specific weight within a training batch. In contrast,
uncertainty quantifies the fluctuations in the loss, given by
$U^{t}(w_{ij})=|I^{t}(w_{ij})-\bar{I}^{t-1}(w_{ij})|.$ The importance score is
computed by balancing the two factors via
$\bar{I}^{t}(w_{ij})\bar{U}^{t}(w_{ij})$.

####  4.3.3 Cubic budget schedule with adaptive rank sampling

We introduce a global budget, $b_{t}$, which diminishes following a cubic
budget schedule defined as $b^{t}=b^{T}+(b^{0}-b^{T})(1-\frac{t}{T})^{3}$. For
the adaptive rank sampling process, we define masks $R_{k,ii}^{t}$ for pruning
$\lambda_{k}^{t}$ and re-introducing the pruned $\lambda_{k}^{t}$. These masks
are random variables derived from a Bernoulli distribution with parameter $p$,
$R_{k,ii}^{t}\sim\text{Bernoulli}(p)$. The singular values to be retained are
updated based on the following updating rule:

$\hat{\Lambda}_{k,ii}^{t}=\begin{cases}\Lambda_{k,ii}^{t}\cdot(1-R_{k,ii}^{t})&\text{if
}S_{k,i}^{t}\text{ is in the top }b^{t}\text{ of }S^{t},\\\
\Lambda_{k,ii}^{t}\cdot R_{k,ii}^{t}&\text{otherwise.}\end{cases}$ (4)

Note that we here denote $\lambda_{k}^{t}$ as $\Lambda_{k,ii}^{t}$ to more
conveniently assign masks based on their position index $i$. Only those
singular values, $\hat{\Lambda}_{k,ii}^{t}$, that meet both the adaptive rank
sampling criteria and have their importance score $S_{k_{i}}^{t}$ is in the
top $b^{t}$ of all scores $S^{t}$ are retained. By introducing randomness
rather than strictly adhering to a deterministic cutoff, the process becomes
more robust against potential suboptimal initial states and inaccuracies in
importance scores.

In Algorithm 1, we summarize the steps for the adaptive rank sampling
algorithm. For each timestep $t$, we first compute the binary cross-entropy
loss $\mathcal{L}^{t}$ for a batch of data $\mathcal{D}$. Then, every $\Delta
T$ timesteps, we compute the importance score for each $k$ and $i$, and update
$P_{k}^{t}$ and $Q_{k}^{t}$. Finally, we update the singular values
$\hat{\Lambda}_{k,ii}^{t}$ based on the cubic budget schedule with adaptive
rank sampling.

Algorithm 1 Adaptive rank sampling

1:A batch of data $\mathcal{D}$; budget $b_{t}$; hyper-parameters
$\eta,\lambda,\beta_{1},\beta_{2}$; final timesteps $T_{\text{final}}$;
timesteps $T$ and $\Delta T$ for low rank approximation

2:$\Delta W_{k}$

3:for $t=1,...,T_{\text{final}}$ do

4: Compute the binary cross-entropy loss $\mathcal{L}^{t}$ for a batch of data
$\mathcal{D}$

5: if $t\,\%\,\Delta T=0$ and $t<T$ then

6: Compute the stabilized sensitivity $\bar{I}^{t}(w_{ij})$ via Equation (2)
and uncertainty $\bar{U}^{t}(w_{ij})$ via Equation (3)

7: Compute $S_{k_{i}}$ for all $k$ and $i$ via Equation (1)

8: Update $P_{k}^{t}=P_{k}^{t-1}-\eta\nabla_{P_{k}}\mathcal{L}^{t}-\lambda
P_{k}^{t-1}$

9: Update $Q_{k}^{t}=Q_{k}^{t-1}-\eta\nabla_{Q_{k}}\mathcal{L}^{t}-\lambda
Q_{k}^{t-1}$

10: Update
$\Lambda_{k,ii}^{t}=\Lambda_{k}^{t-1}-\eta\nabla_{\Lambda_{k}}\mathcal{L}^{t}-\lambda\Lambda_{k}^{t-1}$

11: Update $\hat{\Lambda}_{k,ii}^{t}$ via Equation (4)

12: end if

13:end for

###  4.4 Analysis setup

For FMFT, LoRA, AdaLoRA, and adaptive rank sampling, we use a batch size of
$8$, while the evaluation batch size is set to $16$. The model is trained over
$5$ epochs on all datasets for both tasks. We employ the Adam optimizer [36],
with a learning rate of $\eta=3e-5$. Additionally, we implement a warmup ratio
of $0.1$ followed by linear decay. The L2 regularization weight decay is set
at $\lambda=5e-3$. For LoRA, we set the rank =$8$. For AdaLoRA and adaptive
rank sampling, the additional hyper-parameters are shown in Supplementary
Table S1. In this Table, Avg. $b_{0}$ and Avg. $b_{T}$ denote the average
number of singular values in each matrix.
$(T_{\text{total}}-T)/T_{\text{total}}$ denotes the final fine-tune ratio
after pruning and reintroducing, and $\Delta T$ indicates the intervals at
which pruning and reintroducing are performed. The additoinal hyper-parameters
for AdaLoRA are shown in gray columns and the additoinal hyper-parameters for
adaptive rank sampling is shown in all columns including the random sampling
ratio $p$.

In the Histone (Human) dataset, to facilitate a rigorous comparison, we
calibrated the performance metrics of OPT-350M and NT-500M, which require
FLOPs${}_{\text{OPT}-350\text{M}}$ and FLOPs${}_{500\text{M}-1000\text{G}}$
respectively. This calibration involved adjusting their performance steps to
the computational effort equivalent to FLOPs${}_{\text{OPT-125}\text{M}}$,
thereby normalizing their outputs to a unified computational standard. We
calibrated the performance metrics of OPT-350M and NT-500M for a rigorous
comparison, , which require FLOPs${}_{\text{OPT}-350\text{M}}$ and
FLOPs${}_{500\text{M}-1000\text{G}}$ respectively. This calibration involved
adjusting their performance steps to the computational effort equivalent to
FLOPs${}_{\text{OPT-125}\text{M}}$, thereby normalizing their outputs to a
unified computational cost. Specifically, the FLOPs-adjusted training steps
($I$) for OPT-350M and NT-500M, denoted as $n_{\text{OPT-350M}}$ and
$n_{500\text{M}-1000\text{G}}$ respectively, were computed as follows:
$I_{\text{OPT-350M}}=I_{\text{OPT-125M}}\times\frac{\text{FLOPs}_{\text{OPT-125M}}}{\text{FLOPs}_{\text{OPT-350M}}}$
and
$I_{500\text{M}-1000\text{G}}=I_{\text{OPT-125M}}\times\frac{\text{FLOPs}_{\text{OPT-125M}}}{\text{FLOPs}_{500\text{M}-1000\text{G}}}.$
This methodology allowed us to directly compare the performance efficacy of
all three models under a common computational constraint, thereby ensuring an
equitable assessment of each model’s capabilities within the parameters of
FLOPs-adjusted efficiency. In this tasks, we evaluate the validation loss
every $1e4$ step, and if the best validation loss has not decreased for $10$
evaluations, we early stop the fine-tuning process.

##  5 Data Availability

The sequences for Histone (Yeast) and Promoter (Human) are obtained from the
GUE framework [2] and the sequences for histone marks prediction in multiple
cell types are extracted from the DeepSEA framework [27].

##  6 Code Availability

Our Lingo code is available on Github at https://github.com/zhanglab-
aim/LINGO.

##  7 Supplementary Information

###  7.1 Datasets and setting-up

Table S0: Genome understanding tasks Task | Num. Datasets | Class | Num. Classes | Sequence Length
---|---|---|---|---
Histone (Yeast) | 10 | Binary | 2 | 500
Promoter (Human) | 3 | Binary | 2 | 300
Histone (Human) | 104 | Multi-label | 104 | 1000

Table S1: Additoinal hyper-parameters for AdaLoRA (in gray) and adaptive rank sampling (all columns). Task | Dataset |  Avg. $b_{0}$ |  Avg. $b_{T}$ |  $(T_{\text{total}}-T)/T_{\text{total}}$ |  $\Delta T$ |  $\beta_{1}$ |  $\beta_{2}$ |  Pruned Matrices |  $p$
---|---|---|---|---|---|---|---|---|---
Histone (Yeast) | H3 |  $12$ |  $6$ |  $0.25$ |  $100$ |  $0.85$ |  $0.85$ |  $W_{q}$, $W_{k}$, $W_{v}$, $W_{f_{1}}$, $W_{f_{2}}$ |  $0.05$
H3K4me1 |  $12$ |  $6$ |  $0.25$ |  $100$ |  $0.85$ |  $0.85$ |  $W_{q}$, $W_{k}$, $W_{v}$, $W_{f_{1}}$, $W_{f_{2}}$ |  $0.05$
H3K4me2 |  $12$ |  $6$ |  $0.25$ |  $100$ |  $0.85$ |  $0.85$ |  $W_{q}$, $W_{k}$, $W_{v}$, $W_{f_{1}}$, $W_{f_{2}}$ |  $0.05$
H3K4me3 |  $12$ |  $6$ |  $0.25$ |  $100$ |  $0.85$ |  $0.85$ |  $W_{q}$, $W_{k}$, $W_{v}$, $W_{f_{1}}$, $W_{f_{2}}$ |  $0.05$
H3K9ac |  $12$ |  $6$ |  $0.25$ |  $100$ |  $0.85$ |  $0.85$ |  $W_{q}$, $W_{k}$, $W_{v}$, $W_{f_{1}}$, $W_{f_{2}}$ |  $0.05$
H3K14ac |  $12$ |  $6$ |  $0.25$ |  $100$ |  $0.85$ |  $0.85$ |  $W_{q}$, $W_{k}$, $W_{v}$, $W_{f_{1}}$, $W_{f_{2}}$ |  $0.05$
H3K36me3 |  $12$ |  $6$ |  $0.30$ |  $100$ |  $0.85$ |  $0.85$ |  $W_{q}$, $W_{k}$, $W_{v}$, $W_{f_{1}}$, $W_{f_{2}}$ |  $0.05$
H3K79me3 |  $12$ |  $6$ |  $0.30$ |  $100$ |  $0.85$ |  $0.85$ |  $W_{q}$, $W_{k}$, $W_{v}$, $W_{f_{1}}$, $W_{f_{2}}$ |  $0.05$
H4 |  $12$ |  $6$ |  $0.30$ |  $100$ |  $0.85$ |  $0.85$ |  $W_{q}$, $W_{k}$, $W_{v}$, $W_{f_{1}}$, $W_{f_{2}}$ |  $0.05$
H4ac |  $12$ |  $6$ |  $0.30$ |  $100$ |  $0.85$ |  $0.85$ |  $W_{q}$, $W_{k}$, $W_{v}$, $W_{f_{1}}$, $W_{f_{2}}$ |  $0.05$
Promoter (Human) | Prom_all |  $8$ |  $6$ |  $0.15$ |  $5000$ |  $0.85$ |  $0.85$ |  $W_{q}$, $W_{k}$, $W_{v}$, $W_{f_{1}}$, $W_{f_{2}}$, $W_{o}$ |  $0.1$
Prom_notata |  $8$ |  $6$ |  $0.15$ |  $5000$ |  $0.85$ |  $0.85$ |  $W_{q}$, $W_{k}$, $W_{v}$, $W_{f_{1}}$, $W_{f_{2}}$, $W_{o}$ |  $0.1$
Prom_tata |  $8$ |  $6$ |  $0.15$ |  $5000$ |  $0.85$ |  $0.85$ |  $W_{q}$, $W_{k}$, $W_{v}$, $W_{f_{1}}$, $W_{f_{2}}$, $W_{o}$ |  $0.1$
Histone (Human) | Histone |  $8$ |  $6$ |  $0.15$ |  $5000$ |  $0.99$ |  $0.99$ |  $W_{q}$, $W_{k}$, $W_{v}$, $W_{f_{1}}$, $W_{f_{2}}$, $W_{o}$ |  $0.1$

###  7.2 Results

a DNABERT-2 on Prom_all

b NT-500M on Prom_all

c OPT-125M on Prom_all

d OPT-350M on Prom_all

e DNABERT-2 on Prom_notata

f NT-500M on Prom_notata

g OPT-125M on Prom_notata

h OPT-350M on Prom_notata

i DNABERT-2 on Prom_tata

j NT-500M on Prom_tata

k OPT-125M on Prom_tata

l OPT-350M on Prom_tata

m DNABERT-2 on H3

n NT-500M on H3 (Yeast)

o OPT-125M on H3

p OPT-350M on H3

q DNABERT-2 on H3K4me1

r NT-500M on H3K4me1

s OPT-125M on H3K4me1

t OPT-350M on H3K4me1

a DNABERT-2 on H3K4me2

b NT-500M on H3K4me2

c OPT-125M on H3K4me2

d OPT-350M on H3K4me2

e DNABERT-2 on H3K4me3

f NT-500M on H3K4me3

g OPT-125M on H3K4me3

h OPT-350M on H3K4me3

i DNABERT-2 on H3K9ac

j NT-500M on H3K9ac

k OPT-125M on H3K9ac

l OPT-350M on H3K9ac

m DNABERT-2 on H3K14ac

n NT-500M on H3K14ac

o OPT-125M on H3K14ac

p OPT-350M on H3K14ac

q DNABERT-2 on H3K36me3

r NT-500M on H3K36me3

s OPT-125M on H3K36me3

t OPT-350M on H3K36me3

a DNABERT-2 on H3K79me3

b NT-500M on H3K79me3

c OPT-125M on H3K79me3

d OPT-350M on H3K79me3

e DNABERT-2 on H4

f NT-500M on H4

g OPT-125M on H4

h OPT-350M on H4

i DNABERT-2 on H4ac

j NT-500M on H4ac

k OPT-125M on H4ac

l OPT-350M on H4ac

m AUC for OPT-125M on the Promoter (Human) task.

n AUC for OPT-350M on the Promoter (Human) task.

o MCC for OPT-125M on the Histone (Yeast) task.

p MCC for OPT-350M on the Histone (Yeast) task.

Figure S2: AUCs for various models and methods on the Promoter (Human) task.

Table S2: MCCs for various models and methods on the Histone (Yeast) task.

Model |  Method |  # Train. Params. |  H3 |  H3K4me1 |  H3K4me2 |  H3K4me3 |  H3K9ac |  H3K14ac |  H3K36me3 |  H3K79me3 |  H4 |  H4ac
---|---|---|---|---|---|---|---|---|---|---|---|---
|  |  |  |  |  |  |  |  |  |  |  |
DNABERT-$2$ |  FMFT1 |  $117$M |  $\mathit{0.783}$ |  $\mathbf{0.505}$ |  $\mathit{0.311}$ |  $\mathbf{0.363}$ |  $\mathbf{0.556}$ |  $0.526$ |  $\mathbf{0.569}$ |  $\mathbf{0.674}$ |  $\mathbf{0.807}$ |  $\mathbf{0.504}$
|  LoRA |  $1.6$M |  $\mathbf{0.791}$ |  $\mathit{0.451}$ |  $\mathbf{0.342}$ |  $0$ |  $\mathit{0.543}$ |  $\mathit{0.523}$ |  $\mathit{0.565}$ |  $0.605$ |  $0.799$ |  $\mathit{0.456}$
|  AdaLoRA |  $1.0$M |  $0.508$ |  $0.334$ |  $0.136$ |  $0.199$ |  $0.431$ |  $0.394$ |  $0.461$ |  $0.578$ |  $0.738$ |  $0.270$
|  Adaptive rank sampling |  $1.0$M |  $0.734$ |  $0.450$ |  $0.300$ |  $\mathit{0.214}$ |  $0.479$ |  $\mathbf{0.531}$ |  $0.557$ |  $\mathit{0.614}$ |  $\mathit{0.801}$ |  $0.449$
NT-$500$M |  FMFT2 |  $500$M |  $\mathbf{0.756}$ |  $\mathbf{0.379}$ |  $\mathit{0.288}$ |  $\mathbf{0.288}$ |  $\mathbf{0.488}$ |  $\mathbf{0.399}$ |  $\mathbf{0.461}$ |  $\mathbf{0.579}$ |  $\mathbf{0.752}$ |  $\mathit{0.342}$
|  LoRA |  $7$M |  $0.725$ |  $0.354$ |  $0.240$ |  $0.267$ |  $0.452$ |  $0.387$ |  $0.442$ |  $0.515$ |  $0.685$ |  $0.331$
|  AdaLoRA |  $6.9$M |  $0.734$ |  $0.365$ |  $0.284$ |  $\mathit{0.281}$ |  $0.469$ |  $0.390$ |  $0.452$ |  $0.557$ |  $0.704$ |  $0.340$
|  Adaptive rank sampling |  $6.9$M |  $\mathit{0.749}$ |  $\mathit{0.367}$ |  $\mathbf{0.289}$ |  $0.280$ |  $\mathit{0.474}$ |  $\mathit{0.395}$ |  $\mathit{0.460}$ |  $\mathit{0.562}$ |  $\mathit{0.717}$ |  $\mathbf{0.343}$
OPT-$125$M |  FMFT |  $125$M |  $\mathbf{0.740}$ |  $\mathbf{0.406}$ |  $0.216$ |  $\mathbf{0.305}$ |  $\mathit{0.439}$ |  $0.522$ |  $\mathit{0.474}$ |  $0.505$ |  $\mathbf{0.750}$ |  $\mathbf{0.460}$
|  LoRA |  $1.1$M |  $0.505$ |  $0.327$ |  $0.252$ |  $0.274$ |  $0.429$ |  $0.510$ |  $0.429$ |  $0.503$ |  $0.725$ |  $0.349$
|  AdaLoRA |  $1.0$M |  $0.562$ |  $0.339$ |  $\mathit{0.275}$ |  $0.266$ |  $0.416$ |  $0.523$ |  $0.431$ |  $0.513$ |  $0.734$ |  $0.404$
|  Adaptive rank sampling |  $1.0$M |  $0.571$ |  $\mathit{0.345}$ |  $\mathbf{0.281}$ |  $0.275$ |  $0.428$ |  $\mathit{0.531}$ |  $0.453$ |  $\mathbf{0.527}$ |  $\mathit{0.736}$ |  $0.415$
|  Lingo |  $1.0$M |  $\mathit{0.63}$ |  $\mathbf{0.406}$ |  $\mathit{0.275}$ |  $\mathit{0.282}$ |  $\mathbf{0.447}$ |  $\mathbf{0.554}$ |  $\mathbf{0.502}$ |  $\mathit{0.523}$ |  $0.735$ |  $\mathit{0.424}$
OPT-$350$M |  FMFT |  $350$M |  $\mathbf{0.801}$ |  $\mathbf{0.463}$ |  $\mathit{0.287}$ |  $\mathit{0.289}$ |  $\mathbf{0.504}$ |  $\mathbf{0.517}$ |  $\mathbf{0.505}$ |  $\mathbf{0.645}$ |  $\mathbf{0.789}$ |  $\mathbf{0.471}$
|  LoRA |  $6.3$M |  $0.619$ |  $0.406$ |  $0.203$ |  $0.209$ |  $0.439$ |  $0.447$ |  $0.449$ |  $0.504$ |  $0.722$ |  $0.437$
|  AdaLoRA |  $6.2$M |  $0.627$ |  $0.409$ |  $0.209$ |  $0.217$ |  $0.446$ |  $0.459$ |  $0.451$ |  $0.525$ |  $0.731$ |  $0.448$
|  Adaptive rank sampling |  $6.2$M |  $0.636$ |  $0.411$ |  $0.213$ |  $0.229$ |  $0.457$ |  $0.463$ |  $0.455$ |  $0.527$ |  $0.741$ |  $0.451$
|  Lingo |  $6.2$M |  $\mathit{0.645}$ |  $\mathit{0.42}$ |  $\mathbf{0.291}$ |  $\mathbf{0.297}$ |  $\mathit{0.478}$ |  $\mathit{0.479}$ |  $\mathit{0.481}$ |  $\mathit{0.538}$ |  $\mathit{0.753}$ |  $\mathit{0.461}$

11footnotetext: Model performances (without further pre-training) are taken
from the DNABERT-2 paper.22footnotetext: Model performances are taken from the
NT paper.

33footnotetext: The best results for each foundation model are highlighted in
bold, while the second best are in italic.

Table S3: AUCs for various models and methods on the Promoter (Human) task. The best results for each foundation model are highlighted in bold, while the second best are in italic.  Model | Method |  # Train. Params. |  Prom_all |  Prom_notata |  Prom_tata
---|---|---|---|---|---
DNABERT-$2$ | FMFT |  $117$M |  $0.908$ |  $0.950$ |  $0.804$
LoRA |  $1.6$M |  $\mathit{0.918}$ |  $\mathbf{0.971}$ |  $\mathit{0.812}$
AdaLoRA |  $1.0$M |  $0.912$ |  $0.954$ |  $0.802$
Adaptive rank sampling |  $1.0$M |  $\mathbf{0.920}$ |  $\mathit{0.964}$ |  $\mathbf{0.815}$
NT-$500$M | FMFT |  $500$M |  $\mathbf{0.950}$ |  $\mathbf{0.951}$ |  $\mathbf{0.939}$
LoRA |  $7$M |  $0.921$ |  $0.942$ |  $0.899$
AdaLoRA |  $6.9$M |  $0.924$ |  $0.949$ |  $0.871$
Adaptive rank sampling |  $6.9$M |  $\mathit{0.926}$ |  $\mathbf{0.951}$ |  $\mathit{0.918}$
OPT-$125$M | FMFT |  $125$M |  $0.898$ |  $0.947$ |  $0.864$
LoRA |  $1.1$M |  $0.887$ |  $0.907$ |  $0.853$
AdaLoRA |  $1.0$M |  $0.902$ |  $0.931$ |  $0.886$
Adaptive rank sampling |  $1.0$M |  $\mathbf{0.959}$ |  $\mathit{0.962}$ |  $\mathbf{0.928}$
Lingo |  $1.0$M |  $\mathit{0.954}$ |  $\mathbf{0.98}$ |  $\mathit{0.895}$
OPT-$350$M | FMFT |  $350$M |  $0.894$ |  $0.923$ |  $0.866$
LoRA |  $6.3$M |  $0.917$ |  $0.928$ |  $0.904$
AdaLoRA |  $6.2$M |  $0.922$ |  $0.947$ |  $\mathit{0.911}$
Adaptive rank sampling |  $6.2$M |  $\mathit{0.938}$ |  $\mathit{0.956}$ |  $\mathbf{0.929}$
Lingo |  $6.2$M |  $\mathbf{0.957}$ |  $\mathbf{0.983}$ |  $0.890$

## References

  *   * Ji et al. [2021] Ji, Y., Zhou, Z., Liu, H., Davuluri, R.V.: Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics 37(15), 2112–2120 (2021)
  * Zhou et al. [2023] Zhou, Z., Ji, Y., Li, W., Dutta, P., Davuluri, R., Liu, H.: Dnabert-2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2306.15006 (2023)
  * Dalla-Torre et al. [2023] Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, J., Carranza, N.L., Grzywaczewski, A.H., Oteri, F., Dallago, C., Trop, E., Sirelkhatim, H., Richard, G., et al.: The nucleotide transformer: Building and evaluating robust foundation models for human genomics. bioRxiv, 2023–01 (2023)
  * Chen et al. [2022] Chen, K.M., Wong, A.K., Troyanskaya, O.G., Zhou, J.: A sequence-based global map of regulatory activity for deciphering human genetics. Nature genetics 54(7), 940–949 (2022)
  * Ding et al. [2023] Ding, N., Qin, Y., Yang, G., Wei, F., Yang, Z., Su, Y., Hu, S., Chen, Y., Chan, C.-M., Chen, W., et al.: Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence 5(3), 220–235 (2023)
  * Valipour et al. [2022] Valipour, M., Rezagholizadeh, M., Kobyzev, I., Ghodsi, A.: Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07558 (2022)
  * Ma et al. [2023] Ma, X., Fang, G., Wang, X.: Llm-pruner: On the structural pruning of large language models. arXiv preprint arXiv:2305.11627 (2023)
  * Hu et al. [2021] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)
  * Zhang et al. [2023] Zhang, Q., Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., Zhao, T.: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512 (2023)
  * Edalati et al. [2022] Edalati, A., Tahaei, M., Kobyzev, I., Nia, V.P., Clark, J.J., Rezagholizadeh, M.: Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212.10650 (2022)
  * Hyeon-Woo et al. [2021] Hyeon-Woo, N., Ye-Bin, M., Oh, T.-H.: Fedpara: Low-rank hadamard product for communication-efficient federated learning. arXiv preprint arXiv:2108.06098 (2021)
  * Floridi and Chiriatti [2020] Floridi, L., Chiriatti, M.: Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines 30, 681–694 (2020)
  * Byrska-Bishop et al. [2022] Byrska-Bishop, M., Evani, U.S., Zhao, X., Basile, A.O., Abel, H.J., Regier, A.A., Corvelo, A., Clarke, W.E., Musunuri, R., Nagulapalli, K., et al.: High-coverage whole-genome sequencing of the expanded 1000 genomes project cohort including 602 trios. Cell 185(18), 3426–3440 (2022)
  * Karczewski et al. [2020] Karczewski, K.J., Francioli, L.C., Tiao, G., Cummings, B.B., Alföldi, J., Wang, Q., Collins, R.L., Laricchia, K.M., Ganna, A., Birnbaum, D.P., et al.: The mutational constraint spectrum quantified from variation in 141,456 humans. Nature 581(7809), 434–443 (2020)
  * Tang and Koo [2023] Tang, Z., Koo, P.K.: Building foundation models for regulatory genomics requires rethinking large language models. Proceedings of the ICML Workshop on Computational Biology (2023)
  * Hupkes et al. [2023] Hupkes, D., Giulianelli, M., Dankers, V., Artetxe, M., Elazar, Y., Pimentel, T., Christodoulopoulos, C., Lasri, K., Saphra, N., Sinclair, A., et al.: A taxonomy and review of generalization research in nlp. Nature Machine Intelligence 5(10), 1161–1174 (2023)
  * Touvron et al. [2023] Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
  * Wang et al. [2023] Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al.: Internimage: Exploring large-scale vision foundation models with deformable convolutions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14408–14419 (2023)
  * Coskun et al. [2023] Coskun, B., Ocakoglu, G., Yetemen, M., Kaygisiz, O.: Can chatgpt, an artificial intelligence language model, provide accurate and high-quality patient information on prostate cancer? Urology (2023)
  * Hu et al. [2023] Hu, Y., Ameer, I., Zuo, X., Peng, X., Zhou, Y., Li, Z., Li, Y., Li, J., Jiang, X., Xu, H.: Zero-shot clinical entity recognition using chatgpt. arXiv preprint arXiv:2303.16416 (2023)
  * Dinh et al. [2022] Dinh, T., Zeng, Y., Zhang, R., Lin, Z., Gira, M., Rajput, S., Sohn, J.-y., Papailiopoulos, D., Lee, K.: Lift: Language-interfaced fine-tuning for non-language machine learning tasks. Advances in Neural Information Processing Systems 35, 11763–11784 (2022)
  * Delétang et al. [2023] Delétang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, L.K., Aitchison, M., Orseau, L., et al.: Language modeling is compression. arXiv preprint arXiv:2309.10668 (2023)
  * Dotan et al. [2023] Dotan, E., Jaschek, G., Pupko, T., Belinkov, Y.: Effect of tokenization on transformers for biological sequences. bioRxiv, 2023–08 (2023)
  * Stewart [1993] Stewart, G.W.: On the early history of the singular value decomposition. SIAM review 35(4), 551–566 (1993)
  * Chicco and Jurman [2020] Chicco, D., Jurman, G.: The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. BMC genomics 21(1), 1–13 (2020)
  * Moore et al. [2020] Moore, J.E., Purcaro, M.J., Pratt, H.E., Epstein, C.B., Shoresh, N., Adrian, J., Kawli, T., Davis, C.A., Dobin, A., et al.: Expanded encyclopaedias of dna elements in the human and mouse genomes. Nature 583(7818), 699–710 (2020)
  * Zhou and Troyanskaya [2015] Zhou, J., Troyanskaya, O.G.: Predicting effects of noncoding variants with deep learning–based sequence model. Nature methods 12(10), 931–934 (2015)
  * Przybyla and Gilbert [2022] Przybyla, L., Gilbert, L.A.: A new era in functional genomics screens. Nature Reviews Genetics 23(2), 89–103 (2022)
  * Cohen et al. [2009] Cohen, I., Huang, Y., Chen, J., Benesty, J., Benesty, J., Chen, J., Huang, Y., Cohen, I.: Pearson correlation coefficient. Noise reduction in speech processing, 1–4 (2009)
  * Li and Liang [2021] Li, X.L., Liang, P.: Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 (2021)
  * Liu et al. [2021] Liu, X., Ji, K., Fu, Y., Tam, W.L., Du, Z., Yang, Z., Tang, J.: P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.07602 (2021)
  * Lester et al. [2021] Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691 (2021)
  * Fu et al. [2023] Fu, Z., Yang, H., So, A.M.-C., Lam, W., Bing, L., Collier, N.: On the effectiveness of parameter-efficient fine-tuning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 12799–12807 (2023)
  * Radford et al. [2019] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
  * Zhang et al. [2022] Zhang, Q., Zuo, S., Liang, C., Bukharin, A., He, P., Chen, W., Zhao, T.: Platon: Pruning large transformer models with upper confidence bound of weight importance. In: International Conference on Machine Learning, pp. 26809–26823 (2022). PMLR
  * Kingma and Ba [2014] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)