11institutetext: Department of Physics & Astronomy, Purdue University, West
Lafayette, IN 47907, USA 22institutetext: Purdue University Institute for
Cancer Research, West Lafayette, IN 47907, USA 33institutetext: Elmore Family
School of Electrical and Computer Engineering, Purdue University, West
Lafayette, IN 47907, USA
44institutetext: Department of Comparative Pathobiology, Purdue University,
West Lafayette, IN 47907, USA 55institutetext: School of Industrial
Engineering, Purdue University, West Lafayette, IN 47907, USA
55email: {malusare, hkothand, dtamboli, natallah<EMAIL_ADDRESS>
# Understanding the Natural Language of DNA using Encoder-Decoder Foundation
Models with Byte-level Precision

Aditya Malusare 11 Harish Kothandaraman 22 Dipesh Tamboli 33 Nadia A. Lanman
22 4 4 Vaneet Aggarwal 33 5 5

###### Abstract

This paper presents the Ensemble Nucleotide Byte-level Encoder-Decoder (ENBED)
foundation model, analyzing DNA sequences at byte-level precision with an
encoder-decoder Transformer architecture. ENBED uses a sub-quadratic
implementation of attention to develop an efficient model capable of sequence-
to-sequence transformations, generalizing previous genomic models with
encoder-only or decoder-only architectures. We use Masked Language Modeling to
pre-train the foundation model using reference genome sequences and apply it
in the following downstream tasks: (1) identification of enhancers, promotors
and splice sites, (2) identification of biological function annotations of
genomic sequences, (3) recognition of sequences containing base call
mismatches and insertion/deletion errors, an advantage over tokenization
schemes involving multiple base pairs, which lose the ability to analyze with
byte-level precision, and (4) generating mutations of the Influenza virus
using the encoder-decoder architecture and validating them against real-world
observations. In each of these tasks, we demonstrate significant improvement
as compared to the existing state-of-the-art results.

###### Keywords:

DNA Transformers Attention Error Detection Mutations Gene Annotation

##  1 Introduction

The rise of foundation models in recent years has led to tremendous
developments in understanding natural languages [1]. Although they were
originally developed to process and generate written text, these models have
transcended their initial purpose due to their generalizable nature and wide
applicability. Foundation models have shown great potential in the field of
bioinformatics [2], since their capacity to be trained on vast amounts of
unlabeled data and their adaptability enable them to achieve state-of-the-art
performance in a variety of tasks.

Early applications of foundation models in bioinformatics can be seen in
analyzing protein sequences [3, 4], which were then trained on diverse
applications like calculation of protein structure, prediction of mutation
effects and the understanding of phylogenetic structure [5, 6, 7]. These
models have since evolved beyond proteins into DNA and RNA analysis, and have
demonstrated the ability to surpass previous benchmarks in identifying
regulatory elements, predicting chromatin profiles, analyzing evolution from
genomic sequence data and predicting the impacts of mutations in DNA [8, 9,
10, 11, 12]. The ability to visualize and interpret the internal model
structure [13] and to derive key insights of the underlying biological
processes [14] demonstrate the unique advantages offered by foundation models
in the field of bioinformatics.

###  1.1 Limitations of previous work

####  1.1.1 Architecture.

Prior work on Transformer-based models for DNA sequence analysis exists in two
forms: (i) Encoder-only models [8, 15, 14, 9] that focus on classification and
regression-based downstream tasks and (ii) Decoder-only models [16, 10] that
are capable of classification, regression as well as generative tasks that
involve design and synthesis.

A combination of encoder and decoder blocks enables the model to perform
sequence-to-sequence transformations. One of the fundamental processes
undergone by DNA is its transcription into an RNA sequence and subsequent
translation into protein sequences, the building blocks of all living
organisms. Understanding sequence-to-sequence processes like these is crucial
to advancing our knowledge of genetics, and developing an encoder-decoder
model is an important step in this direction. Although decoder-only models can
perform sequence-to-sequence transformations, the combination of encoder and
decoder blocks offers significant advantages like: (i) bi-directional context-
aware decoding, (ii) variable length inputs and outputs, and (iii) multi-head
attention, which allows the model to focus on different parts of the input
sequence simultaneously.

####  1.1.2 Tokenization.

Biological sequences like DNA are encoded using a vocabulary of four symbols
(A, C, T, G) representing nucleic acids. These sequences are converted into a
Transformer-compatible format by a tokenizer, which generates a list of tokens
for any given input. Since these models were initially developed for
applications in natural languages, the most prevalent forms of tokenization
are sentence-piece or word-piece, where the language vocabulary is built using
natural ideas like words or syllables. In the absence of typical indicators of
linguistic order in DNA, like spaces and punctuation, these tokenization
schemes use statistical techniques to determine the ‘words’ that make up the
vocabulary of the input sequences. A few examples of previously used
tokenizers are: k-mer [8], SentencePiece [9], and byte-pair encoding (BPE)
[15] tokenization. While such techniques identify optimal encoding methods by
constructing tokens having multiple base pairs, they are vulnerable to any
type of noise present in the sequence. A single variation in a base pair will
result in the fragment being mapped to a completely different word in the
vocabulary, resulting in an outsized impact from a small perturbation [17].

###  1.2 Our contributions

In this paper, we develop the Ensemble Nucleotide Byte-level Encoder-Decoder
(ENBED) Transformer, a foundation model that analyzes nucleotide sequences
with Transformers using byte-level tokenization and an encoder-decoder model.
This implementation bridges the gap between existing models that are either
encoder-only or decoder-only implementations and presents the possibility of
sequence-to-sequence analysis tasks. Using sliding-window and global attention
we obtain a sub-quadratic implementation of attention, and demonstrate the
performance improvements over dense attention. The foundation model is pre-
trained using an ensemble of high-quality reference genomes from NCBI RefSeq,
including the telomere-to-telomere assemblies of Human and Maize DNA, data
from the 1000 Genomes Project and a mix of widely studied organisms like E.
coli, D. melanogaster, M. musculus and P. vivax (Sec 5). This process is
implemented by giving the model a self-supervised goal of internalizing the
structure of the language of nucleotide sequences.

ENBED is built using a byte-level tokenizer. In order to avoid the issues
created by single nucleotide variants and their downstream impacts, we side-
step the problem of determining the tokenization scheme entirely by working
with single nucleotides as tokens. This leads to increased computational
costs, but grants resilience to the types of variations and noise commonly
encountered in DNA sequences. This resilience to noise additionally confers
the ability to correct errors in the nucleotide space without significant
recompute. In order to offset the impact of increased computations, we
implement sub-quadratic attention layers in order to scale up the model
efficiently.

####  1.2.1 Evaluation of performance on Genomic Benchmark datasets.

We evaluate the performance of the ENBED foundation model on sequence-level
classification tasks and compare it’s accuracy against contemporary foundation
models. We show that ENBED outperforms the state-of-the-art in 22 of the 25
benchmarks devised by the authors of the Nucleotide Transformer [9] and
Genomic Benchmarks [18] datasets. These benchmarks consist of tasks like
identifying enhancers, promotors, splice sites and histone marks in multi-
species data comprising of genomic sequences from human, mouse, yeast, fruit
fly and worm DNA.

####  1.2.2 Identifying sequencing noise.

Long-read sequencing using Nanopores is used to study telomeres, which are
protective caps found at chromosomal ends and have long repetitive elements.
It has been found that telomeres in many organisms are frequently miscalled
[19], referring to errors in the process that translates electrical signals
into the alphabet of DNA. We illustrate how ENBED can focus on fragments that
look incorrect or out of place, demonstrating the model’s ability of
distinguishing between noisy and accurate data. In a synthetic dataset
constructed using noise distributions found in real-world raw sequence data,
we demonstrate that our model can identify sequences containing noise with an
accuracy of 97.1%, leveraging the information internalized by bring pretrained
on the telomere-to-telomere reference sequences.

####  1.2.3 Biological function annotations.

Mapping the complete human genome was a significant milestone in modern
biology, and it has produced a new set of challenges in identifying the
functions and interactions of different parts of the genome. We fine-tune our
model to solve a version of this problem by identifying the biological
functions of genomic sequences among the ten most common functional classes
using a fine-tuned model, achieving an $F_{1}$ score of 74.1.

####  1.2.4 Studying mutations as a sequence-to-sequence process.

Exploring mutations is essential as it sheds light on the mechanisms driving
genetic diversity which enhance the overall resilience of living organisms in
a changing environment. The encoder-decoder architecture confers the ability
to rapidly iterate mutagenization of genomic segments. We study mutations in
the Influenza virus, using the NCBI Influenza Virus Resource. By constructing
a dataset with a phylogenetic tree, we obtain parent-child pairs of mutated
sequences and show the effectiveness of our encoder-decoder architecture in
analyzing and predicting these mutations.

##  2 Methods

###  2.1 Encoder-Decoder Model Architecture

Figure 1: Model Architecture. The model is constructed using encoder and decoder blocks with a ratio of 2:1. Both types of blocks consist of attention and feed-forward layers, with the decoder blocks additionally incorporating the embeddings in encoder-decoder attention layers. (Sec 2.1) Configuration | Base | Large
---|---|---
Number of parameters | 580M | 1.2B
$d_{ff}$ | 3584 | 3850
$d_{kv}$ | 64 | 64
$d_{model}$ | 1536 | 1536
Encoder layers | 8 | 24
Decoder layers | 4 | 12
Attention heads | 12 | 16
Dropout rate | 0.1 | 0.1
Table 1: Model Configurations. $d_{model}$ denotes the size of the encoder
layers, and the pooler layer, $d_{kv}$ is the size of the key, query, and
value projections per attention head and $d_{ff}$ is the size of the
intermediate feed-forward layer in each Transformer block.

ENBED is built using an encoder-decoder architecture (Fig. 1) consisting of
encoder and decoder blocks, each comprised of two subcomponents: an attention
layer and a feed-forward neural network. The attention layers process a
sequence by replacing each element with a weighted average of the rest of the
sequence, after which they are normalized and passed through the feed-forward
neural network. Dropout is applied to the feed-forward network, the attention
weights, and the input and output of the entire stack. The implementation is
written using JAX [20] and the Flax-former library [21].

We formulate two model sizes with 580M, and 1.2B parameters labeled Base and
Large respectively, with the configurations specified in Table 1. The model is
encoder-heavy since idiosyncratic relationships among tokens are better
encoded by devoting a larger share of parameters to these blocks. We found
that adjusting the encoder-to-decoder ratio to 2:1 improved performance, with
a 1% increase in Masked Language Modeling (MLM) accuracy for all model sizes
over the 3:1 ratio chosen by the authors of ByT5 [22], a similar architecture
built to process token-free text-to-text transformations. We also find that
reducing the masked span length, which is the average number of tokens masked
during pre-training, from 40 down to 20 helps in faster convergence owing to
the significantly smaller vocabulary of DNA.

###  2.2 Tokenization

Sequences are tokenized by breaking down the input into tokens consisting of
single nucleotides. The vocabulary size is fixed at 384, with 256 ASCII
characters and additional tokens added to function as `MASK`, `PAD` and
`UNKNOWN` tokens during the training process. We require multiple `MASK`
tokens in order to index the positions where masking has occurred and to label
the targets with these indices. Although the alphabet of DNA only comprises of
the four nucleic acids Adenine (A), Cytosine (C), Guanine (G), and Thymine
(T), we choose to keep the whole set of extended ASCII characters since they
could aid in future tasks like sequence-to-sequence transformations involving
targets beyond just DNA sequences.

This approach needs more floating-point operations (FLOPs) since it increases
the tokenized sequence length for the same input DNA sequence compared to
previous methods, resulting in higher resource requirements. Although this
limits us to dealing with short- to medium-length sequences, we can overcome
these constraints and scale up the model by focusing on reducing the
complexity of attention layers as described below.

###  2.3 Attention

Complexity | Base | Large
---|---|---
Full Attention | 65 | 68
Linearized Attention | 69 | 74
Table 2: MLM Accuracy. Sub-quadratic attention outperforms full $O(N^{2})$
attention when evaluated with Masked Language Modeling (MLM) accuracy (%).

Attention can be understood as a soft-lookup of a query Q in a dictionary of
stored keys K and values V. Attention scores are generated by calculating the
similarity between Q and K, each having a dimension $d$, with scaled dot-
product attention
$\left(\text{Softmax}\left(\frac{\textbf{Q}\textbf{K}^{T}}{\sqrt{d}}\right)V\right)$
being the most common implementation. Increasing the sequence length $L$ can
be a challenge, since this type of attention has a complexity of $O(L^{2})$.

In order to reduce the complexity while preserving function, we modify the
architecture to replace dense attention with a combination of two sub-
quadratic variants of attention: (i) sliding-window attention and (ii) global
attention.

####  2.3.1 Sliding-window attention.

Local context is crucial in analyzing DNA, since biological processes like
transcription and translation work within continuous regions of a sequence.
Tokens within a sliding window of radius $r$ are used to calculate the
attention scores, bringing the complexity down to $O(L\times r)$. We fix
$r=64$ for the initial three layers and increase to $r=128$ in the final
layers, which allows them to learn higher-level representations while having
the lower layers focus on local information.

####  2.3.2 Global attention.

For tasks that involve classifying or annotating whole sequences, we need a
mechanism that aggregates global information from the inputs, in addition to
the local scores. We divide the input sequence into $k$ blocks and calculate a
global token by summing and normalizing the embeddings for every token in the
block. Scores are then computed for every input token by letting it attend to
the neighboring tokens (as described above) and all the global tokens, which
has a total complexity of $O(L(r+k))$.

Hence, by choosing appropriate values for $r$ and $k$ relative to $L$, we
implement a scheme to calculate attention with a sub-quadratic complexity
which allows us to set an input and output length of $16,384$, a significant
improvement over the limit of $512$ tokens using dense attention with the same
GPU hardware.

###  2.4 Applications of Foundation Models using Transfer Learning

####  2.4.1 Building the foundation model.

The first step in building our foundation model is pre-training it on high-
quality reference sequences. We use a procedure called Masked Language
Modeling (MLM). The objective is to reconstruct tokens that have been deleted
and replaced with a `MASK` token. This task develops the ability to understand
the context and vocabulary to identify the correct elements that belong in the
masked segments. Utilizing a large corpus of unlabeled data allows us to
impart the model with generalizable knowledge that can be fine-tuned for
specific downstream tasks. The genomic corpus is constructed by concatenating
FASTA files from the NCBI sources mentioned in Sec 5, removing any
descriptions starting with ‘$>$’ and ‘N’ bases that are a result of hard-
masking. We choose a masking rate of $15\%$ over the course of pre-training.
The entire corpus is supplied to a collator that handles masking, padding, and
truncation to ensure that the input length is maintained. We follow a linear
schedule with warmup (5% of the total training steps) using the AdamW
optimizer ($\beta_{1}=0.9,\beta_{2}=0.99,\epsilon=10^{-6}$) with a learning
rate of 1e-5, a cross-entropy loss function and softmax as the activation
function. We train all versions of the model with maximum input and output
lengths of $16,384$ tokens (base pairs). Convergence takes 120-480 GPU-hours
with 8 NVIDIA A100 GPUs, determined by model size and input length.

####  2.4.2 Fine-tuning for downstream tasks.

We fine-tune the model by modifying the final layers into a task-specific
configuration. This is called the ‘head’ of the model and is attached to the
final layer of the pre-trained model. Layers are gradually unfrozen in reverse
order during the course of fine-tuning, allowing the Transformer to integrate
with the attached head while retaining the initial layers, thus enabling the
transfer of pre-trained knowledge for downstream applications.

####  2.4.3 Classification head.

A fully connected (dense) layer is usually added to the output of the base
model, followed by a softmax activation to produce class probabilities,
typically used in sequence-level classification tasks.

####  2.4.4 Language modeling head.

A language modeling head comprises of a single feedforward neural network
layer followed by a softmax activation function. This layer takes hidden
representations from the preceding layers and outputs a probability
distribution over the vocabulary.The objective is to estimate the estimate the
probability of a token given the previous words in a sentence. The softmax
function transforms the raw output scores into probabilities, representing the
likelihood of each word or token in the vocabulary at any particular position.
We use these to perform sequence-to-sequence transformations.

###  2.5 Application Domains

####  2.5.1 Genomic Benchmarks.

The Genomic Benchmarks (GB) dataset consists of sequences from four organisms:
Human, mouse (Mus musculus), roundworm (Caenorhabditis elegans) and fruit fly
(Drosophila melanogaster). The dataset comprises of: (i) Human enhancers from
Cohn et. al.[23] and Ensembl [24], (ii) Open Chromatin Region classifications
from the Ensembl build, (iii) Computationally generated data for coding and
non-coding sequences (iv) Multi-class data composed of three regulatory
elements (promotors, enhancers and Open Chromatin Regions), (v) Non-TATA
promotor sequences imported from Umarov et. al. [25].

####  2.5.2 Nucleotide Transformer Benchmarks.

The Nucleotide Transformer (NT) benchmarks consist of five data sources: (i)
Epigenetic marks in the yeast genome, which use experimentally obtained
nucleosome occupancy values processed into positive and negative observations
and to provide the following histone marks datasets: {H3, H4, H3K9ac, H3K14ac,
H4ac, H3K4me1, H3K4me2, H3K4me3, H3K36me3, and H3K79me3}, (ii) A dataset [26]
consisting of a mix of strong, weak and non-enhancers. (iii) Promotor
sequences $300$ base pairs in length around transcription start sites, divided
on the basis of TATA and non-TATA box promotors. (iv) Splice site datasets
composed of donor, acceptor and non-splice site sequences from
phylogenetically diverse organisms.

####  2.5.3 Noise identification.

We generate a synthetic dataset with segments of $512$ nucleotides selected at
random from the T2T-CHM13v2 assembly, introducing noise based on real-world
raw DNA sequencing data to generate negative samples. Specifically, we sample
60% of our sequences from telomere regions with the rest being randomly
selected from the entire assembly, introducing insertion, deletion and base-
call mismatch errors with a uniform distribution of 2% each, resulting in a
simulated read accuracy of 94%. The model is fine-tuned on a sequence
classification task with this labeled dataset. This process can be likened to
out-of-distribution detection[27], since the negative samples would represent
data that does not belong to the distribution of the training dataset.

####  2.5.4 Biological function annotation.

We can formulate the process of annotating genes as a sequence-to-sequence
task, with the input being a DNA sequence fragment and the output containing
information regarding the name, type and function of the fragment. For
evaluating our model, we train it to output the biological function annotation
of a given genomic input sequence up to 512 base pairs in length. We choose
the following annotation types for our experiment: Coding Sequences, IncRNA,
snoRNA, miscRNA, miRNA, snRNA, TEC, Processed and Unprocessed Pseudogenes.
These annotations are obtained from the Ensembl dataset [24].

####  2.5.5 Mutation generation.

Figure 2: Phylogenetic Tree. Circular cladogram visualization of the
Influzenza H1 gene sequences. Nodes are represented by yellow dots.

To create the Influenza virus mutation dataset, we obtain our source data from
[28] and subset the HA1 nucleotide sequence of the H3N2 Influenza virus
between 300 to 799 bp (100-266 amino-acids) to capture the Antigenic site A
and B. The selected region is a part of the globular domain that occurs in a
jelly-roll fold of eight-stranded anti-parallel beta-sheets, containing the
most commonly mutating amino-acid residues around the receptor binding site.
The HA1 head also accumulates N-linked glycosylation sites over time, which
are thought to mask antigenic sites from immune recognition. The glycosylation
of the HA1 globular domain modulates receptor binding, stimulates host
antibody responses, and shields key antigenic sites to facilitate immune
evasion of the virus. By focusing on the HA1 subdomain, we aimed to evaluate
the sequence-to-sequence model on a functionally important region of influenza
HA that experiences significant antigenic drift and glycosylation changes. We
visualize the H1 gene sequences using a circular cladogram (Fig. 2).

Candidate sequences are generated using a language modeling head (Sec 2.4.2)
with the parent sequence supplied as the input. Using a beam search
($N_{beams}$=5), we obtain five candidate sequences which are autoregressively
generated to a length of 499 bp (equal to the input). We rank the sequences
using the noise identification pipeline above, and select the sequence least
likely to be identified as having noise present. We identify mutations by
measuring the Levenshtein distance between parent and child sequences. This
metric accounts for insertion, deletion as well as in-place modifications.

##  3 Results

Upon convergence, the pre-training process yields a foundation model ready to
be applied to downstream tasks. The initial layers in the pre-trained model
are frozen since they contain generalizable information that helps the model
build versatile internal representations of the data. We visualize these
internal representations by extracting the encoder output layer and plotting
attention maps in Fig. 3.

Figure 3: Interpreting Attention Layers. We visualize the 12 attention heads
of the pre-trained ENBED (Base) foundation model. These maps are generated
using the outputs from the final encoder block. The use of multiple attention
heads grants the model the ability to simultaneously use a diverse range of
patterns to analyze input sequences. We observe that some heads are dedicated
to analyzing close neighbors (3, 9, 10) while others display a more dilated
version of this phenomenon (1, 2, 5, 11). Additionally, there are heads which
attempt to exclude local information and focus on a more global view of the
input sequence (4, 6, 8, 12).

###  3.1 ENBED outperforms state-of-the-art on genomic benchmark datasets

We finetune the model using a classification head (Sec. 2.4.2) using the
embedding outputs from the final encoder block, on the datasets constructed by
the authors of the Nucleotide Transformer (NT) benchmarks [9] and Genomic
Benchmarks (GB) [18]. The results of evaluating the model on the test dataset
of NT and GB are presented in Tables 3 and 4, respectively. For evaluation on
the NT benchmarks, we compare our performance against the Nucleotide
Transformer (v2) and HyenaDNA [9, 10], which are encoder-only and decoder-only
models respectively. For the GB datasets, we use the performance of the
Convolutional Neural Network (CNN) model developed by the authors of the
dataset [18] as a baseline. We also include the performance of the HyenaDNA
model and the baseline Transformer developed by its authors [10].

NT Benchmark | Nucleotide Transformer (v2) [9] | HyenaDNA[10] | ENBED (Large)
---|---|---|---
H3 | 89.5 | 88.9 | 90.6
H3K14ac | 76.9 | 80.9 | 81.4
H3K36me3 | 81.3 | 80.8 | 82.7
H3K4me1 | 77.7 | 75.8 | 77.9
H3K4me2 | 67.6 | 73.9 | 75.7
H3K4me3 | 69.5 | 77.5 | 77.9
H3K79me3 | 81.3 | 83.7 | 85.4
H3K9ac | 78.0 | 79.3 | 82.6
H4 | 90.5 | 88.2 | 90.8
H4ac | 74.9 | 78.4 | 80.5
Promotor (all) | 97.6 | 96.0 | 98.0
Promotor (non-TATA) | 97.6 | 96.0 | 98.0
Promotor (TATA) | 96.6 | 94.1 | 96.8
Splice acceptor | 98.7 | 95.8 | 95.8
Splice donor | 98.7 | 95.8 | 95.4
Enhancer | 77.3 | 75.9 | 78.3
Enhancer Types | 62.6 | 59.5 | 70.0
Table 3: Nucleotide Transformer (NT) Benchmarks. We evaluate our model using the Accuracy (%) score of the best performing variants of the Nucleotide Transformer v2 [9] and HyenaDNA [10], highlighting the best and second-best scores. The scores are sourced from a leaderboard maintained by the authors of [9] on the Hugging Face platform [29]. Genomic Benchmark | CNN | Transformer | HyenaDNA [10] | ENBED (Large)
---|---|---|---|---
Mouse Enhancers | 69.0 | 80.1 | 84.3 | 90.3
Human Enhancers (Cohn) | 69.5 | 70.5 | 70.6 | 71.2
Human Enhancers (Ensembl) | 68.9 | 83.5 | 89.2 | 92.2
Coding vs Intergenomic | 87.6 | 88.8 | 87.6 | 93.0
Human vs Worm | 93.0 | 95.6 | 96.5 | 97.3
Human Regulatory Elements | 93.3 | 91.5 | 93.8 | 90.2
Human Promoter (Non-TATA) | 84.6 | 87.7 | 96.6 | 96.8
Human OCR (Ensembl) | 68.0 | 73.0 | 80.9 | 81.9
Table 4: Genomic Benchmarks. Accuracy (%) scores of the best and second-best
model in the Genomic Benchmarks datasets [18]. The baseline CNN and
Transformer scores was calculated by the authors of [18] and [10]
respectively.

The combination of byte-level analysis, higher quality reference sequences and
improved pre-training methodology helps us exceed state-of-the-art performance
in 15 out of 17 of the Nucleotide Transformer (NT) benchmarks and 7 out of 8
of the Genomic Benchmarks (GB) datasets. Byte-level tokenization helps the
model internalize patterns better in the presence of variations like single
nucleotide polymorphisms, and the combination of encoder and decoder blocks
achieves superior performance over decoder-only methods due to it’s ability to
focus on multiple sections of the input simultaneously and to process inputs
in a context-aware manner.

###  3.2 ENBED identifies noise in genomic sequences

Model | Parameters | Reference |  $F_{1}$ Score
---|---|---|---
DNABERT | 110M | [8] | 84.6
Nucleotide Transformer | 2.5B | [9] | 91.2
ENBED (Base) | 580M | This paper | 91.8
ENBED (Large) | 1.2B | This paper | 97.1
Table 5: Erroneous Sequence Identification.

Table 5 shows the results of the sequence-level classification on erroneous
sequences using our synthetic dataset. Since competing models are trained
using the GRCh38 reference assembly, they often lack information about
repetitive regions due to hard-masking. We use the telomere-to-telomere
assembly for pre-training the model, resulting in a signifcant performance
improvement and on overall accuracy of 97.1% in the sequence-level
classification task of identifying erroneous genomic data, which is
significant improvement as compared to the baselines of DNABERT [8] (84.6%)
and Nucleotide Transformer [9] (91.2%).

###  3.3 ENBED identifies biological function annotations

Model | Parameters | Reference |  $F_{1}$ Score
---|---|---|---
DNABERT | 110M | [8] | 63.2
Nucleotide Transformer | 500M | [9] | 67.5
HyenaDNA | 7M | [10] | 72.8
ENBED (Base) | 580M | This paper | 70.1
ENBED (Large) | 1.2B | This paper | 74.1
Table 6: Biological Function Identification.

ENBED is trained to identify the ten most common base-level annotations of the
Human reference assembly. As shown in Table 6, we achieve an $F_{1}$ score of
74.1 in this classification task, an improved score compared to contemporary
models DNABERT [8] (63.2), Nucleotide Transformer [9] (67.5), and HyenaDNA
[10] (72.8).

###  3.4 ENBED generates mutations using sequence-to-sequence transformation

We evaluate the accuracy of ENBED in generating mutations, using an encoder-
decoder Transformer with Byte-Pair Encoding (BPE) tokenization (used in
previous genomic models [15]) as a baseline. Top-1 and Top-5 Accuracy (%)
scores are calculated by comparing predictions with real-world data from the
Influenza Virus Resource [30], with any deviation from an exact match being
classified as incorrect. Top-5 scores are calculated by selecting the best
candidate from the procedure described in Sec 2.5. The mean Levenshtein
distance of our model predictions from real-world mutated sequences is 2.3
edits over a length of 500 bp, resulting in an average similarity of 99.5%. We
can attribute the significant increase in accuracy to byte-level tokenization,
since other schemes with tokens involving multiple base pairs will be unable
to capture edits involving single nucleotides effectively.

Model | Top-1 Accuracy | Top-5 Accuracy | Mean LD | Median LD
---|---|---|---|---
Transformer (BPE) | 32.0 | 56.1 | 30.6 | 24
ENBED (Large) | 76.9 | 95.4 | 2.3 | 1
Table 7: Mutation Generation. Accuracy (%) scores of Top-1 and Top-5
candidates with the mean and median Levenshtein Distance (LD) between
predicted and child sequences.

##  4 Conclusion

We presented the ENBED foundation model, a byte-level encoder-decoder
Transformer pre-trained on an ensemble of reference genomes, with a sub-
quadratic attention implementation and an input length of $16,384$. We
demonstrated that it generalizes well in applications involving multiple
species, noisy data, and can perform a wide range of downstream tasks as
summarized below:

  * •

Achieved state-of-the-art scores in 7 out of 8 categories on the Genomic
Benchmarks datasets and 15 out of 17 categories in the Nucleotide Transformer
datasets.

  * •

Demonstrated the ability to identify insertion, deletion and mismatch noise in
sequences (Acc = 97.1%).

  * •

Identified biological function annotations from sequence data with an $F_{1}$
score of 74.1

  * •

Generated mutations in Influenza with a Top-1 and Top-5 accuracy of 76.9% and
95.4% and mean Levenshtein distance of 2.3 bp from real-world influenza
mutations in 500 bp generated sequences.

##  5 Data and Code Availability

The Telomere-to-telomere reference sequences for Human (GCF_009914755.1) and
Maize (GCA_022117705.1) and the reference sequences for E. coli
(GCF_000008865.2), D. melanogaster (GCF_000001215.4), M. musculus
(GCF_000001635.27) and P. vivax (GCF_000002415.2) were obtained from NCBI
RefSeq [31] in FASTA format. Variant Calling Files (VCFs) for the 1000 Genomes
Project [32] were obtained from the European Bioinformatics Institute. Gene
annotations were obtained from GENCODE [33] and Ensembl [24]. The mutation
tree was derived from the data assembled by the authors of [28], sourced from
the NCBI’s Influenza Virus Resource [30].
The source code used to develop and fine-tune the foundation model has been
released on Github.111https://github.itap.purdue.edu/Clan-labs/ENBED

##  6 Acknowledgement

This work uses the Anvil supercomputer [34] at Purdue University through
allocation CIS230228 from the Advanced Cyberinfrastructure Coordination
Ecosystem: Services & Support (ACCESS) program, which is supported by National
Science Foundation grants #2138259, #2138286, #2138307, #2137603, and
#2138296. This work was supported in part by the National Science Foundation
under grant FW-HTF-R-2129097.

## References

  * [1] G. Paaß and S. Giesselbach, Foundation Models for Natural Language Processing.  Springer International Publishing, 2023.
  * [2] S. Zhang, R. Fan, Y. Liu, S. Chen, Q. Liu, and W. Zeng, “Applications of transformer-based language models in bioinformatics: a survey,” Bioinformatics Advances, vol. 3, Jan. 2023.
  * [3] A. Elnaggar, M. Heinzinger, C. Dallago, G. Rehawi, Y. Wang, L. Jones, T. Gibbs, T. Feher, C. Angerer, M. Steinegger, D. Bhowmik, and B. Rost, “ProtTrans: Toward understanding the language of life through self-supervised learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, pp. 7112–7127, Oct. 2022.
  * [4] A. Rives, J. Meier, T. Sercu, S. Goyal, Z. Lin, J. Liu, D. Guo, M. Ott, C. L. Zitnick, J. Ma, and R. Fergus, “Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences,” Proceedings of the National Academy of Sciences, vol. 118, Apr. 2021.
  * [5] U. Lupo, D. Sgarbossa, and A.-F. Bitbol, “Protein language models trained on multiple sequence alignments learn phylogenetic relationships,” Nature Communications, vol. 13, 2022.
  * [6] X. Fang, F. Wang, L. Liu, J. He, D. Lin, Y. Xiang, X. Zhang, H.-H. Wu, H. Li, and L. Song, “A method for multiple-sequence-alignment-free protein structure prediction using a protein language model,” Nature Machine Intelligence, vol. 5, pp. 1087 – 1096, 2022.
  * [7] E. Nijkamp, J. A. Ruffolo, E. N. Weinstein, N. V. Naik, and A. Madani, “Progen2: Exploring the boundaries of protein language models,” ArXiv, vol. abs/2206.13517, 2022.
  * [8] Y. Ji, Z. Zhou, H. Liu, and R. V. Davuluri, “DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome,” Bioinformatics, vol. 37, pp. 2112–2120, Aug. 2021.
  * [9] H. Dalla-Torre, L. Gonzalez, J. M. Revilla, N. L. Carranza, A. H. Grzywaczewski, F. Oteri, C. Dallago, E. Trop, H. Sirelkhatim, G. Richard, M. J. Skwark, K. Beguir, M. Lopez, and T. Pierrot, “The nucleotide transformer: Building and evaluating robust foundation models for human genomics,” bioRxiv, 2023.
  * [10] E. D. Nguyen, M. Poli, M. Faizi, A. W. Thomas, C. J. Birch-sykes, M. Wornow, A. Patel, C. M. Rabideau, S. Massaroli, Y. Bengio, S. Ermon, S. A. Baccus, and C. Ré, “Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution,” ArXiv, 2023.
  * [11] M. Zvyagin, A. Brace, K. Hippe, Y. Deng, B. Zhang, C. O. Bohorquez, A. R. Clyde, B. Kale, D. Perez-Rivera, H. Ma, C. M. Mann, M. W. Irvin, J. G. Pauloski, L. T. Ward, V. Hayot, M. K. Emani, S. Foreman, Z. Xie, D. Lin, M. Shukla, W. Nie, J. Romero, C. Dallago, A. Vahdat, C. Xiao, T. Gibbs, I. T. Foster, J. J. Davis, M. E. Papka, T. S. Brettin, R. L. Stevens, A. Anandkumar, V. Vishwanath, and A. Ramanathan, “Genslms: Genome-scale language models reveal sars-cov-2 evolutionary dynamics,” bioRxiv, 2022.
  * [12] K. Yamada and M. Hamada, “Prediction of rna–protein interactions using a nucleotide language model,” Bioinformatics Advances, vol. 2, 2021.
  * [13] J. Vig, A. Madani, L. R. Varshney, C. Xiong, R. Socher, and N. Rajani, “Bertology meets biology: Interpreting attention in protein language models,” bioRxiv, 2020.
  * [14] T.-H. Zhang, M. M. Hasib, Y.-C. Chiu, Z.-F. Han, Y.-F. Jin, M. Flores, Y. Chen, and Y. Huang, “Transformer for Gene Expression Modeling (T-GEM): An Interpretable Deep Learning Model for Gene Expression-Based Phenotype Predictions,” Cancers, vol. 14, p. 4763, Sept. 2022.
  * [15] V. Fishman, Y. Kuratov, M. Petrov, A. Shmelev, D. Shepelin, N. Chekanov, O. Kardymon, and M. Burtsev, “GENA-LM: A Family of Open-Source Foundational Models for Long DNA Sequences,” preprint, Bioinformatics, June 2023.
  * [16] D. Zhang, W. Zhang, B. He, J. Zhang, C. Qin, and J. Yao, “DNAGPT: A Generalized Pretrained Tool for Multiple DNA Sequence Analysis Tasks,” bioRxiv, 2023.  Publisher: Cold Spring Harbor Laboratory _eprint: https://www.biorxiv.org/content/early/2023/07/12/2023.07.11.548628.full.pdf.
  * [17] E. Dotan, G. Jaschek, T. Pupko, and Y. Belinkov, “Effect of tokenization on transformers for biological sequences,” bioRxiv, 2023.
  * [18] K. Grevsova, V. Martinek, D. Cechak, P. Simecek, and P. Alexiou, “Genomic benchmarks: a collection of datasets for genomic sequence classification,” BMC Genomic Data, vol. 24, 2022.
  * [19] K.-T. Tan, M. K. Slevin, M. Meyerson, and H. Li, “Identifying and correcting repeat-calling errors in nanopore sequencing of telomeres,” Genome Biology, vol. 23, Aug. 2022.
  * [20] J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang, “JAX: composable transformations of Python+NumPy programs,” 2018.
  * [21] J. Heek, A. Levskaya, A. Oliver, M. Ritter, B. Rondepierre, A. Steiner, and M. van Zee, “Flax: A neural network library and ecosystem for JAX,” 2023.
  * [22] L. Xue, A. Barua, N. Constant, R. Al-Rfou, S. Narang, M. Kale, A. Roberts, and C. Raffel, “ByT5: Towards a token-free future with pre-trained byte-to-byte models,” Mar. 2022.  arXiv:2105.13626 [cs].
  * [23] D. Cohn, O. Zuk, and T. Kaplan, “Enhancer identification using transfer and adversarial deep learning of dna sequences,” bioRxiv, 2018.
  * [24] F. J. Martin, M. R. Amode, A. Aneja, O. Austine-Orimoloye, A. G. Azov, I. H. A. Barnes, A. Becker, R. Bennett, A. E. Berry, J. Bhai, S. K. Bhurji, A. Bignell, S. Boddu, P. R. B. Lins, L. Brooks, S. B. Ramaraju, M. Charkhchi, A. Cockburn, L. D. R. Fioretto, C. Davidson, K. J. Dodiya, S. M. Donaldson, B. E. Houdaigui, T. E. Naboulsi, R. Fatima, C. García-Girón, T. A. L. Genez, G. S. Ghattaoraya, J. G. Martinez, C. P. Guijarro, M. Hardy, Z. Hollis, T. Hourlier, T. Hunt, M. P. Kay, V. Kaykala, T. Le, D. Lemos, D. Marques-Coelho, J. C. Marugán, G. A. Merino, L. P. Mirabueno, A. Mushtaq, S. N. Hossain, D. N. Ogeh, M. P. Sakthivel, A. Parker, M. Perry, I. Pilizota, I. Prosovetskaia, J. G. Pérez-Silva, A. I. A. Salam, N. Saraiva-Agostinho, H. Schuilenburg, D. Sheppard, S. Sinha, B. Sipos, W. Stark, E. Steed, R. Sukumaran, D. Sumathipala, M.-M. Suner, L. Surapaneni, K. Sutinen, M. Szpak, F. F. Tricomi, D. Urbina-Gómez, A. Veidenberg, T. A. Walsh, B. Walts, E. Wass, N. Willhoft, J. Allen, J. Álvarez, M. Chakiachvili, B. Flint, S. Giorgetti, L. Haggerty, G. R. Ilsley, J. E. Loveland, B. Moore, J. M. Mudge, J. G. Tate, D. Thybert, S. J. Trevanion, A. Winterbottom, A. Frankish, S. E. Hunt, M. Ruffier, F. Cunningham, S. Dyer, R. D. Finn, K. L. Howe, P. W. Harrison, A. D. Yates, and P. Flicek, “Ensembl 2023,” Nucleic Acids Research, vol. 51, pp. D933 – D941, 2022.
  * [25] R. K. Umarov and V. V. Solovyev, “Recognition of prokaryotic and eukaryotic promoters using convolutional deep learning neural networks,” PloS one, vol. 12, no. 2, p. e0171410, 2017.
  * [26] Q. Geng, R. Yang, and L. Zhang, “A deep learning framework for enhancer prediction using word embedding and sequence generation,” Biophysical Chemistry, vol. 286, p. 106822, 2022.
  * [27] S. Fort, J. J. Ren, and B. Lakshminarayanan, “Exploring the limits of out-of-distribution detection,” in Neural Information Processing Systems, 2021.
  * [28] D. S. Berman, C. Howser, T. Mehoke, and J. D. Evans, “Mutagan: A sequence-to-sequence gan framework to predict mutations of evolving protein populations,” Virus Evolution, vol. 9, 2020.
  * [29] InstaDeepAI, “Nucleotide transformer benchmark,” Hugging Face, Sep 2023.  URL: https://huggingface.co/spaces/InstaDeepAI/nucleotide_transformer_benchmark.
  * [30] Y. Bao, P. Bolotov, D. Dernovoy, B. Kiryutin, L. Zaslavsky, T. Tatusova, J. Ostell, and D. Lipman, “The influenza virus resource at the national center for biotechnology information,” Journal of Virology, vol. 82, pp. 596–601, Jan. 2008.
  * [31] N. A. O'Leary, M. W. Wright, J. R. Brister, S. Ciufo, D. Haddad, R. McVeigh, B. Rajput, B. Robbertse, B. Smith-White, D. Ako-Adjei, A. Astashyn, A. Badretdin, Y. Bao, O. Blinkova, V. Brover, V. Chetvernin, J. Choi, E. Cox, O. Ermolaeva, C. M. Farrell, T. Goldfarb, T. Gupta, D. Haft, E. Hatcher, W. Hlavina, V. S. Joardar, V. K. Kodali, W. Li, D. Maglott, P. Masterson, K. M. McGarvey, M. R. Murphy, K. O'Neill, S. Pujar, S. H. Rangwala, D. Rausch, L. D. Riddick, C. Schoch, A. Shkeda, S. S. Storz, H. Sun, F. Thibaud-Nissen, I. Tolstoy, R. E. Tully, A. R. Vatsan, C. Wallin, D. Webb, W. Wu, M. J. Landrum, A. Kimchi, T. Tatusova, M. DiCuccio, P. Kitts, T. D. Murphy, and K. D. Pruitt, “Reference sequence (RefSeq) database at NCBI: current status, taxonomic expansion, and functional annotation,” Nucleic Acids Research, vol. 44, pp. D733–D745, Nov. 2015.
  * [32] . G. P. Consortium, “A global reference for human genetic variation,” Nature, vol. 526, pp. 68–74, Sept. 2015.
  * [33] J. L. Harrow, A. Frankish, J. M. Gonzalez, E. Tapanari, M. E. Diekhans, F. Kokocinski, B. L. Aken, D. Barrell, A. Zadissa, S. M. J. Searle, I. H. A. Barnes, A. Bignell, V. Boychenko, T. Hunt, M. P. Kay, G. Mukherjee, J. Rajan, G. Despacio-Reyes, G. I. Saunders, C. A. Steward, R. A. Harte, M. F. Lin, C. Howald, A. Tanzer, T. L. Derrien, J. Chrast, N. Walters, S. Balasubramanian, B. Pei, M. L. Tress, J. M. Rodríguez, I. Ezkurdia, J. van Baren, M. R. Brent, D. Haussler, M. Kellis, A. Valencia, A. Reymond, M. B. Gerstein, R. Guigó, and T. J. P. Hubbard, “Gencode: The reference human genome annotation for the encode project,” Genome Research, vol. 22, pp. 1760 – 1774, 2012.
  * [34] X. C. Song, P. Smith, R. Kalyanam, X. Zhu, E. Adams, K. Colby, P. Finnegan, E. Gough, E. Hillery, R. Irvine, A. Maji, and J. St. John, “Anvil - system architecture and experiences from deployment and early user operations,” in Practice and Experience in Advanced Research Computing, PEARC ’22, (New York, NY, USA), Association for Computing Machinery, 2022.