#  A Fine-tuning Dataset and Benchmark for Large Language Models for Protein
Understanding

Yiqing Shen1,2,$\dagger$, Zan Chen1,$\dagger$, Michail Mamalakis3,$\dagger$,
Luhan He1,$\dagger$, Haiyang Xia1,4,$\dagger$,

Tianbin Li5, Yanzhou Su5, Junjun He5, Yu Guang Wang1,5,6,7

1Toursun Synbio 2Johns Hopkins University 3University of Cambridge

4Shanghai Institute for Biomedical and Pharmaceutical Technologies

5Shanghai AI Laboratory 6Shanghai Jiao Tong University 7UNSW Syndey
Corresponding author, E-mail<EMAIL_ADDRESS>$\dagger$Equal
contribution.

###### Abstract

The high similarities between protein sequences and natural language,
particularly in their sequential data structures, have driven parallel
advancements in deep learning models for both domains. In natural language
processing (NLP), large language models (LLMs) have achieved remarkable
success in tasks such as text generation, translation, and conversational
agents, owing to their extensive training on diverse datasets that enable them
to capture complex language patterns and generate human-like text. Inspired by
these advancements, researchers have attempted to adapt LLMs for protein
understanding by integrating a protein sequence encoder with a pre-trained
LLM, following designs like LLaVa. However, this adaptation raises a
fundamental question: “Can LLMs, originally designed for NLP, effectively
comprehend protein sequences as a form of language?” Current datasets fall
short in addressing this question due to the lack of a direct correlation
between protein sequences and corresponding text descriptions, limiting the
ability to train and evaluate LLMs for protein understanding effectively. To
bridge this gap, we introduce ProteinLMDataset, a dataset specifically
designed for further self-supervised pretraining and supervised fine-tuning
(SFT) of LLMs to enhance their capability for protein sequence comprehension.
Specifically, ProteinLMDataset includes 17.46 billion tokens for pretraining
and 893K instructions for SFT. Additionally, we present ProteinLMBench, the
first benchmark dataset consisting of 944 manually verified multiple-choice
questions for assessing the protein understanding capabilities of LLMs.
ProteinLMBench incorporates protein-related details and sequences in multiple
languages, establishing a new standard for evaluating LLMs’ abilities in
protein comprehension. The large language model InternLM2-7B, pretrained and
fine-tuned on the ProteinLMDataset, outperforms GPT-4 on ProteinLMBench,
achieving the highest accuracy score. The dataset and the benchmark are
available at https://huggingface.co/datasets/tsynbio/ProteinLMBench.

##  1 Introduction

Protein science offers insights into biological processes at the molecular
level, driving progress in medicine, biotechnology, and our understanding of
life. Given the similar sequential data structures of protein sequences and
natural language, there have been parallel advancements in deep learning
models, such as protein language models [23] and large language models (LLMs)
[20]. LLMs have already demonstrated their strong capabilities in text
understanding across various tasks, prompting researchers to explore their
potential in protein understanding from the perspective of multi-modal
language models (MMLMs) by integrating protein sequences or structures with
textual content using separate encoders. Specifically, they encode each
modality (protein sequence or protein structure) independently before
combining them with a frozen LLM [34, 37]. However, this approach poses
challenges in fully leveraging the intricate connections between protein
sequences and textual information. As far as we can ascertain, there is no
comprehensive dataset that seamlessly integrates protein sequences and their
corresponding textual descriptions, enabling effective training of language
models to comprehend protein information. This gap limits the ability to train
LLMs to fully utilize the intricate connections between protein sequences and
textual information.

To address these limitations, we introduce a comprehensive large-scale protein
sequence and text (seq-text) hybrid dataset named ProteinLMDataset. This
dataset is designed to train LLMs to learn the correspondence between textual
descriptions and protein sequences, enabling them to effectively understand
and leverage the intricate connections between these two forms of data.
Furthermore, we propose a novel benchmark ProteinLMBench comprising
meticulously curated multiple-choice questions to rigorously evaluate the
LLMs’ proficiency to comprehend protein sequences. To the best of our
knowledge, this is the first dataset of its kind, combining the largest
dataset for both self-supervised learning (SSL) and supervised fine-tuning
(SFT) specifically in the protein science domain, along with the first
manually annotated multiple-choice questions benchmark for evaluating protein
understanding capability with LLMs.

The ProteinLMDataset comprises 17.46 billion tokens for self-supervised
learning and 893K instructions for supervised fine-tuning. ProteinLMBench
contains 944 manually annotated multiple-choice questions for evaluation. The
self-supervised dataset is structured into three segments: 0.69% Chinese-
English text pairs in protein science, 41.51% protein sequence-English text
pairs, and 57.80% protein-related English text. The supervised fine-tuning
dataset includes seven tasks: a novel enzyme chain of thought (ECoT), protein
functionality, induction, disease involvement, post-translational
modifications, sub-unit structure, and tissue specificity. The ECoT approach
aims to innovatively enable LLMs to think step-by-step and generate well-
founded protein knowledge, thereby enhancing the reliability and accuracy of
outputs in protein science and engineering [32]. Unlike the plain chain of
thought, ECoT incorporates prior knowledge of protein science by guiding LLMs
to infer protein functions based on the reactions they are involved in. This
approach enhances logical reasoning capabilities in the protein domain.
ProteinLMDataset aims to bridge existing dataset gaps and unlock the potential
of LLMs in protein data analysis by seamlessly integrating protein sequences
with their relevant textual information.

The major contributions of this work are three-fold. Firstly, we present the
first large-scale protein-text dataset designed to enable LLMs to comprehend
protein sequences without an extra encoder. Secondly, we introduce the Enzyme
Chain of Thought (ECoT), a novel mechanism specifically for protein
understanding, allowing LLMs to generate reliable and accurate protein
knowledge step-by-step. Thirdly, we introduce ProteinLMBench, the first
comprehensive, manually annotated benchmark for thoroughly evaluating LLMs’
comprehension and representation of protein sequences.

##  2 Related work

An extended version of the literature review and related work is presented in
supplementary material: ’Thoroughly Related Work’. In this section, we
summarize the main limitations found in the literature.

Existing Chinese-English datasets. Traditional benchmarks in the English
language primarily focus on evaluating specific LLM abilities related to
singular tasks, such as providing answers in general question answering, text
summarization, or general linguistic understanding [31, 22, 18]. The existing
literature on English benchmarks primarily focuses on specific model abilities
for singular linguistic tasks, with limited extension to multi-domain and
multi-task assessments [12, 27, 17]. However, in the context of Chinese NLP
benchmarks, significant limitations emerge, including rational biases with
geographic restrictions to China regions dataset collection, absence of
analytical documentation and statistics, and lack of interpretability with
limited insights into reasoning abilities [13, 35]. Overall, these limitations
underscore the need for more comprehensive and diverse multi-task language
datasets, addressing a broader geographic scope of collection, incorporating
varied evaluation metrics, and ensuring coverage of distinct writing styles,
including scientific content. Notably, there is currently no dataset known to
the authors that provides text-to-text pairs for English and Chinese
translation.

Existing protein sequence datasets. Several crucial datasets related to
protein sequences, including UniProtKB [8], PEER [33], RefSeq [19], PDB [6],
CATH (Class, Architecture, Topology, Homology) [21], and CASP15 [26, 16],
cover a broad spectrum of different protein sequences. However, despite their
breadth, limitations exist, such as the focus on sequence-based data in the
PEER dataset [33]. Extending beyond these datasets presents challenges,
highlighting the need for ongoing community collaboration to ensure sustained
effectiveness. The literature reveals a lack of representation of the full
diversity of protein structures, as evidenced by limitations in RefSeq [6],
and limited applicability for real-world tasks, as noted in [10]. Addressing
these limitations requires further research to optimize instruction tuning
methods, refine the model’s ability to understand and follow instructions, and
improve its generalization across various protein and biomolecular tasks, as
highlighted in previous works [6, 26, 16]. Besides, it is essential to
acknowledge potential biases inherent in community contributions, which may
affect the representativeness of the dataset. The utilization of automated
systems introduces the possibility of errors, impacting the accuracy of
annotations, as noted in the limitations of UniProtKB [8]. Furthermore,
annotating proteins that are yet to be characterized presents inherent
challenges, with limitations persisting in achieving comprehensive
characterization, as observed in the PEER dataset [33]. Managing the influx of
whole-genome sequencing data and integrating diverse external resources also
present significant challenges, potentially complicating the coherent
presentation of data.

Protein design databases. KEGG [15] serves as a curated resource, offering
researchers a comprehensive repository of accurately annotated biological
entities, including molecular networks, genes, proteins, chemicals, and
health-related data. Despite its strengths, KEGG has limitations, such as a
predominant focus on disease-related perturbations in human cells, potential
biases in data inclusion, and the resource-intensive nature of manual
curation. In contrast, the STRING database [29] excels in its extensive
integration of known and predicted protein associations, covering both
physical interactions and functional relationships across over 14,000
organisms. However, STRING faces challenges like potential biases in text
mining, reliance on predictive methods, and the lack of options for tissue-
specific network pruning. Lastly, InterPro [5], a leading protein
classification database, provides comprehensive annotations through the
integration of predictive models, offering insights into families, domains,
and conserved sites. Despite its valuable contributions, InterPro’s
limitations include infrequent updates, challenges in interpretability, and
concerns about sustainability.

##  3 Dataset and Benchmark Composition

###  3.1 ProteinLMDataset

The ProteinLMDataset includes both self-supervised and supervised fine-tuning
components. The self-supervised part is strategically categorized into three
primary segments, each contributing distinct dimensions to our objectives of
empowering the LLMs to understand protein sequence. The first segment focuses
on comparing biological science in Chinese and English, encompassing 7,000
entries and yielding over 120 million tokens for training and tuning purposes.
The second segment contains protein sequence and English text pairs, derived
from a weakly linked PMC full-text dataset. Associations are established using
the Structure Integration with Function, Taxonomy, and Sequence (SIFTS)
database (https://www.ebi.ac.uk/pdbe/docs/sifts/), which maps PDB to UniProt
IDs and subsequently to protein names, guiding our search for relevant
literature within the PMC database. The full-text component comprises 320,000
entries, providing an extensive corpus exceeding 2.9 billion tokens. Moreover,
the PubMed abstract contributes over 30 million entries and 7.0 billion tokens
for comprehensive training and tuning. For protein sequence and text pairs, we
leverage a highly relevant PMC full-text dataset totaling 18,000 entries and
surpassing 195 million tokens for nuanced training and tuning. The integration
of the UniProtKB Swiss-Prot dataset further enriches this category with an
additional 349 million tokens. The final segment involves extracting insights
from a PubMed abstract dataset, boasting more than 3 million entries, and
generating over 7.0 billion tokens for robust training and tuning. This
strategic categorization ensures a diverse and versatile dataset, addressing
the limitations of existing benchmarks and facilitating advancements in cross-
lingual language models and protein sequence understanding.

The supervised fine-tuning component in ProteinLMDataset is a collection of
893,000 instructions, spanning seven different segments: enzyme Chain of
Thought (10.8k), protein functionality (465k), induction (25.4k), disease
involvement (5.58k), post-translational modifications (45.8k), sub-unit
structure (291k), and tissue specificity (50.3k). It includes analytical
instructions, input tokens, and ground truth prediction outputs for fine-
tuning and performance score determination. The primary source of data
collection is UniProtKB.

###  3.2 ProteinLMBench

Our proposed benchmark, ProteinLMBench, features an evaluation dataset
consisting of 944 six-choice questions, each accompanied by an explanation of
the correct answer. These questions cover a range of topics including protein-
based property prediction, protein descriptions, and protein sequence
understanding. This benchmark is designed to assess the ability of large
language models to interpret and analyze protein sequences in conjunction with
their associated textual descriptions.

##  4 Collection Methodology

###  4.1 The self-supervised dataset collection

##### Biology Chinese and English Text Pair Collection.

We extracted Chinese-English text pairs from articles published in the Chinese
Journal of Biotechnology and the Chinese Journal of Cell Biology. This
extraction process ensures a diverse and authentic representation of
scientific language. The selected time interval, 2013-2024, enhances the
dataset’s relevance by capturing contemporary scientific discourse in both
Chinese and English languages.

Figure 1: Process for pairing proteins with the corresponding PMC articles’
text

##### Protein sequence and English text pair.

We retrieved Protein Data Bank (PDB) IDs of proteins and their corresponding
PubMed IDs from the SIFTS database (https://www.ebi.ac.uk/pdbe/docs/sifts/).
Using a weakly linked PMC full-text dataset, we established associations
through the protein_name_to_pmcid.json index file derived from
pdb_chain_uniprot.csv. This mapping, connecting PDB to UniProt IDs and
subsequently to protein names, guided our literature search within the PMC
database. In this manner, we can associate a specific protein sequence with
the complete texts of the most relevant articles in PMC. We examined the
PubMed articles for PubMed Central (PMC) IDs; when available, we accessed the
full text. This enabled us to acquire protein sequences using PDB IDs and
retrieved detailed research articles on the proteins, creating pairs to
describe the protein sequences in human language (Fig. 1).

Figure 2: Process for inserting protein sequences into text

Protein sequence collection. We employed BERN2 [28] which is a Named Entity
Recognition (NER) model to extract NERs from the titles and abstracts of all
PubMed articles, identifying nouns potentially referring to proteins or genes
(since it is challenging to distinguish whether a single word refers to a
protein or gene without context) and recorded their confidence scores for
further analysis (Fig. 2). We then linked all extracted nouns to specific
entities in the NCBI Gene database [24] through Named Entity Normalization
(NEN) and obtained their NCBI Gene IDs. For these Gene IDs, we could map them
to specific protein sequences via the NCBI’s RefSeq database. Afterward, we
extracted all article titles and the results from the NER & NEN models with a
confidence score above a certain threshold (here, 98%), inputting them into a
large language model to determine whether these nouns are referred to as
proteins or genes in the given context. We assumed that protein names
appearing in titles are likely the focus of the research and used the
abstracts to help describe these proteins. Finally, we located the first
occurrence of each noun in the abstracts that matched the protein names
identified in the titles with the same NCBI Gene ID and inserted the
previously obtained protein sequences after these nouns. This way, we created
a dataset of textual descriptions wrapping protein sequences.

###  4.2 The supervised fine-tuning dataset collection

The primary source for the supervised fine-tuning dataset is UniProtKB.
Through a comprehensive analysis of the locally downloaded raw data, we
identified a substantial corpus of curated and verified protein information.
This encompasses a diverse range of attributes, including protein
functionality, induction, disease associations, post-translational
modifications, sub-unit structures, and tissue specificity. For each protein,
we crafted a dedicated template. The protein sequence was embedded within the
question template, while its corresponding properties were incorporated into
the answer template. This approach facilitated the creation of effective
question-answer pairs to understand protein and its properties. In total, we
generated approximately 883k such data entries.

To create the Chain of Thought (CoT) data, we utilized the IUBMB Enzyme
Nomenclature as the raw source. We extracted the name, nickname, reaction
formula, and comments (typically describing the enzyme’s function) for each
enzyme entry. The response generation process followed a systematic thinking
process: first, the model outputted the reactions the enzyme would catalyze,
and then provided the enzyme’s comments, forming a chain of thought. During
data creation, we generated multiple suitable templates and inserted the
obtained information, resulting in an effective CoT supervised fine-tuning
dataset comprising 10,800 entries.

###  4.3 The benchmark multi-choice questionnaire collection

The evaluation questionnaire, ProteinLMBench, comprises six choices for each
question, accompanied by an explanation of the correct answer. The questions
span various domains, including protein-based property prediction, protein
descriptions, and protein sequence understanding. The questionnaire consists
of a synthetic dataset generated by Retrieval-Augmented Generation (RAG; [11])
and GPT-4 [20]. We implemented a two-round verification scheme involving RAG
to generate the initial questions, choices, and answers, followed by GPT-4 to
validate the answers. Questions with inconsistencies between the two rounds
were discarded, ensuring the consistency, robustness, and reproducibility of
the questionnaire. Despite starting with 1,000 questions, the verification
process yielded a final set of 944 questions.

###  4.4 Text input filtering and tokenization procedure

To preprocess the text inputs from three collections, we employed a filter
cleaner to eliminate citations, references, tables, figures, commas, and any
symbols unrelated to the task of interest. This filter was systematically
applied to both titles and abstracts within the collected manuscripts. The
resulting filtered outputs were stored in ".json" format. Subsequently, this
json files serve as input for extracting tokens of various sizes associated
with the collection and the task of interest, such as protein sequences, pairs
of English and Chinese text, protein descriptions, and design elements. The
implementation source code for the filters is available on our GitHub
repository at the following link:
https://github.com/tsynbio/Protein_LM/blob/main/src/PMC_data_collector.py. The
tokenization is performed using the open-source coding of the Interlm
tokenizer based on byte-level byte-pair-encoding
(https://huggingface.co/internlm/internlm-7b/blob/main/tokenization_internlm.py),
which processes the filtered text inputs.

Table 1: The overall statistics of extracted tokens from the proposed fine-
tuning dataset represent the token length per document. The dataset is
categorized into seven segments: Enzyme Chain of Thought (ECoT), Protein
Functionality, Induction of Protein Expression, Disease Involvement, Post-
Translational Modifications, Sub-Unit Structure, and Tissue Specificity. These
statistics are aggregated across the entire cohort.

Tokens statistic of the fine-tuning dataset
---
Segment | Scope | Min Len. | Average Len. | Max Len.
Enzyme CoT | molecule’s expression | 84 | 800 | 15300
UniProt Function | protein functionality | 74 | 3610 | 70500
UniProt Induction | protein induction | 82 | 1650 | 31300
UniProt Involvement in disease | protein diseases | 88 | 3495 | 68800
Uniprot Post-translational modification | protein translation | 88 | 361 | 6420
UniProt Sub-unit structure | protein structure | 74 | 3560 | 70500
UniProt Tissue specificity | tissue gene expression | 65 | 145 | 2310

##  5 Quality of dataset and benchmark

##### Diversity and statistics of ProteinLMDataset.

The diversity and multi-source curation of the dataset are presented in Tab.
2, highlighting the document collection based on the token length. The three
segments of the proposed dataset includes manuscripts, abstracts, protein
sequences, and taxonomy, divided into five collections: ’Biology
Chinese/English text pair’, ’PMC full-text manuscripts’, ’PubMed Abstracts’,
’UniProtKB Swiss-Prot’, and ’Protein Sequence/PubMed Abstracts’ and a
variability ranges from minimum lengths of 49 to 549 characters to maximum
lengths of 39,239 to 8,390,807 characters. Tab. 2 highlights the extracted
tokens from the documentation characters after the filtering and noise
reduction process (citations, references, tables, etc. unrelated to the task
of interest). The produced tokens exhibit variability, ranging from minimum
lengths of 21 to 101 characters to maximum lengths of 1,017 to 2,351,194
characters. These variations across different tokens, texts and combinations
validate the unbiased and comprehensive content of our dataset.

Table 2: The overall statistics of extracted tokens from the proposed self-
supervised dataset. The numbers are representation of the Length (Len.) token
per document. The dataset classify in three segments-Biology Chinese/English
(Chi/Eng) text pair, Protein Sequence and English (Seq/Eng) text pair and
Protein Sequence (Seq)-. The statistics are aggregated over the entire cohort.
Tokens statistic of the self-supervised dataset
---
Segment | Collection | Min Len. | Average Len. | Max Len.
Biology Chi/Eng text pair | Chinese/English text | 54/48 | 245/267 | 1017/1165
Protein Seq/Eng text pair | PMC Full text | 101 | 9954 | 2351194
| PubMed Abstract | 21 | 282 | 13983
| UniProtKB Swiss-Prot | 69 | 591 | 35751
Protein Seq | Protein Seq | 65 | 1650 | 70500

Fig. 3 illustrates the statistics regarding the different sources and ratios
within our three segment datasets. Our dataset predominantly represents
protein sequences, with the primary source being PubMed abstracts (6.9B),
followed by UniProtKB Swiss-Prot collection (349M). Regarding protein design
and information, the primary resources are derived from PubMed abstracts of
scientific manuscripts (7.0B), followed by weakly linked PMC full manuscripts
and highly relevant PMC full texts (2.9B; see Fig. 3a.). Additionally, the
Chinese-English collection comprises a substantial of 200M tokens. To delve
deeper, the UniProtKB Swiss-Prot collection consists of 62% protein sequences
and 38% text relevant to protein design and scientific biology documentation
tokens, contrasting with the Protein Seq/PubMed Abstract collection, which
includes 16% text documentation and 84% protein sequences tokens (see Fig.
3b).

(a) Statistics for all tokens

(b) Statistics for tokens from UniProtKB and Seq/PubMed

Figure 3: Statistics of the sources and ratios of the three segments of
ProteinLMDataset: Chinese-English pairs, protein sequences, and protein
sequence-text pairs.

Tab. 1 presents the diversity of multi-design topics and curation of the fine-
tuning dataset, highlighting the different tasks and analyzing the statistics
based on token length. The proposed benchmark comprises seven segments:
’Enzyme CoT’, ’UniProt Function’, ’UniProt Induction’, ’UniProt Involvement in
disease’, ’UniProt Post-translational modification’, ’UniProt Sub-unit
structure’, and ’UniProt Tissue specificity’. These segments exhibit a wide
range of token lengths, varying from a minimum of 65 to 88 characters to a
maximum of 2,310 to 70,500 tokens. Tab. 1 highlights the extracted token
counts for each segment. After filtering and noise reduction, the instructions
and output of the benchmark are provided on the website
https://huggingface.co/datasets/tsynbio/ProteinLMBench. The variations across
different token lengths, fine-tuning tasks, and combinations validate the
unbiased and comprehensive nature of our benchmark.

##### Diversity and statistics of the ProteinLMBench

The evaluation dataset consists of 944 questions with a median string length
of 98-124 tokens, with a minimum of 46 and a maximum of 305 tokens. There are
six answer choices, with an average string length of 77-114 tokens, with a
minimum of 3 and a maximum of 368 tokens. The answers are well-balanced, with
the correct answer being option 1 in 16.3% of the cases, option 2 in 17.9%,
option 3 in 19.9%, option 4 in 16.1%, option 5 in 15.6%, and option 6 in 14.0%
of the cases.

##### Safety during collection

In ensuring the safety and reliability of our proposed dataset, we have
implemented rigorous measures and levels of filtering to address potential
issues related to data integrity and quality. Given the inherent imperfections
in structural and textual data, we acknowledge the importance of safeguarding
against corrupted or erroneous information. Our dataset curation process
includes thorough validation checks and quality control procedures to identify
and rectify any anomalies that may compromise the integrity of the data. We
are committed to transparency in our approach, encouraging users to report any
instances of suspected data corruption or safety concerns through our
designated channels. This proactive engagement with the research community is
integral to maintaining the credibility of the dataset and fostering a
collaborative environment for continual improvement. Additionally, our
strategic categorization approach facilitates the identification and isolation
of potential outliers, contributing to the overall safety and reliability of
the dataset by providing users with a clear understanding of the distinct
dimensions and sources within each segment. The ethical statement of this work
is presented in Appendix Section B.2.

Figure 4: Model performence of LLMs on ProteinLMBench. The purple box plot
represents the model performance combined training on ProteinLMDataset SSL and
SFT datasets. The pink box plot represents the model performance obtained by
only conducting SFT without SSL training. The green box plots represent the
performance of other comparative models.

##### Technical Limitations

Even when systematically collecting protein sequence data through scientific
filtering and established resources, inherent technical limitations persist.
Achieving perfection in structural data is rare, as experimental uncertainties
stem from the limited resolution of techniques like X-ray crystallography or
electron cryo-microscopy.

##  6 Experiment

##### Experiment setup

We conducted three different experiments: no training, supervised fine-tuning
training (SFT), and self-supervised learning combined with supervised fine-
tuning training (SSL-SFT) using ProteinLMDataset. For the no training
experiment, we employed various large language models, including Falcon-7b
[2], Qwen1.5-7B [3], Moonshot [36], Mistral-7B-Instruct-v0.2 [14],
Baichuan2-7B-Chat [4], Llama-2-7B-Chat-hf [30], InternLM-Chat-20B,
InternLM2-Chat-7B, and variations [7], ChatGLM3-6B [9], Yi-6B-Chat [1],
GPT3.5-turbo, and GPT4.0-turbo [20]. For the SFT and SSL-SFT experiments, we
utilized the InternLM2-7B model, naming them InternLM2-Protein-7B (w/o SSL)
and InternLM2-Protein-7B, respectively. All the experiments were evaluated for
accuracy using our proposed ProteinLMBench. For detailed training procedures
and LLM architectures, please refer to Appendix C.

##### Experiment results

The results of all experiments are presented in Fig. 4. In the no-training
experiment (depicted by green box plots), GPT-4.0-turbo achieved the highest
accuracy with a correct rate of 57.94%, followed by InternLM2-20B with a score
of 57.52%. Falcon-7B exhibited the poorest performance, achieving a correct
rate of 19.17%. In the supervised fine-tuning experiment (illustrated by pink
box plots), InternLM2-7B delivered accuracy scores of 58.26% (Fig. 4;
InternLM2-Protein-7B (w/o SSL)). This is an improvement from the no training
experiment, where its accuracy was 54.98%. Notably, when we initially trained
InternLM2-7B model on the self-supervised dataset and subsequently fine-tuned
it (Fig. 4; InternLM2-Protein-7B), the accuracy increased significantly to
62.18%. This underscores the importance of the proposed ProteinLMDataset for
more efficient and accurate protein understanding.

##  7 Conclusion

Our proposed dataset-benchmark tandem offers a comprehensive platform for
training, fine-tuning, and evaluating language models on both Chinese-English
text pairs and protein sequences, addressing limitations in existing datasets.
The strategic classification approach ensures diversity and adaptability,
tackling constraints within protein science and language model benchmarking.
By curating a diverse collection from various sources, we reinforce the
impartiality of our dataset. Incorporating both Chinese and English in
detailing protein characteristics introduces a novel cross-lingual
perspective, mitigating biases and enhancing multilingual understanding.
Moreover, our proposed fine-tuning dataset enhances language models’
capabilities in understanding, designing, and analyzing protein sequences.
Experimentation within our ProteinLMBench benchmark validates the efficacy of
our dataset. The InternLM2-7B language model achieved an accuracy of less than
55.00% without further training on the ProteinLMDataset. When fine-tuned, its
accuracy increased to 58.26%. Notably, when InternLM2-7B was trained on both
the self-supervised and fine-tuning datasets of ProteinLMDataset, it exhibited
an exceptional improvement in accuracy, reaching 62.18%. Our dataset and
benchmark contribute to advancing artificial intelligence for biological
sciences. It enables groundbreaking research and applications with the
potential to transform diverse fields.

## References

  * 01.AI et al. [2024] 01.AI, :, A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, K. Yu, P. Liu, Q. Liu, S. Yue, S. Yang, S. Yang, T. Yu, W. Xie, W. Huang, X. Hu, X. Ren, X. Niu, P. Nie, Y. Xu, Y. Liu, Y. Wang, Y. Cai, Z. Gu, Z. Liu, and Z. Dai.  Yi: Open Foundation Models by 01.AI.  _arXiv preprint arXiv:2403.04652_ , 2024.
  * Almazrouei et al. [2023] E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Debbah, Étienne Goffinet, D. Hesslow, J. Launay, Q. Malartic, D. Mazzotta, B. Noune, B. Pannier, and G. Penedo.  The Falcon Series of Open Language Models.  _arXiv preprint arXiv:2311.16867_ , 2023.
  * Bai et al. [2023] J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. Yang, S. Yang, Y. Yao, B. Yu, H. Yuan, Z. Yuan, J. Zhang, X. Zhang, Y. Zhang, Z. Zhang, C. Zhou, J. Zhou, X. Zhou, and T. Zhu.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_ , 2023.
  * Baichuan [2023] Baichuan.  Baichuan 2: Open large-scale language models.  _arXiv preprint arXiv:2309.10305_ , 2023.  URL https://arxiv.org/abs/2309.10305.
  * Blum et al. [2020] M. Blum, H.-Y. Chang, S. Chuguransky, T. Grego, S. Kandasaamy, A. Mitchell, G. Nuka, T. Paysan-Lafosse, M. Qureshi, S. Raj, L. Richardson, G. A. Salazar, L. Williams, P. Bork, A. Bridge, J. Gough, D. H. Haft, I. Letunic, A. Marchler-Bauer, H. Mi, D. A. Natale, M. Necci, C. A. Orengo, A. P. Pandurangan, C. Rivoire, C. J. A. Sigrist, I. Sillitoe, N. Thanki, P. D. Thomas, S. C. E. Tosatto, C. H. Wu, A. Bateman, and R. D. Finn.  The InterPro protein families and domains database: 20 years on.  _Nucleic Acids Research_ , 49(D1):D344–D354, 11 2020.  ISSN 0305-1048.  doi: 10.1093/nar/gkaa977.  URL https://doi.org/10.1093/nar/gkaa977.
  * Burley et al. [2020] S. K. Burley, C. Bhikadiya, C. Bi, S. Bittrich, L. Chen, G. V. Crichlow, C. H. Christie, K. Dalenberg, L. Di Costanzo, J. M. Duarte, S. Dutta, Z. Feng, S. Ganesan, D. S. Goodsell, S. Ghosh, R. K. Green, V. Guranović, D. Guzenko, B. P. Hudson, C. Lawson, Y. Liang, R. Lowe, H. Namkoong, E. Peisach, I. Persikova, C. Randle, A. Rose, Y. Rose, A. Sali, J. Segura, M. Sekharan, C. Shao, Y.-P. Tao, M. Voigt, J. Westbrook, J. Y. Young, C. Zardecki, and M. Zhuravleva.  RCSB Protein Data Bank: powerful new tools for exploring 3D structures of biological macromolecules for basic and applied research and education in fundamental biology, biomedicine, biotechnology, bioengineering and energy sciences.  _Nucleic Acids Research_ , 49(D1):D437–D451, 11 2020.  ISSN 0305-1048.  doi: 10.1093/nar/gkaa1038.  URL https://doi.org/10.1093/nar/gkaa1038.
  * Cai et al. [2024] Z. Cai, M. Cao, H. Chen, K. Chen, K. Chen, X. Chen, X. Chen, Z. Chen, Z. Chen, P. Chu, X. Dong, H. Duan, Q. Fan, Z. Fei, Y. Gao, J. Ge, C. Gu, Y. Gu, T. Gui, A. Guo, Q. Guo, C. He, Y. Hu, T. Huang, T. Jiang, P. Jiao, Z. Jin, Z. Lei, J. Li, J. Li, L. Li, S. Li, W. Li, Y. Li, H. Liu, J. Liu, J. Hong, K. Liu, K. Liu, X. Liu, C. Lv, H. Lv, K. Lv, L. Ma, R. Ma, Z. Ma, W. Ning, L. Ouyang, J. Qiu, Y. Qu, F. Shang, Y. Shao, D. Song, Z. Song, Z. Sui, P. Sun, Y. Sun, H. Tang, B. Wang, G. Wang, J. Wang, J. Wang, R. Wang, Y. Wang, Z. Wang, X. Wei, Q. Weng, F. Wu, Y. Xiong, C. Xu, R. Xu, H. Yan, Y. Yan, X. Yang, H. Ye, H. Ying, J. Yu, J. Yu, Y. Zang, C. Zhang, L. Zhang, P. Zhang, P. Zhang, R. Zhang, S. Zhang, S. Zhang, W. Zhang, W. Zhang, X. Zhang, X. Zhang, H. Zhao, Q. Zhao, X. Zhao, F. Zhou, Z. Zhou, J. Zhuo, Y. Zou, X. Qiu, Y. Qiao, and D. Lin.  Internlm2 technical report.  _arXiv preprint arXiv:2403.17297_ , 2024.
  * Consortium [2022] T. U. Consortium.  UniProt: the Universal Protein Knowledgebase in 2023.  _Nucleic Acids Research_ , 51(D1):D523–D531, 11 2022.  ISSN 0305-1048.  doi: 10.1093/nar/gkac1052.  URL https://doi.org/10.1093/nar/gkac1052.
  * Du et al. [2022] Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang.  GLM: General Language Model Pretraining with Autoregressive Blank Infilling.  In _ACL (Volume 1: Long Papers)_ , pages 320–335, 2022.
  * Fang et al. [2024] Y. Fang, X. Liang, N. Zhang, K. Liu, R. Huang, Z. Chen, X. Fan, and H. Chen.  Mol-Instructions: A Large-Scale Biomolecular Instruction Dataset for Large Language Models.  In _ICLR_ , 2024.  URL https://openreview.net/forum?id=Tlsdsb6l9n.
  * Gao et al. [2024] Y. Gao et al.  Retrieval-augmented generation for large language models: A survey.  _arXiv preprint arXiv:2312.10997_ , 2024.
  * Hendrycks et al. [2021] D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt.  Measuring massive multitask language understanding.  In _ICLR_ , 2021.  URL https://openreview.net/forum?id=d7KBjmI3GmQ.
  * Huang et al. [2023] Y. Huang, Y. Bai, Z. Zhu, J. Zhang, J. Zhang, T. Su, J. Liu, C. Lv, Y. Zhang, j. lei, Y. Fu, M. Sun, and J. He.  C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models.  In _NeurIPS_ , volume 36, pages 62991–63010, 2023.
  * Jiang et al. [2023] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed.  Mistral 7B.  _arXiv preprint arXiv:2310.06825_ , 2023.
  * Kanehisa et al. [2022] M. Kanehisa, M. Furumichi, Y. Sato, M. Kawashima, and M. Ishiguro-Watanabe.  KEGG for taxonomy-based analysis of pathways and genomes.  _Nucleic Acids Research_ , 51(D1):D587–D592, 10 2022.  ISSN 0305-1048.  doi: 10.1093/nar/gkac963.  URL https://doi.org/10.1093/nar/gkac963.
  * Kinch et al. [2021] L. Kinch, R. Schaeffer, A. Kryshtafovych, and N. Grishin.  Target classification in the 14th round of the critical assessment of protein structure prediction (CASP14).  _Proteins: Structure, Function and Bioinformatics_ , 89(12):1618–1632, Dec. 2021.  ISSN 0887-3585.  doi: 10.1002/prot.26202.
  * Liang et al. [2023] P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. A. Cosgrove, C. D. Manning, C. Re, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. WANG, K. Santhanam, L. Orr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. Kim, N. Guha, N. S. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. A. Chi, S. M. Xie, S. Santurkar, S. Ganguli, T. Hashimoto, T. Icard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Y. Mai, Y. Zhang, and Y. Koreeda.  Holistic Evaluation of Language Models.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=iO4LZibEqW.  Featured Certification, Expert Certification.
  * Narayan et al. [2018] S. Narayan, S. B. Cohen, and M. Lapata.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _EMNLP_ , pages 1797–1807, 2018.  doi: 10.18653/v1/D18-1206.  URL https://aclanthology.org/D18-1206.
  * O’Leary et al. [2015] N. A. O’Leary, M. W. Wright, J. R. Brister, S. Ciufo, D. Haddad, R. McVeigh, B. Rajput, B. Robbertse, B. Smith-White, D. Ako-Adjei, A. Astashyn, A. Badretdin, Y. Bao, O. Blinkova, V. Brover, V. Chetvernin, J. Choi, E. Cox, O. Ermolaeva, C. M. Farrell, T. Goldfarb, T. Gupta, D. Haft, E. Hatcher, W. Hlavina, V. S. Joardar, V. K. Kodali, W. Li, D. Maglott, P. Masterson, K. M. McGarvey, M. R. Murphy, K. O’Neill, S. Pujar, S. H. Rangwala, D. Rausch, L. D. Riddick, C. Schoch, A. Shkeda, S. S. Storz, H. Sun, F. Thibaud-Nissen, I. Tolstoy, R. E. Tully, A. R. Vatsan, C. Wallin, D. Webb, W. Wu, M. J. Landrum, A. Kimchi, T. Tatusova, M. DiCuccio, P. Kitts, T. D. Murphy, and K. D. Pruitt.  Reference sequence (RefSeq) database at NCBI: current status, taxonomic expansion, and functional annotation.  _Nucleic Acids Research_ , 44(D1):D733–D745, 11 2015.  ISSN 0305-1048.  doi: 10.1093/nar/gkv1189.  URL https://doi.org/10.1093/nar/gkv1189.
  * OpenAI et al. [2024] OpenAI et al.  GPT-4 Technical Report.  _arxiv preprint arXiv:2303.08774_ , 2024.
  * Orengo et al. [1997] C. Orengo, A. Michie, S. Jones, D. Jones, M. Swindells, and J. Thornton.  CATH – a hierarchic classification of protein domain structures.  _Structure_ , 5(8):1093–1109, 1997.  ISSN 0969-2126.  doi: https://doi.org/10.1016/S0969-2126(97)00260-8.  URL https://www.sciencedirect.com/science/article/pii/S0969212697002608.
  * Rajpurkar et al. [2018] P. Rajpurkar, R. Jia, and P. Liang.  Know What You Don’t Know: Unanswerable Questions for SQuAD.  In _ACL_ , 2018.
  * Rives et al. [2019] A. Rives, J. Meier, T. Sercu, S. Goyal, Z. Lin, J. Liu, D. Guo, M. Ott, C. L. Zitnick, J. Ma, and R. Fergus.  Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences.  _PNAS_ , 2019.  doi: 10.1101/622803.  URL https://www.biorxiv.org/content/10.1101/622803v4.
  * Sayers et al. [2021] E. W. Sayers, E. E. Bolton, J. R. Brister, K. Canese, J. Chan, D. C. Comeau, R. Connor, K. Funk, C. Kelly, S. Kim, T. Madej, A. Marchler-Bauer, C. Lanczycki, S. Lathrop, Z. Lu, F. Thibaud-Nissen, T. Murphy, L. Phan, Y. Skripchenko, T. Tse, J. Wang, R. Williams, B. W. Trawick, K. D. Pruitt, and S. T. Sherry.  Database resources of the national center for biotechnology information.  _Nucleic Acids Research_ , 50(D1):D20–D26, 12 2021.  ISSN 0305-1048.  doi: 10.1093/nar/gkab1112.  URL https://doi.org/10.1093/nar/gkab1112.
  * Schuhmann et al. [2022] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al.  LAION-5B: an open large-scale dataset for training next generation image-text models.  In _NeurIPS_ , volume 35, pages 25278–25294, 2022.
  * Senior et al. [2019] A. W. Senior, R. Evans, J. Jumper, J. Kirkpatrick, L. Sifre, T. Green, C. Qin, A. Žídek, A. W. R. Nelson, A. Bridgland, H. Penedones, S. Petersen, K. Simonyan, S. Crossan, P. Kohli, D. T. Jones, D. Silver, K. Kavukcuoglu, and D. Hassabis.  Protein structure prediction using multiple deep neural networks in the 13th Critical Assessment of Protein Structure Prediction (CASP13).  _Proteins: Structure, Function, and Bioinformatics_ , 87(12):1141–1148, 2019.  doi: https://doi.org/10.1002/prot.25834.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/prot.25834.
  * Srivastava et al. [2023] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, A. Kluska, A. Lewkowycz, A. Agarwal, A. Power, A. Ray, A. Warstadt, A. W. Kocurek, A. Safaya, A. Tazarv, A. Xiang, A. Parrish, A. Nie, A. Hussain, A. Askell, A. Dsouza, A. Slone, A. Rahane, A. S. Iyer, A. J. Andreassen, A. Madotto, A. Santilli, A. Stuhlmüller, A. M. Dai, A. La, A. Lampinen, A. Zou, A. Jiang, A. Chen, A. Vuong, A. Gupta, A. Gottardi, A. Norelli, A. Venkatesh, A. Gholamidavoodi, A. Tabassum, A. Menezes, A. Kirubarajan, A. Mullokandov, A. Sabharwal, A. Herrick, A. Efrat, A. Erdem, A. Karakaş, B. R. Roberts, B. S. Loe, B. Zoph, B. Bojanowski, B. Özyurt, B. Hedayatnia, B. Neyshabur, B. Inden, B. Stein, B. Ekmekci, B. Y. Lin, B. Howald, B. Orinion, C. Diao, C. Dour, C. Stinson, C. Argueta, C. Ferri, C. Singh, C. Rathkopf, C. Meng, C. Baral, C. Wu, C. Callison-Burch, C. Waites, C. Voigt, C. D. Manning, C. Potts, C. Ramirez, C. E. Rivera, C. Siro, C. Raffel, C. Ashcraft, C. Garbacea, D. Sileo, D. Garrette, D. Hendrycks, D. Kilman, D. Roth, C. D. Freeman, D. Khashabi, D. Levy, D. M. González, D. Perszyk, D. Hernandez, D. Chen, D. Ippolito, D. Gilboa, D. Dohan, D. Drakard, D. Jurgens, D. Datta, D. Ganguli, D. Emelin, D. Kleyko, D. Yuret, D. Chen, D. Tam, D. Hupkes, D. Misra, D. Buzan, D. C. Mollo, D. Yang, D.-H. Lee, D. Schrader, E. Shutova, E. D. Cubuk, E. Segal, E. Hagerman, E. Barnes, E. Donoway, E. Pavlick, E. Rodolà, E. Lam, E. Chu, E. Tang, E. Erdem, E. Chang, E. A. Chi, E. Dyer, E. Jerzak, E. Kim, E. E. Manyasi, E. Zheltonozhskii, F. Xia, F. Siar, F. Martínez-Plumed, F. Happé, F. Chollet, F. Rong, G. Mishra, G. I. Winata, G. de Melo, G. Kruszewski, G. Parascandolo, G. Mariani, G. X. Wang, G. Jaimovitch-Lopez, G. Betz, G. Gur-Ari, H. Galijasevic, H. Kim, H. Rashkin, H. Hajishirzi, H. Mehta, H. Bogar, H. F. A. Shevlin, H. Schuetze, H. Yakura, H. Zhang, H. M. Wong, I. Ng, I. Noble, J. Jumelet, J. Geissinger, J. Kernion, J. Hilton, J. Lee, J. F. Fisac, J. B. Simon, J. Koppel, J. Zheng, J. Zou, J. Kocon, J. Thompson, J. Wingfield, J. Kaplan, J. Radom, J. Sohl-Dickstein, J. Phang, J. Wei, J. Yosinski, J. Novikova, J. Bosscher, J. Marsh, J. Kim, J. Taal, J. Engel, J. Alabi, J. Xu, J. Song, J. Tang, J. Waweru, J. Burden, J. Miller, J. U. Balis, J. Batchelder, J. Berant, J. Frohberg, J. Rozen, J. Hernandez-Orallo, J. Boudeman, J. Guerr, J. Jones, J. B. Tenenbaum, J. S. Rule, J. Chua, K. Kanclerz, K. Livescu, K. Krauth, K. Gopalakrishnan, K. Ignatyeva, K. Markert, K. Dhole, K. Gimpel, K. Omondi, K. W. Mathewson, K. Chiafullo, K. Shkaruta, K. Shridhar, K. McDonell, K. Richardson, L. Reynolds, L. Gao, L. Zhang, L. Dugan, L. Qin, L. Contreras-Ochando, L.-P. Morency, L. Moschella, L. Lam, L. Noble, L. Schmidt, L. He, L. Oliveros-Colón, L. Metz, L. K. Senel, M. Bosma, M. Sap, M. T. Hoeve, M. Farooqi, M. Faruqui, M. Mazeika, M. Baturan, M. Marelli, M. Maru, M. J. Ramirez-Quintana, M. Tolkiehn, M. Giulianelli, M. Lewis, M. Potthast, M. L. Leavitt, M. Hagen, M. Schubert, M. O. Baitemirova, M. Arnaud, M. McElrath, M. A. Yee, M. Cohen, M. Gu, M. Ivanitskiy, M. Starritt, M. Strube, M. Swędrowski, M. Bevilacqua, M. Yasunaga, M. Kale, M. Cain, M. Xu, M. Suzgun, M. Walker, M. Tiwari, M. Bansal, M. Aminnaseri, M. Geva, M. Gheini, M. V. T, N. Peng, N. A. Chi, N. Lee, N. G.-A. Krakover, N. Cameron, N. Roberts, N. Doiron, N. Martinez, N. Nangia, N. Deckers, N. Muennighoff, N. S. Keskar, N. S. Iyer, N. Constant, N. Fiedel, N. Wen, O. Zhang, O. Agha, O. Elbaghdadi, O. Levy, O. Evans, P. A. M. Casares, P. Doshi, P. Fung, P. P. Liang, P. Vicol, P. Alipoormolabashi, P. Liao, P. Liang, P. W. Chang, P. Eckersley, P. M. Htut, P. Hwang, P. Miłkowski, P. Patil, P. Pezeshkpour, P. Oli, Q. Mei, Q. Lyu, Q. Chen, R. Banjade, R. E. Rudolph, R. Gabriel, R. Habacker, R. Risco, R. Millière, R. Garg, R. Barnes, R. A. Saurous, R. Arakawa, R. Raymaekers, R. Frank, R. Sikand, R. Novak, R. Sitelew, R. L. Bras, R. Liu, R. Jacobs, R. Zhang, R. Salakhutdinov, R. A. Chi, S. R. Lee, R. Stovall, R. Teehan, R. Yang, S. Singh, S. M. Mohammad, S. Anand, S. Dillavou, S. Shleifer, S. Wiseman, S. Gruetter, S. R. Bowman, S. S. Schoenholz, S. Han, S. Kwatra, S. A. Rous, S. Ghazarian, S. Ghosh, S. Casey, S. Bischoff, S. Gehrmann, S. Schuster, S. Sadeghi, S. Hamdan, S. Zhou, S. Srivastava, S. Shi, S. Singh, S. Asaadi, S. S. Gu, S. Pachchigar, S. Toshniwal, S. Upadhyay, S. S. Debnath, S. Shakeri, S. Thormeyer, S. Melzi, S. Reddy, S. P. Makini, S.-H. Lee, S. Torene, S. Hatwar, S. Dehaene, S. Divic, S. Ermon, S. Biderman, S. Lin, S. Prasad, S. Piantadosi, S. Shieber, S. Misherghi, S. Kiritchenko, S. Mishra, T. Linzen, T. Schuster, T. Li, T. Yu, T. Ali, T. Hashimoto, T.-L. Wu, T. Desbordes, T. Rothschild, T. Phan, T. Wang, T. Nkinyili, T. Schick, T. Kornev, T. Tunduny, T. Gerstenberg, T. Chang, T. Neeraj, T. Khot, T. Shultz, U. Shaham, V. Misra, V. Demberg, V. Nyamai, V. Raunak, V. V. Ramasesh, vinay uday prabhu, V. Padmakumar, V. Srikumar, W. Fedus, W. Saunders, W. Zhang, W. Vossen, X. Ren, X. Tong, X. Zhao, X. Wu, X. Shen, Y. Yaghoobzadeh, Y. Lakretz, Y. Song, Y. Bahri, Y. Choi, Y. Yang, Y. Hao, Y. Chen, Y. Belinkov, Y. Hou, Y. Hou, Y. Bai, Z. Seid, Z. Zhao, Z. Wang, Z. J. Wang, Z. Wang, and Z. Wu.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=uyTL5Bvosj.
  * Sung et al. [2022] M. Sung, M. Jeong, Y. Choi, D. Kim, J. Lee, and J. Kang.  BERN2: an advanced neural biomedical named entity recognition and normalization tool.  _Bioinformatics_ , 38(20):4837–4839, Sept. 2022.  ISSN 1367-4811.  doi: 10.1093/bioinformatics/btac598.  URL http://dx.doi.org/10.1093/bioinformatics/btac598.
  * Szklarczyk et al. [2020] D. Szklarczyk, A. L. Gable, K. C. Nastou, D. Lyon, R. Kirsch, S. Pyysalo, N. T. Doncheva, M. Legeay, T. Fang, P. Bork, L. J. Jensen, and C. von Mering.  The STRING database in 2021: customizable protein–protein networks, and functional characterization of user-uploaded gene/measurement sets.  _Nucleic Acids Research_ , 49(D1):D605–D612, 11 2020.  ISSN 0305-1048.  doi: 10.1093/nar/gkaa1074.  URL https://doi.org/10.1093/nar/gkaa1074.
  * Touvron et al. [2023] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  _arXiv preprint arXiv:2307.09288_ , 2023.
  * Wang et al. [2019] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _ICLR_ , 2019.  URL https://openreview.net/forum?id=rJ4km2R5t7.
  * Wei et al. [2022] J. Wei, X. Wang, D. Schuurmans, M. Bosma, brian ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou.  Chain of thought prompting elicits reasoning in large language models.  In _NeurIPS_ , 2022.  URL https://openreview.net/forum?id=_VjQlMeSB_J.
  * Xu et al. [2022] M. Xu, Z. Zhang, J. Lu, Z. Zhu, Y. Zhang, M. Chang, R. Liu, and J. Tang.  PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding.  In _NeurIPS_ , volume 35, pages 35156–35173. Curran Associates, Inc., 2022.
  * Xu et al. [2023] M. Xu, X. Yuan, S. Miret, and J. Tang.  ProtST: Multi-modality learning of protein sequences and biomedical texts.  In _ICML_ , volume 202, pages 38749–38767, 2023.  URL https://proceedings.mlr.press/v202/xu23t.html.
  * Xu et al. [2021] S. Xu, Y. Liu, X. Yi, S. Zhou, H. Li, and Y. Wu.  Native Chinese Reader: A Dataset Towards Native-Level Chinese Machine Reading Comprehension.  In _NeurIPS Datasets and Benchmarks Track (Round 2)_ , 2021.  URL https://openreview.net/forum?id=GEcWUTN1v1v.
  * Zhang et al. [2024] D. J. Zhang, D. Li, H. Le, M. Z. Shou, C. Xiong, and D. Sahoo.  Moonshot: Towards controllable video generation and editing with multimodal conditions.  _arXiv preprint arXiv:2401.01827_ , 2024.
  * Zhou et al. [2023] H.-Y. Zhou, Y. Fu, Z. Zhang, B. Cheng, and Y. Yu.  Protein representation learning via knowledge enhanced primary structure reasoning.  In _ICLR_ , 2023.  URL https://openreview.net/forum?id=VbCMhg7MRmj.

## Appendix: A Finetuning Dataset and Benchmark for Large Language Models on
Protein Sequence Understanding

The questionnaire is based on the award paper of NeurIPS 2022 [25].

##  Appendix A Ethics statement

This investigation adhered strictly to ethical guidelines and research best
practices. All protein data utilized were sourced from publicly available
datasets, and no proprietary or confidential information was employed.
Stringent quality control measures and security checks have been implemented
to eliminate any potential presence of harmful or malicious content in our
dataset. It is important to acknowledge the profound implications and
potential risks associated with integrating Large Language Model (LLM) and
biomolecular knowledge. While our primary goal is to contribute positively to
scientific understanding, we are cognizant of the potential misuse of these
tools in the wrong hands. The combination of LLMs and protein sequence data
could be exploited by malicious actors to generate harmful substances, such as
illicit drugs. We strongly emphasize the imperative for users to adhere to the
highest ethical standards when utilizing our dataset, promoting fairness,
transparency, and responsibility in their research. Any utilization of the
dataset that may result in harm or pose a detriment to society is strictly
prohibited.

##  Appendix B Questionnaire

###  B.1 Motivation

Q1 For what purpose was the dataset created? Was there a specific task in
mind? Was there a specific gap that needed to be filled? Please provide a
description.

  1. Our proposed ProteinLMDataset and ProteinLMBench address the limitations of existing resources, offering a comprehensive evaluation platform for natural language processing tasks involving Chinese-English text pairs, protein sequences, protein design, and analysis. The datasets can help large language models (LLMs), typically designed for natural language, effectively understand protein sequences as if they were a foreign language. By incorporating protein-related details and sequences in multiple languages, our dataset and benchmark aim to train LLMs to understand protein sequences in a cross-lingual manner.

ProteinLMDataset comprises 17.46 billion tokens for self-supervised
pretraining, alongside a dataset of 893K instructions for supervised fine-
tuning. Innovatively structured into three segments: 0.69% Chinese-English
text pairs in protein science, 41.51% protein sequence-English text pairs, and
57.80% protein-related English text, ProteinLMDataset can contribute to
enhancing the cross-lingual capabilities of LLMs (English-Chinese-Sequence).

ProteinLMBench is the benchmark designed to evaluate the protein-related
capabilities of LLMs. It has 944 multiple-choice questions, respectively, on
protein tasks. ProteinLMBench is pioneering in its aim to thoroughly assess
LLM performance in comprehending protein sequences.

Q2 Who created the dataset (e.g., which team, research group) and on behalf of
which entity (e.g., company, institution, organization)?

  1. This dataset is jointly presented by Toursun Synbio with top research institutes Johns Hopkins University, University of Cambridge, Shanghai Institute for Biomedical and Pharmaceutical Technologies, Shanghai AI Laboratory, Shanghai Jiao Tong University, and UNSW Syndey.

Toursun Synbio is an AI-driven protein design and manufacturing enterprise
with its core team members coming from world-renowned universities. The
company is committed to leading protein engineering with AI, shaping the
future of synthetic biology, and addressing global health and environmental
challenges.

Q3 Who funded the creation of the dataset?  If there is an associated grant,
please provide the name of the grantor and the grant name and number.

  1. This work was sponsored Toursun Synbio and Shanghai AI Laboratory.

Q4 Any other comments?

  1. No.

###  B.2 Composition

Q5 What do the instances that comprise the dataset represent (e.g., documents,
photos, people, countries)? Are there multiple types of instances (e.g.,
movies, users, and ratings; people and interactions between them; nodes and
edges)? Please provide a description.

  1. ProteinLMDataset is innovatively structured into three segments: 0.69% Chinese-English text pairs in protein science, 41.51% protein sequence-English text pairs, and 57.80% protein-related English text. ProteinLMBench has 944 multi-choice questions on protein tasks respectively.

Q6 How many instances are there in total (of each type, if appropriate)?

  1. ProteinLMDataset comprises 17.46 billion tokens for self-supervised pretraining, alongside a dataset of 893K instructions for supervised fine-tuning. ProteinLMBench has 944 multiple-choice questions respectively on protein-related tasks to evaluate model generalization. This large-scale dataset covers a wealth of protein knowledge and can support the development of large language models to advance natural language processing tasks related to proteins.

Q7 Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set? If the dataset is a
sample, then what is the larger set? Is the sample representative of the
larger set (e.g., geographic coverage)? If so, please describe how this
representativeness was validated/verified. If it is not representative of the
larger set, please describe why not (e.g., to cover a more diverse range of
instances, because instances were withheld or unavailable).

  1. The ProteinLMDataset is full dataset that we present. ProteinLMBench is randomly sampled from a dataset we created with over 100K protein related question and answer pairs. Although larger datasets may contain more possibilities, due to manpower limitations, it is difficult for us to scale up here.

Q8 What data does each instance consist of ? “Raw” data (e.g., unprocessed
text or images) or features? In either case, please provide a description.

  1. Data source for protein:

UniProt: https://www.uniprot.org/help/downloads

RefSeq: https://ftp.ncbi.nlm.nih.gov/refseq/

SIFTS: https://ftp.ebi.ac.uk/pub/databases/msd/sifts/

Enzyme Nomenclature: https://iubmb.qmul.ac.uk/enzyme/

Data sources for academic papers and journals:

PubMed Central: https://www.ncbi.nlm.nih.gov/pmc/tools/ftp/

PubMed: https://pubmed.ncbi.nlm.nih.gov/download/

Q9 Is there a label or target associated with each instance? If so, please
provide a description.

  1. For SSL dataset there is no label. For SFT dataset each data has the answer to the question as the label.

Q10 Is any information missing from individual instances? If so, please
provide a description, explaining why this information is missing (e.g.,
because it was unavailable). This does not include intentionally removed
information, but might include, e.g., redacted text.

  1. No.

Q11 Are relationships between individual instances made explicit (e.g., users’
movie ratings, social network links)? If so, please describe how these
relationships are made explicit.

  1. No.

Q12 Are there recommended data splits (e.g., training, development/validation,
test- ing)? If so, please provide a description of these splits, explaining
the rationale behind them.

  1. No.

Q13 Are there any errors, sources of noise, or redundancies in the dataset? If
so, please provide a description.

  1. There exist near duplicate protein sequences which makes possible a good many to one embedding in certain scenarios. When attaching literature to a protein sequence, two sequences may refer to the same paper that mentioned both sequences.

Q14 Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)? If it links to or
relies on external resources, a) are there guarantees that they will exist,
and remain constant, over time; b) are there official archival versions of the
complete dataset (i.e., including the external resources as they existed at
the time the dataset was created); c) are there any restrictions (e.g.,
licenses, fees) associated with any of the external resources that might apply
to a future user? Please provide descriptions of all external resources and
any restrictions associated with them, as well as links or other access
points, as appropriate.

  1. The dataset is self-contained as the protein sequences and literature are downloaded from the academic datasets such as UniProt, NCBI and PubMed, and then cleaned and well preprocessed.

Q15 Does the dataset contain data that might be considered confidential (e.g.,
data that is protected by legal privilege or by doctor–patient
confidentiality, data that includes the content of individuals’ non-public
communications)? If so, please provide a description.

  1. This dataset was collected using openly available parts of the internet with the assumption that any data found was intended to be shared freely.

Q16 Does the dataset contain data that, if viewed directly, might be
offensive, insulting, threatening, or might otherwise cause anxiety? If so,
please describe why.

  1. The dataset is the biological information and references of proteins and there is no such concern.

Q17 Does the dataset relate to people? If not, you may skip the remaining
questions in this section.

  1. No.

Q18 Does the dataset identify any subpopulations (e.g., by age, gender)?

  1. No

Q19 Is it possible to identify individuals (i.e., one or more natural
persons), either directly or indirectly (i.e., in combination with other data)
from the dataset? If so, please describe how.

  1. No.

Q20 Does the dataset contain data that might be considered sensitive in any
way (e.g., data that reveals racial or ethnic origins, sexual orientations,
religious be- liefs, political opinions or union memberships, or locations;
financial or health data; biometric or genetic data; forms of government
identification, such as social security numbers; criminal history)? If so,
please provide a description.

  1. The protein information contains biological categories which the protein belongs to. But it is not related to racial or ethnic origins, sexual orientions and so on.

Q21 Any other comments?

  1. No

###  B.3 Collection Process

Q22  How was the data associated with each instance acquired? Was the data
directly observable (e.g., raw text, movie ratings), reported by subjects
(e.g., survey responses), or indirectly inferred/derived from other data
(e.g., part-of-speech tags, model-based guesses for age or language)? If data
was reported by subjects or indirectly inferred/derived from other data, was
the data validated/verified? If so, please describe how.

  1. None of the journal text and protein sequence parts in the ProteinLMDataset SSL part have been processed. However, during the concatenation of these two parts, NER and NEN techniques were applied, with detailed procedures described in the main text. For the UniProtKB Swiss-Prot part, the downloaded data was filled according to a predefined template.In ProteinLMDataset SFT part, the data was generated by creating suitable templates with a Large Language Model (LLM) and then filling in the corresponding data.

The ProteinLMBench dataset, used for evaluating the effectiveness of the large
model, was generated by using the LLM to read the downloaded research papers
and create relevant questions. These questions were then validated by both
machines and humans. The specific process is detailed in the main text.

Q23 What mechanisms or procedures were used to collect the data (e.g.,
hardware apparatus or sensor, manual human curation, software program,
software API)? How were these mechanisms or procedures validated?

  1. We ran a preprocessing script in python, over dozens of CPU cores, and 8 Nvidia 4090 cards. They were validated by python program we wrote.

Q24 If the dataset is a sample from a larger set, what was the sampling
strategy (e.g., deterministic, probabilistic with specific sampling
probabilities)?

  1. We deleted some literature that is not related to the protein.

Q25 Who was involved in the data collection process (e.g., students,
crowdworkers, contractors) and how were they compensated (e.g., how much were
crowdworkers paid)?

  1. No crowdworkers were used in the curation of the dataset. Only authors of the paper enabled its creation.

Q26 Over what timeframe was the data collected? Does this timeframe match the
creation timeframe of the data associated with the instances (e.g., recent
crawl of old news articles)? If not, please describe the timeframe in which
the data associated with the instances was created.

  1. The data collected from the database has the same timeframe as when the database is created. The literature may date back to 100 years ago.

Q27 Were any ethical review processes conducted (e.g., by an institutional
review board)? If so, please provide a description of these review processes,
including the outcomes, as well as a link or other access point to any
supporting documentation.

  1. We corresponded with the University of Washington’s Human Subject Division, and as we do not intervene with the people depicted in the data as well as the data being public, they stated that the work did not require IRB review. Furthermore, the NeurIPS ethics review determined that the work has no ethical issues.

Q28 Does the dataset relate to people? If not, you may skip the remaining
questions in this section.

  1. The protein sequence may be used by human being as protein the basic unit of life.

Q29 Did you collect the data from the individuals in question directly, or
obtain it via third parties or other sources (e.g., websites)?

  1. We retrieve the data from public databases including PubMed, UniProt and NCBI, and from the official website of Chinese journals.

Q30 Were the individuals in question notified about the data collection? If
so, please describe (or show with screenshots or other information) how notice
was provided, and provide a link or other access point to, or otherwise
reproduce, the exact language of the notification itself.

  1. Individuals were not notified about the data collection.

Q31 Did the individuals in question consent to the collection and use of their
data? If so, please describe (or show with screenshots or other information)
how consent was requested and provided, and provide a link or other access
point to, or otherwise reproduce, the exact language to which the individuals
consented.

  1. The database and journals are public and free to download and use.

Q32 If consent was obtained, were the consenting individuals provided with a
mechanism to revoke their consent in the future or for certain uses? If so,
please provide a description, as well as a link or other access point to the
mechanism (if appropriate).

  1. Not applicable.

Q33 Has an analysis of the potential impact of the dataset and its use on data
subjects (e.g., a data protection impact analysis) been conducted? If so,
please provide a description of this analysis, including the outcomes, as well
as a link or other access point to any supporting documentation.

  1. The dataset we processed is used to assign and evaluate the capability of understanding proteins for foundation models. No ethical issue will be involved.

Q34 Any other comments?

  1. No

###  B.4 Preprocessing, Cleaning, and/or Labeling

Q35 Was any preprocessing/cleaning/labeling of the data done (e.g.,
discretization or bucketing, tokenization, part-of-speech tagging, SIFT
feature extraction, remova of instances, processing of missing values)? If so,
please provide a description. If not, you may skip the remainder of the
questions in this section.

  1. Yes. Preprocessing/cleaning/labeling is conducted when the raw data is obtained from publica database for protein (PubMed, UniProt, NCBI and Chinese journals). Preprocessing/cleaning/labeling is done without any bias.

Q36 Was the “raw” data saved in addition to the preprocessed/cleaned/labeled
data (e.g., to support unanticipated future uses)? If so, please provide a
link or other access point to the “raw” data.

  1. We save the raw data in our local server. The data are not sensitive and will be repeated used in our study.

Q37 Is the software used to preprocess/clean/label the instances available? If
so, please provide a link or other access point.

  1. None of the preprocessing/cleaning/labeling was done using external software; the entire process was completed using custom Python scripts.

Q38 Any other comments?

  1. No.

###  B.5 Uses

Q39 Has the dataset been used for any tasks already? If so, please provide a
description.

  1. It has been used for research related to proteins. In particular, we use ProteinLMDataset for training a protein language model and ProteinLMBench for evaluating a variety of LLMs in protein tasks.

Q40 Is there a repository that links to any or all papers or systems that use
the dataset? If so, please provide a link or other access point.

  1. Yes, the datasets can be downloaded from https://huggingface.co/datasets/tsynbio/ProteinLMBench.

Q41 What (other) tasks could the dataset be used for?

  1. Our dataset ProteinLMDataset can be used for training an LLM to linking the protein language and natural language (including English and Chinese). The dataset ProteinLMBench is used for evaluating the capability of LLMs in protein tasks including Protein general knowledge, Protein literature understanding, Protein property prediction, Protein function prediction, Protein design.

Q42 Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses? For
example, is there anything that a future user might need to know to avoid uses
that could result in unfair treatment of individuals or groups (e.g.,
stereotyping, quality of service issues) or other undesirable harms (e.g.,
financial harms, legal risks) If so, please provide a description. Is there
anything a future user could do to mitigate these undesirable harms?

  1. The protein sequence and information that we collected from the database may be updated, and new literature about a particular protein may appear in future. Future research can update the dataset with the same preprocessing/cleaning/labeling method that we used while the method per se has no bias toward any individual or species that may violate ethical rule.

Q43 Are there tasks for which the dataset should not be used? If so, please
provide a description.

  1. No. The dataset is for protein related tasks, specified for LLMs supporting Chinese and English. It can be also used for LLMs supporting multilanguage.

Q44 Any other comments?

  1. No.

###  B.6 Distribution

Q45 Will the dataset be distributed to third parties outside of the entity
(e.g., company, institution, organization) on behalf of which the dataset was
created? If so, please provide a description.

  1. Yes, the dataset is open-source.

Q46 How will the dataset be distributed (e.g., tarball on website, API,
GitHub)? Does the dataset have a digital object identifier (DOI)?

  1. The data is available through Huggingface datasets, and can be downloaded from https://huggingface.co/datasets/tsynbio/ProteinLMBench.

Q47 When will the dataset be distributed?

  1. 03/06/2024 and onward.

Q48 Will the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms of use (ToU)? If so,
please describe this license and/or ToU, and provide a link or other access
point to, or otherwise reproduce, any relevant licensing terms or ToU, as well
as any fees associated with these restrictions.

  1. Apache License 2.0

Q49 Have any third parties imposed IP-based or other restrictions on the data
associated with the instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any relevant
licensing terms, as well as any fees associated with these restrictions.

  1. Toursun Synbio owns the metadata and release as CC-BY-4.0.

  2. We do not own the copyright of the protein sequence or the related literature.

Q50 Do any export controls or other regulatory restrictions apply to the
dataset or to individual instances? If so, please describe these restrictions,
and provide a link or other access point to, or otherwise reproduce, any
supporting documentation.

  1. No.

Q51 Any other comments?

  1. No.

###  B.7 Maintenance

Q52 Who will be supporting/hosting/maintaining the dataset?

  1. Huggingface will support hosting of the metadata.

  2. The Toursun Synbio supports hosting of the embeddings and backups of the rest, and will maintain the samples distributed.

Q53 How can the owner/curator/manager of the dataset be contacted (e.g., email
address)?

  1. If any questions, please email to<EMAIL_ADDRESS>
Q54 Is there an erratum? If so, please provide a link or other access point.

  1. There is no erratum for our initial release. Errata will be documented as future releases on the dataset website.

Q55 Will the dataset be updated (e.g., to correct labeling errors, add new
instances, delete instances)? If so, please describe how often, by whom, and
how updates will be communicated to users (e.g., mailing list, GitHub)?

  1. ProteinLMDataset will not be updated. However a future Toursun Synbio-streamed-from-CC may exist for updates. ProteinLMBench will be updated with more multi-choice questions on protein tasks.

Q56 If the dataset relates to people, are there applicable limits on the
retention of the data associated with the instances (e.g., were individuals in
question told that their data would be retained for a fixed period of time and
then deleted)? If so, please describe these limits and explain how they will
be enforced.

  1. The datasets are not related to people. But people may contact us at the<EMAIL_ADDRESS>and add specific samples to a blacklist.

Q57 Will older versions of the dataset continue to be
supported/hosted/maintained? If so, please describe how. If not, please
describe how its obsolescence will be communicated to users.

  1. This the first version of the datasets.

Q58 If others want to extend/augment/build on/contribute to the dataset, is
there a mechanism for them to do so? If so, please provide a description. Will
these contribu- tions be validated/verified? If so, please describe how. If
not, why not? Is there a process for communicating/distributing these
contributions to other users? If so, please provide a description.

  1. No.

Q59 Any other comments?

  1. No.

##  Appendix C Experiments and Training Details

###  C.1 LLMs instruction

We conducted three different experiments: no training, supervised fine-tuning
(SFT), and self-supervised learning combined with supervised fine-tuning (SSL-
SFT) using the ProteinLMDataset. For the no-training experiment, we employed
various large language models, including Falcon-7B [2], Qwen1.5-7B [3],
Moonshot [36], Mistral-7B-Instruct-v0.2 [14], Baichuan2-7B-Chat [4],
Llama-2-7B-Chat-hf [30], InternLM-Chat-20B, InternLM2-Chat-7B, and its
variations [7], ChatGLM3-6B [9], Yi-6B-Chat [1], GPT3.5-turbo, and
GPT4.0-turbo [20]. For the SFT and SSL-SFT experiments, we utilized the
InternLM2-7B model, naming them InternLM2-Protein-7B (w/o SSL) and
InternLM2-Protein-7B, respectively. The accuracy of all models was evaluated
using our proposed ProteinLMBench benchmark.

####  C.1.1 Hyperparameter configuration

Common hyperparameters for both SSL and SFT phases included:

Learning Rate: 0.0002

Batch Size: 16

Optimizer: AdamW

Parameter Scheduler: Alternating between LinearLR and CosineAnnealingLR

Training Method: LoRA (LoRA Alpha = 16, LoRA Dropout = 0.1)

####  C.1.2 Self-Supervised learning (SSL) phase

For the self-supervised learning (SSL) phase, we initialized the model weights
based on the InternLM2 model. We trained on a mixed dataset comprising 50
billion tokens, integrating both our specific protein data and a broad range
of general-purpose data. The training was performed with a batch size of 16
and a fixed learning rate of 0.0002. The same SSL process, involving
initialization from a pre-trained model and training on a mixed dataset, was
followed for the other large language models (LLMs) as highlighted in the main
manuscript.

####  C.1.3 Supervised fine-tuning (SFT) phase

Post-pre-training, the model underwent SFT using 850,000 samples from our
constructed dataset. The fine-tuning process spanned three additional epochs,
refining the weights to achieve optimal performance.

Table 3: Accuracy and inference time of protein language models on
ProteinLMBench. Protein models score
---
Model | Correct Rate (%) | Inference Time (minutes)
GPT4.0-turbo | 57.94 | 15.52
InternLM2-20B | 57.52 | 47.20
GPT3.5-turbo | 55.19 | 21.03
InternLM2-7B | 54.98 | 19.23
InternLM2-Chat-7B | 54.76 | 35.58
InternLM2-Chat-20B | 51.38 | 31.11
Yi-6B | 50.85 | 59.05
Mistral-7B-Instruct-v0.2 | 50.11 | 13.00
ChatGLM3-6B | 48.94 | 8.00
Baichuan2-7B | 44.49 | 16.37
InternLM-Chat-20B | 40.54 | 66.00
Llama2-7B | 39.64 | 64.00
Moonshot | 38.26 | 16.25
Qwen1.5-7B | 21.73 | 13.00
Falcon-7B-Instruct | 20.55 | 25.42
Falcon-7B | 19.17 | 15.55
InternLM2-Protein-7B | 62.18 | 22.34
InternLM2-Protein-7B (w/o SSL) | 58.26 | 21.36

###  C.2 Model performance comparison

Tab. 3 compares the accuracy and inference time of various protein language
models on the ProteinLMBench benchmark. GPT4.0-turbo achieved the highest
accuracy of 57.94% with a relatively fast inference time of 15.52 minutes
among models without fine-tuning on ProteinLMDataset. InternLM2-20B and
GPT3.5-turbo also showed competitive accuracies of 57.52% and 55.19%,
respectively. Notably, InternLM2-Protein-7B, which utilized self-supervised
learning on the ProteinLMDataset followed by supervised fine-tuning, achieved
the highest accuracy of 62.18% among all models, with a reasonable inference
time of 22.34 minutes. Models like Falcon-7B, Qwen1.5-7B, and Moonshot
exhibited lower accuracies, while others like Yi-6B, InternLM-Chat-20B, and
Llama2-7B had considerably longer inference times. The results highlight the
trade-off between accuracy and inference time, with InternLM2-Protein-7B
demonstrating the best balance of high accuracy and reasonable inference time.

##  Appendix D Thoroughly related work

Existing Chinese-English datasets. Traditional benchmarks in the English
language primarily focus on evaluating specific Language Model (LLM) abilities
related to singular tasks, such as providing answers in general question
answering, text summarization, or general linguistic understanding [31, 22,
18].

Recent benchmarks, such as MMLU [12], expand the evaluation scope by
encompassing multi-domain and multi-task assessments derived from real-world
university and high school exams and published manuscripts. BIG-bench [27]
comprises 204 diverse tasks, including some considered beyond current LLM
capabilities, while HELM [17] aggregates 42 tasks evaluating LLM performance
across seven different metrics, ranging from accuracy to robustness. In the
domain of NLP benchmarks specific to the Chinese language, two notable studies
have made significant strides. The first study, as outlined in [13], is
remarkable for its thorough evaluation of various Language Model (LLM)
systems. Notably, GPT-4 achieves an impressive accuracy exceeding 60%. The
dataset comprises 13,948 questions and answers, spanning diverse proficiency
levels (middle school, high school, college, professional) and covering a wide
array of topics including STEM, Humanities, Social Science, and more. Despite
its merits, the evaluation has certain limitations. It is confined to regions
within China, predominantly focuses on accuracy metrics, lacks detailed
distribution statistics, and falls short in offering interpretability and
insights conducive to enhanced reasoning. While the inclusion of zero-shot and
five-shot comparisons enriches the assessment, there remains room for
improvement in terms of comprehensiveness and interpretive depth. Conversely,
the second study, detailed in [35], caters specifically to the needs of native
Chinese readers proficient in both modern and classical Chinese. The dataset
encompasses 8,390 documents, each containing 1,024 characters, spanning
classic literature, modern articles, and classical poetry. With 20,477
questions of varying language difficulty, it incorporates a rich variety of
characters in each document style, facilitating diverse language exploration.
The benchmark, featuring models that achieve an impressive 79% accuracy,
stands out for its provision of detailed statistics on dataset distribution,
ensuring a fair evaluation. However, a limitation of this study lies in the
absence of scientific documents, representing a unique writing style not
covered in this benchmark. This gap highlights the need for further
exploration and inclusion of diverse textual genres to provide a more
comprehensive assessment.

The existing literature on English benchmarks primarily focuses on specific
model abilities for singular linguistic tasks, with limited extension to
multi-domain and multi-task assessments. However, in the context of Chinese
NLP benchmarks, significant limitations emerge, including rational biases with
geographic restrictions to China regions dataset collection, absence of
analytical documentation and statistics, and lack of interpretability with
limited insights into reasoning abilities. Overall, these limitations
underscore the need for more comprehensive and diverse multi-task language
datasets, addressing a broader geographic scope of collection, incorporating
varied evaluation metrics, and ensuring coverage of distinct writing styles,
including scientific content. Notably, there is currently no dataset known to
the authors that provides text-to-text pairs for English and Chinese
translation.

Existing protein sequence datasets. The UniProtKB [8] provides a comprehensive
repository of protein sequences along with functional annotations. With a vast
collection exceeding 227 million sequences, it serves as a vital resource for
a wide range of research applications. Ongoing efforts are directed towards
establishing a reference proteome for each taxonomic group, ensuring
inclusivity across diverse organisms. Continuous literature extraction
mechanisms ensure the entries remain current, while unreviewed entries benefit
from automated system annotations facilitated by machine learning techniques,
thereby enhancing overall coverage. UniProt demonstrates remarkable
adaptability to the evolving information landscape by continually enhancing
its data processing pipeline and website interface. However, it is imperative
to acknowledge potential biases inherent in community contributions, which may
affect the representativeness of the dataset. Moreover, the utilization of
automated systems introduces the possibility of errors, thereby impacting the
accuracy of annotations.

Annotating proteins that are yet to be characterized presents inherent
challenges, and limitations persist in achieving comprehensive
characterization. Efficient management of the influx of whole genome
sequencing data poses significant challenges, and integrating diverse external
resources may complicate the coherent presentation of data. The PEER dataset
[33] fills a crucial gap by providing a standardized benchmark for evaluating
deep learning methods in protein sequence understanding. Its openly available
datasets and source codes enhance transparency and reproducibility in
research. Covering diverse tasks like function prediction, localization,
structure, protein-protein, and protein-ligand interactions, PEER evaluates
various sequence-based methods, with a focus on the effectiveness of large-
scale pre-trained protein language models, notably ESM-1b. Multi-task learning
significantly boosts performance across tasks. However, extending beyond
sequence-based datasets presents challenges, necessitating ongoing community
collaboration to ensure sustained effectiveness.

Several important datasets related to protein sequences include RefSeq [19],
PDB [6], CATH (Class, Architecture, Topology, Homology) [21], and CASP15 [26,
16]. RefSeq provides a comprehensive, curated collection of reference
sequences for various organisms, though it primarily focuses on single-chain
protein design. CATH, while valuable, has limitations in representing the full
diversity of protein structures [6]. Incorporating de novo proteins from
CASP15 [26, 16], the TS45 dataset comprises diverse protein targets, including
Free Modeling (FM) and Template-Based Modeling (TBM), divided into TBM-easy
and TBM-hard proteins. TS45 serves as a benchmark for de novo protein design,
offering dissimilarity to known structures and mitigating biases present in
training sets. Lastly, the study by [10] introduces a biomolecular instruction
dataset aimed at enhancing LLMs’ understanding and prediction capabilities in
this specialized field. While promising, it’s important to note that this
benchmark is still in its early stages and has limited applicability for real-
world tasks. Further research is required to optimize instruction tuning
methods, refine the model’s ability to understand and follow instructions, and
improve its generalization across various protein and biomolecular tasks.

Protein design databases. KEGG [15] represents a curated resource, offering
researchers a comprehensive repository of accurately annotated biological
entities, spanning molecular networks, genes, proteins, chemicals, and health-
related data. Despite its strengths, KEGG faces limitations such as a current
focus on disease-related perturbations in human cells, potential biases in
data inclusion, and the resource-intensive nature of manual curation.
Meanwhile, the STRING database [29] distinguishes itself through its extensive
integration of known and predicted protein associations, encompassing both
physical interactions and functional relationships across over 14,000
organisms. Despite its comprehensive coverage, STRING encounters challenges
including potential biases in text mining, reliance on predictive methods, and
the absence of options for tissue-specific network pruning. Lastly, InterPro
[5], a leading protein classification database, offers comprehensive
annotations through the integration of predictive models, providing insights
into families, domains, and conserved sites. However, its limitations include
a lack of updates, interpretability, and sustainability.

While the protein-related databases and datasets mentioned offer invaluable
resources for various aspects of biological research, they are not without
their limitations. These constraints include potential biases stemming from
community contributions and automated systems in UniProtKB, challenges in
extending beyond sequence-based datasets in PEER, a focus solely on single-
chain structures in CATH, limitations in representing the complete diversity
of protein structures in PDB, applicability primarily to designs present in
training sets in CASP15, a disease-centric focus and resource-intensive manual
curation in KEGG, potential biases in text mining and reliance on predictive
methods in STRING, and challenges related to funding growth and dependencies
on individual resources in InterPro. Recognizing these limitations is crucial
for researchers to navigate and interpret the data effectively and highlights
the ongoing efforts needed to improve the comprehensiveness and sustainability
of these resources.