# IDP-Bert: Predicting Properties of Intrinsically Disordered Proteins (IDP)
Using Large Language Models

Parisa Mollaei  [  Chakradhar Guntuboina  [  Danush Sadasivam  Department of
Chemical Engineering, Carnegie Mellon University, 15213, USA  Amir Barati
Farimani<EMAIL_ADDRESS>[

###### keywords:

American Chemical Society, LaTeX

meche] Department of Mechanical Engineering, Carnegie Mellon University,
15213, USA ece] Department of Electrical and Computer Engineering, Carnegie
Mellon University, 15213, USA meche] Department of Mechanical Engineering,
Carnegie Mellon University, 15213, USA [biomed] Department of Biomedical
Engineering, Carnegie Mellon University, 15213, USA [mld] Machine Learning
Department, Carnegie Mellon University, 15213, USA IR,NMR,UV

Intrinsically Disordered Proteins (IDPs) constitute a large and structure-less
class of proteins with significant functions. The existence of IDPs challenges
the conventional notion that the biological functions of proteins rely on
their three-dimensional structures. Despite lacking well-defined spatial
arrangements, they exhibit diverse biological functions, influencing cellular
processes and shedding light on the disease mechanisms. However, it is
expensive to run experiments or simulations to characterize this class of
proteins. Consequently, we designed an ML model that relies solely on amino
acid sequences. In this study, we introduce IDP-Bert model, a deep-learning
architecture leveraging Transformers and Protein Language Models (PLMs) to map
sequences directly to IDPs properties. Our experiments demonstrate accurate
predictions of IDPs properties, including Radius of Gyration, end-to-end
Decorrelation Time, and Heat Capacity.

##  1 Introduction

Within the complex frame of molecular biology, the long-standing belief that
proteins’ function strongly depends on their three-dimensional structures
faces a compelling challenge due to the advent of a distinct class of
biomolecules; Intrinsically Disordered Proteins (IDPs). IDPs challenge
conventional structural norms, existing as dynamic ensembles of conformations
that lack a well-defined spatial arrangement1, 2, 3, 4, 5. Despite their
structural diversity, IDPs are often evolutionarily conserved across different
species6, 7. This suggests that such proteins may play fundamental roles in
biological processes and underscore their significance in the context of
evolution8, 9. IDPs with their ability to adopt diverse shapes, are vital for
essential cellular activities like gene expression and cell signaling10, 11,
12, which is crucial for cell growth and response to stimuli. They often
engage in complex interactions with other biomolecules13, 14 that facilitate
the formation of macromolecular complexes and cellular structures. These
interactions are disrupted without IDPs, impacting cellular function. Thus,
investigating IDPs properties can yield insights into disease mechanisms and
avenues for therapeutic intervention15, 16, 17, 18, 19, 20. However,
performing experiments to characterize the properties of IDPs is expensive and
is not feasible for large-scale screening. Moreover, running molecular
dynamics (MD) simulations for analyzing IDPs’ properties requires time-
consuming and expensive computations. Therefore, a surrogate model is needed
for rapid IDPs’ properties prediction.

In the last decade, machine learning (ML) models were successful in chemistry
and building fast surrogate models for property prediction21, 22, 23, 24, 25,
26, 27. For proteins with well-known structures, several geometrical deep
learning methods exist, such as graph neural networks (GNNs), the Equivariant
Graph Neural Nets (EGNN), etc28, 29, 22, 21. Inspired by these advances, we
developed an ML model to predict the IDPs properties. Given the absence of
structural features in IDPs, our model relies primarily on the arrangement of
amino acids in protein sequences. Although ML models such as Bag of Words and
fingerprinted features have been used to model the IDPs properties30, they
often fall short in prediction accuracy. These methods lack the ability to
adequately capture the sequential relationships between amino acids,
particularly the long-range connections, which are crucial for accurately
predicting the proteins’ properties.

The advent of Transformers31, 32 and Protein Language Models (PLMs)33 has
catalyzed the development of deep learning architectures in modeling amino
acid sequences as analogous to words and sentences in languages34, 23, 26. The
attention mechanism inherent in PLMs enables them to adeptly capture both
immediate and intricate connections among various elements of textual data31.
Specifically, combination of pre-training with transformers leads to creation
of ”BERT”35 which is a powerful language encoder. These advancements have
sparked a renaissance in bioinformatics, as protein sequences, much like
languages, reveal intricate interactions among their constituent amino acids.
With the capabilities offered by PLMs and Transformers, we are able to explore
the contributions of amino acids to the features of proteins23, 26. Through
the utilization of pre-trained models like ProtBERT36, we fine-tuned it to
accurately forecast the IDPs’ properties. The ProtBERT model has been trained
on extensive datasets of protein sequences, enabling them to learn
comprehensive representations of protein sequences36. Previous studies have
demonstrated the efficacy of learning algorithms, particularly PLMs, in
deciphering the language of proteins and uncovering their intrinsic
characteristics23, 37, 38, 26, 39. Our experiments revealed that the PLMs can
also accurately predict structural, dynamic, and thermodynamic properties of
IDPs, including Radius of Gyration, end-to-end Decorrelation Time, and Heat
Capacity.

The Radius of Gyration provides information about the compactness and spatial
extent of the protein structure which is defined as the root mean square
distance of all atoms in a protein from their common center of mass. For IDPs,
which lack a well-defined three-dimensional structure, the Radius of Gyration
provides insight into their conformational flexibility and structural
heterogeneity40. The end-to-end Decorrelation Time measures the timescale over
which the positional correlation between two ends of the protein chain
diminishes, providing insight into the dynamics and flexibility of the
protein41. The Heat Capacity reflects the protein’s ability to absorb and
dissipate heat, which is related to its stability and structural dynamics.
IDPs’ thermal stability and heat capacity are significantly different compared
to structured proteins which highlights their ability to adopt multiple
conformations and undergo conformational changes easily42. Gaining insight
into these properties yields valuable information regarding IDPs’ biological
functionalities and versatility in molecular interactions.

##  2 Methods

Figure 1: The overall IDP-Bert framework illustrates the preprocessing of
input data, tokenization, integration with ProtBERT, fine-tuning process, and
regression tasks aimed at predicting the Radius of Gyration, end-to-end
Decorrelation Time, and Heat Capacity of IDPs.

The schematic representation of IDP-Bert’s overall architecture is depicted in
Figure1. IDP-Bert utilizes ProtBERT36, a transformer-based model with 16
attention heads and 30 hidden layers as its backbone. ProtBERT is pre-trained
on Big Fantastic Database (BFD)43, 44, 45, a vast protein sequence corpus
comprising over 217 million unique sequences. An adaptation of the original
BERT46 developed by Google, ProtBERT adopts the encoder segment structure of
the transformer model, featuring multiple sequential layers of an attention-
feed-forward network, as illustrated in Figure1. Within the attention
mechanism, each token is encoded into an input embedding and then transformed
into keys, queries, and values. Keys and queries are combined through matrix
multiplication to construct the attention map, which subsequently undergoes a
softmax function47 to produce a probability distribution. Following this, the
resulting distribution is employed to scale (multiply) the value vectors. The
feed-forward layer within each Transformer layer aids in capturing intricate
patterns within the input, while the attention mechanism encodes relationships
among various tokens. The multi-head attention structure partitions the input
across multiple parallel attention layers, or ”heads”, allowing each head to
independently specialize in detecting diverse patterns and relationships31.
This configuration of the Transformer encoder in ProtBERT enables the model to
learn context-aware representations of amino acids in protein sequences. One
significant advantage of ProtBERT is its superior performance across multiple
benchmarks36. For the fine-tuning process, a series of experiments were
conducted to determine the optimal configuration for the ProtBERT backbone and
the architecture for the regression head, resulting in the selection of a
design comprising two fully connected layers (refer to Supplementary
Information section Experiments).

###  2.1 Data preprocessing and model

The dataset used in our study contains properties derived from simulations for
2585 IDPs, each with a degree of polymerization ranging from 20 to 600 amino
acids48. These IDPs are characterized as linear and stochastic polymers due to
their sequences. The sequences were sourced from version 9.0 of the DisProt
database49, 50 to ensure uniqueness. Properties of the IDPs were computed
using molecular dynamics simulations at 300 K with the LAMMPS simulation
package51, employing the improved hydropathy scale (HPS) CG model52. Key
properties, including Radius of Gyration ($R_{g}$), end-to-end Decorrelation
Time ($\tau_{N}$), and Heat Capacity ($C_{v}$), were determined for use as
labels in regression tasks. About 279 protein sequences in this dataset were
found to be composed of more than 200 amino acids. During our initial
experimentation, we noticed that our trained model struggled to effectively
learn the properties of these long sequences. This challenge arose from the
limited number of such sequences in the dataset (refer to Supplementary
Information section Original and augmented data). Consequently, we opted to
exclude all protein sequences exceeding 200 amino acids. This led to a refined
dataset comprising 2306, which was subsequently divided into three distinct
subsets; train, validation, and test sets, each non-overlapping with the
others.

To transform raw protein sequences into a format that can be effectively
processed by ProtBERT, they need to be tokenized (Figure1). Each amino acid is
represented by a number (known as a token) in the range of 5 to 24
(inclusive), with numbers 0 through 4 being reserved for special tokens (0:
[PAD], 1: [UNK], 2: [CLS], 3: [SEP], and 4: [MASK]). These tokens were fed to
IDP-Bert and three separate models were trained (one for each IDP property).
We employed the Mean Squared Error (MSE) loss to train our model over 100
epochs and used $R^{2}$ scores to assess the performance of our model. We used
a modified configuration for ProtBERT with 20 hidden layers (each with a size
of 400), a hidden layer dropout probability of 0.15, and 20 attention heads.
We attach a regression head on top of this which consists of 2 fully connected
layers, each followed by a ReLU activation. The model was optimized using the
AdamW optimizer53 with an initial learning rate of $1*10^{-5}$.
ReduceLROnPlateau scheduler54 was used to improve the convergence of the loss.
The scheduler parameters were set to reduce the learning rate by a factor of
0.1 if there was no improvement in the loss over 4 epochs.

Upon dividing the dataset into buckets based on the length of the sequences,
we noticed that buckets with fewer data points exhibited higher average MSE
values (Figure2(a) through (c)). To address this and enhance our model’s
generalizability, we used a data augmentation55 technique known as
oversampling. Oversampling involves artificially augmenting the number of data
points in the under-represented class by duplicating them. Specifically, we
determined the bucket with the highest number of points denoted as N-max. For
each bucket with a lower number of points, denoted as N-bucket, we calculated
the multiplier by rounding the ratio of N-max/N-bucket down to the nearest
integer. Subsequently, each bucket with fewer points was duplicated to
increase its size, resulting in a balanced dataset (Figure2(d) through (f))
with a similar number of data points across all buckets. It is important to
note that this augmentation is only applied to the training set.

##  3 Results and discussion

IDP’s property Average of ${R^{2}\pm SD}$ Radius of Gyration ${0.98\pm 0.005}$
end-to-end Decorrelation Time ${0.87\pm 0.025}$ Heat Capacity ${0.83\pm
0.023}$

Table 1: The performance of IDP-Bert model in predicting the Radius of
Gyration, end-to-end Decorrelation Time, and Heat Capacity properties of IDPs
with standard deviations (SD).

The average $R^{2}$ with corresponding standard deviations have been obtained
for predicted properties values using the IDP-Bert model. Table 1 shows the
$R^{2}$ is 0.98 for the Radius of Gyration, indicating approximately 98% of
the variance in Radius of Gyration can be explained by the independent
variable(s) modeled by IDP-Bert. The standard deviation associated with this
is 0.005. The average $R^{2}$ for end-to-end Decorrelation Time and Heat
Capacity are 0.87 and 0.83, respectively, with corresponding standard
deviations of 0.025 and 0.023 (Table 1). These results suggest a strong
relationship between the features used in IDP-Bert model and the IDP’s
properties, underscoring a high degree of predictability of the model.

Figure 2: Distribution of original and augmented IDP samples based on the
length of their amino acid sequences. The Mean Squared Error (MSE) for Radius
of Gyration is depicted for both (a) original and (d) augmented datasets,
followed by the MSE for end-to-end Decorrelation Time shown for (b) original
and (e) augmented datasets. Lastly, the MSE for Heat Capacity is presented for
(c) original and (f) augmented datasets.

To evaluate the effect of augmentation process on the results, we mapped the
MSE values onto each range of sequence length for both the original and
augmented dataset across each IDP property: Radius of Gyration (Figure2
(a),(d)), end-to-end Decorrelation Time (Figure2(b),(e)), and Heat Capacity
(Figure2 (c),(f)). This information provides insights into the impact of both
number of samples and length of IDP sequences on the accuracy of the predicted
values. As depicted in Figure2, it is evident that the MSE values for large
sequences (those exceeding 180 amino acids) exhibit notably higher values.
This may be attributed to several reasons, such as larger IDP sequences tend
to have more complex structures and conformational variability that make
predicting IDP properties more challenging. Furthermore, the limited dataset
of longer sequences in the training dataset can lead to poorer performance of
IDP-Bert model. However, data augmentation improved the model’s performance
for the large sequences (compare Figure2(b) with (e) or Figure2(c) with (f)).

Figure 3: Visualization of the correlation between predicted and ground truth
values, accompanied by their corresponding $R^{2}$ coefficients for the train
and test sets. The Radius of Gyration (a) and (d), end-to-end Decorrelation
Time (b) and (e), and Heat Capacity (c) and (f).

With an emphasis on reliability and efficiency of IDP-Bert model in capturing
the essential features of IDPs, our analysis reveals a robust correlation
between the predicted properties and corresponding ground truth values in both
train and test sets (Figure3). The slight differences between $R^{2}$ values
in test and train sets, specifically for the Radius of Gyration property,
indicate that the model has learned the underlying pattern within the date
without memorizing the noise in the training set, allowing it to make accurate
predictions on unseen data. Additionally, it validates the generalizability of
our model across different datasets.

Figure 4: Ground truth values and sequence lengths mapped onto the t-SNE
representation for Radius of Gyration (a,d), end-to-end Decorrelation Time
(b,e), and Heat Capacity (c,f)

Building upon the strong correlation observed between predicted and ground
truth values, we further explored the features extracted from the penultimate
layer of the IDP-Bert framework (within fine-tuning part in Figure 1).
Utilizing these features, we generated t-distributed Stochastic Neighbor
Embedding (t-SNE) visualization to gain deeper insights into the underlying
relationships in the data. The t-SNE is a technique for reducing the
dimensionality of data while preserving local structure, making it useful for
visualizing complex datasets in lower dimensions56. In our analysis, the t-SNE
interpretation reflects the relationship between IDP properties, sequence
lengths, and their structural and functional characteristics. Figure4(a), (b),
and (c) show the ground truth values mapped onto the t-SNE representations for
Radius of Gyration, end-to-end Decorrelation Time, and Heat Capacity,
respectively. Notably, the figure exhibits a discernible pattern, gracefully
transitioning from minimum to maximum values of the ground truth across the
representations. This smooth progression suggests that the fine-tuned results
used for the t-SNE algorithm have effectively preserved the underlying
structure and relationships in the original high-dimensional data.
Furthermore, the patterns observed in the t-SNE representations suggest that
there may be inherent structural similarities among IDPs with similar
properties. To justify this, we mapped the lengths of sequences onto the t-SNE
representations (as shown in Figure 4 (d), (e), and (f)). The comparison of
ground truth values and sequence lengths on the t-SNE representations (Figure
4) verifies that longer sequences, which often exhibit greater structural
complexities, are associated with higher property values than shorter ones.

In addition, we delve deeper into understanding the contribution of individual
amino acids in determining the properties of IDPs. This investigation is
facilitated by attention weights, enabling the IDP-Bert model to effectively
capture contextual relationships between tokens in a sequence that is
correlated to the protein’s properties. By examining attention weights, we
sought to uncover how the model captures these contextual relationships at the
level of single amino acids.

Figure 5 presents the attention scores visualization for one sequence in the
dataset. The intensity of the color for each amino acid represents the average
attention score from other amino acids within the same sequence toward that
particular amino acid. A darker hue of an amino acid indicates a higher
contribution of this amino acid in determining the property of the given
sequence. By analyzing these attention weights, we can understand which parts
of the protein sequence are more influential in dictating a specific IDP
property. This offers valuable information for a wide range of applications,
from protein engineering and drug design to elucidating fundamental biological
processing and structural biology studies. Ultimately, the insights obtained
from this analysis will open avenues for the development of innovative
biotechnological and therapeutic solutions.

Figure 5: Average attention weights heatmap for a) Radius of Gyration, b) end-
to-end Decorrelation Time, and c) Heat Capacity properties of IDP sequence.

##  4 Conclusion

This study introduces IDP-Bert model, a fine-tuned protein language model
(PLM), to effectively predict Intrinsically Disordered Protein (IDPs)
properties. Given the inherent lack of well-defined structures in IDPs, this
model is designed to decipher the language of IDPs and precisely predict their
intrinsic characteristics based solely on amino acid sequences. Through our
experiments, accurate predictions of structural, dynamic, and thermodynamic
properties, including Radius of Gyration, end-to-end Decorrelation Time, and
Heat Capacity, were achieved. Additionally, analysis of attention weights
assigned to individual amino acids within IDPs illuminates their contributions
to determining protein properties that facilitate protein engineering and drug
design efforts.

##  5 Data and software availability

The necessary information containing the codes and data for downstream tasks
used in this study is available here: https://github.com/ChakradharG/IDP-BERT

This work is supported by the Center for Machine Learning in Health (CMLH) at
Carnegie Mellon University and a start-up fund from the Mechanical Engineering
Department at CMU.

## References

  * Wright and Dyson 2015 Wright, P. E.; Dyson, H. J. Intrinsically disordered proteins in cellular signalling and regulation. _Nature reviews Molecular cell biology_ 2015, _16_ , 18–29
  * Uversky 2013 Uversky, V. N. Unusual biophysics of intrinsically disordered proteins. _Biochimica et Biophysica Acta (BBA)-Proteins and Proteomics_ 2013, _1834_ , 932–951
  * Oldfield and Dunker 2014 Oldfield, C. J.; Dunker, A. K. Intrinsically disordered proteins and intrinsically disordered protein regions. _Annual review of biochemistry_ 2014, _83_ , 553–584
  * Dunker et al. 2001 Dunker, A. K.; Lawson, J. D.; Brown, C. J.; Williams, R. M.; Romero, P.; Oh, J. S.; Oldfield, C. J.; Campen, A. M.; Ratliff, C. M.; Hipps, K. W.; others Intrinsically disordered protein. _Journal of molecular graphics and modelling_ 2001, _19_ , 26–59
  * Tompa 2012 Tompa, P. Intrinsically disordered proteins: a 10-year recap. _Trends in biochemical sciences_ 2012, _37_ , 509–516
  * Ahrens et al. 2017 Ahrens, J. B.; Nunez-Castilla, J.; Siltberg-Liberles, J. Evolution of intrinsic disorder in eukaryotic proteins. _Cellular and Molecular Life Sciences_ 2017, _74_ , 3163–3174
  * Chen et al. 2006 Chen, J. W.; Romero, P.; Uversky, V. N.; Dunker, A. K. Conservation of intrinsic disorder in protein domains and families: II. functions of conserved disorder. _Journal of proteome research_ 2006, _5_ , 888–898
  * Handa et al. 2023 Handa, T.; Kundu, D.; Dubey, V. K. Perspectives on evolutionary and functional importance of intrinsically disordered proteins. _International Journal of Biological Macromolecules_ 2023, _224_ , 243–255
  * Chen et al. 2006 Chen, J. W.; Romero, P.; Uversky, V. N.; Dunker, A. K. Conservation of intrinsic disorder in protein domains and families: I. A database of conserved predicted disordered regions. _Journal of proteome research_ 2006, _5_ , 879–887
  * Bondos et al. 2022 Bondos, S. E.; Dunker, A. K.; Uversky, V. N. Intrinsically disordered proteins play diverse roles in cell signaling. _Cell Communication and Signaling_ 2022, _20_ , 20
  * Uversky 2016 Uversky, V. N. Dancing protein clouds: the strange biology and chaotic physics of intrinsically disordered proteins. _Journal of Biological Chemistry_ 2016, _291_ , 6681–6688
  * Chakrabarti and Chakravarty 2022 Chakrabarti, P.; Chakravarty, D. Intrinsically disordered proteins/regions and insight into their biomolecular interactions. _Biophysical Chemistry_ 2022, _283_ , 106769
  * Uversky 2019 Uversky, V. N. Intrinsically disordered proteins and their “mysterious”(meta) physics. _Frontiers in Physics_ 2019, _7_ , 10
  * Arai et al. 2024 Arai, M.; Suetaka, S.; Ooka, K. Dynamics and interactions of intrinsically disordered proteins. _Current Opinion in Structural Biology_ 2024, _84_ , 102734
  * Saurabh et al. 2023 Saurabh, S.; Nadendla, K.; Purohit, S. S.; Sivakumar, P. M.; Cetinel, S. Fuzzy drug targets: disordered proteins in the drug-discovery realm. _ACS omega_ 2023, _8_ , 9729–9747
  * Rezaei-Ghaleh et al. 2012 Rezaei-Ghaleh, N.; Blackledge, M.; Zweckstetter, M. Intrinsically disordered proteins: from sequence and conformational properties toward drug discovery. _ChemBioChem_ 2012, _13_ , 930–950
  * Wang et al. 2011 Wang, J.; Cao, Z.; Zhao, L.; Li, S. Novel strategies for drug discovery based on intrinsically disordered proteins (IDPs). _International journal of molecular sciences_ 2011, _12_ , 3205–3219
  * Uversky 2012 Uversky, V. N. Intrinsically disordered proteins and novel strategies for drug discovery. _Expert opinion on drug discovery_ 2012, _7_ , 475–488
  * Ambadipudi and Zweckstetter 2016 Ambadipudi, S.; Zweckstetter, M. Targeting intrinsically disordered proteins in rational drug discovery. _Expert opinion on drug discovery_ 2016, _11_ , 65–77
  * Joshi and Vendruscolo 2015 Joshi, P.; Vendruscolo, M. Druggability of intrinsically disordered proteins. _Intrinsically disordered proteins studied by NMR spectroscopy_ 2015, 383–400
  * Li et al. 2022 Li, Z.; Meidani, K.; Yadav, P.; Barati Farimani, A. Graph neural networks accelerated molecular dynamics. _The Journal of Chemical Physics_ 2022, _156_
  * Wang et al. 2022 Wang, Y.; Wang, J.; Cao, Z.; Barati Farimani, A. Molecular contrastive learning of representations via graph neural networks. _Nature Machine Intelligence_ 2022, _4_ , 279–287
  * Guntuboina et al. 2023 Guntuboina, C.; Das, A.; Mollaei, P.; Kim, S.; Barati Farimani, A. PeptideBERT: A language model based on transformers for peptide property prediction. _The Journal of Physical Chemistry Letters_ 2023, _14_ , 10427–10434
  * Yadav et al. 2022 Yadav, P.; Mollaei, P.; Cao, Z.; Wang, Y.; Farimani, A. B. Prediction of GPCR activity using machine learning. _Computational and Structural Biotechnology Journal_ 2022, _20_ , 2564–2573
  * Mollaei and Barati Farimani 2023 Mollaei, P.; Barati Farimani, A. Activity Map and Transition Pathways of G Protein-Coupled Receptor Revealed by Machine Learning. _Journal of Chemical Information and Modeling_ 2023, _63_ , 2296–2304
  * Kim et al. 2024 Kim, S.; Mollaei, P.; Antony, A.; Magar, R.; Barati Farimani, A. GPCR-BERT: Interpreting Sequential Design of G Protein-Coupled Receptors Using Protein Language Models. _Journal of Chemical Information and Modeling_ 2024,
  * Mollaei and Barati Farimani 2023 Mollaei, P.; Barati Farimani, A. Unveiling Switching Function of Amino Acids in Proteins Using a Machine Learning Approach. _Journal of Chemical Theory and Computation_ 2023, _19_ , 8472–8480
  * Liao et al. 2023 Liao, Y.-L.; Wood, B.; Das, A.; Smidt, T. Equiformerv2: Improved equivariant transformer for scaling to higher-degree representations. _arXiv preprint arXiv:2306.12059_ 2023,
  * Wang et al. 2023 Wang, Y.; Li, Z.; Barati Farimani, A. _Machine Learning in Molecular Sciences_ ; Springer, 2023; pp 21–66
  * Goldberg 2022 Goldberg, Y. _Neural network methods for natural language processing_ ; Springer Nature, 2022
  * Vaswani et al. 2017 Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; Polosukhin, I. Attention is all you need. _Advances in neural information processing systems_ 2017, _30_
  * Wolf et al. 2020 Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; others Transformers: State-of-the-art natural language processing. Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations. 2020; pp 38–45
  * Thirunavukarasu et al. 2023 Thirunavukarasu, A. J.; Ting, D. S. J.; Elangovan, K.; Gutierrez, L.; Tan, T. F.; Ting, D. S. W. Large language models in medicine. _Nature medicine_ 2023, _29_ , 1930–1940
  * Defresne et al. 2021 Defresne, M.; Barbe, S.; Schiex, T. Protein design with deep learning. _International Journal of Molecular Sciences_ 2021, _22_ , 11741
  * Devlin et al. 2019 Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. 2019
  * Elnaggar et al. 2021 Elnaggar, A.; Heinzinger, M.; Dallago, C.; Rehawi, G.; Wang, Y.; Jones, L.; Gibbs, T.; Feher, T.; Angerer, C.; Steinegger, M.; others Prottrans: Toward understanding the language of life through self-supervised learning. _IEEE transactions on pattern analysis and machine intelligence_ 2021, _44_ , 7112–7127
  * Ock et al. 2023 Ock, J.; Guntuboina, C.; Barati Farimani, A. Catalyst Energy Prediction with CatBERTa: Unveiling Feature Exploration Strategies through Large Language Models. _ACS Catalysis_ 2023, _13_ , 16032–16044
  * Ramadoss et al. 2023 Ramadoss, R.; Kumar, J. K.; others Pre-Trained Language Models Based Sequence Prediction of Wnt-Sclerostin Protein Sequences in Alveolar Bone Formation. _Journal of Pioneering Medical Sciences_ 2023, _12_
  * Patil et al. 2023 Patil, S.; Mollaei, P.; Farimani, A. B. Forecasting COVID-19 New Cases Using Transformer Deep Learning Model. _medRxiv_ 2023, 2023–11
  * Lobanov et al. 2008 Lobanov, M. Y.; Bogatyreva, N.; Galzitskaya, O. Radius of gyration as an indicator of protein structure compactness. _Molecular Biology_ 2008, _42_ , 623–628
  * Trément et al. 2014 Trément, S.; Schnell, B.; Petitjean, L.; Couty, M.; Rousseau, B. Conservative and dissipative force field for simulation of coarse-grained alkane molecules: A bottom-up approach. _The Journal of chemical physics_ 2014, _140_
  * Uversky 2017 Uversky, V. N. Paradoxes and wonders of intrinsic disorder: Stability of instability. _Intrinsically disordered proteins_ 2017, _5_ , e1327757
  * Jumper et al. 2021 Jumper, J.; Evans, R.; Pritzel, A.; Green, T.; Figurnov, M.; Ronneberger, O.; Tunyasuvunakool, K.; Bates, R.; Žídek, A.; Potapenko, A.; others Highly accurate protein structure prediction with AlphaFold. _Nature_ 2021, _596_ , 583–589
  * Steinegger et al. 2019 Steinegger, M.; Mirdita, M.; Söding, J. Protein-level assembly increases protein sequence recovery from metagenomic samples manyfold. _Nature methods_ 2019, _16_ , 603–606
  * Steinegger and Söding 2018 Steinegger, M.; Söding, J. Clustering huge protein sequence sets in linear time. _Nature communications_ 2018, _9_ , 2542
  * Devlin et al. 2018 Devlin, J.; Chang, M.-W.; Lee, K.; Toutanova, K. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_ 2018,
  * Bridle 1990 Bridle, J. S. _Neurocomputing: Algorithms, architectures and applications_ ; Springer, 1990; pp 227–236
  * Patel et al. 2022 Patel, R. A.; Borca, C. H.; Webb, M. A. Featurization strategies for polymer sequence or composition design by machine learning. _Molecular Systems Design & Engineering_ 2022, _7_ , 661–676
  * Hatos et al. 2020 Hatos, A.; Hajdu-Soltész, B.; Monzon, A. M.; Palopoli, N.; Álvarez, L.; Aykac-Fas, B.; Bassot, C.; Benítez, G. I.; Bevilacqua, M.; Chasapi, A.; others DisProt: intrinsic protein disorder annotation in 2020. _Nucleic acids research_ 2020, _48_ , D269–D276
  * Piovesan et al. 2017 Piovesan, D.; Tabaro, F.; Mičetić, I.; Necci, M.; Quaglia, F.; Oldfield, C. J.; Aspromonte, M. C.; Davey, N. E.; Davidović, R.; Dosztányi, Z.; others DisProt 7.0: a major update of the database of disordered proteins. _Nucleic acids research_ 2017, _45_ , D219–D227
  * Thompson et al. 2022 Thompson, A. P.; Aktulga, H. M.; Berger, R.; Bolintineanu, D. S.; Brown, W. M.; Crozier, P. S.; in’t Veld, P. J.; Kohlmeyer, A.; Moore, S. G.; Nguyen, T. D.; others LAMMPS-a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales. _Computer Physics Communications_ 2022, _271_ , 108171
  * Regy et al. 2021 Regy, R. M.; Thompson, J.; Kim, Y. C.; Mittal, J. Improved coarse-grained model for studying sequence dependent phase separation of disordered proteins. _Protein Science_ 2021, _30_ , 1371–1379
  * Loshchilov and Hutter 2017 Loshchilov, I.; Hutter, F. Decoupled weight decay regularization. _arXiv preprint arXiv:1711.05101_ 2017,
  * 54 PyTorch: ReduceLROnPlateau — PyTorch 1.9.0 documentation. https://pytorch.org/docs/stable/generated/torch.optim.lr_scheduler.ReduceLROnPlateau.html#torch.optim.lr_scheduler.ReduceLROnPlateau, Accessed January 2024
  * Van Dyk and Meng 2001 Van Dyk, D. A.; Meng, X.-L. The art of data augmentation. _Journal of Computational and Graphical Statistics_ 2001, _10_ , 1–50
  * Van der Maaten and Hinton 2008 Van der Maaten, L.; Hinton, G. Visualizing data using t-SNE. _Journal of machine learning research_ 2008, _9_