# A large language model for predicting T cell receptor-antigen binding
specificity ††thanks: This work was supported by National Natural Science
Foundation of China (No. 62072058, No. 62372229), Natural Science Foundation
of Jiangsu Province (No. BK20231271).

1st Xing Fang  College of Computer and Information Engineering
Nanjing Tech University
Nanjing, China
<EMAIL_ADDRESS>2nd Chenpeng Yu  College of Computer and
Information Engineering
Nanjing Tech University
Nanjing, China
<EMAIL_ADDRESS>3rd Shiye Tian  College of Computer and
Information Engineering
Nanjing Tech University
Nanjing, China
<EMAIL_ADDRESS>4th Hui Liu  College of Computer and Information
Engineering
Nanjing Tech University
Nanjing, China
Correspondence<EMAIL_ADDRESS>
###### Abstract

The human immune response depends on the binding of T-cell receptors (TCRs) to
antigens (pTCR), which elicits the T cells to eliminate viruses, tumor cells,
and other pathogens. The ability of human immunity system responding to
unknown viruses and bacteria stems from the TCR diversity. However, this vast
diversity poses challenges on the TCR-antigen binding prediction methods. In
this study, we propose a Masked Language Model (MLM), referred to as tcrLM, to
overcome limitations in model generalization. Specifically, we randomly masked
sequence segments and train tcrLM to infer the masked segment, thereby extract
expressive feature from TCR sequences. Meanwhile, we introduced virtual
adversarial training techniques to enhance the model’s robustness. We built
the largest TCR CDR3 sequence dataset to date (comprising 2,277,773,840
residuals), and pre-trained tcrLM on this dataset. Our extensive experimental
results demonstrate that tcrLM achieved AUC values of 0.937 and 0.933 on
independent test sets and external validation sets, respectively, which
remarkably outperformed four previously published prediction methods. On a
large-scale COVID-19 pTCR binding test set, our method outperforms the current
state-of-the-art method by at least 8%, highlighting the generalizability of
our method. Furthermore, we validated that our approach effectively predicts
immunotherapy response and clinical outcomes on a clinical cohorts. These
findings clearly indicate that tcrLM exhibits significant potential in
predicting antigenic immunogenicity.
The source code and datasets used in this study are available at:
https://github.com/hliulab/tcrLM.

###### Index Terms:

Immune Response, Large Language Model, Transformer, Virtual Adversarial
Training, T Cell Receptor, Tumor Antigen

##  I Introduction

Antigen recognition is a prerequisite for triggering the T cell-mediated anti-
cancer immune response. T cells interact via a dimeric surface protein, the
T-cell receptor (TCR), with an antigen presented on a major histocompatibility
complex (MHC) located on the surface of antigen-presenting cells (APC). Their
interaction triggers the activation of T-cells, leading to their clonal
expansion and differentiation into effector T-cells, such as cytotoxic T
lymphocytes (CTLs). These CTLs can migrate to the tumor microenvironment and
directly kill tumor cells through secreting toxic molecules, like perforin and
granzyme, or indirectly promote tumor cell death by modulating the immune
environment. In addition, a portion of activated T-cells further differentiate
into memory T-cells, which survive long-term in the body, providing the host
with long-term immune memory and a rapid response capability against tumors.

The T-cell receptor (TCR) comprises $\alpha$ and $\beta$ chains that are
generated through genetic recombination, leading to an expansive TCR
repertoire. Previous studies have documented that humans have the potential to
generate approximately 1015 to 1020 distinct TCR sequences. This diversity
predominantly manifests in the complementarity determining region 3 (CDR3)
[1], which engages directly with the peptide-MHC complex, thereby dictating
the binding specificity of the TCR [2, 3]. The antigens recognized by TCRs are
peptide segments composed of 8-14 amino acids. Essentially, both the TCR CDR3
and antigens can be expressed as a sequence of amino acids, exhibiting
striking similarity to human natural language. Consequently, we can draw
inspiration from methods in the field of in Natural Language Processing, and
utilize advanced language models to learn and analyze these biological
sequences.

This marks the first application of large language model technology in the
field of immunology, promising unprecedented breakthroughs, a deeper
understanding of TCR complexity, and potentially opening a new chapter in
immunotherapy.

##  II Related Work

###  II-A Predicting pTCR binding specificity

Quite a few computational methods have been developed to predict the binding
specificity of T-cell receptors (TCRs) to peptide-MHC complexes (pMHCs). These
methods roughly fall into three main categories: 1) Clustering-based methods
measure the similarities between TCRs and try to understand underlying
patterns in binding to antigens. Representative methods include TCRdist [4],
DeepTCR [5], GIANA [6], iSMART [7], GLIPH [8], and ELATE [9] models. 2)
Peptide-specific models focus on predicting the binding of specific peptide
segments to TCRs, including TCRGP [10], TCRex [11], and NetTCR-2 [12]. 3)
Generic binding prediction models are not limited to specific peptides but
require models trained on known TCR bindings, including PanPep [13], pMTnet
[14], DLpTCR [15], ERGO2 [16], and TITAN [17]. Although these methods have
demonstrated promising accuracy in specific scenarios, they have limitations
in generalizing to unseen peptides, which is crucial for identifying the
binding specificity of neoantigens or exogenous antigens. Therefore,
accurately identifying the pTCR bindings remains a challenging task.

(a) TCR Frequency (b) Antigen Frequency

Figure 1: Statistical frequency distributions of TCR and antigen sequences
with respect to length

###  II-B Protein language model

Large language models have gained increasing attention in the field of protein
modeling in recent years, and yield to new insights and capabilities for
understanding the structures and functions of proteins. Most protein language
models are built upon the Transformer encoder, which can encode protein
sequences or structures into fixed-length latent representations. The
representations have been proven to boost the performance of downstream tasks
related to proteins, such as the prediction of structure, functions, and
protein-drug interactions. The masked language models (MLM), such as ESM1b
[18], ESM-1v [19], ProtFlash [20], and ProtTrans [21], aim to reconstruct
masked tokens based on surrounding sequences. Their success in protein tasks
has inspired us to make advantage of MLM to accommodate the diversity of TCR
sequences.

(a) Masked language model (b) pTCR prediction network

Figure 2: Illustrative diagram of tcrLM model and its encoder architecture, as
well as the network for pTCR binding prediction

##  III MATERIALS AND METHODS

###  III-A Data source and preprocessing

For model pre-training, we set about to establish a largest TCR CDR3 sequences
to date. For this purpose, we collected 113,888,692 distinct TCR CDR3
sequences from more than ten databases and publications. With over 100 million
sequences, our model is able to capture underlying feature from TCR sequences.
The frequency distribution of TCR CDR3 sequences is illustrated in Fig. 1(a),
which reveals that most CDR3 sequences range between 10 to 20 amino acids in
length.

For fine-tuning, we build a benchmark dataset of pTCR bindings by collecting a
substantial amount of pTCR binding data from various databases. We consider
both the $\alpha$ and $\beta$ chains of TCR and treat them as independent CDR3
sequences, as previous studies have demonstrated both of them play critical
role in antigen recognition. The pTCR dataset consists of 109,554 bindings,
covering 1,377 unique antigens and 104,623 unique TCR CDR3 sequences. To our
best knowledge, this the largest pTCR binding dataset to date.

The frequency distribution of antigen sequences is shown in Fig. 1(b),
indicating that most antigens 9mer in length. Next, we randomly mismatched TCR
and peptide sequences to generate the same number of negative pTCR samples.
During the fine-tuning stage, we randomly select 10% of the pTCR binding
dataset as an independent test set, while the remaining 90% is used as the
training set. The training set encompassed 881 distinct antigens and 93,727
unique CDR3 sequences, while the test set included 496 distinct antigens and
10,896 unique CDR3 sequences. The five-fold cross-validation is used to
optimize the model hyperparameters.

###  III-B tcrLM model

The BERT model [22] pre-trains deep bidirectional representations through the
Masked Language Model (MLM) task, which allows to freely capture information
from all tokens in the context, whether they appear before or after the
current word. Therefore, we apply the BERT-based language model, named tcrLM,
to utilize Transformer encoder and its training methodology for the masked
language model to effectively process and parse TCR sequences.

As illustrated in Fig. 2(a), each TCR CDR3 sequence is padded to a fixed
length of 20, since all CDR3 sequences do not exceed 20. We randomly generate
a mask of length between 3 to 5 to cover a continuous segment in the TCR
sequence. The masked sequence passes through an embedding layer that converts
each token into a high-dimensional embedding. The embeddings are then taken as
input to the encoder and ultimately mapped into a 20x512 matrix, aiming to
capture the underlying biological features within the TCR sequences. Next, the
matrix is flattened into a 2176-dimensional vector, which subsequently passes
through three fully connected layers with node counts of 256, 64, and 420
(20x21), activated by the ReLU function. As a result, the vector is reshaped
into a 20x21 matrix that represents the amino acid distribution at each
position of a TCR sequence. We introduced the rotary positional embedding
(RoPE)[23] into tcrLM. This method encodes absolute positions using rotation
matrices and integrates explicit relative positional dependencies into self-
attention calculations. RoPE offers several advantages, such as flexibility in
sequence length, decaying dependencies between tokens as their relative
distance increases, and the capability to enhance linear self-attention with
relative positional encoding. Furthermore, we incorporated the mixed chunk
attention (MCA) [24] to combine the advantages of both local attention and
linear attention mechanisms.

The language model is trained to recover the masked segment using the
surrounding sequences on the collected more than 100 million TCR sequences. In
our practice, we applied different learning rates to distinct components of
the model. For the encoder, the learning rate was set to 1e-6, while for the
subsequent fully connected layers, it was adjusted to 1e-4. The optimizer was
set to Adam. The pre-training was conducted on a CentOS Linux 8.2.2004 (Core)
system, equipped with an Intel(R) Xeon(R) Silver 4210R CPU running at 2.40GHz,
along with a GeForce RTX 4090 GPU and 128GB of RAM. The model was implemented
using the PyTorch 2.2.1 framework, and accelerated by the DeepSpeed 0.13.5
library. The entire training process spanned four days, resulting in a
reduction of the perplexity from 10.4 to 6.4.

###  III-C Predicting pTCR binding

The pre-trained encoder is used in the prediction network for pTCR binding. As
shown in Fig. 2(b), the antigen sequences and TCR sequences are padded to the
maximum length of 15 and 20, respectively. After passing through an embedding
layer, these two sequences are taken as input of the pre-trained encoder, and
transformed into matrices of dimensions 15x512 and 20x512. Subsequently, these
two matrices are concatenated along the feature dimension to form a 35x512
matrix, which is then flattened into a 17,920-dimensional vector. This vector
passes through a fully-connected layer and is projected to a 2-dimensional
vector, followed by softmax layer to obtain the final predicted probabilities.
During the training process, the pre-trained encoder is frozen. For pTCR
binding prediction task, we use the cross-entropy as the loss function:

$\mathcal{L}_{cls}=-\frac{1}{N}\sum_{i=1}^{N}\left[y_{i}\cdot\log(p_{i})+(1-y_{i})\cdot\log(1-p_{i})\right]$
(1)

in which $y_{i}$ and $p_{i}$ represent the actual and predicted pTCR binding,
and $N$ is the total number of training samples.

(a) ROC curves (b) Performance comparison

(c) PPV

Figure 3: Performance evaluation of tcrLM and four comparative methods on
independent test set

###  III-D Virtual adversarial training

Given the vast diversity of the TCR repertoire, the training data currently
available is still limited and even biased. This poses a tough challenge on
the development of robust prediction models for pTCR binding. To address this
challenge, we employs virtual adversarial training to enhance the model’s
generalizability. Since protein sequence are discrete, we introduce
perturbations in the embedding layer of the sequences [25]. For this purpose,
we create adversarial examples designed to maximize the loss function by
applying adversarial perturbations to the sequence embeddings, namely, the
adversaries are generated in the direction of gradient ascend and constrained
by L2 norm. Formally, we define the adversarial loss as below:

$\displaystyle L_{\mathrm{vadv}}(x,\theta)$
$\displaystyle=D\left[p(y|x,\hat{\theta}),p(y|x+r_{\mathrm{vadv}},\theta)\right]),$
(2) $\displaystyle\text{ where }r_{\mathrm{vadv}}$
$\displaystyle=\arg\max_{r;\|r\|\leq\epsilon}D\left[p(y|x_{*},\hat{\theta}),p(y|x+r)\right],$

$D$ represents the function that measures the divergence between two
distributions, $p(y|x)$ denotes the probability of the model predicting label
$y$ given input $x$, $r_{\mathrm{vadv}}$ is a virtual adversarial perturbation
regarding the input sample $x$. This perturbation strives to maximize the
divergence between $p(y|x_{*},\hat{\theta})$ and $p(y|x+r)$ by choosing the
direction of gradient ascent.

Virtual adversarial training requires the model to minimize not only the risk
on actual observed data but also the risk arising from adversarial loss. This
approach helps reduce the model’s sensitivity to slight input variations,
thereby enhancing its generalizability. By incorporating virtual adversarial
training, we aim to build a more robust and reliable prediction model, capable
of handling the vast diversity of the TCR repertoire even with limited and
potentially biased training data.

##  IV Results

###  IV-A Evaluation on independent test set

We first assessed the performance of tcrLM on the 10% hold-out independent
test set. To benchmark the performance in predicting pTCR binding specificity,
we conducted a comparative analysis with four currently state-of-the-art
methods: PanPep [13], ERGO2 [16], pMTnet [14], and DLpTCR [15]. For the former
three methods, we executed their executable codes using their recommended
parameters on the same workstation. For DLpTCR, we utilized its web server to
make predictions on the test data.

We found that tcrLM significantly outperforms all other methods, as shown in
Fig. 3(a-c). Specifically, tcrLM achieved an AUROC of 0.937 and an AUPR of
0.933, highlighting its exceptional predictive capability in determining pTCR
binding specificity. Among the compared methods, only ERGO2 demonstrated
moderate performance, with an AUROC of 0.704 and an AUPR of 0.747. The
remaining methods performed close to random guessing, indicating their limited
performance in predicting pTCR binding. These previous methods may showed
promising performance on small datasets, but their performance declined
significantly when tested on large-scale datasets. This suggests that they
have weak generalizability and struggle to adapt to real-world, large-scale
data scenarios.

To further examine our model’s ability to prioritize pTCR bindings, we
calculated the positive predictive value (PPV) for the top-ranked predicted
pTCR samples on two different datasets. Specifically, we evaluated the PPV for
the top 100, top 1000, and top 5000 predictions. As shown in Fig. 3(d), tcrLM
achieved PPV values of 99%, 98.7%, and 96.74% for the top 100, top 1000, and
top 5000 predictions, respectively. In comparison, other methods demonstrated
inferior prioritization capacity compared to tcrLM.

(a) Performance on external set (b) Performance on COVID-19 set

(c) PPV

Figure 4: Performance evaluation of tcrLM and four comparative methods on
external and COVID-19 test set

###  IV-B Evaluation on external set

To further evaluate the performance of our model, we constructed an external
test set collected from tens of publications. The external set included 63,324
pTCR binding samples across 998 unique peptides and 53,439 CDR3 sequences,
which allowed us to assess the capacity of our method in real-world scenarios.
As shown in Fig. 4(a), tcrLM achieved AUROC and AUPR values exceeding 0.9, and
significantly outperformed the second-best method, ERGO2, which had AUROC and
AUPR values of only about 0.7. This observation reflected the limitations of
previously published methods, while in turn validated the superior performance
of tcrLM in predicting pTCR binding specificity between unseen peptides and
TCR sequences.

###  IV-C Performance evaluation on COVID-19 dataset

To validate the generalizability of tcrLM, we tested its capacity to predict
binding between virus-derived antigens and TCRs. We collected a total of
520,000 binding samples between COVID-19 virus antigens and human TCRs from
the ImmuneCODE database [26]. For class balance, we generated equal number of
negative samples through random shuffle. As a result, we created a million-
scale COVID-19 test set, which is the largest pTCR binding test set to date.

For objective performance evaluation, we again compared tcrLM with previously
published methods, including PanPep, ERGO2, DLpTCR, and pMTnet [14]. As shown
in Fig.4(b), tcrLM achieved AUROC and AUPR values of 0.595 and 0.602,
respectively, while the comparative methods obtained AUROC and AUPR values
only slightly above 0.5, which is nearly close to random guessing.
Furthermore, we calculated the positive predictive value (PPV) of the top 100,
top 1000, and top 5000 predictions made by each method. Our model consistently
achieved 93% PPV value, significantly better than all competitors, whose PPV
values remained below 65% (Fig.4c). Overall, the significant advantage over
previous methods strongly validates the robust generalizability of tcrLM and
highlights its potential in facilitating effective immune-based therapies and
vaccine design targeting the COVID-19 virus.

###  IV-D tcrLM predicts immunotherapy outcomes

To further explore the predictive capacity of tcrLM, we conducted an
exploratory analysis of pTCR in a cohort of patients with advanced melanoma.
This cohort consists of 29 patients who underwent treatment by immune
checkpoint inhibitors. Based on the TCR-seq and genomic sequencing data, we
took the missense mutations as anchors, and generated all possible 9-mer
peptides covering the anchors. After extracting CDR3 sequences from the TCR-
seq data, we created all conceivable peptide-CDR3 pairs for each patient,
yielding a total of 81,851,486 pTCR candidates.

We scored these samples using tcrLM and selected the top 25,000 highest-scored
pairs for each patient. Subsequently, we categorized the patients into four
categories according to RECIST criteria: Complete Remission (CR), Partial
Remission (PR), Stable Disease (SD), and Progressive Disease (PD) groups. As
shown in Fig. 5(a), the boxplots of predicted scores of four groups differed
significantly from each other. The analysis of variance verified the
statistical differences among the groups (F-test, p-value¡0.01). By
stratifying the patients into benefit, non-benefit, and long-term survival
groups, we observed that patients in the long-term survival group had higher
predicted scores compared to the other groups (Fig. 5b). This findings
validated the correlation between higher pTCR predicted scores and better
immunotherapy outcomes, offering a new perspective and a possible clinical
indicator for tumor immunotherapy.

(a) Boxplots of RECIST grouping (b) Violin plots of long-term grouping

Figure 5: Combined Box Plot and Violin Plot

###  IV-E Ablation experiments

To verify the effectiveness of the pre-trained encoder, we conducted ablation
experiments to evaluate its impact on the performance in predicting pTCR
binding. In particular, we removed the pre-trained encoder as the antigen
sequence encoder, the TCR sequence encoder, or both. The performance
comparison results of the three ablated models were shown in Table I. We found
that removal of the pre-trained encoder from the model always led to
performance decline. In particular, when both the antigen and TCR encoders
were removed simultaneously, the AUC value dropped to 0.85 AUC value,
demonstrating the significance of the pre-trained encoder.

We also tested the impact virtual adversarial training, and found that the
ablated model without virtual adversarial led to 2% performance reduction.

TABLE I: Performance evaluation of ablated models

Ablated model | AUROC | Accuracy | MCC | F1 | AUPR
---|---|---|---|---|---
tcrLM | 0.9404 | 0.8612 | 0.7234 | 0.8574 | 0.9331
Without TCR encoder | 0.9213 | 0.8474 | 0.6998 | 0.8554 | 0.9213
Without antigen encoder | 0.8617 | 0.7905 | 0.5812 | 0.7880 | 0.8551
Without TCR&antigen Encoder | 0.8522 | 0.7737 | 0.5478 | 0.7702 | 0.8765
Without Virtual Adversarial | 0.9266 | 0.8390 | 0.7082 | 0.8508 | 0.9180

##  V Conclusion

This study proposed a BERT-based large language model, named tcrLM, to enhance
the accuracy of predicting pTCR binding specificity by utilizing the strong
capability in feature extraction of large language models. We have pre-trained
the model on more than 100 million TCR sequence, and verified its performance
on downstream tasks. Our experimental results demonstrated that tcrLM
exhibited superior performance compared to the current four state-of-the-art
models, on both independent and external test sets. Moreover, in the real-
world scenario of COVID-19 and patient-derived datasets, tcrLM still yielded
promising performance, highlighting its potential applicability in vaccine
design and immunotherapy treatment.

## References

  * [1] Shu-Qi Zhang, Patricia Parker, Ke-Yue Ma, Chenfeng He, Qian Shi, Zhonghao Cui, Chad M Williams, Ben S Wendel, Amanda I Meriwether, Mary Alice Salazar, et al.  Direct measurement of t cell receptor affinity and sequence from naïve antiviral t cells.  Science translational medicine, 8(341):341ra77–341ra77, 2016.
  * [2] Mark M Davis and Pamela J Bjorkman.  T-cell antigen receptor genes and t-cell recognition.  Nature, 334(6181):395–402, 1988.
  * [3] Michelle Krogsgaard and Mark M Davis.  How t cells’ see’antigen.  Nature immunology, 6(3):239–245, 2005.
  * [4] Pradyot Dash, Andrew J Fiore-Gartland, Tomer Hertz, George C Wang, Shalini Sharma, Aisha Souquette, Jeremy Chase Crawford, E Bridie Clemens, Thi HO Nguyen, Katherine Kedzierska, et al.  Quantifiable predictive features define epitope-specific t cell receptor repertoires.  Nature, 547(7661):89–93, 2017.
  * [5] John-William Sidhom, H Benjamin Larman, Drew M Pardoll, and Alexander S Baras.  Deeptcr is a deep learning framework for revealing sequence concepts within t-cell repertoires.  Nature communications, 12(1):1605, 2021.
  * [6] Hongyi Zhang, Xiaowei Zhan, and Bo Li.  Giana allows computationally-efficient tcr clustering and multi-disease repertoire classification by isometric transformation.  Nature communications, 12(1):4699, 2021.
  * [7] Hongyi Zhang, Longchao Liu, Jian Zhang, Jiahui Chen, Jianfeng Ye, Sachet Shukla, Jian Qiao, Xiaowei Zhan, Hao Chen, Catherine J Wu, et al.  Investigation of antigen-specific t-cell receptor clusters in human cancers.  Clinical Cancer Research, 26(6):1359–1371, 2020.
  * [8] Jacob Glanville, Huang Huang, Allison Nau, Olivia Hatton, Lisa E Wagar, Florian Rubelt, Xuhuai Ji, Arnold Han, Sheri M Krams, Christina Pettus, et al.  Identifying specificity groups in the t cell receptor repertoire.  Nature, 547(7661):94–98, 2017.
  * [9] Shirit Dvorkin, Reut Levi, and Yoram Louzoun.  Autoencoder based local t cell repertoire density can be used to classify samples and t cell receptors.  PLoS Computational Biology, 17(7):e1009225, 2021.
  * [10] Emmi Jokinen, Jani Huuhtanen, Satu Mustjoki, Markus Heinonen, and Harri Lähdesmäki.  Predicting recognition between t cell receptors and epitopes with tcrgp.  PLoS computational biology, 17(3):e1008814, 2021.
  * [11] Sofie Gielis, Pieter Moris, Wout Bittremieux, Nicolas De Neuter, Benson Ogunjimi, Kris Laukens, and Pieter Meysman.  Detection of enriched t cell epitope specificity in full t cell receptor sequence repertoires.  Frontiers in immunology, 10:2820, 2019.
  * [12] Alessandro Montemurro, Viktoria Schuster, Helle Rus Povlsen, Amalie Kai Bentzen, Vanessa Jurtz, William D Chronister, Austin Crinklaw, Sine R Hadrup, Ole Winther, Bjoern Peters, et al.  Nettcr-2.0 enables accurate prediction of tcr-peptide binding by using paired tcr$\alpha$ and $\beta$ sequence data.  Communications biology, 4(1):1060, 2021.
  * [13] Yicheng Gao, Yuli Gao, Yuxiao Fan, Chengyu Zhu, Zhiting Wei, Chi Zhou, Guohui Chuai, Qinchang Chen, He Zhang, and Qi Liu.  Pan-peptide meta learning for t-cell receptor–antigen binding recognition.  Nature Machine Intelligence, 5(3):236–249, 2023.
  * [14] Tianshi Lu, Ze Zhang, James Zhu, Yunguan Wang, Peixin Jiang, Xue Xiao, Chantale Bernatchez, John V Heymach, Don L Gibbons, Jun Wang, et al.  Deep learning-based prediction of the t cell receptor–antigen binding specificity.  Nature machine intelligence, 3(10):864–875, 2021.
  * [15] Zhaochun Xu, Meng Luo, Weizhong Lin, Guangfu Xue, Pingping Wang, Xiyun Jin, Chang Xu, Wenyang Zhou, Yideng Cai, Wenyi Yang, et al.  Dlptcr: an ensemble deep learning framework for predicting immunogenic peptide recognized by t cell receptor.  Briefings in Bioinformatics, 22(6):bbab335, 2021.
  * [16] Ido Springer, Hanan Besser, Nili Tickotsky-Moskovitz, Shirit Dvorkin, and Yoram Louzoun.  Prediction of specific tcr-peptide binding from large dictionaries of tcr-peptide pairs.  Frontiers in immunology, 11:534364, 2020.
  * [17] Anna Weber, Jannis Born, and María Rodriguez Martínez.  Titan: T-cell receptor specificity prediction with bimodal attention networks.  Bioinformatics, 37(Supplement_1):i237–i244, 2021.
  * [18] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al.  Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences.  Proceedings of the National Academy of Sciences, 118(15):e2016239118, 2021.
  * [19] Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives.  Language models enable zero-shot prediction of the effects of mutations on protein function.  Advances in neural information processing systems, 34:29287–29303, 2021.
  * [20] Lei Wang, Hui Zhang, Wei Xu, Zhidong Xue, and Yan Wang.  Deciphering the protein landscape with protflash, a lightweight language model.  Cell Reports Physical Science, 4(10), 2023.
  * [21] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al.  Prottrans: Toward understanding the language of life through self-supervised learning.  IEEE transactions on pattern analysis and machine intelligence, 44(10):7112–7127, 2021.
  * [22] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [23] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu.  Roformer: Enhanced transformer with rotary position embedding.  Neurocomputing, 568:127063, 2024.
  * [24] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quoc Le.  Transformer quality in linear time.  In International conference on machine learning, pages 9099–9117. PMLR, 2022.
  * [25] Takeru Miyato, Andrew M Dai, and Ian Goodfellow.  Adversarial training methods for semi-supervised text classification.  arXiv preprint arXiv:1605.07725, 2016.
  * [26] Sean Nolan, Marissa Vignali, Mark Klinger, Jennifer N Dines, Ian M Kaplan, Emily Svejnoha, Tracy Craft, Katie Boland, Mitch Pesesky, Rachel M Gittelman, et al.  A large-scale database of t-cell receptor beta (tcr$\beta$) sequences and binding associations from natural and synthetic exposure to sars-cov-2.  Research square, 2020.