* [193] Shitong Luo, Yufeng Su, Xingang Peng, Sheng Wang, Jian Peng, and Jianzhu Ma.  Antigen-specific antibody design and optimization with diffusion-based generative models for protein structures.  Advances in Neural Information Processing Systems, 35:9754–9767, 2022.

##  8 Appendix

####  8.0.1 Protein Representation Learning

##### Overview

Protein representation learning involves learning embeddings to convert raw
protein data into latent space representations, extracting meaningful features
and chemical attributes. More specifically, given a protein
$x=[o_{1},o_{2},\dots o_{L}]$, where each $o_{1}$ represents an amino acid
(sequence-based) or atom coordinate (structure-based), learn an embedding
$z=[h_{1},h_{2},\dots h_{L}]$, where each $h_{i}\in\mathbb{R}^{d}$ represents
a $d-$dimensional token representation for amino acid $o_{i}$. This general
framework can be extended to a multitude of tasks— for token property
prediction tasks, additionally allocate a prediction $p(y_{i}|o_{i})$ in the
embedding, and for pairwise prediction tasks, allocate a prediction
$p(y_{ij}|o_{i},o_{j})$. For overall protein property prediction, allocate
predictions $p(y|x)$ or $p(y|x_{1},x_{2})$ for individual and protein-protein
properties, respectively.

These embeddings can be seen as creating “richer” data spaces for various
models to work on; whereas raw atom graphs and amino acid sequences are
entirely arbitrary, these embeddings are aimed to capture various chemical
traits and properties (i.e. two proteins with similar properties should be
represented with similar embeddings, even if their raw forms are very
different). Thus, representation learning models do not directly perform any
tasks on their own, but instead are incorporated in other generative models to
improve their ability to model these important chemical features.

##### Datasets

Note that a plethora of datasets exist for all various downstream tasks; we do
not describe all such datasets. We focus on datasets used for pre-training in
protein representation learning, described below. Note that UniRef, UniParc,
and ProteinKG are typically used for sequence-based learning, and PDB and
AlphafoldDB are primarily used for structure-based learing. Pfam has been used
in both types of models.

  * •

UniRef [160] \- A clustered version of the Unified Protein KnowledgeBase
(UniProtKB), part of the central resource UniProt, which is a curated and
labeled set of protein sequences and their functions

  * •

UniParc [160] \- A larger dataset of protein sequences, part of the central
resource Uniprot, which includes UniProtKB and adds proteins from a variety of
other sources

  * •

ProteinKG [38] \- A dataset created by Zhang et al. (OntoProtein), aligning
biological knowledge with protein sequences with knowledge graphs (KG), used
for directly injecting biological knowledge in the representation learning
process

  * •

PDB [95] \- Protein Data Bank, a central archive for all experimentally
determined protein structures, widely used in almost all protein structure-
related tasks.

  * •

AlphaFoldDB [136] \- AlphaFold Protein Structure Database, a large dataset
created by using AlphaFold2 to predict structures from sequence datasets, such
as UniProt and Swiss-Prot.

  * •

Pfam [161] \- A collection of protein families, used in multiple sequence
alignments (MSA)

##### Tasks

Representation learning models do not directly address tasks on their own but
rather supplement a wide range of tasks by refining a latent space. Thus, the
protein representation learning field in particular lacks uniformity in
testing and training methods, making it hard to make generalized evaluations.
Some methods like Tasks Assessing Protein Embeddings (TAPE) [162] and, more
recently, Protein sEquence undERstanding (PEER) [163] seek to address this by
compiling various testing metrics into a standardized benchmark for evaluating
these language models. A few of the most prominent tasks include:

  * •

Contact Prediction - Given two amino acid residues, predict the probability
that they “contact”, or are within some 3D distance threshold. Measured using
precision of top $L/5$ medium-range and long-range contacts (contact between
sequentially distant amino acids) according to CASP standards [164].

  * •

Fold Classification - Also known as remote homology prediction. Given an amino
acid sequence, predict the corresponding fold class [165]. Used for finding
structural similarities for distantly related inputs. Tested using the dataset
proposed by Hou et al. [166], which comprises three versions of increasing
difficulty: Family (test set in the same family as the training set),
Superfamily (same superfamily as training) and Fold (same fold as training).
Measured using accuracy.

  * •

Stability Prediction - Given a protein, output a label $y\in\mathbb{R}$
representing the most extreme circumstances under which a protein maintains
its fold above a concentration threshold. Measured using Spearman’s $\rho$.

  * •

PPI - Protein-Protein Interaction, given two proteins, predict whether or not
they interact (binary classification). Measured using accuracy.

##### Metrics

The following metrics are used to evaluate the above tasks:

  * •

Accuracy - Ratio between correct classifications to total classifications.

  * •

Precision - Accuracy within the set of selected elements; in contact
prediction, this refers to the following: of the top $L/5$ contacts with the
highest predicted probability, find the percentage of these which are ground-
truth contacts. $L$ refers to the length of a protein sequence.

  * •

Spearman’s $\mathbf{\rho}$ \-  Spearman’s rank correlation coefficient, found
by sorting the ground-truth and predicted output values into ranked order (so
each protein has a true rank $y_{i}\in\mathbb{N}$ and predicted rank
$x_{i}\in\mathbb{N}$) and computing:

${\displaystyle\rho=1-{\frac{6\sum d_{i}^{2}}{n(n^{2}-1)}},}$

where $d_{i}$ refers to the numerical difference in ranks for each
$y_{i},x_{i}$, and $n$ refers to the total number of elements in each set.

##### Models

Table 8: An overview of the most relevant sequence-based protein representation learning models. All metrics are self-reported unless otherwise noted; metrics reported by different papers may be incomparable due to different test settings. Note that OntoProtein and KeAP have two separate architectures for protein, and knowledge encoding respectively. [**] denotes the current SOTA. Model | Type of Model | Dataset | Contact ($\uparrow$) | Fold Classification ($\uparrow$) | Stability ($\uparrow$)
---|---|---|---|---|---
Family | Superfam. | Fold
UniRep [167] | LSTM RNN | UniRef50 | \ | \ | \ | \ | \
ProtBERT [116] | BERT | BFD100, UniRef100 |  0.556 [40] |  0.528 [40] |  0.192 [40] |  0.170 [40] |  0.651 [40]
ESM-1B [117] | Transformer | UniParc | 0.458 | 0.978 | 0.601 |  0.268 [168] |  0.695 [168]
MSA Trans. [169] | Transformer | 26M MSAs |  0.618 [40] |  0.958 [40] |  0.503 [40] |  0.235 [40] |  0.796 [40]
RSA** [40] | Transformer | Pfam | 0.717 | 0.987 | 0.677 | 0.267 | 0.987
OntoProtein** [38] | ProtBERT, BERT/Trans. | ProteinKG* | 0.40 |  0.96 [40] | \ | 0.24 |  0.75 [40]
KeAP** [39] | BERT, Transformer | ProteinKG | 0.62 | \ | \ | 0.29 | 0.82
GearNET** [168] | Geo-EGNN | AlphaFoldDB | \ | 0.995 | 0.703 | 0.483 | \
DeepFRI [170] | LSTM, GCNN | Pfam | \ |  0.732 [171] |  0.206 [171] |  0.153 [171] | \
IEConv [171] | CNN | PDB | \ | 0.997 | 0.806 | 0.503 | \

As mentioned above, protein representation learning models can be divided into
two broad categories: sequence-based and structure-based. This distinction
simply refers to the type of input these models learn representations for; we
begin by discussing sequence-based learning models. Both amino acid sequences
and protein structures carry valuable information about a protein’s function
and binding properties, and thus these two types of learning can be seen as
complements to each other.

Unified Representation (UniRep) [167] used a multiplicative long-short-term
memory recurrent neural network (mLSTM RNN) to approach the task of
representation learning. By iterating through each amino acid and comparing
its prediction with the true amino acid type, adjusting its parameters at each
step, UniRep gradually learns a better representation of the sequence. While
the LSTM model has been proven inferior to newer transformer and natural
language models, the basic concepts laid an important groundwork for future
representation learning models.

ProtBERT [116] improved upon performance by applying a BERT model to amino
acid sequences, treating each residue as a word. BERT stands for Bidirectional
Encoder Representations from Transformers, which is a pre-trained language
model for neural language processing. Several variations, such as RoBERTa and
DeBERTa, improve upon BERT by enhancing parameters and attention mechanisms
and are used in models discussed later. ESM-1B [117] trained a deep
transformer with 33 layers and 650 million parameters, using the masked
language modeling (MLM) objective for training. ESM-1b considered residue
contexts across the entire sequence through its stacked attention layers, and
its complexity and size allowed it to outperform other SOTA models on a wide
range of tasks, as shown below. ESM-1b’s massive size also resulted in
expensive training costs, however, and smaller research groups would had to
find more unique approaches to compete with ESM-1b on representation learning.

Following ESM-1b’s breakthrough, a series of models applied the concept of
direct knowledge injection. Previous models had simply treated these sequences
as generic sequences of tokens, failing to take advantage of the vast
biological knowledge in existence today. One avenue for knowledge injection
comes from the analysis of Multiple Sequence Alignments (MSA), which groups
proteins concerning their evolutionary families. Because inherent patterns
appear as a result of the evolutionary process, analyzing MSAs can result in a
richer understanding of protein sequences. MSA Transformer [169] was the first
model to intersect a protein language model with MSA analysis, pretraining a
transformer model on MSA input. While MSAs can provide large benefits, the
alignment process is computationally costly. Retrieved Sequence Augmentation
(RSA) [40] provides a solution to this, using a dense sequence retriever to
directly augment input proteins with corresponding evolutionary information
from the Pfam database. This allows RSA to incorporate similar information as
traditional MSA methods, without the computationally costly MSA alignment
process. OntoProtein [38] incorporates direct knowledge injection by
pretraining on gene ontology (GO) knowledge graphs (KG). OntoProtein uses the
ProtBERT architecture for encoding protein sequences but introduces a second
encoder for training on the GO input. To perform this hybrid encoding, Zhang
et al. [38] create a new dataset, ProteinKG25, which aligns protein sequences
with their respective GO annotations in text format. Knowledge-exploited Auto-
encoder for Protein (KeAP) [39] extends upon this by exploring protein and
annotated text on a more granular level; while OntoProtein models
relationships over proteins and texts as a whole, KeAP uses a cross-attention
mechanism to perform token-level exploration on individual amino acids and
words.

Now, we discuss a few structure-based learning models. IEConv [171]
incorporates both intrinsic distance (based on connectivity in a node-graph
representation) and extrinsic distance (based on physical 3D distance)
information in a convolutional neural network (CNN), using contrastive
learning to pre-train. Additionally, IEConv uses protein-specific pooling
methods, like reducing each amino acid to its alpha carbon, to reduce
computational complexity and allow for more learned features per atom under
memory constraints. DeepFRI [170] combines the idea of sequence representation
learning with structural representation learning, incorporating both a pre-
trained LSTM to learn sequence data and a GCN to learn from contact map input
(matrix representing 3D distances between residues). GearNET [168] applies a
GNN to a graph of residues, with three types of directed edges defined for
sequential/physical distance and K-nearest neighbors. While previous models
had only considered message-passing between residues, GearNET applied direct
message-passing between edges in its variant model GearNET-Edge, which led to
even further improved performance on various metrics. In addition, GearNET-
IEConv added an intrinsic-extrinsic convolutional layer inspired by Hermosilla
et al. [171] to improve performance, with GearNET-Edge-IEConv combining both
of these features.

###  8.1 Antibody

####  8.1.1 Antibody Task Background

Antibodies are Y-shaped proteins used by the immune system to identify and
neutralize foreign objects like bacteria and viruses. Each antibody contains
two identical sets of chains, with each set containing one heavy chain and one
light chain. The variable domains of these chains control binding specificity
for various antigens, containing the binding surface, known as the paratope.
The paratope binds with specificity to a target antigen at its respective
binding region, known as the epitope. The paratope is typically located within
six Complementarity-Determining Regions (CDRs) on the antibody: three from the
light chain (CDR-L1, CDR-L2, CDR-L3) and three from the heavy chain (CDR-H1,
CDR-H2, CDR-H3). The Complementary-Determining Region 3 (CDR-H3) is the most
diverse in terms of amino acid sequence and length— due to its lack of
constraints, it is often the most complex to produce, making its generation
the primary focus for modern generative models.

The end goal of machine learning is to generate antibodies in silico. In
approaching this goal, the typical pipeline often includes an amino acid
sequence input, sequence representation learning, structure prediction,
paratope-epitope prediction, antibody-antigen docking, CDR generation, and
evaluation, typically in the form of affinity prediction. The final output is
a generated antibody, either in the form of a 1D sequence, 3D structure, or
both. Sometimes, a 3D structure is input directly, with the sequence being
predicted from the structure. Machine learning methods have been applied to
various fragments of the process, with some more recent models [172], [159],
[158] handling both sequence and structure generation simultaneously, and
others [159] even addressing the entire end-to-end process. Other subtasks
like drug-applicable attribute prediction and affinity prediction have been
explored using predictive AI models, but these tasks are not generative by
nature and not be discussed in detail.

####  8.1.2 Antibody Representation Learning

##### Overview

This task is identical to the protein sequence representation learning, but
language models in this field pretrain on antibody-specific data, in
particular, outperforming general protein representation models on antibody-
based tasks.

Table 9: An overview of relevant antibody representation learning models. Due to the lack of standardization in benchmarking metrics, we note each model’s benchmarking findings. Model | Type of Model | Dataset | AAR (%, $\uparrow$) | Additional Self-Benchmarking Notes
---|---|---|---|---
BERTTransformer [173] | BERT | OAS | \ | Improved binding affinity prediction over generic CNN
AntiBERTy [120] | BERT | OAS |  26.00 [121] | Reveals evolutionary trajectories for affinity maturation
AbLang [174] | RoBERTa | OAS |  33.60 [121] | Clustering better represents distinction between B-cell type
|  |  |  | and other metrics (compared to ESM-1b)
PARA [121] | DeBERTa | OAS |  34.20 [121] | Improved heavy/light chain matching (compared to AntiBERTy)
AntiBERTa [122] | RoBERTa | OAS | \ | Clustering better represents distinction between B-cell type
|  |  |  | (compared to ProtBERT), improves paratope prediction

##### Datasets

All reported methods pre-train on the OAS dataset. In all methods, OAS was
divided between heavy and light chains, with some [121, 174] also using
Linclust to cluster the data. Leem et. al also includes pre-training on the
general Pfam database for protein sequences, as a comparison to pre-training
on the antibody-specific OAS dataset.

  * •

OAS [175] \- Observed Antibody Space, a compilation of over one billion raw
antibody sequences.

##### Tasks

Given the variety of applications for representation learning, the benchmarks
used for this task vary widely; some use Pearson correlation to measure
affinity binding prediction [122], while others [121] use amino acid recovery
(AAR) to measure accuracy in a CDR-H3 sequence prediction task. Some use a
masked language modeling task (MLM) to measure the quality of BCR sequence
representation [122], while others measure accuracy in paratope prediction
[122, 120]. However, we do not have a standardized benchmarking system as we
did with protein representation learning, so we only report CDR-H3 sequence
prediction in Table 9, which will be elaborated on more in the Antibody CDR
Generation section (see page 8.1.4). Other evaluations are left as individual
notes for each model.

##### Metrics

As noted above, we only report results on the CDR-H3 prediction task, which is
measured with amino acid recovery (AAR), which represents the percentage of
matching amino acids between ground truth and generated amino acid sequences.

##### Models

Li et al. [173] created the BERTTransformer model, serving as the first
representation learning model aimed specifically at antibody learning. Li et
al. find that pre-training on Pfam results in improved performance on affinity
binding prediction when compared to pre-training on OAS, suggesting some merit
to general protein datasets in pre-training for antibody representation
learning. AntiBERTy [120] is a BERT-based model, and produces valuable
embeddings used in structure prediction by IgFold [134]. AntiBERTa [122] and
AbLang [174] both use the more optimized RoBERTa architecture, while the most
recent model PARA [121] uses the current state-of-the-art DeBERTa
architecture, and outperforms previous pre-trained models in AAR.

A comprehensive overview of representation learning models can be seen in
Table 9. As representation learning serves as a general tool, rather than the
optimization for a specific task, these methods all use a variety of self-
chosen applications to measure performance. PARA tests amino acid recovery
against AbLang and ‘ on the CDR-H3 prediction task, reporting improved
accuracy. AntiBERTy and AntiBERTa report improved paratope prediction, while
AbLang and AntiBERTa focus on improved clustering to distinguish between naive
and memory B-Cells. AntiBERTa also explores clustering quality in terms of
drug-applicable traits, such as origin, closest human V gene identity, and
anti-drug antibody (ADA) response scores.

####  8.1.3 Antibody Structure Prediction

##### Overview

Antibody structure prediction extends general protein structure prediction
methods, with the key difference lying in the reliance on multiple sequence
alignment (MSA) of homologous proteins mapping evolutionary relationships
between genetically related sequences. Because the relevant evolutionary
histories for CDR-H3 loop sequences are lacking, MSAs in antibodies are not
always available. This makes models like AlphaFold [130], designed for general
protein structure prediction, highly inefficient and slow. Thus, antibody-
specific structural prediction methods seek to predict antibody structure
without the need for an input MSA [134].

##### Datasets

Models in this field all train on the Structural Antibody Database (SAbDab),
with the Rosetta Antibody Benchmark (RAB), used as a validation set in some
cases [176, 177]. Note that “canonical conformations” do not exist for the
highly variable H3 region due to high variability, which remains a challenge
to structurally model.

  * •

SAbDab [178] - Structural Antibody Database, a collection of all antibody
structures in PDB [95]. Structures are annotated with antibody-specific
structural data like canonical conformations for complementarity determining
regions (CDR), orientation between the variable domains on the light and heavy
chains, and the presence of constant domains in the structure.

  * •

RAB [179] - Rosetta Antibody Benchmark, a hand-selected set of 60 antibody-
antigen complexes, chosen to be as diverse in CDR lengths and clusters as
possible.

##### Task

Given an antibody amino acid sequence (organized into heavy chain, light
chain, and linker sequences), generate a set of 3D point coordinates for each
amino acid residue.

##### Metrics

For benchmarking, models compare similarities between the predicted and ground
truth structures through the following metrics:

  * •

RMSD \- Root-mean-square deviation, measures distances between ground truth
and generated residue coordinates. Also used in protein structure prediction
(see page 4.2.3)

  * •

OCD [180] \- Orientational coordinate distance, measures similarity between
light-heavy orientational coordinates (LHOC) of ground truth and generated
residues. LHOC is defined by Marze et al. [180], consisting of four metrics to
measure orientation between light and heavy chains.

used root-mean-square deviation (RMSD) to measure the discrepancy between
predicted and ground truth 3D structures, while some models [134], [181],
[135] also used orientational coordinate distance (OCD) to measure the
accuracy of predicted relative position between heavy and light chains in some
cases.

##### Models

Model | Type of Model | Dataset |  H3 RMSD (Å, $\downarrow$) | OCD ($\downarrow$) |  Generation Time (s, $\downarrow)$) |
---|---|---|---|---|---|---
IgFold** [134] | BERT, Graph | OAS, SAbDab | 2.99 |  3.77 [134] |  0.46 [134] |
DeepAB [181] | LSTM, NN | OAS, SAbDab | 3.28 |  3.6 [134] |  1620.22 [135], 600 [134] |
ABLooper [176] | EGNN | SAbDab, RAB | 3.2 |  4.53 [134] |  30-60 [134] |
ImmuneBuilder [177] | AlphaFold - Multimer | SAbDab, RAB | 2.81 |  4.9 [134] |  5* |
tFold-Ab** [135] | AlphaFold - Multimer | SAbDab | 2.74 |  3.21 [134] |  2.23* [135] |
xTrimoABFold [182] | AlphaFold - Multimer | PDB | 1.25 | \ | \ |
Table 10: An overview of the most relevant antibody structure prediction
models. All benchmarking metrics are self-reported unless otherwise noted.
Igfold and tFold-Ab use slightly different benchmarking methods, so their
results may not be comparable. [*] denotes significant discrepancies which
result from varying benchmarking methods. [**] denotes the current SOTA.

tFold-Ab [135], xTrimoABFold [182], and ABodyBuilder2 [177] all apply similar
methods to AlphaFold, replacing the unnecessary MSA searching component.
AbLooper [176] uses five EGNNS to each individually predict CDR structures,
outputting the positional average of the five structures— this allows for the
evaluation of confidence as the deviation between these predicted structures.
Some models do not predict structures directly: DeepH3 [183] uses a deep
residual neural network to first predict structural restraints, using Rosetta
to then output atomic structures. SimpleDH3 [184] adds ELMo embeddings to
DeepH3, and DeepAb [181], the most recent version, adds interpretable
attention layers to enhance output from the neural network.

Igfold [134] takes advantage of AntiBERTy’s [120] sequence embeddings to
predict antibody structures using invariant point attention. While some other
models, such as tFold-Ab [135], xTrimoABFold [182], and ABodyBuilder2 [177],
demonstrate more accurate structures, IgFold produces structures at a superior
speed, making it the established state-of-the-art model. Igfold’s downside is
its reliance on PyRosetta to produce side-chain conformations, as the model
Igfold itself only outputs backbone atoms. This adds significant time to the
full process, as it takes around 0.46 seconds for Igfold to produce backbone
atoms, but an additional 21.86 seconds, on average, for PyRosetta to convert
these into full-atom structures [135]. By contrast, tFold-Ab [135] generates
full-atom structures in less time than the Igfold + PyRosetta pipeline (2.23
seconds compared to 22.32 seconds).

Notably, in the training process, ImmuneBuilder and xTrimoABFold choose to
include antibodies with identical sequences to expose models to antibodies
with multiple conformations (identical sequences with different structures).
On the other hand, DeepAB includes an entirely non-redundant set of sequences,
seeking to expose its model only to unique items.

Table 10 presents detailed results for the performance of the discussed
models. While IgFold does not have the absolute highest performance in terms
of RMSD and OCD when compared to other methods, its speed makes it the state-
of-the-art model. This speed advantage is contested only by tFold-Ab, which
generates full-atom structures in less time than IgFold, as it circumvents
IgFold’s reliance on computationally expensive Rosetta energy functions for
side-chain prediction. Note that discrepancies in time results may result from
varying benchmarking methods— time results from tFold-ab were reported on
their own SAbDab-22H1-Ab/Nano dataset, while time results from IgFold were
reported using their IgFold-Ab benchmark.

####  8.1.4 Antibody CDR Generation

Figure 5: A comprehensive overview of the antibody generation pipeline for
CDR-H3 design. The inputs are a target antigen and antibody information
(without CDR-H3), and the output is an antibody-antigen complex with a
designed CDR-H3 sequence. Note that while most antibody CDR-H3 generation
methods only generate the CDR-H3 region, needing a docked structure as input,
some methods like DockGPT [185], HERN [186], and dyMEAN [159] perform multiple
steps of the pipeline on their own. Images taken from [20, 23, 187, 188, 189].

##### Overview

CDR generation involves the generation of the variable regions of the
antibody, directly determining binding affinity and making it the core of
antibody generation. As mentioned above, the CDR-H3 region is the most
variable, making it the most difficult region for models to generate and thus
the primary region of focus.

##### Datasets

The primary datasets used in recent models are listed below. Note that SAbDab
is the main dataset typically used for training, whereas RAbD and SKEMPI are
specialized datasets used for testing specific downstream tasks.

  * •

SAbDab [178] - The Structural Antibody Database, an annotated dataset of all
antibody structures contained in PDB [95], used in structure prediction (see
page 8.1.3)

  * •

RAB [179] - Rosetta Antibody Benchmark, a hand-selected group of 60 diverse
antibody-antigen complexes, also used in antibody structure prediction (see
page 8.1.3).

  * •

SKEMPI [190] - Structural Database of Kinetics and Energetics of Mutant
Protein Interactions, a dataset of energy changes for mutations of protein-
protein interactions.

##### Tasks

Three general tasks are used to benchmark models in this field:

  * •

Sequence and Structure Modeling - Given an antibody structure, predict its
corresponding sequence (or vice-versa). AAR is used to evaluate generated
sequences, while RMSD is used to evaluate generated structures. Testing for
this task is done on the SAbDab dataset.

  * •

CDR-H3 Generation - Given a target antigen, generate the binding antibody
CDR-H3 region. AAR is used to evaluate sequences, while RMSD and TM-Score are
used to evaluate structures. Testing is done on the RAbD dataset.

  * •

Affinity Optimization - Given an antibody-antigen pair, make changes to the
CDR region to optimize its binding affinity with the target antigen.
Performance is measured by $\Delta\Delta G$, and testing is done on SKEMPI
v2.0.

##### Metrics

The following metrics are used to evaluate CDR generation models:

  * •

AAR - Amino acid recovery, a comparison between ground truth and generated
amino acid sequences, also measured in antibody representation learning (see
8.1.2).

  * •

RMSD \- Root-mean-square deviation, measures distances between ground truth
and generated residue coordinates. Also used in protein structure prediction
(see page 4.2.3)

  * •

TM-score [128] \- Template Modeling Score, a distance measurement capturing
both local and global similarities, also used in protein structure prediction
(see page 4.2.3).

  * •

$\mathbf{\Delta\Delta G}$ \- Change in binding energy after affinity
optimization. Binding energy is predicted by the geometric network proposed by
Shan et al. [191].

##### Models

While some approaches to the CDR generation problem have been sequence-based,
such as the LSTM used by Akbar et al. [157], the field has seen movement
towards structure-based design and even more recently, sequence-structure co-
design, as this allows models to incorporate both 1D amino acid sequence
information and 3D structure information in the generation process. RefineGNN
[158] was the first to introduce structure/sequence co-design, representing
both structural and sequential information in a graph and performing iterative
refinement. A Multi-channel Equivariant Attention Network (MEAN) [172]
improves efficiency by predicting all amino acids within a CDR region at once,
allowing for fewer iterations than RefineGNN’s prediction for each amino acid
residue. AntiDesigner [192] implements a Protein Complex Invariant Embedding
(PCIE) and dual Multi-Layer Perceptrons (MLPs) to design structures and
sequences in a one-shot manner, removing the need for iterative refinement
entirely. DiffAb [193] defines diffusion processes for both amino acid
sequences and 3D backbone coordinates, serving as one of the first DDPMs in
the field of antibody generation.

Table 11: A summary of relevant CDR generation models. All benchmarking metrics are self-reported unless otherwise noted. dyMEAN and DiffAB use slightly different evaluation conditions, so their results may not be fully comparable. [**] denotes the current SOTA. Model | Type of Model | Dataset | Modeling | CDR-H3 Design | Affinity
---|---|---|---|---|---
AAR (%, $\uparrow$) |  RMSD ($\downarrow$) |  AAR (%, $\uparrow$) |  TM-Score ($\uparrow$) |  RMSD ($\downarrow$) | $\mathbf{\Delta\Delta G}(\downarrow)$
LSTM [157] | LSTM | Self-made |  15.69 [172] | \ |  22.36 [172] | \ | \ |  -1.48 [172]
RefineGNN [158] | GNN | SAbDab |  21.13 [172] |  6 [172] |  29.79 [172] |  0.8308 [172] |  7.55 [172] |  -3.98 [172]
MEAN [172] | EGNN | SAbDab | 36.38 | 2.21 | 36.77 | 0.9812 | 1.81 | -5.33
AntiDesigner** [192] | MLP | SAbDab | 37.37 | 1.97 | 40.94 | 0.985 | 1.55 | -10.78
DiffAB [193] | Diffusion | SAbDab | 26.78 | 3.597 |  35.31 [159] |  0.9695 [159] | \ |  -2.17 [159]
DockGPT [185] | Transformer | SAbDab | \ | \ | \ | \ | 1.88 | \
HERN [186] | EGNN | SAbDab | \ | \ | 34.1 | \ | \ | \
dyMEAN** [159] | EGNN | SAbDab | \ | \ | 43.65 | 0.9726 | \ | -7.31

While the above models all show significant progress in the specific CDR
generation task, these models still require the input of fully docked
antibody-antigen structures; thus, in real-world applications where only
sequence and/or structure data is available, these models must be paired with
structure prediction and docking methods to operate. As a result, some methods
simultaneously implement other parts of the antibody generation process in
their model. The deep learning method DockGPT [185] includes an adaptation of
their protein-protein docking method for antibody CDR-H3 docking and design.
By adding an encoding feature to distinguish CDR residues during training and
providing structures with missing CDR regions as input, DockGPT simultaneously
docks and designs all CDR regions. HERN [186] uses hierarchical equivariant
refinement to both generate and dock paratopes for target antigens. However,
HERN requires a defined epitope region as input and needs to be combined with
epitope prediction models to dock with antigens whose epitopes are not
identified. Dynamic Multichannel Equivariant Graph Network (dyMEAN) [159]
extends these ideas even further by creating the first end-to-end method,
incorporating structure prediction, docking, and CDR generation in a singular
model. To demonstrate its improved performance, dyMEAN benchmarks against
pipelines combining models for each subtask (Igfold $\Rightarrow$ HDock
$\Rightarrow$ MEAN to perform structure prediction, docking, and CDR
generation, for example).