# Deep Generative Modeling for Protein Design

Alexey Strokach  Philip M. Kim<EMAIL_ADDRESS>
###### Abstract

Deep learning approaches have produced substantial breakthroughs in fields
such as image classification and natural language processing and are making
rapid inroads in the area of protein design. Many generative models of
proteins have been developed that encompass all known protein sequences, model
specific protein families, or extrapolate the dynamics of individual proteins.
Those generative models can learn protein representations that are often more
informative of protein structure and function than hand-engineered features.
Furthermore, they can be used to quickly propose millions of novel proteins
that resemble the native counterparts in terms of expression level, stability,
or other attributes. The protein design process can further be guided by
discriminative oracles to select candidates with the highest probability of
having the desired properties. In this review, we discuss five classes of
generative models that have been most successful at modeling proteins and
provide a framework for model guided protein design.

###### keywords:

artificial intelligence , machine learning , representation learning , neural
networks , protein optimization , protein design

††journal: Current Opinion in Structural Biology

[label1] organization=Department of Computer Science, University of Toronto,
addressline=40 St. George Street, city=Toronto, postcode=M5S 2E4,
state=Ontario, country=Canada

[label2] organization=Donnelly Centre for Cellular and Biomolecular Research,
University of Toronto, addressline=160 College Street, city=Toronto,
postcode=M5S 3E1, state=Ontario, country=Canada

[label3] organization=Department of Molecular Genetics, University of Toronto,
addressline=1 King’s College Circle, city=Toronto, postcode=M5S 1A8,
state=Ontario, country=Canada

Machine learning is becoming a key component of the protein design process

Deep generative models can produce novel protein sequences and structures

Conditioned generative models can produce proteins with specific properties

Discriminative oracles can be used to further fine-tune the design process

##  1 Introduction

Figure 1: Overview of different machine learning approaches for protein
design. (a) Generative models trained on protein sequences or structures can
learn probability density landscapes defined by the training data and can be
used to generate new proteins that are expressed, are stable and, optionally,
have a specific structure and function. These models can loosely be subdivided
into self-supervised models, which use supervised learning to learn one aspect
of the data from another, and latent variable models, which learn a mapping
from latent variables with defined structure to the data. (b) Discriminative
models learn to predict specific properties of a protein from its sequence or
structure. In some cases, especially when limited training data are available,
discriminative models can show a substantial boost in performance when they
leverage protein representations learned by the generative models. (c) Model
guided protein design involves a generative model, which proposes new protein
sequences or structures, and one or more discriminative oracles, which assign
a score to the proposed proteins based on their predicted ability to meet
specific objectives.

The optimization of existing proteins, the generation of new proteins with
specific shapes and functions, as well as other aspects of protein design,
remain key challenges in structural biology. Traditionally, computational
protein design has been carried out using tools which use different sampling
techniques to explore the energy landscape defined by molecular mechanics
force fields or semi-empirical energy functions [1]. Recently, the use of
machine learning and artificial intelligence has led to breakthroughs in a
number of areas [2, 3, 4], including the accurate prediction of protein
structures using AlphaFold2 [5, 4]. Given the continued growth in the number
of available protein sequences and structural information, machine learning
approaches are also posed to become indispensable for efficient and successful
protein design.

The goal of this review is to formalize the different machine learning
algorithms used for protein design and to delineate a framework for generating
novel proteins meeting specific objectives (Figure 1). We first describe five
classes of generative models that can be used to produce new protein sequences
and structures or learn meaningful representations thereof (Figure 1a). Where
appropriate, we also describe supervised models that achieved superior
performance because they leveraged representations learned by the trained
generative models (Figure 1b). Supervised models, however, are not the focus
of this review, as they have been described in detail elsewhere [2] and their
utility is well-established. Finally, we describe strategies that have been
used to combine generative models with supervised models, simulations, and
domain expertise, to produce the desired proteins (Figure 1c).

##  2 Generative models of protein sequences and structures

Figure 2: Overview of the five generative model architectures that are covered
in this review. (a) Autoregressive models learn to predict the identities of
tokens (e.g. amino acids) making up a protein from the identities of preceding
or surrounding tokens. (b) Variational autoencoders (VAEs) comprise an encoder
trained to parameterize the distribution over the latent variables $\bm{z}$
and a decoder trained to reconstruct the inputs using samples from the
distribution defined by the encoder. (c) Normalizing flows (NFs) use a
bijective model to map inputs to and from a latent representation. The model
parameters are optimized such that the probability of the training data in the
latent space is high while the amount of “warp” required to map the data back
to the input space is low. (d) Energy-based models (EBMs) learn an energy
function that assigns low energies to probable states, including the training
data, and high energies to improbable states, often generated by perturbing
the training data. Noise-contrastive estimation (NCE) is a training strategy
for EBMs where fake examples are sampled from a predefined distribution, and
the model is trained to distinguish between real and fake examples. (e)
Generative adversarial networks (GANs) comprise a generator trained to produce
examples which appear real to the discriminator and a discriminator trained to
distinguish between real and generated examples. Table 1: Advantages and
disadvantages of different generative models used for protein design.

| Advantages | Disadvantages
---|---|---
Autoregressive models |  • Native support for modeling categorical variables*. • Native support for modeling data of varying dimensionality. • Gives the exact log-likelihood of the data. • Training is relatively stable. • Can be applied to graph-structured data. |  • Limited support for modeling continuous variables* (requires discretization, etc.). • No support for domains that cannot be modeled as sequences or graphs. • Inference is performed one token at a time, making it slow.
VAEs |  • Native support for modeling categorical and continuous variables*. • Gives the lower bound (ELBO) of the log-likelihood of the data. |  • Generated examples tend to be more blurry than with GANs. • Limited support for modeling data of varying dimensionality.
Normalizing flows |  • Native support for modeling continuous variables*. • Gives the exact log-likelihood of the data. |  • Little to no support for modeling categorical variables*. • Deeper models, with more parameters, are needed to achieve performance comparable to VAEs / GANs. • The underlying model needs to be bijective. • Calculating the trace of the Jacobian matrix can be computationally expensive.
Energy-based models |  • Native support for modeling categorical and continuous variables*. • Flexible in terms of the types of data that can be modeled. |  • Obtaining negative examples can be difficult and computationally expensive. • The model can be biased by the sampling strategy used to generate negative examples. • Calculating probabilities requires evaluating every possible alternative, which can be exceedingly slow or intractable. • Generation is performed through sampling (e.g. MCMC).
GANs |  • Native support for modeling continuous variables*. • Can generate the most realistic examples (at least in the case of images). |  • Special care is needed to model categorical variables* (e.g. using the straight-through Gumbel max trick [6, 7] to back-propagate through outputs). • Training can be less stable than with other models. • Generated examples can be of low diversity (mode collapse). • No native support for mapping existing data to latent space or calculating log-likelihoods. • Limited support for modeling data of varying dimensionality.

  * •

*In the context of protein design, DNA and amino acid sequences are usually represented as categorical variables, while protein structures are usually represented as continuous variables. This is an important factor that needs to be considered when evaluating models for a particular application.

Deep generative models have gained wide adoption in recent years due to their
ability to learn from massive unlabeled datasets, produce meaningful
representations and density estimates of the data and generate new examples of
striking coherence. Many generative models have been developed [8] (Figure 2),
with various trade-offs and limitations that make them well-suited to
different aspects of protein design (Table 1). Here we provide an overview of
five classes of deep generative models, including their notable applications
to protein design. A more detailed description of each model class can be
found in A.

###  2.1 Autoregressive models

Autoregressive models are trained to predict the next token given the previous
tokens or to predict masked tokens given unmasked tokens (Figure 2a). These
models have received much attention in recent years due to their success in
natural language processing (NLP). Similar to language, protein sequences are
readily represented as a succession of tokens, which has allowed state-of-the-
art NLP models to be applied to protein sequences with few modifications.

The first autoregressive models to be applied to protein sequences used
recurrent neural networks (RNNs) with long short-term memory (LSTM) layers [9,
10] or dilated convolutions [11] to predict the identity of an amino acid
given the preceding amino acids. More recently, the transformer architecture
[12] has gained popularity [13, 14, 15, 16], as it generally produces higher
reconstruction accuracies and better performance on downstream tasks,
including remote homology detection, secondary structure prediction, contact
prediction [15, 17, 18], and mutation effect prediction [19]. The power of
transformers comes primarily from their use of multi-head attention, which
allows every element to have direct access to the information stored in every
other element in the sequence. Furthermore, transformer models are more
efficient to train than RNNs, as they process entire sequences in parallel
rather than one element at a time.

Graph neural networks can be viewed as an extension to the transformer
architecture [20, 21], allowing the use of “edge” attributes encoding the
relationships between pairs of tokens, and making it possible to limit the
neighbors with which the information is shared in multi-head attention.
Ingraham et al. [22] developed Structured Transformer, a graph neural network
with an encoder-decoder architecture, where the encoder takes as input the
protein structure, defined by the backbone torsion angles and the distances
and relative translations and rotations between pairs of residues, and the
decoder generates the amino acid sequences with self-attention to the
preceding residues and attention to the embeddings generated by the encoder.
Structured Transformer assigns native amino acids higher probabilities than
sequence-only autoregressive models, and it is able to recover correct amino
acids in NMR protein structures with higher accuracy than Rosetta [23].
Strokach et al. [24] developed ProteinSolver, a graph neural network where the
input node and edge attributes define the identities and the distances between
pairs of amino acids, respectively, and the network is trained to reconstruct
the identities of masked amino acids. ProteinSolver generates novel sequences
that fold into stable proteins with the desired topologies, as confirmed by an
array of computational validation techniques and the circular dichroism
spectra of expressed and purified proteins, and it is better able to predict
changes in protein stability and affinity than transformers that do not
leverage structural information [25].

The transformer architecture has also been extended in other ways. Rao et al.
[16] used axial attention to leverage the information present in multiple
sequence alignments, improving the reconstruction accuracies and achieving
better performance on contact prediction, secondary structure prediction, and
mutation effect prediction [16, 19]. Madani et al. [26, 27] used an additional
input token to encode the function of the protein and trained a conditional
transformer to generate novel protein sequences with predetermined functions.
The authors validated their model by generating novel lysozymes and showed
experimentally that the generated proteins have lysozyme activity and fold
into structures that are characteristic of existing lysozymes. Finally, fine-
tuning a pre-trained model using sequences of proteins with the target
function or topology can be a simple way of improving model performance on
tasks that are specific to those proteins [9, 19].

###  2.2 Variational autoencoders (VAEs)

Variational autoencoders (VAEs) use an encoder network to map the inputs to a
low-dimensional latent space and a decoder network to reconstruct the inputs
using a sample from that latent space (Figure 2b). VAEs are trained to
minimize the distance between the original and the reconstructed inputs while
constraining the latent space to approximate a standard Gaussian to improve
generalizability.

VAEs were some of the first unsupervised methods for mutation effect
prediction [28, 29] and have been used to generate novel protein sequences
with predetermined functions. Greener et al. [30] trained a conditional VAE,
which incorporated a rough topology of the protein as an additional input, on
$\sim$4,000 short monomeric structures in the PDB, as well as their homologs
from UniRef, and showed that the resulting model can generate new protein
sequences that correspond to a specified topology. Hawkins-Hooker et al. [31]
trained a VAE on $\sim$70,000 luciferase sequences and showed that the
proteins generated by the model are also often luminescent. Das et al. [32]
trained a VAE on peptide sequences from UniProt and, using controlled
generation and screening, they were able to produce novel peptides with
antimicrobial activity.

VAEs have also been used to generate backbones of proteins with predetermined
topologies. Eguchi et al. [33] trained a VAE to take as input a distance
matrix and to generate 3D coordinates matching both the input distance matrix
and the torsion angles of the corresponding protein structure. After training
the VAE on $\sim$11,000 structures of immunoglobulins, the resulting model was
able to generate novel immunoglobulin backbones matching the expected bond
lengths, bond angles, and torsion angles and to learn a meaningful latent
representation that could be explored to find backbones with desired shapes
and characteristics.

###  2.3 Normalizing flows (NFs)

Normalizing flows (NFs) use an invertible neural network to learn a
bidirectional mapping between the inputs and the latent representation (Figure
2c). The use of an invertible network makes it possible to calculate the exact
probability of the training data given the model parameters and to optimize
the model parameters accordingly, although it also imposes substantial
constraints on the types of neural network architectures that can be used.

The most notable application of normalizing flows to protein design has been
the modeling of protein dynamics [34]. Noé et al. [34] introduced Boltzmann
generators: neural networks which learn a mapping between configurations of a
many-body system and a latent representation. The authors showed that, after
training a Boltzmann generator using a set of protein conformations and
energies predicted by a molecular mechanics force-field, it is possible to
generate new conformations that can be confirmed by molecular dynamics
simulations, and to accurately model transitions and energy differences
between known states.

###  2.4 Energy-based models (EBMs)

Energy-based models (EBMs) are a large class of models that, in lieu of
learning a probability density function over the input space, are simply
trained to assign low values (or “energies”) to observed states and high
values to unobserved or improbable states (Figure 2d). Training EBMs requires
a strategy for efficiently sampling a representative set of improbable states,
and different strategies are often employed for different applications.

EBMs have been used extensively for learning meaningful representations of
protein sequences [35, 36, 37] and structures [38]. Gainza et al. [38]
introduced MaSIF, a model trained to map protein surface meshes into compact
representations called “fingerprints” such that complementary surfaces of
known binders have complementary fingerprints (i.e., have low Euclidean
distance when one of the two fingerprints is negated). The resulting
fingerprints can be used to perform protein-protein interaction prediction and
protein docking significantly faster than traditional approaches while
achieving comparable accuracy. This approach can further be extended by
incorporating the feature generation step into the model architecture,
allowing the model to be trained end-to-end [39].

EBMs have also been used for fixed backbone design. Du et al. [40] introduced
Atom Transformer, a model trained to predict whether an amino acid rotamer
matches the context defined by the identity and position of $k$ nearest atoms.
The model is trained to assign low energies when contexts are paired with
native rotamers and high energies when contexts are paired with non-native
rotamers. The non-native rotamers are selected at random from a rotamer
library, after conditioning on the backbone torsion angles and the amino acid
types. The resulting model achieves comparable accuracy to Rosetta [23] in
recovering native rotamers, and it supports continuous rotamer
representations, which would not be possible if rotamer placement was framed
as a classification problem [41]. However, in order to assign a rotamer to a
given context, all possible rotamers have to be evaluated, which makes
inference relatively slow.

###  2.5 Generative adversarial networks (GANs)

Generative adversarial networks (GANs) are a subset of EBMs where a generator
network is trained to propose challenging negative examples and a
discriminator network is trained to distinguish between the real and the
generated examples (Figure 2e). The concomitant training of the generator
network allows GANs to be efficient at generating new examples, in contrast to
many other EBMs where the generation of new examples requires extensive
sampling.

GANs have been used to generate [42] and refine [43] distance matrices and to
generate novel protein sequences with specific folds [44] and functions [45].
Anand and Huang [42] trained a GAN model, employing 2D convolution, pooling,
and upsampling layers, to generate distance matrices corresponding to novel
protein folds. Protein backbones could be reconstructed from the distance
matrices either by solving a convex optimization objective [42] or by using a
model trained to map distance matrices to coordinates [46]. Repecka et al.
[45] trained a GAN model, employing convolution and attention layers, on a
dataset of malate dehydrogenase (MDH) sequences. Sequences generated by the
resulting model were validated experimentally and possessed enzymatic activity
in $\sim$24% of cases.

##  3 Model guided protein design

In model guided protein design [47], a pretrained deep generative model,
preferably conditioned on the structure [22, 24] or function [26, 9] of the
target protein, is used to generate the initial pool of candidates.
Discriminative oracles are then used to independently validate the generated
candidates [43, 48], to prioritize them for experimental validation [10, 32],
or to guide the generator to produce sequences or structures that are more
desirable [34, 49, 50, 51, 52]. Ultimately, the generative model, which can be
trained on vast amounts of unlabeled data, increases the probability that the
candidates correspond to valid sequences or structures, while the
discriminative oracles, which can include molecular mechanics simulations or
models trained on domain-specific datasets, increase the probability that the
candidates have the desired functionality.

##  4 Conclusions

In this review, we described a number of protein design scenarios where deep
generative models successfully produced novel proteins [22, 41, 15] often
orders of magnitude faster than traditional approaches [24, 32]. Continued
growth in the number of protein sequences and structures that are available
[53, 54], coupled with the development of protein-specific machine learning
libraries [55, 56], and network architectures [16, 20, 57, 58], are likely to
result in further improvements in the future.

##  5 Acknowledgements

AS acknowledges support from an NSERC PGS-D graduate scholarship. PMK
acknowledges support from an NSERC Discovery grant (RGPIN-2017-064) and a CIHR
Project grant (PJT-166008).

##  6 Conflicts of interest

PMK is a cofounder of Resolute Bio Inc. and serves on the scientific advisory
board of ProteinQure.

##  Appendix A Model descriptions

###  A.1 Autoregressive models

Autoregressive models operate on sequences of tokens and are typically trained
either using the causal language modeling objective function or using the
masked language modeling objective function (Figure 2a).

In causal language modelling (CLM), the goal is to predict the identity of
each amino acid given the preceding amino acids in the input sequence
(Equation 1). Models trained using the CLM objective are particularly well-
suited for generating novel protein sequences, since this task closely
resembles the objective function used to train the models.

$\mathcal{L}_{CLM}=\mathbb{E}_{\bm{x}\sim p_{data}(\bm{x})}\left[\log
p_{\theta}(\bm{x}_{0})+\sum_{i=1}^{N-1}p_{\theta}(\bm{x}_{i}|\bm{x}_{0},\dotsc,\bm{x}_{i-1})\right]\\\
$ (1)

In masked language modelling (MLM), the goal is to predict the identity for a
fraction of randomly selected and masked amino acids in the input sequence
(Equation 2). Models trained used the MLM objective are bidirectional and
therefore are particularly well-suited for optimizing specific regions in a
protein and for providing representations for each residue which capture
information about both the preceding and the succeeding regions in the
sequence. These models have also been used to generate entire protein
sequences using sampling, beam search, or other strategies.

$\mathcal{L}_{MLM}=\mathbb{E}_{\bm{x}\sim
p_{data}(\bm{x})}\left[\mathbb{E}_{M}\sum_{i}^{M}\log
p_{\theta}(\bm{x}_{i}|\bm{x}_{\notin M})\right]$ (2)

###  A.2 Variational autoencoders (VAEs)

Traditional autoencoders comprise an encoder network
$q_{\phi}(\bm{z}|\bm{x})$, which maps an input $\bm{x}$ to a latent
representation $\bm{z}$, and a decoder network $p_{\theta}(\bm{x}|\bm{z})$,
which maps a latent representation $\bm{z}$ to a reconstructed input
$\bm{\hat{x}}$. Variational autoencoders (VAEs) [59] are similar, but instead
of predicting the latent variables $\bm{z}$, VAEs predict parameters of a
distribution over the latent variables, or $\bm{\mu}$ and $\bm{\sigma}$ in the
case where latent variables are modeled as independent Gaussians. The decoder
then takes a sample from the predicted distribution, using a
reparameterization trick to make the sampling process differentiable [59], and
maps that sample to the output $\bm{\hat{x}}$ (Figure 2b).

VAEs cannot be trained by minimizing the negative marginal probability
$p_{\theta}(\bm{x})$ directly because calculating the marginal probability
requires taking an integral over the latent space (Equation 3), which is
intractable in most cases.

$p_{\theta}(\bm{x})=\int p_{\theta}(\bm{x}|\bm{z})p(\bm{z})dz$ (3)

Instead, VAEs are trained by minimizing the negative evidence lower bound
(ELBO) of the data given the model parameters (Equation 4). The first term is
the reconstruction loss between the input and the output (e.g. cross-entropy
loss in the case of categorical data or mean squared error in the case of
continuous data). The second term is the Kullback–Leibler distance between the
predicted parameters of the latent distribution and the prior over the latent
distribution (e.g.
$\frac{1}{2}\sum_{j=1}^{k}\left[\sigma_{j}+\mu_{j}^{2}-1-\log\sigma_{j}\right]$
in the case where latent variables are modeled as $k$ independent Gaussians).

$\mathcal{L}_{VAE}=\mathbb{E}_{q_{\phi}(\bm{z}|\bm{x})}\left[\log
p_{\theta}(\bm{x}|\bm{z})\right]-D_{KL}\left(q_{\phi}(\bm{z}|\bm{x})\mathrel{\|}p(\bm{z})\right)\\\
$ (4)

###  A.3 Normalizing flows (NFs)

Normalizing flows use a bijective model $f_{\theta}^{-1}(\bm{x})$ to map
inputs $\bm{x}$ to latent variables $\bm{z}\sim p(\bm{z})$ and its inverse
$f_{\theta}(\bm{z})$ to map latent variables back to inputs (Figure 2c) [60].
Using the change of variables rule, this allows the marginal probability of
the data to be calculated as the marginal probability of the latent variables
times the determinant of the model mapping data between the two distributions
(Equation 5).

$\begin{split}p(\bm{x})&=p(\bm{z})\left|\det\frac{\partial
f_{\theta}^{-1}}{\partial\bm{z}}\right|\\\ &=p(\bm{z})\left|\det\frac{\partial
f_{\theta}}{\partial\bm{z}}\right|^{-1}\end{split}$ (5)

The model $f_{\theta}$ can itself be a product of multiple submodels
$f_{1},\dots,f_{K}$ applied consecutively (Equation 6). As long as each
submodel is invertible and has a tractable Jacobian determinant, the change of
variables rule can be applied repeatedly to calculate the probability of each
variable $\bm{z}_{0},\dots,\bm{z}_{K}$ forming the “flow” (Equation 7).

$\bm{x}=\bm{z}_{K}=f_{\theta}(\bm{z}_{0})=f_{K}\circ f_{K-1}\circ\dots\circ
f_{1}(\bm{z}_{0})$ (6)

$\log p(\bm{x})=\log p(\bm{z}_{0})-\sum_{j=1}^{K}\log\left|\det\frac{\partial
f_{i}}{\partial\bm{z}_{i-1}}\right|$ (7)

The goal when training normalizing flows is to optimize the parameters of the
model $f_{\theta}$ such that the negative log-likelihood of the data is
minimized (Equation 8).

$\displaystyle\mathcal{L}_{NF}$ $\displaystyle=\mathbb{E}_{\bm{x}}\left[-\log
p(\bm{x})\right]$ (8) $\displaystyle=\mathbb{E}_{\bm{x}}\left[-\log
p(f_{\theta}^{-1}(\bm{x}))+\log\left|\det\frac{\partial
f_{\theta}}{\partial\bm{z}}\right|\right]$ (9)

A key limitation of normalizing flows is the need for the model $f_{\theta}$
to be invertible and for the Jacobian determinant $\left|\det\frac{\partial
f_{\theta}}{\partial\bm{z}}\right|$ to be tractable. One approach to get
around those limitations is to model $f_{\theta}$ as a neural network which,
in each step, applies affine transformations on a fraction of the channels
using information provided by the other channels [61, 62, 63]. Affine
transformations are invertible, and the resulting Jacobian matrix is
triangular, which makes the determinant quick to calculate. Another approach
is to model $f_{\theta}$ using a neural ODE [64] and to approximate the trace
of the Jacobian matrix using the Hutchinson’s trace estimator [65]. Other
approaches have also been proposed [66, 67, 68], and this remains an active
area of research [69, 70].

###  A.4 Energy-based models (EBMs)

The goal of energy-based models (EBMs) is to learn an energy function
$E_{\theta}$ which outputs low values when the inputs correspond to probable
(or “low-energy”) states and high values when the inputs correspond to
improbable (or “high-energy”) states [71, 8] (Figure 2d). In contrast to other
approaches, the objective is not to maximize the log-likelihood of the data,
which allows EBMs to sidestep the considerable challenge of approximating the
partition function $Z_{\theta}$ during training (Equation 10). However, since
the energies predicted by EBMs are not normalized, special care needs to be
taken to make sure that the model does not assign arbitrarily low energies to
all possible inputs [71].

$p_{\theta}(\bm{x})=\frac{\exp(-E_{\theta}(\bm{x}))}{Z_{\theta}}$ (10)

Contrastive divergence is a training strategy where fake examples are
generated using Gibbs sampling, stochastic gradient Langevin dynamics (SGLD),
or another Markov-chain Monte Carlo (MCMC) method, and the model parameters
are updated such that real examples are assigned lower energies while fake
examples are assigned higher energies [72] (Equation 11). A major strength of
contrastive divergence is that it can be applied to almost any domain while
major weaknesses are computational cost of generating fake examples with every
iteration and training instability due to the evolving nature of the
distribution from which the fake examples are sampled.

$\nabla\mathcal{L}_{CD}=\mathbb{E}_{\bm{x}_{+}\sim p_{data}(\bm{x})}[\nabla
E_{\theta}(\bm{x}_{+})]-\mathbb{E}_{\bm{x}_{-}\sim p_{\theta}(\bm{x})}[\nabla
E_{\theta}(\bm{x}_{-})]$ (11)

Noise-contrastive estimation (NCE) is a training strategy where fake examples
are sampled from a predefined distribution, and the model is trained to
distinguish between real and fake examples akin to logistic regression [73,
74] (Equation 12). Energies $E_{\theta}(\bm{x})$ are mapped to probabilities
$p_{\theta}(\bm{x})$ by introducing a learnable parameter standing for the
logarithm of the partition function (Equation 10), and the key challenge of
NCE is defining a tractable and representative distribution of negative
examples $q(\bm{x})$. NCE can be particularly effective when the goal is to
assign energies to a pairing between some categorical variable and its
context, since the negative distribution can be defined by all categories that
are not found in the given context. Furthermore, if the model is trained to
learn vector representations for both the variable and the context, with the
similarity between those vectors corresponding to the probability
$p_{\theta}(\bm{x})$, then the learned embeddings are likely to carry useful
information for downstream tasks.

$\mathcal{L}_{NCE}=\mathbb{E}_{\bm{x}_{+}\sim
p_{data}(\bm{x})}\left[\frac{p_{\theta}(\bm{x}_{+})}{p_{\theta}(\bm{x}_{+})+q(\bm{x}_{+})}\right]+\mathbb{E}_{\bm{x}_{-}\sim
q(\bm{x})}\left[\frac{q(\bm{x}_{-})}{p_{\theta}(\bm{x}_{-})+q(\bm{x}_{-})}\right]$
(12)

###  A.5 Generative adversarial networks (GANs)

A generative adversarial network (GAN) comprises a generator and a
discriminator that are trained in tandem through a minimax game (Figure 2e)
[75]. Several flavors of GANs have been proposed that differ in the
nonlinearities that are applied to the discriminator outputs and in the loss
functions that are minimized [8]. The simplest and most popular is the
Wasserstein GAN [76], where the discriminator is trained to assign low values
to real examples and high values to generated examples (Equation 13), while
the generator is trained to produce examples that are assigned low values by
the discriminator (Equation 14). GANs can be viewed as EBMs, where the
discriminator corresponds to $E_{\theta}$ while the generator corresponds to
the function that proposes negative examples [8].

$\mathcal{L}_{D}=\mathbb{E}_{\bm{x}\sim
p_{data}(\bm{x})}[D(\bm{x})]-\mathbb{E}_{\bm{z}\sim p(\bm{z})}[D(G(\bm{z}))]$
(13)

$\mathcal{L}_{G}=\mathbb{E}[D(G(\bm{z}))]$ (14)

## References

  * Huang et al. [2016] P.-S. Huang, S. E. Boyken, D. Baker,  The coming of age of de novo protein design,  Nature 537 (2016) 320–327. doi:10.1038/nature19946.
  * Gao et al. [2020] W. Gao, S. P. Mahajan, J. Sulam, J. J. Gray,  Deep Learning in Protein Structural Modeling and Design,  Patterns 1 (2020) 100142. doi:10.1016/j.patter.2020.100142.
  * Wu et al. [2021] Z. Wu, K. E. Johnston, F. H. Arnold, K. K. Yang,  Protein sequence design with deep generative models,  arXiv:2104.04457 [cs, q-bio, stat] (2021). URL: http://arxiv.org/abs/2104.04457. arXiv:2104.04457.
  * AlQuraishi [2021] M. AlQuraishi,  Machine learning in protein structure prediction,  Current Opinion in Chemical Biology 65 (2021) 1–8. doi:10.1016/j.cbpa.2021.04.005.
  * Jumper et al. [2021] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, A. Bridgland, C. Meyer, S. A. A. Kohl, A. J. Ballard, A. Cowie, B. Romera-Paredes, S. Nikolov, R. Jain, J. Adler, T. Back, S. Petersen, D. Reiman, E. Clancy, M. Zielinski, M. Steinegger, M. Pacholska, T. Berghammer, S. Bodenstein, D. Silver, O. Vinyals, A. W. Senior, K. Kavukcuoglu, P. Kohli, D. Hassabis,  Highly accurate protein structure prediction with AlphaFold,  Nature 596 (2021) 583–589. doi:10.1038/s41586-021-03819-2.
  * Maddison et al. [2017] C. J. Maddison, A. Mnih, Y. W. Teh,  The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables,  arXiv:1611.00712 [cs, stat] (2017). URL: http://arxiv.org/abs/1611.00712. arXiv:1611.00712.
  * Jang et al. [2017] E. Jang, S. Gu, B. Poole,  Categorical Reparameterization with Gumbel-Softmax,  arXiv:1611.01144 [cs, stat] (2017). URL: http://arxiv.org/abs/1611.01144. arXiv:1611.01144.
  * Bond-Taylor et al. [2021] S. Bond-Taylor, A. Leach, Y. Long, C. G. Willcocks,  Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models,  arXiv:2103.04922 [cs, stat] (2021). URL: http://arxiv.org/abs/2103.04922. arXiv:2103.04922.
  * Alley et al. [2019] E. C. Alley, G. Khimulya, S. Biswas, M. AlQuraishi, G. M. Church,  Unified rational protein engineering with sequence-based deep representation learning,  Nature Methods 16 (2019) 1315–1322. doi:10.1038/s41592-019-0598-1.
  * Biswas et al. [2021] S. Biswas, G. Khimulya, E. C. Alley, K. M. Esvelt, G. M. Church,  Low- N protein engineering with data-efficient deep learning,  Nature Methods 18 (2021) 389–396. doi:10.1038/s41592-021-01100-y.
  * Shin et al. [2021] J.-E. Shin, A. J. Riesselman, A. W. Kollasch, C. McMahon, E. Simon, C. Sander, A. Manglik, A. C. Kruse, D. S. Marks,  Protein design and variant prediction using autoregressive generative models,  Nature Communications 12 (2021) 2403. doi:10.1038/s41467-021-22732-w.
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin,  Attention is All you Need,  Advances in Neural Information Processing Systems 30 (2017). URL: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
  * Rao et al. [2019] R. Rao, N. Bhattacharya, N. Thomas, Y. Duan, X. Chen, J. Canny, P. Abbeel, Y. S. Song,  Evaluating Protein Transfer Learning with TAPE,  arXiv:1906.08230 [cs, q-bio, stat] (2019). URL: http://arxiv.org/abs/1906.08230. arXiv:1906.08230.
  * Elnaggar et al. [2020] A. Elnaggar, M. Heinzinger, C. Dallago, G. Rehawi, Y. Wang, L. Jones, T. Gibbs, T. Feher, C. Angerer, M. Steinegger, D. Bhowmik, B. Rost,  ProtTrans: Towards Cracking the Language of Life’s Code Through Self-Supervised Deep Learning and High Performance Computing,  bioRxiv (2020) 2020.07.12.199554. doi:10.1101/2020.07.12.199554.
  * Rives et al. [2021] A. Rives, J. Meier, T. Sercu, S. Goyal, Z. Lin, J. Liu, D. Guo, M. Ott, C. L. Zitnick, J. Ma, R. Fergus,  Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences,  PNAS 118 (2021). doi:10.1073/pnas.2016239118, • The first study to show that transformers trained on amino acid sequences show better accuracy on downstream tasks than other network architectures, such as LSTMs.
  * Rao et al. [2021] R. Rao, J. Liu, R. Verkuil, J. Meier, J. F. Canny, P. Abbeel, T. Sercu, A. Rives,  MSA Transformer,  bioRxiv (2021) 2021.02.12.430858. doi:10.1101/2021.02.12.430858, •• The authors use axial attention to train protein language models which leverage the information stored in multiple sequence alignments to improve the accuracy of the predictions and the quality of the learned embeddings.
  * Bhattacharya et al. [2020] N. Bhattacharya, N. Thomas, R. Rao, J. Dauparas, P. K. Koo, D. Baker, Y. S. Song, S. Ovchinnikov,  Single Layers of Attention Suffice to Predict Protein Contacts,  bioRxiv (2020) 2020.12.21.423882. doi:10.1101/2020.12.21.423882.
  * Rao et al. [2020] R. Rao, J. Meier, T. Sercu, S. Ovchinnikov, A. Rives,  Transformer protein language models are unsupervised structure learners,  bioRxiv (2020) 2020.12.15.422761. doi:10.1101/2020.12.15.422761, •• The authors show that a supervised model that uses only the attention weights of a pretrained protein language model can predict protein inter-residue distances with an accuracy that matches other leading approaches.
  * Meier et al. [2021] J. Meier, R. Rao, R. Verkuil, J. Liu, T. Sercu, A. Rives, Language models enable zero-shot prediction of the effects of mutations on protein function, 2021. doi:10.1101/2021.07.09.450648.
  * Fuchs et al. [2020] F. B. Fuchs, D. E. Worrall, V. Fischer, M. Welling,  SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks,  arXiv:2006.10503 [cs, stat] (2020). URL: http://arxiv.org/abs/2006.10503. arXiv:2006.10503.
  * Bronstein et al. [2021] M. M. Bronstein, J. Bruna, T. Cohen, P. Veličković,  Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges,  arXiv:2104.13478 [cs, stat] (2021). URL: http://arxiv.org/abs/2104.13478. arXiv:2104.13478.
  * Ingraham et al. [2019] J. Ingraham, V. Garg, R. Barzilay, T. Jaakkola,  Generative Models for Graph-Based Protein Design,  in: H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, R. Garnett (Eds.), Advances in Neural Information Processing Systems 32, Curran Associates, Inc., 2019, pp. 15820–15831. URL: http://papers.nips.cc/paper/9711-generative-models-for-graph-based-protein-design.pdf, • The authors introduce Structured Transformer, a graph neural network with an encoder-decoder architecture which translates protein structures to amino acid sequences.
  * Leman et al. [2020] J. K. Leman, B. D. Weitzner, S. M. Lewis, J. Adolf-Bryfogle, N. Alam, R. F. Alford, M. Aprahamian, D. Baker, K. A. Barlow, P. Barth, B. Basanta, B. J. Bender, K. Blacklock, J. Bonet, S. E. Boyken, P. Bradley, C. Bystroff, P. Conway, S. Cooper, B. E. Correia, B. Coventry, R. Das, R. M. De Jong, F. DiMaio, L. Dsilva, R. Dunbrack, A. S. Ford, B. Frenz, D. Y. Fu, C. Geniesse, L. Goldschmidt, R. Gowthaman, J. J. Gray, D. Gront, S. Guffy, S. Horowitz, P.-S. Huang, T. Huber, T. M. Jacobs, J. R. Jeliazkov, D. K. Johnson, K. Kappel, J. Karanicolas, H. Khakzad, K. R. Khar, S. D. Khare, F. Khatib, A. Khramushin, I. C. King, R. Kleffner, B. Koepnick, T. Kortemme, G. Kuenze, B. Kuhlman, D. Kuroda, J. W. Labonte, J. K. Lai, G. Lapidoth, A. Leaver-Fay, S. Lindert, T. Linsky, N. London, J. H. Lubin, S. Lyskov, J. Maguire, L. Malmström, E. Marcos, O. Marcu, N. A. Marze, J. Meiler, R. Moretti, V. K. Mulligan, S. Nerli, C. Norn, S. Ó’Conchúir, N. Ollikainen, S. Ovchinnikov, M. S. Pacella, X. Pan, H. Park, R. E. Pavlovicz, M. Pethe, B. G. Pierce, K. B. Pilla, B. Raveh, P. D. Renfrew, S. S. R. Burman, A. Rubenstein, M. F. Sauer, A. Scheck, W. Schief, O. Schueler-Furman, Y. Sedan, A. M. Sevy, N. G. Sgourakis, L. Shi, J. B. Siegel, D.-A. Silva, S. Smith, Y. Song, A. Stein, M. Szegedy, F. D. Teets, S. B. Thyme, R. Y.-R. Wang, A. Watkins, L. Zimmerman, R. Bonneau,  Macromolecular modeling and design in Rosetta: Recent methods and frameworks,  Nature Methods 17 (2020) 665–680. doi:10.1038/s41592-020-0848-2.
  * Strokach et al. [2020] A. Strokach, D. Becerra, C. Corbi-Verge, A. Perez-Riba, P. M. Kim,  Fast and Flexible Protein Design Using Deep Graph Neural Networks,  Cell Systems (2020). doi:10.1016/j.cels.2020.08.016, • The authors introduce ProteinSolver, a graph neural network which can generate amino acid sequences of proteins from a distance matrix listing pairs of interacting residues.
  * Strokach et al. [2021] A. Strokach, T. Y. Lu, P. M. Kim,  ELASPIC2 (EL2): Combining Contextualized Language Models and Graph Neural Networks to Predict Effects of Mutations,  Journal of Molecular Biology 433 (2021) 166810. doi:10.1016/j.jmb.2021.166810.
  * Madani et al. [2020] A. Madani, B. McCann, N. Naik, N. S. Keskar, N. Anand, R. R. Eguchi, P.-S. Huang, R. Socher,  ProGen: Language Modeling for Protein Generation,  bioRxiv (2020) 2020.03.07.982272. doi:10.1101/2020.03.07.982272.
  * Madani et al. [2021] A. Madani, B. Krause, E. R. Greene, S. Subramanian, B. P. Mohr, J. M. Holton, J. L. Olmos, C. Xiong, Z. Z. Sun, R. Socher, J. S. Fraser, N. Naik, Deep neural language modeling enables functional protein generation across families, 2021. doi:10.1101/2021.07.18.452833, •• The authors train a conditional protein language model to generate amino acid sequences of proteins with specific functions.
  * Riesselman et al. [2018] A. J. Riesselman, J. B. Ingraham, D. S. Marks,  Deep generative models of genetic variation capture the effects of mutations,  Nature Methods 15 (2018) 816–822. doi:10.1038/s41592-018-0138-4.
  * Ding et al. [2019] X. Ding, Z. Zou, C. L. Brooks Iii,  Deciphering protein evolution and fitness landscapes with latent space models,  Nature Communications 10 (2019) 1–13. doi:10.1038/s41467-019-13633-0.
  * Greener et al. [2018] J. G. Greener, L. Moffat, D. T. Jones,  Design of metalloproteins and novel protein folds using variational autoencoders,  Scientific Reports 8 (2018) 16189. doi:10.1038/s41598-018-34533-1.
  * Hawkins-Hooker et al. [2021] A. Hawkins-Hooker, F. Depardieu, S. Baur, G. Couairon, A. Chen, D. Bikard,  Generating functional protein variants with variational autoencoders,  PLOS Computational Biology 17 (2021) e1008736. doi:10.1371/journal.pcbi.1008736.
  * Das et al. [2021] P. Das, T. Sercu, K. Wadhawan, I. Padhi, S. Gehrmann, F. Cipcigan, V. Chenthamarakshan, H. Strobelt, C. dos Santos, P.-Y. Chen, Y. Y. Yang, J. P. K. Tan, J. Hedrick, J. Crain, A. Mojsilovic,  Accelerated antimicrobial discovery via deep generative models and molecular dynamics simulations,  Nature Biomedical Engineering (2021) 1–11. doi:10.1038/s41551-021-00689-x, • The authors generate novel antimicrobial peptides by training a VAE using all known peptide sequences and using controlled generation to sample regions of the VAE latent space that are most likely to correspond to peptides with antimicrobial activity.
  * Eguchi et al. [2020] R. R. Eguchi, N. Anand, C. A. Choe, P.-S. Huang,  IG-VAE: Generative Modeling of Immunoglobulin Proteins by Direct 3D Coordinate Generation,  bioRxiv (2020) 2020.08.07.242347. doi:10.1101/2020.08.07.242347, • The authors train a variational autoencoder to directly generate the three-dimensional coordinates of immunoglobulin backbones.
  * Noé et al. [2019] F. Noé, S. Olsson, J. Köhler, H. Wu,  Boltzmann generators: Sampling equilibrium states of many-body systems with deep learning,  Science 365 (2019) eaaw1147. doi:10.1126/science.aaw1147, •• The authors introduce Boltzmann generators, a strategy for training normalizing flows such that they learn, and can be used to generate, different configurations of many-body systems.
  * Asgari and Mofrad [2015] E. Asgari, M. R. K. Mofrad,  Continuous Distributed Representation of Biological Sequences for Deep Proteomics and Genomics,  PLOS ONE 10 (2015) e0141287. doi:10.1371/journal.pone.0141287.
  * Yang et al. [2018] K. K. Yang, Z. Wu, C. N. Bedbrook, F. H. Arnold,  Learned protein embeddings for machine learning,  Bioinformatics 34 (2018) 2642–2648. doi:10.1093/bioinformatics/bty178.
  * Lu et al. [2020] A. X. Lu, H. Zhang, M. Ghassemi, A. Moses,  Self-Supervised Contrastive Learning of Protein Representations By Mutual Information Maximization,  bioRxiv (2020) 2020.09.04.283929. doi:10.1101/2020.09.04.283929, • The authors train an energy-based model to predict the latent representations of the succeeding amino acids given the latent representations of the preceding amino acids. The model shows competitive performance on a number of downstream tasks while using substantially fewer parameters than transformer-based architectures.
  * Gainza et al. [2020] P. Gainza, F. Sverrisson, F. Monti, E. Rodolà, D. Boscaini, M. M. Bronstein, B. E. Correia,  Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning,  Nature Methods 17 (2020) 184–192. doi:10.1038/s41592-019-0666-6, • The authors describe a methodology for learning embeddings or “fingerprints” that describe the geometry and physicochemical properties of protein surfaces.
  * Sverrisson et al. [2020] F. Sverrisson, J. Feydy, B. E. Correia, M. M. Bronstein,  Fast end-to-end learning on protein surfaces,  bioRxiv (2020) 2020.12.28.424589. doi:10.1101/2020.12.28.424589, • The authors describe differentiable transformations and convolutions for extracting and analyzing protein surfaces.
  * Du et al. [2020] Y. Du, J. Meier, J. Ma, R. Fergus, A. Rives,  Energy-based models for atomic-resolution protein conformations,  in: International Conference on Learning Representations, 2020. URL: https://openreview.net/forum?id=S1e_9xrFvS, • The authors train an energy-based model to predict amino acid rotamers from the neighboring atoms.
  * Anand-Achim et al. [2021] N. Anand-Achim, R. R. Eguchi, I. I. Mathews, C. P. Perez, A. Derry, R. B. Altman, P.-S. Huang,  Protein Sequence Design with a Learned Potential,  bioRxiv (2021) 2020.01.06.895466. doi:10.1101/2020.01.06.895466, • The authors describe a hybrid approach which uses simulated annealing and a learned energy function to select favorable amino acids and rotamers for fixed backbone design.
  * Anand and Huang [2018] N. Anand, P. Huang,  Generative modeling for protein structures,  in: S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, R. Garnett (Eds.), Advances in Neural Information Processing Systems 31, Curran Associates, Inc., 2018, pp. 7494–7505. URL: http://papers.nips.cc/paper/7978-generative-modeling-for-protein-structures.pdf, • The authors train a generative adversarial network for generating protein distance matrices and develop an efficient convex optimization algorithm for mapping distance matrices to atom coordinates.
  * Maddhuri Venkata Subramaniya et al. [2021] S. R. Maddhuri Venkata Subramaniya, G. Terashi, A. Jain, Y. Kagaya, D. Kihara,  Protein contact map refinement for improving structure prediction using generative adversarial networks,  Bioinformatics (2021). doi:10.1093/bioinformatics/btab220.
  * Karimi et al. [2020] M. Karimi, S. Zhu, Y. Cao, Y. Shen,  De Novo Protein Design for Novel Folds Using Guided Conditional Wasserstein Generative Adversarial Networks,  J. Chem. Inf. Model. 60 (2020) 5667–5681. doi:10.1021/acs.jcim.0c00593.
  * Repecka et al. [2021] D. Repecka, V. Jauniskis, L. Karpus, E. Rembeza, I. Rokaitis, J. Zrimec, S. Poviloniene, A. Laurynenas, S. Viknander, W. Abuajwa, O. Savolainen, R. Meskys, M. K. M. Engqvist, A. Zelezniak,  Expanding functional protein sequence spaces using generative adversarial networks,  Nature Machine Intelligence (2021) 1–10. doi:10.1038/s42256-021-00310-5, • The authors train a generative adversarial network, containing both convolutional and attention layers, on amino acid sequences of individual protein families, and they show that a substantial portion of the generated sequences maintain enzymatic activity characteristic of the family.
  * Anand et al. [2019] N. Anand, R. R. Eguchi, P.-S. Huang,  Fully differentiable full-atom protein backbone generation,  in: DGS@ICLR, 2019.
  * Biswas et al. [2018] S. Biswas, G. Kuznetsov, P. J. Ogden, N. J. Conway, R. P. Adams, G. M. Church,  Toward machine-guided design of proteins,  bioRxiv (2018) 337154\. doi:10.1101/337154.
  * Strokach et al. [2021] A. Strokach, D. Becerra, C. Corbi-Verge, A. Perez-Riba, P. M. Kim,  Computational generation of proteins with predetermined three-dimensional shapes using ProteinSolver,  STAR Protocols 2 (2021) 100505. doi:10.1016/j.xpro.2021.100505.
  * Gupta and Zou [2019] A. Gupta, J. Zou,  Feedback GAN for DNA optimizes protein functions,  Nature Machine Intelligence 1 (2019) 105–111. doi:10.1038/s42256-019-0017-4.
  * Gómez-Bombarelli et al. [2018] R. Gómez-Bombarelli, J. N. Wei, D. Duvenaud, J. M. Hernández-Lobato, B. Sánchez-Lengeling, D. Sheberla, J. Aguilera-Iparraguirre, T. D. Hirzel, R. P. Adams, A. Aspuru-Guzik,  Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules,  ACS Cent. Sci. 4 (2018) 268–276. doi:10.1021/acscentsci.7b00572.
  * Brookes et al. [2019] D. Brookes, H. Park, J. Listgarten,  Conditioning by adaptive sampling for robust design,  in: International Conference on Machine Learning, PMLR, 2019, pp. 773–782. URL: http://proceedings.mlr.press/v97/brookes19a.html, • The authors describe a strategy for tuning generative models such that the generated examples are scored highly by an oracle. Under certain conditions, the strategy strikes an optimal balance between generating high-scoring examples and staying in regions where predictions are of relatively high confidence.
  * Norn et al. [2021] C. Norn, B. I. M. Wicky, D. Juergens, S. Liu, D. Kim, D. Tischer, B. Koepnick, I. Anishchenko, F. Players, D. Baker, S. Ovchinnikov,  Protein sequence design by conformational landscape optimization,  PNAS 118 (2021). doi:10.1073/pnas.2017228118, • The authors perform activation maximization and MCMC sampling over the inputs to trRosetta, a pretrained model for predicting inter-residue distances and orientations, and produce sequences that correspond to proteins with specific three-dimensional shapes.
  * The UniProt Consortium [2021] The UniProt Consortium,  UniProt: The universal protein knowledgebase in 2021,  Nucleic Acids Research 49 (2021) D480–D489. doi:10.1093/nar/gkaa1100.
  * Burley et al. [2021] S. K. Burley, C. Bhikadiya, C. Bi, S. Bittrich, L. Chen, G. V. Crichlow, C. H. Christie, K. Dalenberg, L. Di Costanzo, J. M. Duarte, S. Dutta, Z. Feng, S. Ganesan, D. S. Goodsell, S. Ghosh, R. K. Green, V. Guranović, D. Guzenko, B. P. Hudson, C. L. Lawson, Y. Liang, R. Lowe, H. Namkoong, E. Peisach, I. Persikova, C. Randle, A. Rose, Y. Rose, A. Sali, J. Segura, M. Sekharan, C. Shao, Y.-P. Tao, M. Voigt, J. D. Westbrook, J. Y. Young, C. Zardecki, M. Zhuravleva,  RCSB Protein Data Bank: Powerful new tools for exploring 3D structures of biological macromolecules for basic and applied research and education in fundamental biology, biomedicine, biotechnology, bioengineering and energy sciences,  Nucleic Acids Research 49 (2021) D437–D451. doi:10.1093/nar/gkaa1038.
  * Jamasb et al. [2020] A. R. Jamasb, P. Lió, T. L. Blundell,  Graphein - a Python Library for Geometric Deep Learning and Network Analysis on Protein Structures,  bioRxiv (2020) 2020.07.15.204701. doi:10.1101/2020.07.15.204701.
  * Pavlović et al. [2021] M. Pavlović, L. Scheffer, K. Motwani, C. Kanduri, R. Kompova, N. Vazov, K. Waagan, F. L. M. Bernal, A. A. Costa, B. Corrie, R. Akbar, G. S. A. Hajj, G. Balaban, T. M. Brusko, M. Chernigovskaya, S. Christley, L. G. Cowell, R. Frank, I. Grytten, S. Gundersen, I. H. Haff, S. Hochreiter, E. Hovig, P.-H. Hsieh, G. Klambauer, M. L. Kuijjer, C. Lund-Andersen, A. Martini, T. Minotto, J. Pensar, K. Rand, E. Riccardi, P. A. Robert, A. Rocha, A. Slabodkin, I. Snapkov, L. M. Sollid, D. Titov, C. R. Weber, M. Widrich, G. Yaari, V. Greiff, G. K. Sandve,  immuneML: An ecosystem for machine learning analysis of adaptive immune receptor repertoires,  bioRxiv (2021) 2021.03.08.433891. doi:10.1101/2021.03.08.433891.
  * Köhler et al. [2020] J. Köhler, L. Klein, F. Noe,  Equivariant flows: Exact likelihood generative learning for symmetric densities,  in: H. D. III, A. Singh (Eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, PMLR, 2020, pp. 5361–5370. URL: http://proceedings.mlr.press/v119/kohler20a.html, •.
  * Hermosilla et al. [2021] P. Hermosilla, M. Schäfer, M. Lang, G. Fackelmann, P.-P. Vázquez, B. Kozlikova, M. Krone, T. Ritschel, T. Ropinski,  Intrinsic-Extrinsic Convolution and Pooling for Learning on 3D Protein Structures,  in: International Conference on Learning Representations, 2021. URL: https://openreview.net/forum?id=l0mSUROpwY.
  * Kingma and Welling [2013] D. P. Kingma, M. Welling,  Auto-Encoding Variational Bayes,  arXiv:1312.6114 [cs, stat] (2013). URL: http://arxiv.org/abs/1312.6114. arXiv:1312.6114.
  * Rezende and Mohamed [2015] D. Rezende, S. Mohamed,  Variational Inference with Normalizing Flows,  in: International Conference on Machine Learning, PMLR, 2015, pp. 1530–1538. URL: http://proceedings.mlr.press/v37/rezende15.html.
  * Dinh et al. [2015] L. Dinh, D. Krueger, Y. Bengio,  NICE: Non-linear Independent Components Estimation,  arXiv:1410.8516 [cs] (2015). URL: http://arxiv.org/abs/1410.8516. arXiv:1410.8516.
  * Dinh et al. [2017] L. Dinh, J. Sohl-Dickstein, S. Bengio,  Density estimation using real NVP,  in: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, OpenReview.net, 2017\. URL: https://openreview.net/forum?id=HkpbnH9lx.
  * Kingma and Dhariwal [2018] D. P. Kingma, P. Dhariwal,  Glow: Generative flow with invertible 1x1 convolutions,  in: Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, Curran Associates Inc., Red Hook, NY, USA, 2018, pp. 10236–10245.
  * Chen et al. [2018] R. T. Q. Chen, Y. Rubanova, J. Bettencourt, D. K. Duvenaud,  Neural ordinary differential equations,  in: S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, R. Garnett (Eds.), Advances in Neural Information Processing Systems, volume 31, Curran Associates, Inc., 2018\. URL: https://proceedings.neurips.cc/paper/2018/file/69386f6bb1dfed68692a24c8686939b9-Paper.pdf.
  * Grathwohl et al. [2018] W. Grathwohl, R. T. Q. Chen, J. Bettencourt, I. Sutskever, D. Duvenaud,  FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models,  arXiv:1810.01367 [cs, stat] (2018). URL: http://arxiv.org/abs/1810.01367. arXiv:1810.01367.
  * Behrmann et al. [2019] J. Behrmann, W. Grathwohl, R. T. Q. Chen, D. Duvenaud, J.-H. Jacobsen,  Invertible residual networks,  in: K. Chaudhuri, R. Salakhutdinov (Eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, PMLR, 2019, pp. 573–582. URL: http://proceedings.mlr.press/v97/behrmann19a.html.
  * Chen et al. [2019] R. T. Q. Chen, J. Behrmann, D. K. Duvenaud, J.-H. Jacobsen,  Residual flows for invertible generative modeling,  in: H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, R. Garnett (Eds.), Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019\. URL: https://proceedings.neurips.cc/paper/2019/file/5d0d5594d24f0f955548f0fc0ff83d10-Paper.pdf.
  * Ho et al. [2019] J. Ho, X. Chen, A. Srinivas, Y. Duan, P. Abbeel,  Flow++: Improving flow-based generative models with variational dequantization and architecture design,  in: K. Chaudhuri, R. Salakhutdinov (Eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, PMLR, 2019, pp. 2722–2730. URL: http://proceedings.mlr.press/v97/ho19a.html.
  * Papamakarios et al. [2019] G. Papamakarios, E. Nalisnick, D. J. Rezende, S. Mohamed, B. Lakshminarayanan,  Normalizing Flows for Probabilistic Modeling and Inference,  arXiv:1912.02762 [cs, stat] (2019). URL: http://arxiv.org/abs/1912.02762. arXiv:1912.02762.
  * Kobyzev et al. [2020] I. Kobyzev, S. Prince, M. A. Brubaker,  Normalizing Flows: An Introduction and Review of Current Methods.,  IEEE transactions on pattern analysis and machine intelligence (2020). doi:10.1109/TPAMI.2020.2992934.
  * Lecun et al. [2006] Y. Lecun, S. Chopra, R. Hadsell, M. A. Ranzato, F. J. Huang,  A tutorial on energy-based learning,  in: G. Bakir, T. Hofman, B. Scholkopt, A. Smola, B. Taskar (Eds.), Predicting structured data, MIT Press, 2006.
  * Hinton [2002] G. E. Hinton,  Training Products of Experts by Minimizing Contrastive Divergence,  Neural Computation 14 (2002) 1771–1800. doi:10.1162/089976602760128018.
  * Gutmann and Hyvärinen [2010] M. Gutmann, A. Hyvärinen,  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,  in: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, JMLR Workshop and Conference Proceedings, 2010, pp. 297–304. URL: http://proceedings.mlr.press/v9/gutmann10a.html.
  * Gao et al. [2020] R. Gao, E. Nijkamp, D. P. Kingma, Z. Xu, A. M. Dai, Y. N. Wu,  Flow Contrastive Estimation of Energy-Based Models,  in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 7518–7528. URL: https://openaccess.thecvf.com/content_CVPR_2020/html/Gao_Flow_Contrastive_Estimation_of_Energy-Based_Models_CVPR_2020_paper.html.
  * Goodfellow et al. [2014] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio,  Generative adversarial nets,  in: Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, K. Q. Weinberger (Eds.), Advances in Neural Information Processing Systems, volume 27, Curran Associates, Inc., 2014. URL: https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf.
  * Arjovsky et al. [2017] M. Arjovsky, S. Chintala, L. Bottou,  Wasserstein GAN,  arXiv:1701.07875 [cs, stat] (2017). URL: http://arxiv.org/abs/1701.07875. arXiv:1701.07875.