# Sifting through the Noise: A Survey of Diffusion Probabilistic Models and
Their Applications to Biomolecules

Trevor Norton  Department of Computer Science, Virginia Tech, Blacksburg,
Virginia, 24061, USA  Debswapna Bhattacharya Correspondence to
<EMAIL_ADDRESS>Department of Computer Science, Virginia Tech,
Blacksburg, Virginia, 24061, USA

###### Abstract

Diffusion probabilistic models have made their way into a number of high-
profile applications since their inception. In particular, there has been a
wave of research into using diffusion models in the prediction and design of
biomolecular structures and sequences. Their growing ubiquity makes it
imperative for researchers in these fields to understand them. This paper
serves as a general overview for the theory behind these models and the
current state of research. We first introduce diffusion models and discuss
common motifs used when applying them to biomolecules. We then present the
significant outcomes achieved through the application of these models in
generative and predictive tasks. This survey aims to provide readers with a
comprehensive understanding of the increasingly critical role of diffusion
models.

##  1 Introduction

Diffusion probabilistic models (or more simply diffusion models) have
attracted attentions from researchers and the public alike for their success
in image generation. After usurping generative adversarial networks (GANs) in
sampling high-quality images [1, 2], we have seen many diffusion-based image
synthesis models introduced. Diffusion models have also been successfully
applied to problems in computer vision [3], audio generation [4, 5, 6],
robotics [7, 8, 9, 10, 11], and many others.

Diffusion models are a category of deep generative models, which seek to
easily sample some underlying distribution $p(x)$. Sampling a distribution can
be difficult in cases where the distribution has many modes or is in a high-
dimensional space. This problem can be especially prevalent when working with
thermodynamic systems, where energy barriers can prevent adequate exploration
of the distribution. Non-machine learning techniques have relied on expensive
search methods, such as Markov chain Monte Carlo (MCMC) simulations, to
accurately sample a distribution [12, 13, 14, 15]. Diffusion models instead
sample a tractable prior (a normal distribution, for example), which is then
transformed into the correct distribution. They do this by adding Gaussian
noise to the data distribution and then learning how to iteratively remove the
noise. This iterative denoising breaks down the generation process into
simpler, learnable steps and allows sampling from very high-dimensional, rough
distributions.

This ability makes diffusion models an attractive choice for the modeling of
biomolecules. Problems in the field typically result in very high-dimensional
data that have rough distributions. Levinthal’s paradox [16] famously noted
this difficulty with the protein folding problem: put simply, if one were to
search every possible conformation of a protein for its native strucutre, it
would take longer than the age of the universe (even if checking the state
only took on the order of picoseconds). The outlook worsens when one considers
that the energy landscape may have several meta-stable states that are not the
native structure. Because of this, straight-forward calculations are usually
intractable, and deep learning methods have provided tremendous leaps forward
in predictive tasks. Most saliently, AlphaFold2 has been able to provide
experimental accuracy of the native folded structures for many proteins [17].
Diffusion models benefit from scalable deep learning architectures and can
circumvent the issues of dimensionality through the iterative process.
Furthermore, the diffusion process progressively smooths out the underlying
distribution, which makes the new, smoother distributions amenable to
approximation via deep neural networks [18].

Research into applications of diffusion models to biomolecules has exploded in
recent years, and there have already been a series of successes. This paper
seeks to give an overview of the current state of diffusion models in the
field and highlight the strengths and weaknesses of the method. We will first
proceed by giving a brief history of diffusion models and their derivations.
Following this, we details some of the most common techniques used when
directly applying diffusion models to problems involving biomolecules. The
remainder of the paper summarizes recent results and looks at possible future
areas of research.

Figure 1: Illustration of a diffusion process in three dimensions. A molecule,
such as a protein or RNA, can be represented by a collection of points in
Euclidean space. The diffusion process gradually adds noise until the
distribution is approximately normal. A diffusion model learns to denoise, so
that samples from a normal distribution can be converted back to samples of
the original distribution.

##  2 Preliminaries on diffusion models

Early diffusion models were heavily inspired by techniques from thermodynamics
and statistics. The common observation among them was that it is possible to
progressively transform a normal distribution into a given data distribution.
However, the theories behind this transformation differed. Some viewed this
process as learning to reverse a noising process, where the model
progressively removes noise from a sample. Others thought about this process
as accurately sampling a continuous family of perturbed distributions. Both
theories can be unified into a flexible framework where the diffusion is
described by continuous processes. In this section, we briefly recall the
early work on diffusion models and the main results.

###  2.1 Denoising diffusion probabilistic modeling

Inspired by techniques in nonequlibrium thermodynamics and statistics [19,
20], Sohl-Dickstein et al. first proposed diffusion models as a way to develop
probabilistic models that are flexible enough to capture complex distributions
while providing exact sampling [21]. The main idea behind the algorithm was to
start with samples from the desired distribution and iterative inject Gaussian
noise until approximately reaching a stationary standard normal distribution.
Training the model then comes down to learning how to reverse the noising
process and recover the initial distribution.

More concretely, we want to approximate the distribution of data given as
$\mathbf{x}_{0}\sim p_{\text{data}}(\mathbf{x}_{0})$. The forward process is a
Markov chain whose transitions are given by Gaussian distributions according
to a variance schedule $0<\beta_{1},\ldots,\beta_{T}<1$:

$q(\mathbf{x}_{t}\mid\mathbf{x}_{t-1}):=\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}\mathbf{I}).$

The transition kernel can be given exactly as

$q(\mathbf{x}_{t}\mid\mathbf{x}_{0})=\mathcal{N}(\mathbf{x}_{t};\sqrt{\alpha_{t}}\mathbf{x}_{0},(1-\alpha_{t})\mathbf{I}),$

where $\alpha_{t}=\Pi_{t=1}^{t}(1-\beta_{t})$, and so the exact distribution
at time $t$ is

$q(\mathbf{x}_{t})=\int
q(\mathbf{x}_{t}\mid\mathbf{x}_{0})p_{\text{data}}(\mathbf{x}_{0})\,\mathrm{d}\mathbf{x}_{0}.$

After sufficiently many steps through the Markov chain, we have that
$q(\mathbf{x}_{T})$ is approximately equal to
$\mathcal{N}(\mathbf{x}_{T}\mathbf{0},\mathbf{I})$, the stationary
distribution of the process.

To recover $p_{\text{data}}$, we want to parameterize a reverse Markov chain
$p_{\theta}$ that will take the distribution
$\mathcal{N}(\mathbf{0},\mathbf{I})$ and iteratively remove the added Gaussian
noise. If the size of the steps $\beta_{t}$ are small enough, the reverse
process can be approximated by Gaussian transitions [21] and so the reverse
process is parameterized as

$p_{\theta}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t}):=\mathcal{N}(\mathbf{x}_{t-1};{\bm{\mu}}_{\theta}(\mathbf{x}_{t},t),\mathbf{\Sigma}_{\theta}(\mathbf{x}_{t},t)).$

Training in [21] is then based on maximizing an evidence lower bound (ELBO).

In [22], this model is updated to have a simplified variational bound which
improves sample quality. Reparameterizing the reverse process by setting

$\bm{\mu}_{\theta}(\mathbf{x}_{t},t)=\frac{1}{\sqrt{1-\beta_{t}}}\left(\mathbf{x}_{t}-\frac{\beta_{t}}{\sqrt{1-\alpha_{t}}}\bm{\epsilon}_{\theta}(\mathbf{x}_{t},t)\right)$

and rescaling the loss gives a simplified training objective

$L_{\text{DDPM}}(\theta):=\mathbb{E}_{t,\mathbf{x}_{0},\bm{\epsilon}}\left[\|\bm{\epsilon}-\bm{\epsilon}_{\theta}(\sqrt{\alpha_{t}}\mathbf{x}_{0}+\sqrt{1-\alpha_{t}}\bm{\epsilon},t)\|^{2}\right],$

where $\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. This diffusion
model with $L_{\text{DDPM}}$ is known as a Denoising Diffusion Probabilistic
Model (DDPM).

###  2.2 Noise conditional score network

In [23], the authors approach the problem of sampling from an unknown
distribution $p_{\text{data}}$ by estimating its (Stein) score
$\nabla_{\mathbf{x}}\log p_{\text{data}}(\mathbf{x})$ at varying noise levels.
The score is related to ideas from statistical mechanics. Namely, if we have
an energy function $E(\mathbf{x})$, then the associated Boltzmann distribution
of the data is given according to

$p(\mathbf{x})=\frac{\exp(-E(\mathbf{x}))}{Z},$

where $Z$ is a normalizing factor that is typically intractable to compute.
Thus determining the score of the distribution is essentially equivalent to
finding $-\nabla E(\mathbf{x})$, from which Langevin dynamics of the system
may be computed.

Theoretically, approximating $\nabla_{\mathbf{x}}\log p_{\text{data}}$ would
allow sampling of the distribution directly through simulations. However, as
Song et al. note, there are two major challenges with this approach:

  1. (1)

Typically data lies on a low-dimensional manifold in the ambient space, in
which case the score may be undefined or difficult to accurately approximate.

  2. (2)

There may be low-density regions of the distribution. This results in poor
initial sampling of the distribution in those spaces making it harder to
estimate the score. Furthermore, when sampling with Langevin dynamics, slow
mixing can occur leading inaccurate estimates of the distribution in the short
term.

Both challenges can be addressed by instead estimating the scores of
perturbations of $p_{\text{data}}$. The perturbation kernel is defined as

$q_{\sigma}(\mathbf{x}\mid\mathbf{x}^{\prime}):=\mathcal{N}(\mathbf{x};\mathbf{x}^{\prime},\sigma^{2}\mathbf{I}),$

so that the data distribution is perturbed to

$q_{\sigma}(\mathbf{x})=\int
q(\mathbf{x}\mid\mathbf{x}^{\prime})p_{\text{data}}(\mathbf{x}^{\prime})\,\mathrm{d}\mathbf{x}^{\prime}.$

The aim of the model is to approximate the score $\nabla_{\mathbf{x}}\log
q_{\sigma}(\mathbf{x})$ for increasing levels of noise
$\sigma_{\text{min}}=\sigma_{1}<\sigma_{2}<\cdots<\sigma_{N}=\sigma_{\text{max}}$.
The levels of noise are chosen so that
$q_{\sigma_{\text{min}}}(\mathbf{x})\approx p_{\text{data}}(\mathbf{x})$ and
$q_{\sigma_{\text{max}}}(\mathbf{x})\approx\mathcal{N}(\mathbf{x};\mathbf{0},\mathbf{I})$.
Thus there will be a family of distributions ending with pure noise and
progressively getting closer to the desired distribution.

A _noise conditional score network_ (NCSN) $s_{\theta}(\mathbf{x},\sigma)$ is
trained to estimate the scores at the specified noise levels. There are
several techniques to approximate the scores, but most implementations use
_denoising score matching_ which seeks to minimize

$\mathbb{E}_{q_{\sigma}(\mathbf{x}|\mathbf{x}^{\prime})p_{\text{data}}(\mathbf{x}^{\prime})}\left[\|s_{\theta}(\mathbf{x},\sigma)-\nabla_{\mathbf{x}}\log
q_{\sigma}(\mathbf{x}|\mathbf{x}^{\prime})\|^{2}\right].$

Since the perturbation kernel is Gaussian, the gradient can be written
exactly. Song et al. use the training objective

$L_{\text{NCSN}}(\theta):=\sum_{i=1}^{N}\sigma_{i}^{2}\mathbb{E}_{p_{\text{data}}(\mathbf{x}^{\prime})}\mathbb{E}_{q_{\sigma_{i}}(\mathbf{x}\mid\mathbf{x}^{\prime})}\left[\left\|s_{\theta}(\mathbf{x},\sigma_{i})+\frac{\mathbf{x}-\mathbf{x}^{\prime}}{\sigma_{i}^{2}}\right\|^{2}\right]$

After training and determining
$\theta^{*}=\mathrm{argmin}_{\theta}L_{\text{NCSN}}(\theta)$, sampling from
the distribution is done by a series of Langevin simulations at each noise
level. That is, $M$ steps of the (overdamped) Langevin dynamics are performed
for each $\sigma_{i}$:

$\mathbf{x}_{i}^{m}=\mathbf{x}_{i}^{m-1}+\varepsilon_{i}s_{\theta^{*}}(\mathbf{x}_{i}^{m-1},\sigma_{i})+\sqrt{2\varepsilon_{i}}\mathbf{z}_{i}^{m},\quad
m=1,2,\ldots,M$

where $\varepsilon_{i}>0$ is the step size and $\mathbf{z}_{i}^{m}$ is
standard normal. Then the final sample at the noise level $\mathbf{x}^{M}_{i}$
is approximately distributed as $q_{\sigma_{i}}(\mathbf{x})$. The sampling is
initialized with
$\mathbf{x}^{0}_{N}\sim\mathcal{N}(\mathbf{x}\mid\mathbf{0},\sigma_{\text{max}}^{2}\mathbf{I})$.
And each noise level is initialized with the terminal value of the previous
noise level: $\mathbf{x}^{0}_{i}=\mathbf{x}^{M}_{i+1}$. The final value,
$\mathbf{x}^{M}_{1}$ is an approximate sample from
$p_{\text{data}}(\mathbf{x})$.

###  2.3 Score based modeling with stochastic differential equations

Previous diffusion models used discrete Gaussian noising process to transform
their distributions into standard normal distributions. However, modeling the
noising process as a stochastic differential equation (SDE), as suggested in
[24], has many advantages. In particular, the framework is flexible and
generalizes DDPMs and NCSNs as discretizations of different equations.

The forward diffusion process is modeled with an Itô SDE:

$\mathrm{d}\mathbf{x}=\mathbf{f}(\mathbf{x},t)\mathrm{d}t+g(t)\mathrm{d}\mathbf{w},$
(1)

where $\mathbf{w}$ is a standard Wiener process,
$\mathbf{f}:\mathbb{R}^{d}\times[0,\infty)\to\mathbb{R}^{d}$ is the _drift_
coefficient and $g:[0,\infty)\to\mathbb{R}$ is the _diffusion_ coefficient.
The initial condition $\mathbf{x}_{0}$ has the distribution of the data,
$p_{0}(\mathbf{x})$, and it is assumed that for a sufficiently long time $T$
that $\mathbf{x}_{T}$ is approximately distributed as the stationary solution
of eq. 1.

Taking the step sizes for DDPMs and NCSNs to zero gives a continuous version
of the models that we can write as SDEs. The DDPM converges to the SDE

$\mathrm{d}\mathbf{x}=-\frac{1}{2}\beta(t)\mathbf{x}\mathrm{d}t+\sqrt{\beta(t)}\mathrm{d}\mathbf{w},$

which we refer to as the variance preserving (VP) SDE. Also, NCSNs converge to
the SDE

$\mathrm{d}\mathbf{x}=\sqrt{\frac{\mathrm{d}[\sigma(t)^{2}]}{\mathrm{d}t}}\mathrm{d}\mathbf{w},$

which we refer to as the variance exploding (VE) SDE.

If the score $\nabla_{\mathbf{x}}\log p_{t}(\mathbf{x})$ is known, then the
diffusion process can be reversed by the following SDE:

$\mathrm{d}\mathbf{x}=\left[\mathbf{f}(\mathbf{x},t)-g(t)^{2}\nabla_{\mathbf{x}}\log
p_{t}(\mathbf{x})\right]\mathrm{d}t+g(t)\mathrm{d}\overline{\mathbf{w}},$ (2)

where $\overline{\mathbf{w}}$ is a standard Wiener process when time flows
backward from $T$ to $0$. Thus the goal is to train a score network
$s_{\theta}(\mathbf{x},t)$ to approximate $\nabla_{\mathbf{x}}\log
p_{t}(\mathbf{x})$. A typical objective for training is by again using
denoising score matching:

$\mathbb{E}_{t}\Big{\\{}\lambda(t)\mathbb{E}_{p_{0}(\mathbf{x}^{\prime})}\mathbb{E}_{p_{0,t}(\mathbf{x}\mid\mathbf{x}^{\prime})}\left[\|s_{\theta}(\mathbf{x},t)-\nabla_{\mathbf{x}}\log
p_{0,t}(\mathbf{x}\mid\mathbf{x}^{\prime})\|^{2}\right]\Big{\\}}.$

Other score-matching objectives can also be used here, such as sliced score
matching [25].

Apart from eq. 2, there is another method for reversing the diffusion process.
The ODE

$\mathrm{d}\mathbf{x}=\left[\mathbf{f}(\mathbf{x},t)-\frac{1}{2}g(t)^{2}\nabla_{\mathbf{x}}\log
p_{t}(\mathbf{x})\right]\mathrm{d}t$ (3)

shares the same marginal probability densities as the SDE. One benefit of eq.
3 is that sampling can be done much quicker using standard ODE solvers versus
the Euler–Maruyama method for integrating eq. 2. Furthermore, integrating the
ODE provides identifiable latent representations of the data and exact
likelihood computations.

[21][23][22][24][2][26, 27][28][29, 30][31, 32][33, 34][35][36] Figure 2:
Timeline of applications of diffusion models to biomolecules.

##  3 Motifs for diffusion applied to biomolecules

Due to the geometry of biomolecules and the common goals of researchers, when
diffusion is applied to problems in the field several techniques and ideas
come up often. Some of these are standard notions for diffusion generally, but
others are more niche. This section covers ideas that appear frequently in
research with the aim of highlighting the most important concepts when
engaging in the literature.

###  3.1 Controllable generation

Unconditionally sampling from the learned distribution of a diffusion model is
usually insufficient for applications. For example, in protein design, one may
desire not just a protein sequence and conformation but also that the final
structure satisfies some prescribed function. The SDE formulation of diffusion
models allows for a straight-forward description of controllable generation.
Given an observation $\mathbf{y}$, the goal is to sample from the distribution
of $\mathbf{x}(0)$ conditioned on $\mathbf{y}$, i.e., the distribution
$p_{0}(\mathbf{x}(0)\mid\mathbf{y})$. The score of this distribution when
perturbed can be rewritten using Bayes’ rule to get

$\nabla_{\mathbf{x}}\log
p_{t}(\mathbf{x}(t)\mid\mathbf{y})=\nabla_{\mathbf{x}}\log
p_{t}(\mathbf{x}(t))+\nabla_{\mathbf{x}}\log p(\mathbf{y}\mid\mathbf{x}(t)).$

Typically, $\nabla_{\mathbf{x}}\log p_{t}(\mathbf{x}(t))$ will be approximated
by a score network and so $\nabla_{\mathbf{x}}\log
p(\mathbf{y}\mid\mathbf{x}(t))$ will need to be estimated.

One option is to train another network to approximate this term. Training of
the networks can often be done simultaneously so that the conditioned and
unconditioned scores can be learned together. In some cases, domain knowledge
or heuristics can be used to get an approximation. In the particular case of
in-painting, the _replacement method_ has been used to approximate the
conditional gradient. However, as pointed out in [32], this leads to an
irreducible error, but a similar method using particle filtering can guarantee
an approximation with the correct limiting distribution.

###  3.2 Equivariant/Invariant score networks

For many problems involving biomolecules, the space is three-dimensional
Euclidean space (that is, $\mathbb{R}^{3}$) and the results should not depend
on the choice of coordinate axes. For example, when predicting the native
structure of a protein, two conformations are equivalent if one can be
transformed into the other by translations and rotations. To be precise, the
solution should be _equivariant_ (or _invariant_) to rigid motions. A function
$f:\mathcal{X}\to\mathcal{Y}$ is equivariant with respect to a group $G$ if
for each $g\in G$ and $x\in\mathcal{X}$

$f(g\cdot x)=g\cdot f(x),$

where $g\cdot x$ and $g\cdot f(x)$ represents a group action on $\mathcal{X}$
and $\mathcal{Y}$, respectively. A function is invariant if $f(g\cdot x)=f(x)$
for each $g\in G$ and $x\in\mathcal{X}$. Restricting machine learning
architectures to the smaller space of equivariant functions speeds up training
and makes models robust to such transformations. Thus, networks that are
equivariant with respect to the special Euclidean group $\mathrm{SE}(3)$ are a
natural choice for problems involving biomolecules111Due to the chirality of
certain biomolecules, one typically does not want equivariance to
_reflections_. Hence, equivariance/invariance is typically with respect to
$\mathrm{SE}(3)$ and not $\mathrm{E}(3)$.

A common choice for incorporating equivariance into diffusion models is to use
an equivariant network for $s_{\theta}(\mathbf{x},t)$. Given that the
diffusion process and the score are both equivariant with respect ot
$\mathrm{SE}(3)$, the reverse diffusion and sampling processes are also
equivariant (c.f. [26, Prop. 1] for instance). Common choices of architecture
for the score network include Invariant Point Attention [17], Tensor Field
Networks [37], and Equivariant Graph Neural Networks [38], among other.

###  3.3 Diffusion on manifolds

While equivariance provides numerous benefits when designing diffusion models,
it may also lead unrealistic generation of molecules. This is due to the
chirality of some biomolecules, which is not preserved under reflections of
$\mathbb{R}^{3}$. Furthermore, for molecules with $n$ particles, the effective
number of degrees of freedom is typically much smaller than $3n$. For
instance, bond lengths and angles are usually fairly rigid in molecules and
thus contribute little to the diversity of conformations. Thus, one can
imagine the configurations of a biomolecule as lying on a smaller submanifold
of the ambient space, and by defining coordinates on that manifold the
configuration of the molecules can be specified. It is possible to define the
diffusion process on these new coordinates, which reduces the dimension of the
problem while enforcing certain geometric priors.

Score-based diffusion models may be carried over to Riemannian manifolds while
retaining many of the characteristics of the models from Cartesian space [28].
A nautral choice for forward diffusion on compact manifolds is the VE SDE

$\mathrm{d}\mathbf{x}=\sqrt{\frac{\mathrm{d}[\sigma(t)^{2}]}{\mathrm{d}t}}\mathrm{d}\mathbf{w}_{\mathcal{M}},$

where $\mathbf{w}_{\mathcal{M}}$ is now Brownian motion on the manifold. This
SDE has a uniform distribution as its stationary distribution. For generic
Riemannian manifolds, the transition kernel cannot be explicitly written, and
so training may be computationally expensive. However, the kernel is known for
some simple manifolds. For rigid bodies in three dimensions, two manifolds are
particularly useful: the torus $\mathbb{T}$, and the set of orientations
$\mathrm{SO}(3)$.

Figure 3: Sample paths for diffusion in $\mathbb{T}$ and $\mathrm{SO}(3)$.
(top) Diffusion in $\mathbb{T}$ is frequently applied to the torsion angle of
four consecutive particles. Since bond lengths and angles are usually fairly
rigid, most of the diversity in conformations can be explained by the
torsional angles. Here an example torsional angle is perturbed by the
diffusion process. (bottom) Diffusion in $\mathrm{SO}(3)$ can be applied to
frame data to perturb the orientation. Here the cube’s center is not moved
while a rigid rotation is applied.

####  3.3.1 Diffusion on $\mathbb{T}$: torsional space

The manifold $\mathbb{T}$ can be used to represent angular data, with the
points being associated with the interval $[0,2\pi)$ [29]. The conformation
for many molecules can almost entirely be described by their torsional angles,
and so modeling the diffusion process on $\mathbb{T}^{m}$ can effectively
reduce the dimensionality without a significant loss of accuracy.

For the VE formulation the transition kernel on $\mathbb{T}^{m}$ is given by
the wrapped Gaussian:

$p_{t}(\bm{\tau}\mid\bm{\tau}^{\prime})\propto\sum_{\mathbf{k}\in\mathbb{Z}^{m}}\exp\left(\frac{-|\bm{\tau}-\bm{\tau}^{\prime}-2\pi\mathbf{k}|^{2}}{2\sigma(t)^{2}}\right)$

Sampling from this distribution amounts from sampling from a normal
distribution in $\mathbb{R}^{m}$ and then modding out by units of $2\pi$.

####  3.3.2 Diffusion on $\mathrm{SO}(3)$: orientations in three dimensions

The manifold $\mathrm{SO}(3)$ is the set of $3\times 3$ orthogonal matrices
with determinant $1$, and so they are used to represent rigid rotations of
$\mathbb{R}^{3}$ or the orientation of three-dimensional objects. Diffusing on
$\mathrm{SO}(3)$ can be useful during generation when the structure being
diffused is not spherical and properties are dependent on its orientation –
for example a ligand or the residue of a protein. The transition kernel for
the VE formulation is given by the $\mathrm{IGSO}(3)$ distribution [39], which
easily computable via a series expansion.

####  3.3.3 Direct product of manifolds

More complicated manifolds can be built from the direct product of simpler
manifolds. The VE forward process diffuses each coordinate independently, and
so for direct product the diffusion can be carried out separately on each
component manifold. This allows for diffusion across multiple coordinates: for
instance, diffusing the position and orientation of $n$ particles amounts to
diffusing on the manifold $\mathbb{R}^{3n}\times\mathrm{SO}(3)^{n}$, for which
we know each of the transition kernels.

###  3.4 Low-temperature sampling

A common problem with training generative models is _overdispersion_. When
this occurs, the coverage of the modes of an underlying distribution is
emphasized more than sample quality, leading to poor quality in the samples.
This can be particularly deleterious when using generative models for
predictions. It is common to use modified sampling to sacrifice diversity for
higher-quality. See for instance shruken encodings in normalizing flows [40].

Low-temperature sampling of a distribution is an attractive way to combat
overdispersion. That is, if $p(x)$ is the original distribution, sampling
instead from $\frac{1}{Z}p(x)^{\lambda}$, where $Z$ is a normalizing factor
and $\lambda>0$ is an inverse temperature parameter, will emphasize high
likelihood areas of the distribution. Taking $\lambda\to\infty$ will lead the
distribution to concentrate around the global maximum of $p(x)$. This process
is analogous to lowering the temperature for a thermodynamic system so that
enthalpy is emphasized over entropy. However, low-temperature sampling in many
cases is intractable or requires expensive Markov chain Monte Carlo (MCMC)
simulations to compute.

For diffusion models, it is tempting to simply increase the score and decrease
the noise, but doing this does not give the correct distribution and is
generally ineffective [2]. Ingraham et al. demonstrate how to derive an
approximate scheme in the case of a normal distribution [41]. One can then
approximate the perturbed score function $\mathbf{s}_{\mathrm{perturb}}$ by
multiplying the original score function $\mathbf{s}$ by a time-dependent
constant $\lambda_{t}$: that is,

$\mathbf{s}_{\mathrm{perturb}}(\mathbf{x},t)\approx\lambda_{t}\mathbf{s}(\mathbf{x},t).$

Replacing $\mathbf{s}$ with $\mathbf{s}_{\mathrm{perturb}}$ leads to new
reverse diffusion SDE that one can use to sample the low-temperature
distribution. This approximating only holds in the case of a Gaussian
distribution for $p_{\mathrm{data}}$, and so applying the rescaling to
arbitrary distribution will not lead to a proper reweighting. However, mixing
the low-temperature reverse SDE with a Langevin SDE allows for proper sampling
of a generic distribution. For more details, we refer readers to [41, Appendix
C].

##  4 Applications

[35][42][41][43][31][32][44][45][46][47][48][49][50][51][52][53][54][55][56][57][58][33][59][60][34][61][62][63][64][65][66][67][36][68]
Figure 4: Summary of diffusion application for the generation and prediction
of biomolecules. Ellipses are used for structure generation/prediction;
rectangles for sequences; and rounded rectangles for co-generation methods.
Blue shapes represent monomeric applications while orange shapes represent
polymeric/complex applications.

Applications for diffusion models can be roughly divided into two categories:
generation/design and prediction. Methods between these two categories are
similar, but the goals are different in terms of sampling the underlying
distribution. For generative applications, the goal is to faithfully sample
the distribution. This means not only having accurate samples but also having
good coverage of all modes. Design tasks are usually downstream from
generation task, where one conditionally samples the distribution of interest.
For predictive applications, the goal is to sample the most likely point in
the probability distribution. This tends to correspond with an optimization
goal, e.g., minimizing the total energy of a system.

This classification is not a strict binary, but rather represent two ends of a
spectrum for sampling a distribution. There is a tension between sample
_diversity_ (how well a method covers all modes of a distribution) and sample
_quality_ (the likelihood of samples from a distribution), and there may be
cases where one might sacrifice sample diversity for better quality or vice
versa. For instance, we discussed how low-temperature sampling gives a
controlled way to ignore smaller modes in the distribution to instead focus on
more likely points. Even in prediction cases, where a single answer is
desired, diversity can still be beneficial. Diversity of sampling can benefit
prediction when (a) there are multiple plausible solutions, (b) there are
close modes which must be discriminated or (c) a measure of flexibility or
certainty is needed.

Figure 5: The difference between generation tasks and prediction tasks is how
samples are to be drawn from the distribution. On the one hand, generation
tasks value faithful sampling of the distribution and the sampling of all
modes. On the other hand, prediction tasks value only the most likely outcomes
from a distribution. The left- and right-hand plots above show this dichonomy
between generation and prediction, respectively. Generated samples from the
methods are shown in orange stars.

###  4.1 Generative modeling

Diffusion models are particularly useful for generating samples from a high-
dimensional distribution, and so it is natural to apply them to the problem of
creating plausible molecular structures. Furthermore, conditional diffusion
allows for additional constraints to be added to the generation process,
allowing for the design of proteins. Despite the recentness of the techniques,
there have already been significant results. Many results focus on generating
the backbone of the protein sequence (for example the locations of the
C$\alpha$ atoms), and so this direction will be the main focus of this
section.

####  4.1.1 Benchmarking

The ability of generative models to effectively sample the desired
distribution is typically measured in three aspects: sample quality, sample
diversity, and novelty. _Sample quality_ looks at whether the generated
samples are likely to come from the underlying distribution. _Sample
diversity_ looks at whether the distribution of samples reasonably spans the
entirety of the underlying distribution. _Novelty_ is a measure of how well
the model generalizes from the training set; a model that simply memorizes the
training data would do well with quality and diversity, but would not be able
to produce new samples from the distribution.

To measure the quality of generated protein structures, one needs to determine
whether a polypeptide chain has a native conformation matching the generated
structure. By design, the generated structures are novel and thus cannot be
directly compared with previous structures. Validating these structures
experimentally is time-intensive, and so _in silico_ methods for determining
the structure – such as AlphaFold2 [17] and ESMFold [69] – are employed to
evaluate the samples. _Self-consistency_ of a method is the measure of whether
a sequence can be found and independently folded into the generated structure.
For many models, no sequence is generated and so a reverse folding method,
such as ProteinMPNN [70], must first be used to find a number of plausible
sequences. The self-consistency score is computed by comparing the folded
structure with the generated structure and taking the closest comparison. The
self-consistency TM (scTM) score computes the TM score between the generated
backbone and each folded backbone and returns the highest result. Another
choice is the self-consistency RMSD (scRMSD) score which similarly compares
structures using the backbone RMSD and returns the smallest result. The scRMSD
is a more stringent requirement, and is a stricter measurement of
designability. A protein is said to be _designable_ if its scRMSD is less than
2 Å or its scTM score is greater than $0.5$.

Assessing diversity and novelty is more straight-forward since this only
involves comparisons with the generated and known structures. The average
pairwise TM score among the generated structures measures the diversity of the
sampling, and the maximum TM score of the generated structure and the training
set measures the novelty of the sampling. Other qualitative measures of
assessing the model exist as well. For unconditional generation, comparing the
distribution of relevant statistics (e.g. dihedral angles and secondary
structure) can help determine whether the sampling does not well approximate
the underlying distribution.

####  4.1.2 Backbone generation models

Trippe et al. created a generative model for the C$\alpha$ backbone of
proteins with a specific focus on controllable generation for the motif-
scaffold problem [32]. Diffusion was carried out on the Cartesian coordinates
and used an $E(3)$ equivariant score network. Despite producing plausible
protein structures, generally the backbones were not designable. Furthermore,
the equivariance of the network to reflections produced incorrect left-handed
$\alpha$-helices, and so the model did not respect the chirality of the
proteins. Wu et al. diffused on the internal angles of the protein backbone in
order to create a generative model that respects chirality [44]. Again, the
generated proteins were plausible and had distributions of secondary
structures similar to native proteins, but most generated backbones were not
designable.

ProteinSGM was one of the first models to be able to generate highly
designable protein backbones [45]. The diffusion is carried out on the six-
dimensional space of inter-residue coordinates as defined in trRossetta [71],
using the VE SDE formulation. Protein backbones are obtained from coordinates
using an adaptation of the trRossetta minimization method, which enables
generation of realistic structures. ProteinSGM is able to generate proteins up
to length 128 with scTM of 90.2%, versus 11.8% and 22.7% as reported by [32]
and [44], respectively.

Ingraham et al. developed Chroma, which generates backbones by diffusing on
the C$\alpha$ atoms [41]. The backbones generated are more likely to be
designable than the previously mentioned methods, with around 50% being
designable by their criterion. Notable features of their model include (1)
choosing a covariance structure of the initial distribution to replicate the
statistics of realistic protein backbones (such as radius of gyration), (2) a
random graph neural score network to maintain subquadratic computational time,
and (3) an annealed sampling scheme that allows the model to sample from
$\frac{1}{Z}p(x)^{\lambda}$ without retraining. Furthermore, the
programmability of the model and code is emphasized in the paper. That is,
several conditioners are created to allow for a wide range of conditions for
protein generation, such as on the three-dimensional shape and point
symmetries of the final structure. The generated protein structures are
diverse and cover all of natural protein space.

Other recent attempts at diffusion models for protein generation carry out the
diffusion process on the coordinates and orientations of the backbone atoms.
That is, for $n$ atoms, diffusion will be carried out on
$\mathbb{R}^{3n}\times\mathrm{SO}(3)^{n}\simeq\mathrm{SE}(3)$. This allows the
models to reason about the three-dimensional structure of the protein during
the generation step while retaining angular information between residues so
that chirality is respected. This paired with $\mathrm{SE}(3)$ equivariant
networks like IPA have been particularly successful.

The most salient example of this method is RFdiffusion, which is considered
the gold standard of backbone generation [35]. As argued by the authors,
previous models lacked deep understanding of protein structure, and so taking
advantage of structure predictions methods could allow for better sample
quality. In particular, pretraining using the weights from RoseTTAFold [72]
provided an enormous benefit to the diffusion model. RFdiffusion has high
designabiltiy of unconditionally generated proteins, with independent
benchmarking showing over $95\%$ of moderately sized backbones being
designable [73, 43] and some of the protein designs being experimentally
verified [35]. Furthermore, the model is capable of conditional generation of
proteins, such as producing oligomers with point symmetries and functional-
motif scaffolding. Recent work has been done on expanding RFdiffusion’s
capabilities to more general biomolecular tasks. RoseTTAFold All-atom (RFAA)
[42] expands RoseTTAFold’s by including atomic-level inputs for other
biomolecular components. This allows RFAA to more accurately predict protein
structure in the context of assemblies of molecules. RFdiffusionAA (similar to
the original RFdiffusion) uses the pretrained weights of RFAA to develop a
protein backbone generation model. Thus RFdiffusionAA can generate protein
structures conditioned on small-molecule binders. Designs from RFdiffusionAA
were experimentally validated and were shown to bind with the target
molecules.

(a)

(b)

(c)

(d)

Figure 6: Example protein backbone generated from diffusion methods. Each
backbone is 250 residues long: (a) RFdiffusion [35]; (b) Chroma [41]; (c)
FrameDiff [46]; (d) Genie [43].

Other models have used $\mathrm{SE}(3)$ diffusion to get designable backbones
without the need to pretrain. Anaand and Achim were able to generate plausible
protein structures diffusing on $\mathrm{SE}(3)$222More accurately the
diffusion took place on $\mathbb{R}^{3}\times\mathrm{SU}(2)$, where
quaternions are used to represent orientation. and capture realistic
distributions of secondary structures [31]. The diffusive process in this case
was approximated by random interpolations in the space. FrameDiff uses a
principled approach of diffusing on $\mathrm{SE}(3)$ where the loss is given
by denoising score matching (as opposed to a Frobenius norm used in [35]). The
score network uses IPA to capture spatial relationships while a transformer
captures interactions along the protein chain. The model is able to generate
designable backbones (using the scTM metric) about $75\%$ of the time, while
still maintaining diversity and novelty of the samples. This model is also
extendable to the protein in-painting problem, as demonstrated by FrameDiPT
[48].

Lin and AlQuraishi carry out a similar idea by using $\mathrm{SE}(3)$
equivariant network [43]. The forward diffusion process is carried out on the
coordinates on the C$\alpha$ atoms while the backward process uses coordinate
frames to predict the noise displacements at each step. This allows for a
cheaper noising process while allowing the full reasoning of IPA when
generating samples. The authors demonstrate improved designability and
diversity of tertiary structures compared with FrameDiff (although FrameDiff
showed better diversity of secondary structure elements).

Designability Diversity Novelty Method % scRMSD $<$ 2.0 Å $\uparrow$ TM-Score
$\downarrow$ % PDBTM $<$ 0.5 $\uparrow$ Number of Parameters RFdiffusion 0.904
0.2894 0.3750 59.8 M Chroma 0.728 0.2779 0.3198 32.4 M FrameDiff 0.484 0.2912
0.157 17.4 M Genie 0.668 0.2674 0.5069 4.1 M

Table 1: Benchmarking of backbone generation models on unconditional
generation.

For convenience, we provided original benchmarking of backbone generation
methods. The results are given in table 1 and further details of the
benchmarking are provided in section A.1. Notably, RFdiffusion performs the
best in designability, which matches the benchmarking done in other works.
Genie performs well with diversity and novelty, despite its relatively small
size compared to the other models. One should note that there is a tradeoff
between designability and diversity/novelty: prioritizing designability will
lead to proteins with more stable structures but may bias toward predictable
designs (e.g. an overemphasis on $\alpha$-helices). However, RFdiffusion still
performs well on all metrics, even given its high designability. The strength
of RFdiffusion can likely be credited to using pretrained weights from
RoseTTAFold, which gives useful information about protein tertiary structure.

####  4.1.3 Other generative models

Although generating proteins based on backbones coordinates is a popular
choice, other approaches have been attempted. One deficit to focusing on
coordinates is that conditional generation based on sequence information is
not possible: the backbone methods above only consider structural information
and obtain the sequences via a reverse folding method. Thus diffusing in
_sequence_ space could be useful for conditioning on sequence information. Up
to this point, we have only discussed diffusion on continuous spaces (either
Euclidean space or a manifold), while diffusing on sequences requires being in
a discrete space. One option for bringing diffusion to discrete space is to
simply embed categorical information into continuous (e.g. using a one-hot
encoding scheme) and then performing a traditional diffusion. A more
principled options is to create a diffusion scheme for discrete data. For
instance, Discrete Denoising Diffusion Probabilistic Models (D3PMs) describe a
corruption process for discrete data that is analogous to Gaussian noise [74].

This more principled approach was taken by the authors of EvoDiff, which is a
protein sequence generation model [55]. EvoDiff carries out discrete diffusion
in a method similar to D3PM, but the authors found it more successful to
generate the sequence autoregressively. The model demonstrates good coverage
of protein sequence space. By embedding the sequences with ProtT5 [75] and
computing the Fréchet distance between the generated set and the test set, the
authors show that EvoDiff provides better coverage than other protein language
models [76] and RFdiffusion [35]. EvoDiff is able to generate proteins with
intrinsically disordered regions, which is something that other structure-
based diffusion methods struggle to do. Furthermore, it is also possible to
condition on MSA data to generate other members of a protein family without
retraining. Overall, many of the strengths of EvoDiff are orthogonal to those
of structure-based diffusion.

Protein Generator is another method that opts to diffuse over sequence space
to generate protein structures [56]. However, the diffusion process is carried
out in continuous space as standard DDPM, where the amino acid sequence is
embedded via a one-hot encoding. Structures are simultaneously generated by
fine-tuning RoseTTAfold [72] to accept noisy sequences and diffusion
times333This method of structure generation echoes RFdiffusion, which uses
pretrained weights from RoseTTAfold to guide training.. This method creates
sequences alongside a physically plausible three-dimensional structure. This
also allows for the conditional generation based on sequence and structure.

Research has also been focused on sequence generation of proteins with
predefined functions. For instance, ForceGen uses a pretrained protein
language (pLM) model to diffuse in probability space [57]. The authors were
particularly focused on generating proteins which matched mechanical folding
responses. To accomplish this, they curated a dataset consisting of protein
sequences as well MD simulation data of the force required to stretch a
protein. Samples are then generated by reversing diffusion in probability
space with the force data as a conditioner. The pLM is then used to decode the
sequence for the final prediction. The generated sequences were tested by
obtaining the three-dimensional structure via OmegaFold [77] and then
simulating the mechanical unfolding. It was shown that the generated sequences
do produce responses close to the conditioned mechanical response. Another
example is CPDiffusion, which is a DDPM treating sequences as categorical data
embedded in Euclidean space [78]. In the study, the authors use CPDiffusion to
generate novel programmable endonucleases. The designed proteins (after an
initial screening) were validated experimentally, with all 27 being soluble.
Furthermore, some of the proteins demonstrated desired properties, with 24 of
them displaying DNA cleavage activity and most having enhanced enzymatic
activity compared to the template protein.

Another novel diffusion method was used in the development on LatentDiff [47].
As the name suggests, diffusion is not carried out in coordinate space but
rather in a latent space. Inspired by image generation methods like
StableDiffusion [79], Fu et al. create an $\mathrm{SE}(3)$ autoencoder to
downsample the geometry of the protein as well as upsample to recover the
protein coordinates. Protein generation is carried out by reversing diffusion
in the latent space and then upsampling to retrieve the predicted protein. In
benchmarking, the method outperforms other small-protein generation methods,
such as FoldingDiff, in designability with comparable (but slightly worse)
results to FrameDiff. However, its sampling speed is around 64 times faster
than FrameDiff, likely due to the reduced dimensionality of the latent space.

Protpardelle differs from the backbone generation methods in that it generates
proteins at an all-atom resolution [52]. An obstacle to all-atom generation is
that the number and types of atoms are determined exactly by the amino acid
sequence, so making a choice on the coordinates to diffuse will also determine
the amino acid sequence and thus the final structure of the protein. To
address this, diffusion is carried out on a “superposition” of possible
structures. That is, the positions of atoms for any choice of amino acid are
tracked during diffusion, allowing the model the flexibility to decide the
sequence during the generation process. On the standard benchmarks,
Protpardelle does quite well. In particular, its scRMSD score is about 90% for
small proteins and around 70% for proteins around 300 amino acids in length.
Also, because the model generates an all-atom representation, it is possible
to condition generation on side-chain information. However, improvements in
conditional generation are needed as the model is prone to ignoring the
conditioning information.

While an overwhelming amount of work has been done on the generation and
design of proteins, relatively less attention has been paid to other
biomolecules. However, there have been some recent efforts to bring diffusion
methods to DNA and RNA generation.

For instance, DiscDiff is a latent diffusion model for generating DNA
sequences [54]. A variational autoencoder is trained to first take DNA
sequences into a latent space. Then a diffusion model is trained in latent
space to produce samples. One notable change that the authors make is during
the decoding step. Often with latent diffusion models, errors may be
introduced when moving from latent space to sequence space. The authors
hypothesize that the latent variable captures the global information of the
sequence well but that there may be regions in the sequence with lower
certainty. To address the authors introduce the Absorb-Escape method, where a
pretrained autoregressive model for DNA sequences is used to correct
nucleotides with lower certainty. When benchmarking against other sequence
generation methods trained on the DNA task, it achieved state-of-the-art
results without post-conditioning and did even better with Absorb-Escape.

Another attempt of bringing diffusion models to other biomolecules can be seen
with MMDiff [53]. This model jointly generates sequences and structures of
nucleic acid and protein complexes. Many of the methods are directly inspired
by the work of diffusion models applied to proteins. The diffusion takes place
in $\mathrm{SE}(3)$ with coordinate frames for each of the residues of the
protein and RNA/DNA structures, similar to that used by Yim et al. [46].
Sequences are simultaneously generated by embedding them into continuous space
via a one-hot encoding, as done by Lisanza et al. for Protein Generator [56].
Benchmarking proceeds analogously to protein generation, where designability,
diversity, and novelty are assessed. However, MMDiff fails to achieve high
designability. For instance, the percentage of designable protein-nucleic acid
complexes is 0.74% (versus 0.37% for random generation). The method does
marginally better with the generation of nucleic acids only, with 8.67% being
designable (1.33% for random generation). The authors suggest that the key to
improvement is more diverse and high-quality training data. This highlights
the current challenges in directly transferring structural generation models
to the problem of RNA and DNA generation.

####  4.1.4 Molecular dynamics and ensemble generation

Molecular dynamics simulations is an important tool for computational biology.
It sees uses in the refinement of predictions and the study of dynamics of
biomolecules [80, 81, 82, 83, 84, 85, 86]. Typically, simulations are carried
out using Langevin dynamics and a prescribed potential energy function.
However, computation of a conventional energy function is expensive (usually
quadratic in the number of particles), and so simulating the dynamics on a
suitably long time-scale is infeasible. Generative models have the ability to
speed up these computations by directly sampling the distribution of future
states conditioned on the current state. This has been applied to the
simulation of molecules with diffusion models and other generative models [87,
88]. However, this direction has not been thoroughly explored for the
simulation of protein dynamics.

Lu et al. made progress in this direction with the introduction of Str2Str, a
model for sampling protein conformations [49]. The method shares similarities
to simulated annealing, where the temperature for MD simulations is raised to
overcome energetic barriers. Here a structure is perturbed by diffusion, and
then the process is reversed to sample a new conformation of the protein. The
noising process is carried out in $\mathrm{SE}(3)$ as described by Yim et al.
[46] and the score network is given by a variant of IPA [17]. Str2Str is able
to recapture many of the statistics of the equilibrium distributions of fast-
folding proteins in negligible amount of time compared to direct MD simulation
(510 GPU seconds versus over 160 GPU days, respectively.)

A similar approach was carried out by Zheng et al. with the development of the
Distributional Graphormer (DiG) [50]. This model samples the equilibrium
distribution of molecular systems using a diffusion model, where the score
architecture is given by a Graphormer. In this work, the equilibrium
distribution of proteins is used during training and validation. Again,
diffusion is carried out over frame data in $\mathrm{SE}(3)$ using the VE
formulation of the SDE. One notable addition to the typical diffusion model is
the use of physics-informed diffusion pre-training (PIDP). Since MD simulation
are expensive to compute, the authors sought a method to utilize prior energy
functions to train the score model. The main idea is to enforce that the score
network satisfies the corresponding Fokker-Plank equation – a partial
differential equation which governs the value of the score through the
diffusion process – and the initial condition given by the gradient of the
prior energy. The PIDP does not require samples from the equilibrium
distribution since information from the energy function will enforce the
correct distributions. A tractable number of samples can be selected during
this process since the equilibrium distribution lies on a much lower
dimensional manifold than the latent space. DiG is shown to recapture protein
equilibirum distributions, as well as generate plausible transition paths
between conformations.

A related direction is the training of coarse-grained models for proteins. The
free-energy landscape for an all-atom representation of a protein is rough and
difficult to simulate, so coarse-grained models look to simplify the
simulations by combining particles. The training of coarse-grained models is
typically done through variational force matching, where the forces on the
virtual particles are chosen to match the expected force before the coarse-
graining process. However, this process is data-inefficient since the coarse-
graining maps many configurations to the same state and so forces must be
averaged out over those configurations. An alternative method is using
relative entropy, but this requires resampling the coarse-grained distribution
during training. Flow-matching is a more data-efficient method of training a
coarse-grained model [89], which first uses a generative model to learn the
equilibrium distribution of a protein that then is used to determine the
forces of the model. This method is built on by Arts et al. who train a
generative and coarse-grained model simultaneously using a denoising diffusion
generative model [51]. Similarly to flow matching, training does not require
the recording of forces. The coarse-grained model and sampler are able to
recapture the equilibrium distribution of the proteins on which they are
trained. The training scheme shows promise in terms of scaling to longer
proteins.

###  4.2 Predictive modeling

Diffusion models being generative models are traditionally designed to sample
the whole probability distribution they are trained on. However, there has
been success with using diffusion models to predict the solution to regression
and/or optimization problems with a single solution. Non-generative machine
learning methods sometimes struggle to account for uncertainty and thus try to
minimize the expected error at the cost of realistic predictions (see [33, §3]
for an example). A generative model instead handles the uncertainty by
sampling over all possibilities, giving a more accurate representation of the
likely solution.

For generating a single prediction, one can repeatedly sample from the
diffusion model and then choose the “best” sample. A typical strategy for
making the final choice is training a separate confidence model which would
rank each of the samples based on the confidence of being the optimal
solution. A training-free alternative would be to use a likelihood estimate or
evidence-based lower bound (ELBO) to determine the most likely sample from the
distribution.

Following this strategy, several diffusion models have been applied to
prediction problems and have achieved state-of-the-art results.

####  4.2.1 Protein side-chains

The protein side-chain packing problem focuses on the case where the backbone
of a protein is fixed and asks for the positions of its side-chain atoms.
Traditional methods for solving the problem involve searching the space of
positions for the side-chains using rotamer libraries and comparing energies
[90, 91, 92, 93, 94, 95, 96, 97]. Due to the complexity of the energy
landscape and the number of possible conformations of the side-chains, this
process can be time-intensive. Machine-learning methods aim to speed up this
process while maintaining precision. Methods such as AttnPacker [98] predict
the positions of the side-chain atoms but do not account for bond lengths and
angles. Other machine learning methods treat side-chain packing as a
regression problem and do not model the diversity of possible conformations in
the energy landscape [99, 12].

DiffPack is a diffusion-based model for side-chain packing that predicts the
torsional angles, $\chi_{1},\chi_{2},\ldots$, of each side-chain [65]. Ideal
bond lengths and angles are assumed for the side-chain atoms, so that the
degrees of freedom are greatly reduced and the samples produced have natural
bond lengths and angles. The diffusion process is carried out on
$\mathbb{T}^{n}$, where a sample corresponds to a choice of torsional angles
for the side-chains. One notable change in the diffusion process is that the
angles are sampled autoregressively: $\chi_{1}$ is first sampled, followed by
$\chi_{2}$ and so on. Since the value of $\chi_{1}$ affects the coordinates of
atoms further down the chain, sampling all angles simultaneously can lead to
excess steric clashes due to the sensitivity of earlier angles on coordinates.
Using an autoregressive generation process mitigates this effect because the
model can self-correct for earlier choices of angles. A confidence model is
trained along side the diffusion model in order to score the sampled
conformations. During inference, multi-round sampling and annealed temperature
sampling are used to choose conformations with lower energies. DiffPack is
able to achieve state-of-the-art performance in the mean absolute error of the
predicted torsional angles when compared to other machine learning approaches
to the side-chain packing problem. Diffpack’s reported results are given in
tables 2 and 3. For details on the benchmarking, see section A.2.

Angle MAE ∘ $\downarrow$ Angle Accuracy % $\uparrow$ Atom RMSD Å $\downarrow$
Method $\chi_{1}$ $\chi_{2}$ $\chi_{3}$ $\chi_{4}$ All Core Surface All Core
Surface SCWRL 27.64 28.97 49.75 61.54 56.2% 71.3% 43.4% 0.934 0.495 1.027
FASPR 27.04 28.41 50.30 60.89 56.4% 70.3% 43.6% 0.910 0.502 1.002
RosettaPacker 25.88 28.25 48.13 59.82 58.6% 75.3% 35.7% 0.872 0.422 1.001
DLPacker 22.18 27.00 51.22 70.04 58.8% 73.9% 45.4% 0.772 0.402 0.876
AttnPacker 18.92 23.17 44.89 58.98 62.1% 73.7% 47.6% 0.669 0.366 0.775
DiffPack 15.35 19.19 37.30 50.19 69.5% 82.7% 57.3% 0.579 0.298 0.696

Table 2: Comparative evaluation of DiffPack and prior methods on CASP13 as
reported in [65].

Angle MAE ∘ $\downarrow$ Angle Accuracy % $\uparrow$ Atom RMSD Å $\downarrow$
Method $\chi_{1}$ $\chi_{2}$ $\chi_{3}$ $\chi_{4}$ All Core Surface All Core
Surface SCWRL 33.50 33.05 51.61 55.28 45.4% 62.5% 33.2% 1.062 0.567 1.216
FASPR 33.04 32.49 50.15 54.82 46.3% 62.4% 34.0% 1.048 0.594 1.205
RosettaPacker 31.79 28.25 50.54 56.16 47.5% 67.2% 33.5% 1.006 0.501 1.183
DLPacker 29.01 33.00 53.98 72.88 48.0% 66.9% 33.9% 0.929 0.476 1.107
AttnPacker 25.34 28.19 48.77 51.92 50.9% 66.2% 36.3% 0.823 0.438 1.001
DiffPack 21.91 25.54 44.27 55.03 57.5% 77.8% 43.5% 0.770 0.356 0.956

Table 3: Comparative evaluation of DiffPack and prior methods on CASP14 as
reported in [65].

A related problem to side-chain packing for apo protein structures is
determining the conformations of side-chains for ligand-docking problems. The
native structure of proteins do not account for the the complex flexibility of
proteins, which adapt their structure to their respective molecular binders.
Thus there is demand for methods which take this flexibility into account.
PackDock seeks to incorporate this flexibility to improve on ligand-protein
docking predictions [66]. The authors note that the dynamics of the protein-
ligand process are commonly described by two mechanisms: conformational
selection and induced fit. A torsional diffusion method – called PackPocket –
is introduced to sample multiple possible conformations for the side-chains,
either with or without the ligand. This is carried out in conjunction with a
docking algorithm to sample ligand conformations in the binding pocket (Vina
[100] is used by the authors, but any docking method can be used). PackDock
was shown to improve docking predictions when considering apo structures or
holo structures form non-homologous ligands.

####  4.2.2 Protein structure prediction

Diffusion models have also been applied to the problem of predicting the
native structure of proteins. In particular, EigenFold predicts the
coordinates of the C$\alpha$ atoms of a protein [67]. Diffusion is carried out
on the Cartesian space $\mathbb{R}^{3n}$, but the diffusion process is defined
so that the prior distribution uses a Harmonic potential for the relative
distances of points. This guarantees that diffused points do not separate too
far and that relative distances more closely reflect observed bond lengths.
Thus the prior distribution models more realistic molecular conformations, but
the stiffness of the diffusion process requires the SDE be carried out with a
basis of eigenvectors. An estimated ELBO is used _in lieu_ of training a score
model. The results reported in table 4 are modest compared to established
machine learning approaches such as AlphaFold2 and ESMFold [17, 69] (see
section A.3 for details). One benefit to using a generative model over a
single-structure prediction methods is that generative models can naturally
capture conformational diversity of a protein. If a protein is flexible, then
sampling the distribution of conformations should result in different possible
poses. Eigenfold shows some moderate results in this direction, with a slight
correlation between the diversity of samples and the flexibility of the
protein. Overall, the model demonstrates the possibilities of diffusion models
to the problem of predicting native structures.

| RMSD${}_{\text{C}\alpha}$ $\downarrow$ | TMScore $\uparrow$ | GDT-TS $\uparrow$ | lDDT${}_{\text{C}\alpha}$ $\uparrow$
---|---|---|---|---
AlphaFold2 | 3.30 / 1.64 | 0.87 / 0.95 | 0.86 / 0.91 | 0.90 / 0.93
ESMFold | 3.99 / 2.03 | 0.85 / 0.93 | 0.83 / 0.88 | 0.87 / 0.90
OmegaFold | 5.26 / 2.62 | 0.80 / 0.89 | 0.77 / 0.84 | 0.83 / 0.89
RoseTTAFold | 5.72 / 3.17 | 0.77 / 0.84 | 0.71 / 0.75 | 0.79 / 0.82
EigenFold | 7.37 / 3.50 | 0.75 / 0.84 | 0.71 / 0.79 | 0.78 / 0.85
Table 4: Single-structure prediction accuracy of EigenFold and baseline
methods on CAMEO targets under 750 residues from Aug 1–Oct 31, 2022. All
metrics are reported as mean / median. Table from [67]

A recent development in protein-structure prediction was the announcement of
AlphaFold 3 (AF3) [36]. The main improvement of AF3 over AF2 [17] and
AlphaFold-Multimer [101] is its capability of modeling general molecules. In
particular, the accelerated review paper cites state-of-the-art results in
ligand-protein complex predictions and advancements in protein-RNA modeling
compared to other automated methods. To accommodate the more general modeling
of biomolecular interactions, the generation process was switched to an all-
atom framework that can generalize the construction process of proteins and
other molecules that do not have amino acid nor nucleic acid residues. Since
frames and invariant point attention cannot be used in this case, the
structure module of AF2 was changed to a diffusion module in AF3. The
diffusion module takes the embedding information created by AF3 in order to
determine the atom positions of the proteins and other molecules, where the
generation process follows a standard iterative process of removing Gaussian
noise from a sample. One notable difference between AF3 and other applications
of diffusion to biomolecules is the lack of invariance/equivariance to
rotations and translations in the architecture. Instead, data augmentation is
used to achieve equivariance.

A related but narrower problem is that of backmapping: the task of determining
an all-atom configuration from a coarse-grained representation. Data-driven
approaches to the problem seek to accurately reconstruct the protein while
avoiding un-physical results such as clashes. While some methods are
deterministic, an additional goal would be to capture the diversity of
different possible all-atom configurations in a thermodynamically consistent
way. Previous state-of-the-art methods, used generative models such as GANs
[1] and variational auto-encoders (VAEs) [102]. In particular, GenZProt uses a
VAE to backmap a C$\alpha$ trace to a full-atom resolution while still
generalizing to molecules outside the training set. Researchers have begun to
use diffusion models to generate backmapping samples. DiAMoNDBack, for
instance, uses diffusion to backmap C$\alpha$ traces [62]. The backmapping is
done residue-by-residue in an autoregressive fashion, which lends to a more
transferable model. Another approach to this problem is BackDiff, which more
generally backmaps any coarse-grained representation [63]. Determining the
position of heavy-atoms given the coarse-grained atoms is treated as an
imputation problem, and training is done by randomly choosing heavy atoms
(apart from the C$\alpha$ atoms) to mask. Conditioning on auxiliary variables
and enforcing bond lengths and angles are handled during generation using a
manifold constraint. Both DiAMoNDBack and BackDiff are shown to outperform
GenZProt in terms of accuracy of full-atom reconstructions, diversity of
samples, and the avoidance of clashes and un-physical bond lengths and angles.

####  4.2.3 Inverse protein folding

As opposed to structural prediction methods, the inverse folding problem looks
at determining an amino acid sequence given a protein’s C$\alpha$ backbone.
GraDe-IF applies a discrete diffusion model to the inverse folding problem
[68]. The diffusion is carried out in the categorical space of amino acid
types using the D3PM framework [74]. Notably, the authors use the Blocks
Substitution Matrix (BLOSUM) [103] to create the transition matrix for the
noising step. This allows the model to explore closely related amino acids
during the latter stage of the denoising process instead of random, unrelated
residue types. The score network is given by an adaptation of an equivariant
graph neural network (EGNN) [38] to account for the roto-translational
invariance of the problem. The model outperforms other inverse folding methods
in the authors’ benchmarking as reported in table 5. Details of the
benchmarking can be found in section A.4.

Model | Perplexity $\downarrow$ | Recovery Rate % $\uparrow$ | CATH version
---|---|---|---
Short | Single-chain | All | Short | Single-chain | All | 4.2 | 4.3
StructGNN | 8.29 | 8.74 | 6.40 | 29.44 | 28.26 | 35.91 | $\checkmark$ |
GraphTrans | 8.39 | 8.83 | 6.63 | 28.14 | 28.46 | 35.82 | $\checkmark$ |
GCA | 7.09 | 7.49 | 6.05 | 32.62 | 31.10 | 37.64 | $\checkmark$ |
GVP | 7.23 | 7.84 | 5.36 | 30.60 | 28.95 | 39.47 | $\checkmark$ |
GVP-large | 7.68 | 6.12 | 6.17 | 32.6 | 39.4 | 39.2 |  | $\checkmark$
AlphaDesign | 7.32 | 7.63 | 6.30 | 34.16 | 32.66 | 41.31 | $\checkmark$ |
ESM-if1 | 8.18 | 6.33 | 6.44 | 31.3 | 38.5 | 38.3 |  | $\checkmark$
ProteinMPNN | 6.21 | 6.68 | 4.57 | 36.35 | 34.43 | 49.87 | $\checkmark$ |
PiFold | $6.04$ | 6.31 | $4.55$ | $39.84$ | 38.53 | 51.66 | $\checkmark$ |
GraDe-IF | $\mathbf{5.49}$ | $\mathbf{6.21}$ | $\mathbf{4.35}$ | $\mathbf{45.27}$ | $\mathbf{42.77}$ | $\mathbf{52.21}$ | $\checkmark$ |

Table 5: Recovery rate performance of CATH on zero-shot models. Table taken
from [68]

####  4.2.4 Protein complexes

Ligands bind to proteins and modulate their biological function, so an
important step in drug design is determining where a ligand will bind to a
protein. Traditionally, computational methods for this problem search for the
ligand position that directly optimizes a score function [100, 14, 104].
However, due to the high-dimensionality of the space and the roughness of the
energy landscape, this process can be slow and give inaccurate results.
Machine learning approaches that treat ligand-binding as a regression problem
significantly speed up the process but still suffer from inaccuracy due to the
uncertainty in the energy landscape [105, 106]. DiffDock [33] instead treats
the finding the ligand pose as a generative problem. Put simply, DiffDock
samples possible ligand poses from an underlying distribution and then makes
an optimal choice based on a learned confidence score. This circumvents the
ruggedness of the energy landscape by enumerating multiple options instead of
averaging possibilities together as done in regression-type approaches.
DiffDock uses a diffusion model to sample different poses and a separate score
model to determine the confidence that the RMSD is below 2 Å. The
dimensionality of the search space is reduced by fixing the conformation of
the protein and assuming ideal bond lengths and bond angles in the ligand.
Thus the degrees of freedom are the position, orientation, and dihedral angles
of the ligand. Then sampling a ligand pose is the same as sampling from the
manifold $\mathbb{R}^{3}\times\mathrm{SO}(3)\times\mathbb{T}^{m}$, where $m$
is the number of torsional angles. DiffDock was benchmarked on the PDBBind
dataset [107] with a time split at 2019 for training and validation. On this
data set, DiffDock reported state-of-the-art performance, and it retained high
accuracy on computationally obtained structures. FusionDock augments
DiffDock’s approach by using physics-informed priors [60].

However, the reported results tell an incomplete story of DiffDock’s strength.
Some noted that the comparison to classical docking methods is unfair since
traditional methods were not designed for blind docking and typically require
a designated pocket to perform well [108]. When given a binding pocket,
traditional methods outperform deep learning methods. But experiments
demonstrated that DiffDock still excels at pocket-finding. Others have pointed
out that the typical time split in PDBBind can lead to data leakage in the
validation; some of the structures in the training set are similar to those in
the validation set, leading to an overestimation of a model’s performance
[109, 110]. Finally, many of the predictions of deep learning docking methods
are physically implausible, and when accounting for docking predictions that
are within 2 Å of the true structure _and_ realistic, deep learning methods
perform substantially worse than classical methods [111]. Recently, Corso et
al. created a new benchmark set, DockGen, that removes the data leakage in
terms of structure and pocket similarlity [59]. In addition, they introduce
DiffDock-L, a larger model trained on a bigger, augmented data set. On both
the PDBBind and DockGen benchmarks, DiffDock-L outperforms all other blind
docking methods. Table 6 details the results of DiffDock-L. More information
on benchmarking is given in section A.5.

| PDBBind |  DockGen-full |  DockGen-clusters | Average
---|---|---|---|---
Method |  %$<$2Å | Med. |  %$<$2Å | Med. |  %$<$2Å | Med. | Runtime (s)
SMINA | 18.7 | 7.1 | 7.9 | 13.8 | 2.4 | 16.4 | 126*
SMINA (ex. 64) | 25.4 | 5.5 | 10.6 | 13.5 | 4.7 | 14.7 | 347*
P2Rank+SMINA | 20.4 | 4.3 | 7.9 | 14.1 | 1.2 | 16.4 | 126*
GNINA | 22.9 | 7.7 | 14.3 | 15.2 | 9.4 | 14.5 | 127
GNINA (ex. 64) | 32.1 | 4.2 | 17.5 | 8.1 | 11.8 | 6.2 | 348
P2Rank+GNINA | 28.8 | 4.9 | 13.8 | 16.2 | 4.7 | 15.3 | 127
EquiBind | 5.5 | 6.2 | 0.0 | 13.3 | 0.0 | 13.3 | 0.04
TANKBind | 20.4 | 4.0 | 0.5 | 11.6 | 0.0 | 11.1 | 0.7
DiffDock (10) | 35.0 | 3.6 | 7.1 | 6.8 | 6.1 | 6.0 | 10
DiffDock (40) | 38.2 | 3.3 | 6.0 | 7.3 | 3.7 | 6.7 | 40
DiffDock-L (10) | 43.0 | 2.8 | 22.6 | 4.3 | 27.6 | 3.7 | 25
Table 6:  Top-1 RMSD performance of different methods on the PDBBind and
DockGen benchmarks. Runtimes were computed as averages over the PDBBind test
set. * run on CPU. Med. indicates the median RMSD. Table from [59].

A similar framework was applied to the protein-protein docking problem with
DiffDock-PP [64]. The diffusion process from DiffDock is adapted by assuming
that the proteins are rigid bodies so that diffusion takes place on
$\mathbb{R}^{3}\times\mathrm{SO}(3)$ with torsional fixed. Benchmarking showed
improvements in accuracy compared to other machine learning methods while
being much faster than traditional search methods when computed on GPU.

NeuralPLexer is another diffusion-based method that predicts ligand-protein
docking complexes [34]. Unlike previous docking prediction models, this method
does not rely on the protein structure as an input but only takes the protein
sequence and ligand molecular graph. NeuralPLexer is comprised of a graph-
based architecture to encode molecular properties, a contact prediction module
that determines the inter-molecular distances, and an equivariant diffusion
module that determines the structure of the complex at an atomic resolution.
When applied to the blind ligand-protein docking problem (a simplification of
the full complex prediction), NeuralPLexer outperformed other methods like
DiffDock.The model also outperforms AlphaFold2 on protein structure prediction
for structures with high flexibility and newly added protein-ligand
complexes.444A recent press release claims improved performance for the newly
trained NeuralPLexer2, although details have yet to be released.

A similar model to NeuralPLexer is the Diffusion model for Protein-Ligand
complexes (DPL) [61]. Like NeuralPLexer, it does not take protein structure as
an input, but only relies on the protein sequence and ligand molecular graph.
Furthermore, DPL does not rely on protein backbone templates during
prediction. The inputs are embedded into a single and pair representation with
information from a protein language model (ESM-2) [76] and these
representations are updated with Folding blocks from ESMFold [69]. Finally,
and equivariant denoising network is used to generate the complex. DPL is able
to generate diverse sets of complexes, but performance is limited for
complexes with less data.

##  5 Conclusion

Diffusion models have made remarkable progress in the design and prediction of
proteins, with many recent models quickly achieving state-of-the-art
performance on tasks. Protein backbone generation methods, such as
RFdiffusion, have demonstrated incredible success at sampling realistic and
novel structures that have been validated experimentally [35]. In addition,
the ability to conditionally generate structures opens up new avenues for
designing proteins with desired properties. Diffusion models have also seen
great success with other generation tasks, such as sequence generation/design
and ensemble sampling.

They have also seen success in predictive tasks. Generative models are better
at sampling many possible modes in a distribution (as opposed to regression
models which average over possibilities leading to unrealistic results), which
gives them an edge in some predictive tasks. Diffusion models in particular
seem well suited to searching a large-dimensional, continuous space, as
partially evidenced by DiffDock’s ability to find pockets for protein-ligand
docking [108].

One direction that we believe remains underexplored is the use of diffusion
models for generating and predicting DNA/RNA structures. As seen with the
results with proteins, diffusion models provide a potentially potent method
for solving problems in this domain. Attempts at structure prediction has led
to mild results [53]. Some success has been achieved with jointly modeling
them with proteins ([36], for instance) where deep knowledge of protein
structure can be leveraged, but there has been little success in monomeric
prediction of these biomolecules alone. To this end, there are some challenges
to directly applying diffusion methods: (1) there is relatively less data for
nucleic acids as compared to proteins, which makes training more difficult;
and (2) RNA structures are more flexible, which mirrors the difficulty with
certain protein generation methods from creating intrinsically disordered
regions.

It seems certain that in the coming years there will be more attention paid to
diffusion models for applications involving biomolecules, but they are not
without competitors in the space. Inspired by the success of diffusion,
researchers have proposed other generative deep learning methods that take an
iterative approach to generation. For instance, Bayesian flow networks (BFNs)
are generative models which iteratively samples noised distributions in a
similar way to diffusion models [112]. The sampling process consists of
continually applying Bayes’ rule to flow toward a desired distribution to
sample. BFNs naturally lend themselves to multimodal problems and have been
used in molecular generation [113]. A more popular approach to generative
modeling has been flow matching [114]. Flow matching is a subset of continuous
normalizing flows where the diffeomorphism between probability distributions
is given by the pushforward of a learned vector field that is trained with
conditional flow matching. Sampling then proceeds by solving the differential
equation given by the vector field. An advantage of this method over diffusion
models is reduced training and sampling time. Flow matching has seen success
in backbone generation [73, 115] and sampling protein ensembles [116].

Regardless of the exact shape of the field in the coming years, it is
undeniable that diffusion models and similar frameworks will continue to be
used in the design and prediction of biomolecules. They have introduced
scalable, generative modeling to the field and have already demonstrated their
advantages over previous traditional and machine-learning approaches by
setting new state-of-the-art results in numerous problems.

##  6 Acknowledgments

This work was partially supported by the National Institute of General Medical
Sciences [R35GM138146 to D.B.] and the National Science Foundation [DBI2208679
to D.B.].

## References

  * [1] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [2] Prafulla Dhariwal and Alex Nichol.  Diffusion models beat GANs on image synthesis, June 2021.  arXiv:2105.05233 [cs, stat].
  * [3] Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor Ionescu, and Mubarak Shah.  Diffusion models in vision: A survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023\.
  * [4] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov, and Jiansheng Wei.  Diffusion-based voice conversion with fast maximum likelihood sampling scheme.  arXiv preprint arXiv:2109.13821, 2021.
  * [5] Shoule Wu and Ziqiang Shi.  ItôTTS and ItôWave: Linear stochastic differential equation is all you need for audio generation.  arXiv preprint arXiv:2105.07583, 2021.
  * [6] Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, and Zhou Zhao.  Diffsinger: Singing voice synthesis via shallow diffusion mechanism.  In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 11020–11028, 2022.
  * [7] Joao Carvalho, Mark Baierl, Julen Urain, and Jan Peters.  Conditioned score-based models for learning collision-free trajectory generation.  In NeurIPS 2022 Workshop on Score-Based Methods, 2022.
  * [8] Joao Carvalho, An T Le, Mark Baierl, Dorothea Koert, and Jan Peters.  Motion planning diffusion: Learning and planning of robot motions with diffusion models.  In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1916–1923. IEEE, 2023.
  * [9] Ivan Kapelyukh, Vitalis Vosylius, and Edward Johns.  Dall-e-bot: Introducing web-scale diffusion models to robotics.  IEEE Robotics and Automation Letters, 2023.
  * [10] Julen Urain, Niklas Funk, Jan Peters, and Georgia Chalvatzaki.  SE (3)-diffusionfields: Learning smooth cost functions for joint grasp and motion optimization through diffusion.  In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5923–5930. IEEE, 2023.
  * [11] Edmund Findlay, Haozheng Zhang, Ziyi Chang, and Hubert PH Shum.  Denoising diffusion probabilistic models for styled walking synthesis.  2022\.
  * [12] Sidhartha Chaudhury, Sergey Lyskov, and Jeffrey J Gray.  PyRosetta: a script-based interface for implementing molecular modeling algorithms using rosetta.  Bioinformatics, 26(5):689–691, 2010.
  * [13] David Ryan Koes, Matthew P Baumgartner, and Carlos J Camacho.  Lessons learned in empirical scoring with smina from the CSAR 2011 benchmarking exercise.  Journal of chemical information and modeling, 53(8):1893–1904, 2013\.
  * [14] Richard A Friesner, Jay L Banks, Robert B Murphy, Thomas A Halgren, Jasna J Klicic, Daniel T Mainz, Matthew P Repasky, Eric H Knoll, Mee Shelley, Jason K Perry, et al.  Glide: a new approach for rapid, accurate docking and scoring. 1. method and assessment of docking accuracy.  Journal of medicinal chemistry, 47(7):1739–1749, 2004.
  * [15] Nana Heilmann, Moritz Wolf, Mariana Kozlowska, Elaheh Sedghamiz, Julia Setzler, Martin Brieg, and Wolfgang Wenzel.  Sampling of the conformational landscape of small proteins with monte carlo methods.  Scientific reports, 10(1):18211, 2020.
  * [16] Cyrus Levinthal.  How to fold graciously.  Mossbauer spectroscopy in biological systems, 67:22–24, 1969.
  * [17] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Žídek, and Anna Potapenko.  Highly accurate protein structure prediction with AlphaFold.  Nature, 596(7873):583–589, 2021.  Publisher: Nature Publishing Group.
  * [18] Jianfeng Lu, Zuowei Shen, Haizhao Yang, and Shijun Zhang.  Deep network approximation for smooth functions.  SIAM Journal on Mathematical Analysis, 53(5):5465–5506, 2021.
  * [19] Radford M Neal.  Annealed importance sampling.  Statistics and computing, 11:125–139, 2001.
  * [20] Christopher Jarzynski.  Equilibrium free-energy differences from nonequilibrium measurements: A master-equation approach.  Physical Review E, 56(5):5018, 1997.
  * [21] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In Proceedings of the 32nd International Conference on Machine Learning, pages 2256–2265. PMLR, June 2015.  ISSN: 1938-7228.
  * [22] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In Advances in Neural Information Processing Systems, volume 33, pages 6840–6851. Curran Associates, Inc., 2020.
  * [23] Yang Song and Stefano Ermon.  Generative modeling by estimating gradients of the data distribution.  In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [24] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations, November 2020.
  * [25] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon.  Sliced score matching: A scalable approach to density and score estimation.  In Uncertainty in Artificial Intelligence, pages 574–584. PMLR, 2020.
  * [26] Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang.  Geodiff: A geometric diffusion model for molecular conformation generation.  arXiv preprint arXiv:2203.02923, 2022.
  * [27] Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang.  Learning gradient fields for molecular conformation generation.  In International conference on machine learning, pages 9558–9568. PMLR, 2021.
  * [28] Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton, Yee Whye Teh, and Arnaud Doucet.  Riemannian score-based generative modelling, November 2022.  arXiv:2202.02763 [cs, math, stat].
  * [29] Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola.  Torsional diffusion for molecular conformer generation, February 2023\.  arXiv:2206.01729 [physics, q-bio].
  * [30] Chin-Wei Huang, Milad Aghajohari, Joey Bose, Prakash Panangaden, and Aaron C Courville.  Riemannian diffusion models.  Advances in Neural Information Processing Systems, 35:2750–2761, 2022.
  * [31] Namrata Anand and Tudor Achim.  Protein structure and sequence generation with equivariant denoising diffusion probabilistic models, May 2022.  arXiv:2205.15019 [cs, q-bio].
  * [32] Brian L. Trippe, Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi Jaakkola.  Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problem, March 2023.  arXiv:2206.04119 [cs, q-bio, stat].
  * [33] Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi Jaakkola.  DiffDock: Diffusion steps, twists, and turns for molecular docking, February 2023.  arXiv:2210.01776 [physics, q-bio] version: 2.
  * [34] Zhuoran Qiao, Weili Nie, Arash Vahdat, Thomas F. Miller III, and Anima Anandkumar.  State-specific protein-ligand complex structure prediction with a multi-scale deep generative model, April 2023.  arXiv:2209.15171 [cs, q-bio].
  * [35] Joseph L. Watson, David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, Andrew J. Borst, Robert J. Ragotte, Lukas F. Milles, Basile I. M. Wicky, Nikita Hanikel, Samuel J. Pellock, Alexis Courbet, William Sheffler, Jue Wang, Preetham Venkatesh, Isaac Sappington, Susana Vázquez Torres, Anna Lauko, Valentin De Bortoli, Emile Mathieu, Sergey Ovchinnikov, Regina Barzilay, Tommi S. Jaakkola, Frank DiMaio, Minkyung Baek, and David Baker.  De novo design of protein structure and function with RFdiffusion.  Nature, 620(7976):1089–1100, August 2023.  Number: 7976 Publisher: Nature Publishing Group.
  * [36] Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al.  Accurate structure prediction of biomolecular interactions with AlphaFold 3.  Nature, pages 1–3, 2024.
  * [37] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley.  Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds.  arXiv preprint arXiv:1802.08219, 2018.
  * [38] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling.  E(n) Equivariant Graph Neural Networks.  In Proceedings of the 38th International Conference on Machine Learning, pages 9323–9332. PMLR, July 2021.  ISSN: 2640-3498.
  * [39] Dmitry I. Nikolayev and Tatjana I. Savyolov.  Normal distribution on the rotation group SO (3).  Texture, Stress, and Microstructure, 29:201–233, 1997.  Publisher: Hindawi.
  * [40] Diederik P. Kingma and Prafulla Dhariwal.  Glow: Generative flow with invertible 1x1 convolutions, July 2018.  arXiv:1807.03039 [cs, stat].
  * [41] John B. Ingraham, Max Baranov, Zak Costello, Karl W. Barber, Wujie Wang, Ahmed Ismail, Vincent Frappier, Dana M. Lord, Christopher Ng-Thow-Hing, Erik R. Van Vlack, Shan Tie, Vincent Xue, Sarah C. Cowles, Alan Leung, João V. Rodrigues, Claudio L. Morales-Perez, Alex M. Ayoub, Robin Green, Katherine Puentes, Frank Oplinger, Nishant V. Panwar, Fritz Obermeyer, Adam R. Root, Andrew L. Beam, Frank J. Poelwijk, and Gevorg Grigoryan.  Illuminating protein space with a programmable generative model.  Nature, 623(7989):1070–1078, November 2023.  Publisher: Nature Publishing Group.
  * [42] Rohith Krishna, Jue Wang, Woody Ahern, Pascal Sturmfels, Preetham Venkatesh, Indrek Kalvet, Gyu Rie Lee, Felix S Morey-Burrows, Ivan Anishchenko, Ian R Humphreys, et al.  Generalized biomolecular modeling and design with RoseTTAFold all-atom.  Science, 384(6693):eadl2528, 2024.
  * [43] Yeqing Lin and Mohammed AlQuraishi.  Generating novel, designable, and diverse protein structures by equivariantly diffusing oriented residue clouds, June 2023.  arXiv:2301.12485 [cs, q-bio].
  * [44] Kevin E. Wu, Kevin K. Yang, Rianne van den Berg, Sarah Alamdari, James Y. Zou, Alex X. Lu, and Ava P. Amini.  Protein structure generation via folding diffusion.  Nature Communications, 15(1):1059, February 2024.  Publisher: Nature Publishing Group.
  * [45] Jin Sub Lee, Jisun Kim, and Philip M. Kim.  Score-based generative modeling for de novo protein design.  Nature Computational Science, 3(5):382–392, May 2023.  Number: 5 Publisher: Nature Publishing Group.
  * [46] Jason Yim, Brian L. Trippe, Valentin De Bortoli, Emile Mathieu, Arnaud Doucet, Regina Barzilay, and Tommi Jaakkola.  SE(3) diffusion model with application to protein backbone generation, May 2023.  arXiv:2302.02277 [cs, q-bio, stat].
  * [47] Cong Fu, Keqiang Yan, Limei Wang, Wing Yee Au, Michael McThrow, Tao Komikado, Koji Maruhashi, Kanji Uchino, Xiaoning Qian, and Shuiwang Ji.  A latent diffusion model for protein structure generation, May 2023.  arXiv:2305.04120 [cs, q-bio].
  * [48] Cheng Zhang, Adam Leach, Thomas Makkink, Miguel Arbesú, Ibtissem Kadri, Daniel Luo, Liron Mizrahi, Sabrine Krichen, Maren Lang, Andrey Tovchigrechko, Nicolas Lopez Carranza, Uğur Şahin, Karim Beguir, Michael Rooney, and Yunguan Fu.  FrameDiPT: SE(3) diffusion model for protein structure inpainting, January 2024.  Pages: 2023.11.21.568057 Section: New Results.
  * [49] Jiarui Lu, Bozitao Zhong, Zuobai Zhang, and Jian Tang.  Str2Str: A score-based framework for zero-shot protein conformation sampling.  October 2023.
  * [50] Shuxin Zheng, Jiyan He, Chang Liu, Yu Shi, Ziheng Lu, Weitao Feng, Fusong Ju, Jiaxi Wang, Jianwei Zhu, Yaosen Min, et al.  Predicting equilibrium distributions for molecular systems with deep learning.  Nature Machine Intelligence, pages 1–10, 2024.
  * [51] Marloes Arts, Victor Garcia Satorras, Chin-Wei Huang, Daniel Zügner, Marco Federici, Cecilia Clementi, Frank Noé, Robert Pinsler, and Rianne van den Berg.  Two for one: Diffusion models and force fields for coarse-grained molecular dynamics.  Journal of Chemical Theory and Computation, 19(18):6151–6159, 2023\.
  * [52] Alexander E. Chu, Lucy Cheng, Gina El Nesr, Minkai Xu, and Po-Ssu Huang.  An all-atom protein generative model, May 2023.  Pages: 2023.05.24.542194 Section: New Results.
  * [53] Alex Morehead, Jeffrey Ruffolo, Aadyot Bhatnagar, and Ali Madani.  Towards joint sequence-structure generation of nucleic acid and protein complexes with SE(3)-discrete diffusion, December 2023.  arXiv:2401.06151 [cs, q-bio].
  * [54] Zehui Li, Yuhao Ni, Tim August B. Huygelen, Akashaditya Das, Guoxuan Xia, Guy-Bart Stan, and Yiren Zhao.  Latent diffusion model for DNA sequence generation, December 2023.  arXiv:2310.06150 [cs].
  * [55] Sarah Alamdari, Nitya Thakkar, Rianne van den Berg, Alex X. Lu, Nicolo Fusi, Ava P. Amini, and Kevin K. Yang.  Protein generation with evolutionary diffusion: sequence is all you need, September 2023.  Pages: 2023.09.11.556673 Section: New Results.
  * [56] Sidney Lyayuga Lisanza, Jake Merle Gershon, Sam Tipps, Lucas Arnoldt, Samuel Hendel, Jeremiah Nelson Sims, Xinting Li, and David Baker.  Joint generation of protein sequence and structure with RoseTTAFold sequence space diffusion, May 2023.  Pages: 2023.05.08.539766 Section: New Results.
  * [57] Bo Ni, David L. Kaplan, and Markus J. Buehler.  ForceGen: End-to-end de novo protein generation based on nonlinear mechanical unfolding responses using a protein language diffusion model, October 2023.  arXiv:2310.10605 [cond-mat, q-bio].
  * [58] Bingxin Zhou, Lirong Zheng, Banghao Wu, Kai Yi, Bozitao Zhong, Pietro Liò, and Liang Hong.  Conditional protein denoising diffusion generates programmable endonucleases, August 2023.  Pages: 2023.08.10.552783 Section: New Results.
  * [59] Gabriele Corso, Arthur Deng, Benjamin Fry, Nicholas Polizzi, Regina Barzilay, and Tommi Jaakkola.  Deep confident steps to new pockets: Strategies for docking generalization.  arXiv preprint arXiv:2402.18396, 2024.
  * [60] Matthew R Masters, Amr H Mahmoud, and Markus A Lill.  Fusiondock: physics-informed diffusion model for molecular docking.  In ICML workshop on Computatioal Biology, 2023.
  * [61] Shuya Nakata, Yoshiharu Mori, and Shigenori Tanaka.  End-to-end protein–ligand complex structure generation with diffusion-based generative models.  BMC Bioinformatics, 24(1):233, June 2023.
  * [62] Michael S Jones, Kirill Shmilovich, and Andrew L Ferguson.  DiAMoNDBack: Diffusion-denoising autoregressive model for non-deterministic backmapping of c$\alpha$ protein traces.  Journal of Chemical Theory and Computation, 19(21):7908–7923, 2023\.
  * [63] Yikai Liu, Guang Lin, and Ming Chen.  Backdiff: a diffusion model for generalized transferable protein backmapping.  2023\.
  * [64] Mohamed Amine Ketata, Cedrik Laue, Ruslan Mammadov, Hannes Stärk, Menghua Wu, Gabriele Corso, Céline Marquet, Regina Barzilay, and Tommi S. Jaakkola.  DiffDock-PP: Rigid protein-protein docking with diffusion models, April 2023.  arXiv:2304.03889 [cs, q-bio].
  * [65] Yangtian Zhang, Zuobai Zhang, Bozitao Zhong, Sanchit Misra, and Jian Tang.  DiffPack: A torsional diffusion model for autoregressive protein side-chain packing, June 2023.  arXiv:2306.01794 [cs, q-bio].
  * [66] Runze Zhang, Xinyu Jiang, Duanhua Cao, Jie Yu, Mingan Chen, Zhehuan Fan, Xiangtai Kong, Jiacheng Xiong, Zimei Zhang, Wei Zhang, et al.  PackDock: a diffusion based side chain packing model for flexible protein-ligand docking.  bioRxiv, pages 2024–01, 2024.
  * [67] Bowen Jing, Ezra Erives, Peter Pao-Huang, Gabriele Corso, Bonnie Berger, and Tommi Jaakkola.  EigenFold: Generative protein structure prediction with diffusion models, April 2023.  arXiv:2304.02198 [physics, q-bio].
  * [68] Kai Yi, Bingxin Zhou, Yiqing Shen, Pietro Lió, and Yuguang Wang.  Graph denoising diffusion for inverse protein folding.  Advances in Neural Information Processing Systems, 36:10238–10257, December 2023.
  * [69] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al.  Evolutionary-scale prediction of atomic-level protein structure with a language model.  Science, 379(6637):1123–1130, 2023.
  * [70] Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al.  Robust deep learning–based protein sequence design using ProteinMPNN.  Science, 378(6615):49–56, 2022.
  * [71] Jianyi Yang, Ivan Anishchenko, Hahnbeom Park, Zhenling Peng, Sergey Ovchinnikov, and David Baker.  Improved protein structure prediction using predicted interresidue orientations.  Proceedings of the National Academy of Sciences, 117(3):1496–1503, 2020.
  * [72] Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, R Dustin Schaeffer, et al.  Accurate prediction of protein structures and interactions using a three-track neural network.  Science, 373(6557):871–876, 2021.
  * [73] Avishek Joey Bose, Tara Akhound-Sadegh, Kilian Fatras, Guillaume Huguet, Jarrid Rector-Brooks, Cheng-Hao Liu, Andrei Cristian Nica, Maksym Korablyov, Michael Bronstein, and Alexander Tong.  SE (3)-stochastic flow matching for protein backbone generation.  arXiv preprint arXiv:2310.02391, 2023.
  * [74] Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg.  Structured denoising diffusion models in discrete state-spaces, February 2023.  arXiv:2107.03006 [cs].
  * [75] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost.  ProtTrans: Toward understanding the language of life through self-supervised learning.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10):7112–7127, October 2022.  Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.
  * [76] Robert Verkuil, Ori Kabeli, Yilun Du, Basile I. M. Wicky, Lukas F. Milles, Justas Dauparas, David Baker, Sergey Ovchinnikov, Tom Sercu, and Alexander Rives.  Language models generalize beyond natural proteins, December 2022.  Pages: 2022.12.21.521521 Section: New Results.
  * [77] Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al.  High-resolution de novo structure prediction from primary sequence.  BioRxiv, pages 2022–07, 2022.
  * [78] Bingxin Zhou, Lirong Zheng, Banghao Wu, Kai Yi, Bozitao Zhong, Pietro Lio, and Liang Hong.  Conditional protein denoising diffusion generates programmable endonucleases.  bioRxiv, pages 2023–08, 2023.
  * [79] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  * [80] Michael Feig and Vahid Mirjalili.  Protein structure refinement via molecular-dynamics simulations: what works and what does not?  Proteins: Structure, Function, and Bioinformatics, 84:282–292, 2016\.
  * [81] Michael Feig.  Computational protein structure refinement: almost there, yet still so far to go.  Wiley Interdisciplinary Reviews: Computational Molecular Science, 7(3):e1307, 2017.
  * [82] Lim Heo and Michael Feig.  Prefmd: a web server for protein structure refinement via molecular dynamics simulations.  Bioinformatics, 34(6):1063–1065, 2018.
  * [83] Lim Heo and Michael Feig.  Experimental accuracy in protein structure refinement via molecular dynamics simulations.  Proceedings of the National Academy of Sciences, 115(52):13276–13281, 2018.
  * [84] Vahid Mirjalili and Michael Feig.  Protein structure refinement through structure selection and averaging from molecular dynamics ensembles.  Journal of chemical theory and computation, 9(2):1294–1303, 2013\.
  * [85] Vahid Mirjalili, Keenan Noyes, and Michael Feig.  Physics-based protein structure refinement through multiple molecular dynamics trajectories and structure averaging.  Proteins: Structure, Function, and Bioinformatics, 82:196–207, 2014\.
  * [86] Kresten Lindorff-Larsen, Stefano Piana, Ron O Dror, and David E Shaw.  How fast-folding proteins fold.  Science, 334(6055):517–520, 2011.
  * [87] Fang Wu and Stan Z. Li.  DiffMD: A geometric diffusion model for molecular dynamics simulations.  Proceedings of the AAAI Conference on Artificial Intelligence, 37(4):5321–5329, June 2023.  Number: 4.
  * [88] Tim Hsu, Babak Sadigh, Vasily Bulatov, and Fei Zhou.  Score dynamics: scaling molecular dynamics with picosecond timesteps via conditional diffusion model, October 2023.  arXiv:2310.01678 [physics].
  * [89] Jonas Köhler, Yaoyi Chen, Andreas Krämer, Cecilia Clementi, and Frank Noé.  Flow-matching – efficient coarse-graining of molecular dynamics without forces.  Journal of Chemical Theory and Computation, 19(3):942–952, February 2023.  arXiv:2203.11167 [physics].
  * [90] Xiaoqiang Huang, Robin Pearce, and Yang Zhang.  FASPR: an open-source tool for fast and accurate protein side-chain packing.  Bioinformatics, 36(12):3758–3765, 2020.
  * [91] Chen Yanover, Ora Schueler-Furman, and Yair Weiss.  Minimizing and learning energy functions for side-chain prediction.  Journal of Computational Biology, 15(7):899–911, 2008.
  * [92] Shide Liang, Dandan Zheng, Chi Zhang, and Daron M Standley.  Fast and accurate prediction of protein side-chain conformations.  Bioinformatics, 27(20):2913–2914, 2011.
  * [93] Aleksandra E Badaczewska-Dawid, Andrzej Kolinski, and Sebastian Kmiecik.  Computational reconstruction of atomistic protein structures from coarse-grained models.  Computational and structural biotechnology journal, 18:162–176, 2020.
  * [94] Rebecca F Alford, Andrew Leaver-Fay, Jeliazko R Jeliazkov, Matthew J O’Meara, Frank P DiMaio, Hahnbeom Park, Maxim V Shapovalov, P Douglas Renfrew, Vikram K Mulligan, Kalli Kappel, et al.  The Rosetta all-atom energy function for macromolecular modeling and design.  Journal of chemical theory and computation, 13(6):3031–3048, 2017\.
  * [95] Georgii G Krivov, Maxim V Shapovalov, and Roland L Dunbrack Jr.  Improved prediction of protein side-chain conformations with SCWRL4.  Proteins: Structure, Function, and Bioinformatics, 77(4):778–795, 2009.
  * [96] Jinbo Xu and Bonnie Berger.  Fast and accurate algorithms for protein side-chain packing.  Journal of the ACM (JACM), 53(4):533–557, 2006.
  * [97] Yang Cao, Lin Song, Zhichao Miao, Yun Hu, Liqing Tian, and Taijiao Jiang.  Improved side-chain modeling by coupling clash-detection guided iterative search with rotamer relaxation.  Bioinformatics, 27(6):785–790, 2011.
  * [98] Matt McPartlon and Jinbo Xu.  An end-to-end deep learning method for rotamer-free protein side-chain packing.  bioRxiv, pages 2022–03, 2022.  Publisher: Cold Spring Harbor Laboratory.
  * [99] Mikita Misiura, Raghav Shroff, Ross Thyer, and Anatoly B Kolomeisky.  DLPacker: Deep learning for prediction of amino acid side chain conformations in proteins.  Proteins: Structure, Function, and Bioinformatics, 90(6):1278–1290, 2022.
  * [100] Oleg Trott and Arthur J Olson.  AutoDock Vina: improving the speed and accuracy of docking with a new scoring function, efficient optimization, and multithreading.  Journal of computational chemistry, 31(2):455–461, 2010.
  * [101] Richard Evans, Michael O’Neill, Alexander Pritzel, Natasha Antropova, Andrew Senior, Tim Green, Augustin Žídek, Russ Bates, Sam Blackwell, Jason Yim, et al.  Protein complex prediction with AlphaFold-Multimer.  biorxiv, pages 2021–10, 2021.
  * [102] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [103] Steven Henikoff and Jorja G Henikoff.  Amino acid substitution matrices from protein blocks.  Proceedings of the National Academy of Sciences, 89(22):10915–10919, 1992.
  * [104] René Thomsen and Mikael H Christensen.  MolDock: a new technique for high-accuracy molecular docking.  Journal of medicinal chemistry, 49(11):3315–3321, 2006.
  * [105] Hannes Stärk, Octavian Ganea, Lagnajit Pattanaik, Regina Barzilay, and Tommi Jaakkola.  Equibind: Geometric deep learning for drug binding structure prediction.  In International conference on machine learning, pages 20503–20521. PMLR, 2022.
  * [106] Wei Lu, Qifeng Wu, Jixian Zhang, Jiahua Rao, Chengtao Li, and Shuangjia Zheng.  Tankbind: Trigonometry-aware neural networks for drug-protein binding structure prediction.  Advances in neural information processing systems, 35:7236–7249, 2022.
  * [107] Zhihai Liu, Minyi Su, Li Han, Jie Liu, Qifan Yang, Yan Li, and Renxiao Wang.  Forging the basis for developing protein–ligand interaction scoring functions.  Accounts of chemical research, 50(2):302–309, 2017.
  * [108] Yuejiang Yu, Shuqi Lu, Zhifeng Gao, Hang Zheng, and Guolin Ke.  Do deep learning models really outperform traditional approaches in molecular docking?  arXiv preprint arXiv:2302.07134, 2023.
  * [109] Ganesh Chandan Kanakala, Rishal Aggarwal, Divya Nayar, and U Deva Priyakumar.  Latent biases in machine learning models for predicting binding affinities using popular data sets.  ACS omega, 8(2):2389–2397, 2023.
  * [110] Jie Li, Xingyi Guan, Oufan Zhang, Kunyang Sun, Yingze Wang, Dorian Bagni, and Teresa Head-Gordon.  Leak proof PDBBind: A reorganized dataset of protein-ligand complexes for more generalizable binding affinity prediction.  ArXiv, 2023.
  * [111] Martin Buttenschoen, Garrett M Morris, and Charlotte M Deane.  PoseBusters: AI-based docking methods fail to generate physically valid poses or generalise to novel sequences.  Chemical Science, 2024.
  * [112] Alex Graves, Rupesh Kumar Srivastava, Timothy Atkinson, and Faustino Gomez.  Bayesian flow networks.  arXiv preprint arXiv:2308.07037, 2023.
  * [113] Yuxuan Song, Jingjing Gong, Hao Zhou, Mingyue Zheng, Jingjing Liu, and Wei-Ying Ma.  Unified generative modeling of 3d molecules with bayesian flow networks.  In The Twelfth International Conference on Learning Representations, 2023.
  * [114] Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le.  Flow matching for generative modeling.  arXiv preprint arXiv:2210.02747, 2022.
  * [115] Jason Yim, Andrew Campbell, Andrew YK Foong, Michael Gastegger, José Jiménez-Luna, Sarah Lewis, Victor Garcia Satorras, Bastiaan S Veeling, Regina Barzilay, Tommi Jaakkola, et al.  Fast protein backbone generation with SE (3) flow matching.  arXiv preprint arXiv:2310.05297, 2023.
  * [116] Bowen Jing, Bonnie Berger, and Tommi Jaakkola.  AlphaFold meets flow matching for generating protein ensembles.  arXiv preprint arXiv:2402.04845, 2024.
  * [117] Yang Zhang and Jeffrey Skolnick.  TM-align: a protein structure alignment algorithm based on the tm-score.  Nucleic acids research, 33(7):2302–2309, 2005.
  * [118] Michel van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger.  Foldseek: fast and accurate protein structure search.  Biorxiv, pages 2022–02, 2022.
  * [119] Helen Berman, Kim Henrick, and Haruki Nakamura.  Announcing the worldwide protein data bank.  Nature structural & molecular biology, 10(12):980–980, 2003.
  * [120] Sheng Wang, Jian Peng, Jianzhu Ma, and Jinbo Xu.  Protein secondary structure prediction using deep convolutional neural fields.  Scientific reports, 6(1):1–11, 2016.
  * [121] Christine A Orengo, Alex D Michie, Susan Jones, David T Jones, Mark B Swindells, and Janet M Thornton.  Cath–a hierarchic classification of protein domain structures.  Structure, 5(8):1093–1109, 1997.
  * [122] John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola.  Generative models for graph-based protein design.  Advances in neural information processing systems, 32, 2019.
  * [123] Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror.  Learning from protein structure with geometric vector perceptrons.  In International Conference on Learning Representations, 2020.
  * [124] Cheng Tan, Zhangyang Gao, Jun Xia, Bozhen Hu, and Stan Z Li.  Generative de novo protein design with global context.  arXiv preprint arXiv:2204.10673, 2022.
  * [125] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives.  Learning inverse folding from millions of predicted structures.  In International conference on machine learning, pages 8946–8970. PMLR, 2022.
  * [126] Zhangyang Gao, Cheng Tan, and Stan Z Li.  Alphadesign: A graph protein design method and benchmark on alphafolddb.  arXiv preprint arXiv:2202.01079, 2022.
  * [127] Zhangyang Gao, Cheng Tan, Pablo Chacón, and Stan Z Li.  PiFold: Toward effective and efficient protein inverse folding.  arXiv preprint arXiv:2209.12643, 2022.
  * [128] Hua Cheng, R Dustin Schaeffer, Yuxing Liao, Lisa N Kinch, Jimin Pei, Shuoyong Shi, Bong-Hyun Kim, and Nick V Grishin.  ECOD: an evolutionary classification of protein domains.  PLoS computational biology, 10(12):e1003926, 2014.
  * [129] Andrew T McNutt, Paul Francoeur, Rishal Aggarwal, Tomohide Masuda, Rocco Meli, Matthew Ragoza, Jocelyn Sunseri, and David Ryan Koes.  GNINA 1.0: molecular docking with deep learning.  Journal of cheminformatics, 13(1):43, 2021.
  * [130] Radoslav Krivák and David Hoksza.  P2Rank: machine learning based tool for rapid and accurate prediction of ligand binding sites from protein structure.  Journal of cheminformatics, 10:1–12, 2018.

##  Appendix A Benchmarking details

###  A.1 Backbone generation methods

We focus on methods that were able to generate proteins up to 250 amino acids
long. We generated 50 proteins each for lengths 50, 100, 150, 200, and 250
(giving a total of 250 generated samples). To test designability, we used
ProteinMPNN [70] at a temperature of 0.1 to create 8 possible sequences, which
were then folded using ESMFold [69]. The minimum difference between the RMSD
of the folded C$\alpha$ backbones and the corresponding sample was recorded as
the scRMSD for the structure. A protein is considered designable if scRMSD is
less than 2.0 Å, and the percentage of designable proteins is reported for
each method. To measure diversity, TM-align [117] is used to compare each pair
of designable proteins from a method (normalizing with the length of the
longer protein). The average TM-Score is recorded for each method. To measure
novelty, we used FoldSeek [118] to compare the designable proteins against the
PDB [119]. The TM-Score is computed against proteins in the PDB and the
maximum is recorded as PDBTM. If PDBTM is less than 0.5, then the generated
structure is considered novel. The percentage of novel proteins is recorded.

###  A.2 DiffPack

Training and validation used BC40 [120] and followed the dataset split given
in AttnPacker [98]. The models were then evaluated on CASP13 and CASP14
datasets. The training set was curated to remove sequences with 40% or more
similarity to those in the test set. Both deep learning methods, like
AttnPacker [98] and DLPacker [99], and traditional methods, like SCWRL4 [95],
FASPR [90], and RosettaPacker [12], are used as baselines for DiffPack.

For each model the angle MAE, angle accuracy, and atom RMSD are computed.
Angle MAE is the mean absolute error of the predicted side-chain torsional
angles versus the actual angles. Angle accuracy is the percentage of angles
that are within 20$\deg$ of the actual angle. Atom RMSD is the average RMSD of
the side-chain atoms for each residue. Results are further categorized by
“Core” and “Surface” residues. Residues are labeled as core residues if at
least 20 $C_{\beta}$ atoms are within a 10 Å radius. Surface residues have at
most 15 $C_{\beta}$ atoms within the same radius.

###  A.3 Eigenfold

The test set in the Eigenfold paper [67] is CAMEO targets released between
August 1st, 2022 and October 31st, 2022 with lengths of less than 750
residues. The Eigenfold method was benchmarked against other structure
prediction methods: namely, AlphaFold2 [17], ESMFold [69], OmegaFold [77], and
RoseTTAFold [72]. For each method, five protein structures were generated and
ranked based on an approximate ELBO. The top-ranked structure was taken as the
final prediction and compared against the native structure. The metrics in
table 4 are the results of these comparisons.

###  A.4 GraDe-IF

The inverse folding methods were tested on recovering protein sequences in
CATH [121]. The training-validation-training split follows the CATH
v4.2.0-based partitioning as done for GraphTrans [122] and GVP [123]. Testing
was further split into three categories: short, single-chain, and all. Short
refers to proteins of length less than 100 residues. Single-chain refers to
proteins with only one chain. The methods were evaluated on perplexity and
recovery rate. Perplexity measures how well the predicted amino acid
probabilities match the actual amino acids at each position. A lower
perplexity means that the model better fits the distribution of the data. The
recovery rate measures the percentage of amino acids that are correctly
predicted from the original sequences. A higher recovery rate indicates the
model is better at predicting the original sequence. In addition, the training
data set for each model is given in table 5: CATH version 4.2 or 4.3.

GraDe-IF was benchmarked against other inverse folding methods, including
StructGNN [122], GraphTrans [122], GCA [124], GVP [123], GVP-large [125],
AlphaDesign [126], ESM-if1 [125], ProteinMPNN [70], and PiFold [127].

###  A.5 DiffDock-L

The benchmarking in [59] was performed on the PDBBind dataset [107] and the
authors’ new dataset, DockGen. The main aim of the dataset was to provide
better benchmarks for the protein-ligand docking problem by preventing data
leakage between training and testing data. For instance, proteins with less
than 30% sequence similarity may nonetheless share very similar binding
pockets, and so benchmarking must account for these structural similarities
when doing a split of training and testing. The authors use the ECOD [128]
classification to determine the protein domain of the chain making the most
contacts with the ligand. This is used to cluster the complexes with an equal
number of clusters being assigned to the training and testing datasets.
Further filtering steps are taken (such as removing protein-ligand complexes
with multiple ligands binding to the same pocket) to reach the final benchmark
dataset.

The benchmarking was carried out on both search-based and ML methods. The
search-based methods included SMINA [89] and GNINA [129]. Since search-based
methods have improved blind docking results by first selecting a pocket with a
pocket finder method, results when using P2Rank [130] prior to docking are
also recorded. The ML methods include Equibind [105], TankBind [106], and
DiffDock [33]. For the DiffDock models, the number of poses sampled is also
provided by the number in parentheses.