# Persistent topological Laplacian analysis of SARS-CoV-2 variants

Xiaoqi Wei1111 E-mail<EMAIL_ADDRESS>Jiahui Chen1222E-mail:
<EMAIL_ADDRESS>and Guo-Wei Wei1,2,3333Corresponding author. E-mail:
<EMAIL_ADDRESS>
1 Department of Mathematics,
Michigan State University, MI 48824, USA.
2 Department of Electrical and Computer Engineering,
Michigan State University, MI 48824, USA.
3 Department of Biochemistry and Molecular Biology,
Michigan State University, MI 48824, USA.

###### Abstract

Topological data analysis (TDA) is an emerging field in mathematics and data
science. Its central technique, persistent homology, has had tremendous
success in many science and engineering disciplines. However, persistent
homology has limitations, including its inability to handle heterogeneous
information, such as multiple types of geometric objects; being qualitative
rather than quantitative, e.g., counting a 5-member ring the same as a
6-member ring, and a failure to describe non-topological changes, such as
homotopic changes in protein-protein binding. Persistent topological
Laplacians (PTLs), such as persistent Laplacian and persistent sheaf
Laplacian, were proposed to overcome the limitations of persistent homology.
In this work, we examine the modeling and analysis power of PTLs in the study
of the protein structures of the severe acute respiratory syndrome coronavirus
2 (SARS-CoV-2) spike receptor binding domain (RBD). First, we employ PTLs to
study how the RBD mutation-induced structural changes of RBD-angiotensin-
converting enzyme 2 (ACE2) binding complexes are captured in the changes of
spectra of the PTLs among SARS-CoV-2 variants. Additionally, we use PTLs to
analyze the binding of RBD and ACE2-induced structural changes of various
SARS-CoV-2 variants. Finally, we explore the impacts of computationally
generated RBD structures on a topological deep learning paradigm and
predictions of deep mutational scanning datasets for the SARS-CoV-2 Omicron
BA.2 variant. Our results indicate that PTLs have advantages over persistent
homology in analyzing protein structural changes and provide a powerful new
TDA tool for data science.

Key words: Mutation and binding induced protein structural changes, Persistent
Laplacian, Persistent sheaf Laplacian, Topological data analysis, Topological
deep learning, Spectral data analysis.

###### Contents

  1. 1 Introduction
  2. 2 Results
    1. 2.1 PTL analysis of RBD structural changes induced by mutations
    2. 2.2 PTL analysis of RBD structural changes induced by its binding to ACE2
    3. 2.3 Impacts of computationally generated mutant structures on PTL-based topological deep learning predictions
  3. 3 Theories and methods
    1. 3.1 Persistent topological Laplacians
    2. 3.2 TopLapGBT and TopLapNet
  4. 4 Concluding remarks
  5. 5 Appendix

##  1 Introduction

Severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is the cause of
the ongoing global coronavirus disease 2019 (COVID-19) pandemic. Its evolution
and future direction are of major concern. It was well established that the
emergence of SARS-CoV-2 new variants is dictated by mutation-induced
infectivity strengthening [11] and antibody resistance (or vaccine
breakthrough) [41], two molecular mechanisms that determined the natural
selection at the population scale. More specifically, the binding of the viral
spike protein, particularly the receptor-binding domain (RBD), to the human
receptor angiotensin-converting enzyme 2 (ACE2) facilitates the entry of the
virus into host cells [20, 38]. In early 2020, it was hypothesized that
natural selection favors those SARS-CoV-2 RBD mutations that strengthen the
RBD-ACE2 binding, which leads to higher viral infectivity [11]. The hypothesis
was initially supported by the frequency analysis of 89 single RBD mutations
found from the genotyping of 15,140 complete SARS-CoV-2 genome samples [11]
and later confirmed beyond doubt by the evolution pattern of 651 RBD mutations
found from the genotyping of 506,768 SARS-CoV-2 genomes extracted from
COVID-19 patients up to early 2021 [40].

The vaccine breakthrough mechanism was not discovered until vaccines became
widely available in industrialized countries in the summer of 2021. It was
found that an RBD mutation that weakens the viral infectivity had an unusually
high observed frequency in 2,298,349 complete SARS-CoV-2 genomes isolated from
patients. This abnormal statistics was found to strongly correlate with the
vaccination rates in a few industrialized countries, including Denmark, the
United Kingdom, France, Bulgaria, the United States, etc [41]. To understand
this correlation, the mutational impact of a set of 130 antibodies extracted
from Covid patients that targets the RBD was studied. It was found that the
abnormal mutation on the RBD has a very strong ability to disrupt the binding
of most antibody-RBD complexes, which gives rise to antibody resistance (or
vaccine breakthrough) at the population scale [41].

As discussed above, the reveal of the natural selection mechanisms of SARS-
CoV-2 evolution is a typical example of a data-driven discovery that cannot be
achieved by individual experimental laboratories. In fact, the discovery
utilized results from tens of thousands of experimental laboratories around
the world [11, 41]. Machine learning, including deep learning, and also data-
driven approach, played an essential role in the discovery. Deep learning
methods can offer some of the most accurate predictions of biomolecular
properties, including the binding affinity of protein-protein interactions
(PPIs). This approach becomes particularly advantageous and outperforms other
methods when good-quality experimental data are available. However, structure-
based machine learning, including deep learning methods encounter difficulties
in PPI predictions due to their intricate structural complexity and high
dimensionality.

Advanced mathematics, such as topological data analysis (TDA), can provide an
effective abstraction of PPIs [39]. TDA is an emerging mathematical field that
utilizes algebraic topology approaches to analyze data. Its main tool is
persistent homology [5, 14, 15, 22, 25, 37, 49, 52], which integrates
classical homology and filtration to create a multiscale analysis of data,
resulting in a family of topological invariants. Through analyzing the
signature and change of topological invariants during filtration, one can
infer the shape of data [5]. However, persistent homology has limitations.
Firstly, it is insensitive to homotopic shape evolution that does not involve
any topological change. Secondly, roughy speaking, it cannot distinguish
between a five-member ring and a six-member ring. Thirdly, it is incapable of
differentiating different types of atoms, unable to describe directed
relations, and indifferent to structured data such as functional groups. To
overcome the first two limitations, persistent spectral graph [42], also known
as persistent Laplacian [30, 44], was proposed. This method not only returns
the full set of topological invariants as persistent homology does but also
captures additional homotopic shape evolution of data and is more quantitative
in its non-harmonic spectra. In addition to mathematical analysis [30],
computational algorithms, such as HERMES software package [44] and homotopy
continuation [47], were developed to facilitate topological deep learning, an
emerging paradigm first introduced in 2017 [2, 3] for biomolecular studies,
i.e., persistent Laplacian-assisted protein-ligand binding [31] and protein-
protein binding [9, 45]. Neither persistent homology nor persistent Laplacian
is sensitive to heterogeneous information in data. The element-specific
persistent homology was designed to alleviate this difficulty. This approach
has had tremendous success in deciphering biomolecules [3, 2] and in worldwide
computer-aided drug design competitions [32]. Inspired by this success,
various new TDA methods have been proposed [1, 18, 26, 27]. Recently, a more
elegant theory, persistent sheaf Laplacian, was proposed to embed
heterogeneous information, such as geometry and partial charges, in
topological analysis [48], utilizing the theory of cellular sheaves [19, 53].
Both persistent Laplacian and persistent sheaf Laplacian belong to a class of
persistent topological Laplacians (PTLs) [46]. PTLs are a family of multiscale
topological spectral methods, including continuous (evolutionary) Hodge
Laplacians defined on manifolds [13] and all other discrete multiscale
topological Laplacians, namely, persistent sheaf Laplacians [48], persistent
spectral graphs [42], persistent path Laplacians [43], persistent topological
hypergraph Laplacians [6], persistent hyperdigraph Laplacians [6], etc. Among
them, persistent path Laplacians were designed to describe directed graphs
(digraphs) and directed networks, while persistent topological hypergraph
Laplacians and persistent hyperdigraph Laplacians can further deal with
structured data. These new TDA methods can generate efficient mathematical
representations of macromolecules either being used to model molecular
structures or being used jointly with machine learning models for predicting
various properties of molecules [9]. In the past three years, TDA approaches
have been applied to SARS-CoV-2 related databases to predict PPI binding free
energy (BFE) changes of RBD-ACE2 and RBD-antibody complexes induced by RBD
mutations [7, 8]. Particularly, the non-harmonic spectra of PTLs can further
unveil the homotopic geometric deformation induced by RBD mutations.

Although sequence-based approaches offer good predictions of mutational
impacts on proteins, structure-based methods outperform other approaches [34].
In machine-learning-assisted directed evolution and protein engineering and
machine-learning-based PPI and protein folding stability predictions, mutant
structures are typically not available and are conventionally created by
computational means for the machine learning predictions [3, 2, 7, 8, 27],
which is a source of errors. It is interesting and important to quantify such
errors. Fortunately, since SARS-COV-2 variants are some of the most studied
subjects, some of their three-dimensional (3D) structures are available in the
literature, which offers an opportunity for in-depth analysis and comparison.

Our objectives for this work are three-fold. We are interested in both the
structural changes of the wild type RBD induced by mutations and the
structural changes of the wild type RBD or mutant RBDs induced by their
binding to ACE2. To quantify structural changes we first perform alignment of
structures and calculate the distances between corresponding atoms (e.g., Cα).
Then, we compute PTLs of different structures to further characterize their
structural changes. Finally, we study how the difference between
experimentally determined mutant structures and computationally generated
mutant structures affects PTL-based machine learning and topological deep
learning predictions of PPIs. This is important because we want to understand
machine learning models’ stability with respect to structural perturbations
and approximations. To this end, we utilize the 3D structures of SARS-CoV-2
RBD-ACE2 complexes of the wild type and mutants such as Alpha, Beta, Gamma,
and Omicron BA1 and BA2. We also employed the 3D spike protein structures of
the wild type and mutants such as Alpha, Beta, and Omicron BA.1 and BA.2.
Persistent Laplacian and persistent sheaf Laplacian are tested in our studies.
To quantitatively analyze the influence of computationally generated
structures on machine learning models, we used two topological machine
learning models, namely TopLapGBT and TopLapNet [9] and a deep mutational
scanning (DMS) dataset based on Omicron BA.2 [36]. We found that for this
dataset, the effects introduced by computationally generated structure on
TopLapGBT are not significant. However, they may slightly reduce the accuracy
of TopLapNet.

##  2 Results

Figure 1:  Sequence alignment of RBDs of the wild type, Alpha, Beta, Gamma,
BA.1, and BA.2. Alpha has one RBD mutation N501Y. Beta has three RBD mutations
K417N, E484K, and N501Y. Gamma has three RBD mutations K417T, E484K, and
N501Y. BA.1 has 15 RBD mutations G339D, S371L, S373P, S375F, K417N, N440K,
G446S, S477N, T478K, E484A, Q493R, G496S, Q498R, N501Y, and Y505H. BA.2 has 16
RBD mutations G339D, S371F, S373P, S375F, T376A, D405N, R408S, K417N, N440K,
S477N, T478K, E484A, Q493R, Q498R, N501Y, and Y505H.

###  2.1 PTL analysis of RBD structural changes induced by mutations

Figure 2: (a) Wild type RBD-ACE2 complex. The RBD is colored by light grey and
mutated residues in Alpha, Beta, Gamma, BA.1 and BA.2 are marked. (b, c, d, e,
f) Atoms of the wild type RBD are colored by their distances to corresponding
atoms in a mutant RBD. Subfigures (a), (b), (c), (b), and (f) corresponds to
the Alpha, Beta, Gamma, BA.1, and BA.2 variants, respectively. Pink and red
corresponds to 0Å and 14.32Å respectively. For each mutant we record the
residues that have at least one atom whose distance to the corresponding atom
in the wild type RBD is larger than 7.16Å. In Alpha, Beta, and Gamma, such
residue is R346. In BA.1, such residue is K386. In BA.2, such residues are
N370, A372, K378, and K386. These residues are marked in (g). (Plots generated
by ChimeraX [33].) Figure 3: Atoms of the wild type RBD are colored by their
distances to corresponding atoms in a mutant RBD. Subfigures (a), (b), (c),
and (b) corresponds to the Alpha, Beta, Gamma, and BA.1, variants,
respectively. Each alignment has its own color range. For each mutant, we
record the residues that have at least one atom whose distance to the
corresponding atom in the wild type RBD is more than half of the maximal
distance (4.57Å, 4.67Å, 4.94Å, and 3.72Å) between corresponding atoms. In
Alpha, such residues are T333, R346, K378, K386, R408, and N450. In Beta and
Gamma, such residues are T333, R346, K378, K386, and R408. In BA.1, such
residues are T333, N334, E340, R346, N360, D364, Y369, K378, K386, F392, R408,
K424, N450, K462, and H519. These residues are marked in (e). (Plots generated
by ChimeraX [33].)

To understand the structural differences of RBD between the wild type and
mutants in RBD-ACE2 complex, we align the RBDs of SARS-CoV-2 variants Alpha
(PDB ID: 8DLK[28]), Beta (PDB ID: 8DLN[28]), Gamma (PDB ID: 8DLQ[28]), BA.1
(PDB ID: 7T9L[29]), and BA.2 (PDB ID: 7XB0[24]) along with the wild type RBD
(PDB ID: 6M0J[23]) in Figures 2 and 3. For Alpha, Beta, Gamma, BA.1, and BA.2,
the maximal distances between corresponding atoms of mutant RBDs and the wild-
type RBD are 9.14Å, 9.33Å, 9.87Å, 7.44Å, and 14.32Å respectively. For each
mutant, the residues are recorded if they have at least one atom whose
distance to the corresponding atom in wild-type RBD is more than 7.16Å, which
is half of the maximal distance, 14.32Å. For variants Alpha, Beta, and Gamma,
such a residue is R346, while in BA.1 such residue is K386. BA.2 has most such
residues, which are N370, A372, K378, and K386, containing atoms deviating
from the wild type. However, these residues are not in the receptor-binding
motif (RBM, residues 438-506) that interacts directly with ACE2.

Alternatively, for Alpha, Beta, Gamma, and BA.1 variants, we also change the
threshold from 7.16Å to the half of maximal distance (4.57Å, 4.67Å, 4.94Å, and
3.72Å, respectively). Then in the Alpha variant, such residues are T333, R346,
K378, K386, R408, and N450. In Beta and Gamma variants, such residues are
T333, R346, K378, K386, and R408. In BA.1 such residues are T333, N334, E340,
R346, N360, D364, Y369, K378, K386, F392, R408, K424, N450, K462, and H519.
Also most large Cα structural changes occur at the coil regions of the RBD.
For the BA.2 variant, the half of maximal distance is 7.16Å and we have
recorded such residues that have at least one atom whose distance to the
corresponding atom in the wild-type RBD is more than 7.16Å.

To quantify the total structural differences between the wild type and
mutants, we calculate the sum of squares of distances between corresponding Cα
atoms. The results of Alpha, Beta, Gamma, BA.1, and BA.2 are 69 Å2, 70 Å2,
67Å2, 93Å2, and 255Å2, respectively as shown in Figure 4. The large values for
BA.1 and BA.2 are consistent with fact that BA.1 and BA.2 are strongly
antibody disruptive [10, 12]. The large structural changes induced by BA.2
mutations create significant mismatch between antibodies and antigens, making
BA.2 one of the most antibody resistant variants [12]. Arguably, the amount of
mutation-induced structural changes in RBD-ACE2 complexes also strongly
correlates with viral infectivity changes.

Figure 4: The total structural changes of RBD between the wild type and
mutants in RBD-ACE2 complex. Given an alignment of a mutant RBD to the wild
type RBD, the total structural changes is defined to be the sum of squares of
distances between corresponding Cα atoms in RBD.  Figure 5: Illustration of
persistent (sheaf) Betti numbers of element nonspecific persistent Laplacian
(PL) and persistent sheaf Laplacian (PSL) of the residue 501 mutation site at
different filtration values, i.e., radii (unit: Å). The wild type (PDB ID:
6M0J) and Alpha (PDB ID: 8DLK) are given in the first row. The Beta (PDB ID:
8DLN) and Gamma (PDB ID: 8DLQ) are given in the second row. BA.1 (PDB ID:
7T9L) and BA.2 (PDB ID: 7XB0) are given the third row.  Figure 6: Illustration
of the first nonzero eigenvalues of element nonspecific persistent Laplacian
(PL) and persistent sheaf Laplacian (PSL) of the residue 501 mutation site at
different filtration values, i.e., radii (unit: Å). The wild type (PDB ID:
6M0J) and Alpha (PDB ID: 8DLK) are given in the first row. The Beta (PDB ID:
8DLN) and Gamma (PDB ID: 8DLQ) are given in the second row. BA.1 (PDB ID:
7T9L) and BA.2 (PDB ID: 7XB0) are given the third row.  Figure 7: Illustration
of persistent Betti numbers (red line) and the first nonzero eigenvalues (blue
line) of element nonspecific persistent Laplacians of the wild type N501
mutation site at different filtration values, i.e., radii (unit: Å). Alpha
filtration is used. The graphs from top to bottom represent the results of
dimension-0, dimension-1, and dimension-2 Laplacians.

We are also interested in the topological characterization of the mutation-
induced conformational changes. To this end, we employ persistent Laplacian
(PL) and persistent sheaf Laplacian (PSL) to examine the local RBD structural
changes induced by the mutation N501Y (a common mutation that exists in Alpha,
Beta, Gamma, BA.1, and BA.2). For the wild type and mutants, the residue 501
mutation site is defined as the set of neighborhood heavy atoms (C, N, and O)
in RBD such that the distance of any atom in the set to the residue 501 Cα is
smaller than 10Å. We calculate persistent Laplacians and persistent sheaf
Laplacians for mutation sites of the wild type and variants and compare the
persistent (sheaf) Betti number and the smallest nonzero eigenvalues of
spectra at different filtration values. Persistent Laplacians and persistent
sheaf Laplacians can be calculated as either element non-specifically or
element specifically (i.e., considering carbon, nitrogen, and oxygen atoms
separately). We first employ the element non-specific approach and compare the
results of the wild type and variants. The results of persistent Laplacian and
persistent sheaf Laplacian are shown in Figures 5 and 6. The $x$ axis
represents the filtration values of Rips filtration, such that at a filtration
value $r$ the Rips complex is constructed by considering balls of radius $r$.
The sudden changes of persistent (sheaf) Betti number and the first nonzero
eigenvalues near $r=0.65$Å reflect the fact that most neighboring atoms are
about 1.3Å away from each other. In Figure 5, The number of atoms is reflected
in the initial 0-th Betti numbers. The 0-th Betti number dramatically
decreases around 0.65 Å because covalent bond distances are about 1.5Å. The
0-th Betti number decreases further from 1.2Å to 1.7Å due to other many non-
covalent bonds.

In Figure 6, the results of the wild type and mutants almost coincide, except
that the first nonzero eigenvalues of persistent sheaf Laplacians of BA.1 and
BA.2 near $r=0.65$Å have very different values. The results of persistent
Laplacians are quite different from those of persistent sheaf Laplacians at
large filtration values. The significant changes around $r=0.65$Å are due to
the topological changes.

We are also interested in understanding whether higher dimensional persistent
Laplacians can offer an additional characterization of biomolecules. Figure 7
presents the higher dimensional persistent Laplacian analysis of the wide type
RBD near the N501 residue. Obviously, higher dimensional persistent Laplacian
offers significant structural information about the distributions of circles
and cavities of the macromolecule. Most dimension-1 circles occur in the range
of 1.5-2.4Å, whereas most 2-dimensional cavities locate around 1.8-2.8Å.
2-dimensional cavities are short-lived in the filtration, indicating the lack
of multiple large cavities in the structure (at most one large cavity in the
structure). This distribution can be used to understand interaction forces.
For example, the length of hydrogen bonds ranges from 2-3.6Å(corresponding to
1-1.8 Å in the filtration radii). This information is valuable for the design
of machine learning representations, including the selection of the set of
filtration intervals. We also note that the peak of $\lambda_{2}^{r,0}$ is at
the left of $\beta_{2}^{r,0}$. It’s possible that when $r$ is in the range of
1.2Å-1.5Å, many 2-simplices are born but no 2-cycles are formed yet.

Figure 8: Illustration of the first nonzero eigenvalues of element-specific
persistent Laplacian of the residue 501 mutation site at different filtration
values, i.e., radii (unit: Å). The wild type (PDB ID: 6M0J) and Alpha (PDB ID:
8DLK) are given in the first row. The Beta (PDB ID: 8DLN) and Gamma (PDB ID:
8DLQ) are given in the second row. BA.1 (PDB ID: 7T9L) and BA.2 (PDB ID: 7XB0)
are given the third row.  Figure 9: Illustration of the first nonzero
eigenvalues of element-specific persistent sheaf Laplacian of the residue 501
mutation site at different filtration values, i.e., radii (unit: Å). The wild
type (PDB ID: 6M0J) and Alpha (PDB ID: 8DLK) are given in the first row. The
Beta (PDB ID: 8DLN) and Gamma (PDB ID: 8DLQ) are given in the second row. BA.1
(PDB ID: 7T9L) and BA.2 (PDB ID: 7XB0) are given the third row.

The element-specific results of the residue 501 mutation site of the wild
type, and variants Alpha, Beta, Gamma, BA.1, and BA.2 are shown in Figures 8
and 9, as well as in Figures 13 and 14 in the Appendix. We observe that the
difference between the first nonzero eigenvalues is much more obvious. For
instance, in Figure 8 there is a higher spike near 0.7Å in the graph of Alpha
carbon atoms, and two spikes near 1.3Å and 1.7Å disappear in the graph of the
Alpha variant’s oxygen atoms. In Figure 8, all results of carbon atoms have
similar shapes, implying a relatively stable RBD carbon atom structure. In the
results of nitrogen atoms, we notice that the results of Alpha, Beta, and
Gamma variants resemble each other, and the same can be said of the results of
BA.1 and BA.2 variants. In the results of oxygen atoms, the results of Alpha,
Beta, and Gamma still resemble each other, but the results of BA.1 and BA.2
are quite different. The results of the wild type are unique in the sense that
it has one or two spikes near 1.3Å or 1.7Å. These results indicate that
element-specific persistent Laplacians and element-specific persistent sheaf
Laplacians are better approaches in characterizing SARS-CoV-2 variants than
element-non-specific approaches. We know that nitrogen and oxygen atoms are
sparser in a protein, so if we use element nonspecific approach, nitrogen
atoms and oxygen atoms will first form edges with neighboring carbon atoms,
and we are not able to infer distances between nitrogen atoms or oxygen atoms.
This explains why element specific approach outperforms element nonspecific
approach.

###  2.2 PTL analysis of RBD structural changes induced by its binding to ACE2

We investigate how binding to ACE2 changes the spike protein RBD structure
from the closed state to the open state for the wild type, Alpha, Beta, BA.1,
and BA.2 variants. The PDB IDs of the spike protein of wild type, Alpha, Beta,
BA.1 and BA.2 used in this section are 7DF3 [51], 7LWS [17], 7LYM [17], 7TF8
[16] and 7XIX [4]. The analysis of the Gamma variant is eliminated due to the
lack of experimental structure. We first align each of the three RBDs in the
closed-state spike protein to the RBD in the RBD-ACE2 complex. The maximal
distances between corresponding atoms in the RBM of the three alignments of
BA.1 are 8.76Å, 13.49Å, and 9.44Å, which are larger than those of alignments
of the wild type and other mutants. For each alignment, we record the RBM
residues that have at least one atom whose distance to the corresponding atom
is larger than 5.28Å, i.e., half of the mean maximal distances between
corresponding atoms in RBM of the three alignments of BA.1. In wild-type RBD,
such residues are K444 and K458. In Alpha there are no such residues; In Beta,
chains A and B have K458; chain C has T478 and P479. In BA.1, each chain has
different such residues: chain A has K440, Y453, K458, K478, and F486; chain B
has K440, Y453, R457, K458, R466, Y473, Q474, K478, F486, F490, R493; and
chain C has K440, Y453, Y473, K478, F486. In BA.2 such residues are E465,
K478, and G482.

Figure 10: The total structural changes of the RBM between the closed state
RBD and the open state RBD induced by ACE2 binding. Here the total structural
changes are defined to be the sum of squares of distances between Cα atoms in
the RBM.

We also calculate the total structural changes of the RBM between the closed
state RBD and the open state RBD induced by its binding to the human ACE2.
Here, the total structural changes are defined to be the sum of squares of
distances between Cα atoms in the RBM. Since spike protein is a trimer, we
calculate the total structural changes for each chain and report the average
(see Figure 10). It turns out that the average total structural changes
induced by binding to ACE2 do not increase too much with respect to the number
of RBD mutations.

Figure 11: Illustration of persistent Betti numbers (red line) and the first
nonzero eigenvalues (blue line) of persistent Laplacian of the RBD binding
site of the wild type RBD-ACE2 complex (PDB ID: 6M0J) and closed state spike
protein (PDB ID: 7DF3, Chain ID: A) at different filtration values, i.e.,
radii (unit: Å). The graphs from top to bottom represent the results of carbon
atoms, nitrogen atoms, and oxygen atoms, respectively.

Now, we calculate persistent Laplacians and persistent sheaf Laplacians for
the RBD binding site in the closed state spike protein and the RBD-ACE2
complex. For the wild type and mutants, we define the RBD binding site as the
set of RBD residues whose Cαs are within 10Å from the Cαs of ACE2 residues. We
choose 10Å as the cutoff distance, because if we used 11Å then the RBD binding
site would include non-RBM residues. Spike protein as a trimer has three
chains. In the results of alignments, the recorded residues of the wild type,
Alpha, and BA.2 are the same for the three chains. Therefore, for the wild
type, Alpha and BA.2 we only use chain A, and for Beta and BA.1, we use all
three chains. The study was carried out in an element-specific manner for
carbon atoms, nitrogen atoms, and oxygen atoms. The results of the wild type
are shown in Figure 11. We noted that persistent Betti numbers cannot
distinguish two structures. However, the first nonzero eigenvalues of the
persistent Laplacian capture the difference, demonstrating the advantage of
persistent Laplacian over persistent homology in protein structure analysis.

Additional analysis is presented in Figures 15, 16, 17, 18, 19, 20, 21, 22,
23, and 24 in the Appendix. In Figure 15, the results of the wild type, Alpha,
Beta, BA.1, and BA.2 RBD binding sites are quite similar except that the wild
type RBD binding site has relatively lower first nonzero eigenvalues near
$r=0.7$Å. It is seen that peak appears or disappears in the graph of the
nitrogen atoms, whereas for BA.1 and BA.2, the results of the nitrogen atoms
resemble each other, sometimes even coincide.

In general, the first nonzero eigenvalues of the persistent Laplacian are able
to distinguish the structural difference before and after the complex
formation in various variants. In contrast, the harmonic spectra, or
equivalently, persistent homology, cannot always capture the structural
changes.

The results of persistent Laplacians and persistent sheaf Laplacians are
similar in this work. However, this similarity is due to the specific
implementation of persistent sheaf Laplacians. In general, persistent sheaf
Laplacians enable the embedding of non-geometric chemical and physical
information of biomeolecules in topological and spectral representations.

###  2.3 Impacts of computationally generated mutant structures on PTL-based
topological deep learning predictions

Figure 12: Atoms of BA.2 RBD (PDB ID: 7XB0) are colored by their distances to
corresponding atoms in the computationally generated structure. Blue, white,
and red corresponds to 0Å, 7.57Å, and 15.14Å respectively. We record the
residues that have at least one atom whose distance to the corresponding atom
in wild type RBD is more than 7.57Å. Such residues are 370, 375, 378, 386,
387, and 519.

The understanding of PPIs is a vital task in computational biology. With the
availability of large amounts of good quality data, machine learning
approaches have demonstrated their unique capability [27]. This is
specifically true for the prediction of SARS-CoV-2 infectivity and antibody
resistance using topological deep learning [10]. In this work, we explore the
impact of computationally generated structures on the predictive accuracy of
our topological deep learning framework. We use a BA.2 RBD deep mutational
scanning dataset which involves the systematical mutations of each residue on
the BA.2 RBD to 19 others and records corresponding binding affinity changes
[36].

The deep mutational scanning covers the RBD residues from 333 to 527. In order
to apply machine learning models, such as TopLapGBT and TopLapNet [9], to this
dataset, BA.2 RBD mutants need to be computationally generated based on a BA.2
RBD structure and the choice of the BA.2 RBD structure can affect the
performance of machine learning models. We can employ either an experimentally
determined BA.2 RBD-ACE2 complex structure or a BA.2 RBD-ACE2 complex
structure computationally generated based on an experimentally determined BA.1
RBD-ACE2 complex structure. These two complexes are systematically mutated to
all possible mutants in the deep mutational scanning dataset [36]. Two deep
learning models, namely TopLapGBT and TopLapNet, are used to predict the
binding affinity changes induced by all BA.2 RBD mutations. The results from
these complexes are compared to examine the performance of computationally
generated mutants. Here, we computationally generate mutant structures for
mutations L371F, T376A, D405N, R408S, S446G, and S496G.

When the given BA.2 RBD structure is experimentally determined (PDB ID: 7XB0),
and the resulting models are referred to as ExpTopLapGBT (experimental
TopLapGBT) and ExpTopLapNet. When the BA.2 RBD structure is computationally
generated from BA.1 RBD (PDB ID: 7T9L) by Jackal [50], the resulting model is
referred to as ComTopLapGBT (computational TopLapGBT) or ComTopLapNet. The
distances of corresponding atoms between the experimentally determined RBD
(PDB ID: 7XB0) and the RBD generated computationally from BA.1 RBD (PDB ID:
7T9L) is shown in Figure 12.

Method | $R_{p}(Exp,True)$ | $R_{p}(Com,True)$ | $R_{p}(Exp,Com)$
---|---|---|---
TopLapGBT | 0.901 | 0.898 | 0.990
TopLapNet | 0.879 | 0.849 | 0.925

Table 1:  $R_{p}(Exp,True)$ is the correlation coefficient between predictions
of ExpTopLapGBT (or ExpTopLapNet) and true affinity changes. Here,
$R_{p}(Com,True)$ is the correlation coefficient between predictions of
ComTopLapGBT (or ComTopLapNet) and true affinity changes. $R_{p}(Exp,Com)$ is
Pearson the correlation coefficient between the predictions of ExpTopLapGBT
and ComTopLapGBT (or between ExpTopLapNet and ComTopLapNet). A random state
affects the 10-fold splitting and the training of GBT and neural networks.

To evaluate the validity of computationally generated BA.2 complex structure,
we compare the results of two topological deep learning methods, ExpTopLapGBT
and ComTopLapGBT, on the predictions of the RBD deep mutational scanning
dataset. We split the dataset into 10 folds, and for each fold, we use the
other 9 folds as the training set to build a machine learning model, which is
used to predict ACE2-binding affinity changes for the fold. Therefore, for a
given 10-fold splitting we get the ExpTopLapGBT and ComTopLapGBT predictions
of RBD-ACE2 binding affinity changes for the deep mutational scanning dataset.
We denote $R_{p}(Exp,True)$ the Pearson correlation coefficient between
ExpTopLapGBT predicted binding affinity changes and experimental binding
affinity changes. Similarly, $R_{p}(Com,True)$ (or $R_{p}(Exp,Com)$) is the
Pearson correlation coefficient between ComTopLapGBT predicted binding
affinity changes and experimental binding affinity changes (or ExpTopLapGBT
predicted binding affinity changes).

The results of TopLapGBT and TopLapNet are shown in Table 1. Generally, the
performance of models using experimentally determined structures is better
than that of models using the computationally generated structure. This is not
surprising since the computationally generated structure is an approximation
of the experimental structure. The performance of ExpTopLapGBT and
ComTopLapGBT are extremely close, whereas the performance of ComTopLapNet
differs very much from that of ExpTopLapNet. We also see that ExpTopLapGBT
outperforms ExpTopLapNet.

##  3 Theories and methods

###  3.1 Persistent topological Laplacians

Persistent topological Laplacians (PTLs) are a family of topological data
analysis methods that are topological, multiscale, and spectral. Loosely
speaking, their kernel space dimensions coincide with the topological
invariants or Betti numbers in each topological dimension and their non-
harmonic spectra describe homotopic shape evolution during filtration or
multiscale analysis. Various discrete PTLs, e.g., persistent Laplacian [42],
persistent sheaf Laplacian [48], persistent path Laplacian [43], and
persistent directed hypergraph Laplacian [6] have been proposed for point
cloud data. For volumetric data, evolutionary de Rham-Hodge method has been
developed [13], which is defined on a family of evolving manifolds.
Evolutionary de Rham-Hodge method is based on differential geometry, algebraic
topology, multiscale analysis, and partial differential equations. In this
work, we focus on persistent Laplacian and persistent sheaf Laplacian.

Suppose $K$ and $L$ are two simplicial complexes and $K$ is a subset of $L$.
We denote by $C^{K}$ and $C^{L}$ the simplicial chain complexes of $K$ and $L$
with real coefficients. As a chain group $C_{q}$ in a simplicial chain complex
is formally generated by simplices, it is naturally a finite-dimensional inner
product space, and the adjoint of boundary map $\partial_{q}$ is well defined.
Let $C_{q+1}^{L,K}$ be the subspace $\\{c\in
C_{q+1}^{L}\mid\partial_{q+1}^{L}(c)\in C_{q}^{K}\\}$ of $C_{q+1}^{L}$ and
$\partial_{q+1}^{L,K}$ the restriction of $\partial^{L}_{q+1}$ to
$C_{q+1}^{L,K}$. The $q$-th persistent Laplacian $\Delta_{q}^{K,L}$ is defined
by

$\displaystyle\partial_{q+1}^{L,K}(\partial_{q+1}^{L,K})^{\ast}+(\partial^{K}_{q})^{\ast}\partial^{K}_{q}.$
(1)

Before we define the persistent sheaf Laplacian, we need to explain what a
cellular sheaf is first. A cellular sheaf $\mathscr{S}$ is a simplicial
complex $X$ (viewed as a cell complex) with an assignment to each cell
$\sigma$ of $X$ a finite-dimensional vector space $\mathscr{S}(\sigma)$
(referred to as the stalk of $\mathscr{S}$ over $\sigma$) and to each face
relation $\sigma\leqslant\tau$ (i.e., $\sigma\subset\overline{\tau}$) a linear
morphism of vector spaces denoted by $\mathscr{S}_{\sigma\leqslant\tau}$
(referred to as the restriction map of the face relation
$\sigma\leqslant\tau$), satisfying the rule

$\displaystyle\rho\leqslant\sigma\leqslant\tau\Rightarrow\mathscr{S}_{\rho\leqslant\tau}=\mathscr{S}_{\sigma\leqslant\tau}\mathscr{S}_{\rho\leqslant\sigma}$

and $\mathscr{S}_{\sigma\leqslant\sigma}$ is the identity map of
$\mathscr{S}(\sigma)$. Like a simplicial complex, a cellular sheaf gives rise
to a sheaf cochain complex. The $q$-th sheaf cochain group
$C^{q}_{\mathscr{S}}$ is the direct sum of stalks over $q$-dimensional cells.
To define coboundary maps, we can globally orient the simplicial complex $X$
and obtain a signed incidence relation, i.e. an assignment to each
$\sigma\leqslant\tau$ an integer $[\sigma:\tau]$. Then the coboundary map
$d^{q}:C^{q}_{\mathscr{S}}\to C^{q+1}_{\mathscr{S}}$ is defined by

$\displaystyle
d^{q}|_{\mathscr{S}(\sigma)}=\sum_{\sigma\leqslant\tau}[\sigma:\tau]\mathscr{S}_{\sigma\leqslant\tau}.$

Now suppose we have two cellular sheaves $\mathscr{S}$ on $K$ and
$\mathscr{T}$ on $L$ such that $K\subseteq L$ and stalks and restriction maps
of $K$ are identical to those of $L$.

${C^{q-1}_{\mathscr{S}}}$${C^{q}_{\mathscr{S}}}$${C^{q+1}_{\mathscr{S}}}$${C^{q+1}_{\mathscr{S},\mathscr{T}}}$${C^{q-1}_{\mathscr{T}}}$${C^{q}_{\mathscr{T}}}$${C^{q+1}_{\mathscr{T}}}$$\scriptstyle{d^{q-1}_{\mathscr{S}}}$$\scriptstyle{d^{q}_{\mathscr{S}}}$$\scriptstyle{(d^{q-1}_{\mathscr{S}})^{\ast}}$$\scriptstyle{d^{q}_{\mathscr{S},\mathscr{T}}}$$\scriptstyle{(d^{q}_{\mathscr{S},\mathscr{T}})^{\ast}}$$\scriptstyle{\pi}$$\scriptstyle{d^{q-1}_{\mathscr{T}}}$$\scriptstyle{d^{q}_{\mathscr{T}}}$$\scriptstyle{\pi}$$\scriptstyle{\pi}$

Let $C_{\mathscr{S},\mathscr{T}}^{q+1}=\\{c\in
C^{q+1}_{\mathscr{T}}\mid(d^{q}_{\mathscr{T}})^{\ast}(c)\in
C^{q}_{\mathscr{S}}\\}$. We denote the adjoint map of
$(d^{q}_{\mathscr{T}})^{\ast}|_{C_{\mathscr{S},\mathscr{T}}^{q+1}}$ as
$d^{q}_{\mathscr{S},\mathscr{T}}$ and define the $q$-th persistent sheaf
Laplacian $\Delta_{q}^{\mathscr{S},\mathscr{T}}$ as

$\displaystyle\Delta_{q}^{\mathscr{S},\mathscr{T}}=(d^{q}_{\mathscr{S},\mathscr{T}})^{\ast}d^{q}_{\mathscr{S},\mathscr{T}}+d^{q-1}_{\mathscr{S}}(d^{q-1}_{\mathscr{S}})^{\ast}.$

Of course, to define adjoint maps cochain groups need to be inner product
spaces. In this work, cellular sheaves are constructed in the same way as in
Section 2.4 of [48]. The non-geometrical information we consider is the set of
atomic partial charges. We employ partial charges from the PDB2PQR package
[21]. We build a Rips filtration of graphs. For each simplicial complex $X$,
we denote each vertex by $v_{i}$, the edge connecting $v_{i}$ and $v_{j}$ by
$e_{ij}$ and the partial charges $q_{i}$. Then the cellular sheaf is such that
each stalk is $\mathbb{R}$, and for face relation $v_{i}\leqslant e_{ij}$ the
morphism is the multiplication by $q_{j}/r_{ij}$, where $r_{ij}$ is the length
of $e_{ij}$. Spectra of persistent sheaf Laplacians are not used in TopLapGBT
and TopLapNet.

###  3.2 TopLapGBT and TopLapNet

TopLapGBT and TopLapNet [9] have been employed to study mutational effects on
protein-protein interaction. Gradient boosted trees are employed in TopLapGBT,
whereas, TopLapNet is based on artificial neural networks. Both models are
constructed by using persistent Laplacians. These methods require the 3D
structures of both wide-type PPI complexes and mutant complexes. For instance,
the AB-Bind S645 dataset[35] includes 645 mutants with experimentally
determined BFE changes across 29 antibody-antigen complexes. Mutant structures
can be computationally generated based on experimentally determined structures
of the wild-type antibody-antigen complexes and mutation information (chain
id, residue id, mutant residue, etc.). Representation of mutant structures,
including the persistent homology and persistent Laplacian representations,
and other auxiliary representations, can be used as feature vectors to train
machine learning models (such as gradient boosting trees and deep neural
networks) that can predict mutation-induced BFE changes.

When we apply persistent homology and persistent Laplacian to the study of
protein-protein interactions, we always extract the atoms within a certain
cutoff distance $r$ of the binding site and construct a distance matrix such
that if two atoms are in the same protein then the distance between them is an
extremely large constant number. If we want to further characterize the
interaction between atoms of certain elements $E_{1}$ and $E_{2}$, we can
consider the point cloud formed by the atoms of an element $E_{1}$ of protein
$A$ within $r$ of the binding site, and the atoms of element $E_{2}$ of
protein $B$ within $r$ of the binding site. After the calculation of
persistent homology and persistent Laplacian, the next step is to transform
the barcodes of persistent homology or spectra of persistent Laplacians into
vector representations of fixed lengths. For barcodes, there are at least two
ways: either we divide the interval $[0,r]$ into bins of even length and count
the occurrence of bars, birth values, and death values in each bin, or we
simply compute statistics such as sum, maximum, minimum, mean, and standard
deviation for bar lengths, birth values, and death values. The former method
is often applied to 0-dimensional barcodes and the latter to 1-dimensional and
2-dimensional barcodes. For the spectrum of a persistent Laplacian, we
separate zero eigenvalues (harmonic spectra) and nonzero eigenvalues (non-
harmonic spectra). We use the number of zero eigenvalues, the sum, the
minimum, the maximum, the mean, the standard deviation, the variance, and the
sum of squares of nonzero eigenvalues.

In this study, we use scikit-learn to build a gradient boosting tree whose
parameters are n_estimators=20000, learning_rate = 0.005, max_features =
‘sqrt’, max_depth = 9, min_samples_split = 3, subsample = 0.4, and
n_iter_no_change=500. Additionally, we use PyTorch to build a neural network
with 7 hidden layers and each layer has 8000 neurons.

##  4 Concluding remarks

Persistent topological Laplacians (PTLs) are a class of newly proposed
multiscale topological spectral approaches in data science. These methods can
be used either in a discrete setting for point cloud data [42, 48, 43] or in a
continuous setting for volumetric data [13]. Their mathematical underpinnings
for discrete formulations are algebraic topology, sheaf theory, and
combinatorial graphs, while those for the continuous formulations is algebraic
topology, differential geometry, and partial differential equation. Among
mutants Alpha, Beta, Gamma, Omicron BA.1, and Omicron BA.2, BA.2 has the
largest total structural changes from the wild type, which agrees with the
significant antibody escape of the Omicron BA.2 variant. As to the total
structural changes of a closed state RBD induced by its binding to ACE2, total
structural changes of Alpha, Beta, Omicron BA.1, and Omicron BA.2 do not
differ too much. It is noted that most large structural changes of Cα occur at
flexible random coil regions at the epitope.

We also demonstrate how to use PTLs to characterize structural changes induced
by SARS-CoV-2 variant spike protein receptor-binding domain (RBD) mutations
and by its binding to human angiotensin-converting enzyme 2 (ACE2). Two PTLs,
namely persistent Laplacian and persistent sheaf Laplacian, are utilized in
our work. We also analyze two implementations, i.e., element-nonspecific and
element-specific Laplacian models of persistent Laplacian and persistent sheaf
Laplacian. We show that persistent Laplacian and persistent sheaf Laplacian
provide similar results. These methods capture homotopic shape evolution
information, which persistent homology cannot offer. We expect other
persistent topological Laplacians, such as persistent path Laplacian [43], can
uncover similar information. Additionally, element-specific approaches reveal
more information than element-nonspecific ones as shown in literature [3].

More specifically, the results of persistent Laplacian and persistent sheaf
Laplacian indicate that at the residue 501 mutation site, the structure of RBD
carbon atoms is less affected by mutations than that of RBD nitrogen atoms and
RBD oxygen atoms, partially due to the fact that the bond nature of carbon
atoms is mostly covalent, whereas the distances among oxygen atoms are mostly
non-covalent. The topological Betti numbers of oxygen atoms are associated
with possible hydrogen bonds. Additionally, structural similarity of mutation
sites is observed in the spectra of persistent Laplacian and persistent sheaf
Laplacian.

As for the RBD structural changes induced by its binding to ACE2, for wild
type, Alpha and Beta, a significant difference can be observed in the result
of nitrogen atoms, whereas for BA.1 and BA.2, a significant difference can be
observed in the result of oxygen atoms. We show that the non-harmonic spectra
(the first non-zero eigenvalues) of persistent Laplacian are more sensitive to
structural changes than the harmonic spectra. Therefore, persistent Laplacian
has an advantage over persistent homology in protein analysis.

Finally we test how a computationally generated structure impacts the
prediction of PTL-based machine learning models, i.e., TopLapGBT and TopLapNet
[9]. The results indicate that a computationally generated structure harms the
performance of TopLapGBT and TopLapNet. However, TopLapGBT is much less
affected by a computationally generated structure than TopLapNet, implying a
resistance to the structural approximation from computations. TopLapNet is
more affected by a computationally generated structure probably because neural
networks are more prone to overfitting than gradient-boosted trees.

This work reveals that PTLs are a class of powerful new methods for
topological data analysis (TDA) or more precisely, spectral data analysis
(SDA). These methods can certainly be applied to the data analysis in other
fields and disciplines, including image science, physical science, medical
science, social science, engineering, financial industrial, musical science
[46], etc.

## Data availability

The related datasets studied in this work are available at:
https://github.com/WeilabMSU/PTLvirus.

## Acknowledgment

This work was supported in part by NIH grants R01GM126189 and R01AI164266, NSF
grants DMS-2052983, DMS-1761320, and IIS-1900473, NASA grant 80NSSC21M0023,
MSU Foundation, Bristol-Myers Squibb 65109, and Pfizer.

## References

  * [1] B. Ameneyro, G. Siopsis, and V. Maroulas.  Quantum Persistent Homology.  arXiv preprint arXiv:2202.12965, 2022.
  * [2] Z. Cang, L. Mu, and G.-W. Wei.  Representability of algebraic topology for biomolecules in machine learning based scoring and virtual screening.  PLoS computational biology, 14(1):e1005929, 2018.
  * [3] Z. Cang and G.-W. Wei.  TopologyNet: Topology based deep convolutional and multi-task neural networks for biomolecular property predictions.  PLoS Computational Biology, 13(7):e1005690, 2017.
  * [4] Y. Cao, A. Yisimayi, F. Jian, W. Song, T. Xiao, L. Wang, S. Du, J. Wang, Q. Li, X. Chen, et al.  BA.2.12.1, BA.4 and BA.5 escape antibodies elicited by omicron infection.  Nature, 608(7923):593–602, 2022.
  * [5] G. Carlsson.  Topology and data.  Bulletin of the American Mathematical Society, 46(2):255–308, 2009\.
  * [6] D. Chen, J. Liu, J. Wu, and G.-W. Wei.  Persistent topological hypergraph and hyperdigraph Laplacians.  in preparation, 2023.
  * [7] J. Chen, K. Gao, R. Wang, and G.-W. Wei.  Prediction and mitigation of mutation threats to COVID-19 vaccines and antibody therapies.  Chemical Science, 12(20):6929–6948, 2021.
  * [8] J. Chen, K. Gao, R. Wang, and G.-W. Wei.  Revealing the threat of emerging SARS-CoV-2 mutations to antibody therapies.  Journal of Molecular Biology, 433(7744), 2021.
  * [9] J. Chen, Y. Qiu, R. Wang, and G.-W. Wei.  Persistent Laplacian projected omicron BA.4 and BA.5 to become new dominating variants.  Computers in Biology and Medicine, 151, 106262, 2022.
  * [10] J. Chen, R. Wang, N. B. Gilby, and G.-W. Wei.  Omicron variant (B.1.1.529): infectivity, vaccine breakthrough, and antibody resistance.  Journal of Chemical Information and Modeling, 62(2):412–422, 2022\.
  * [11] J. Chen, R. Wang, M. Wang, and G.-W. Wei.  Mutations strengthened SARS-CoV-2 infectivity.  Journal of Molecular Biology, 432(19):5212–5226, 2020.
  * [12] J. Chen and G.-W. Wei.  Omicron BA. 2 (B. 1.1. 529.2): high potential for becoming the next dominant variant.  The Journal of Physical Chemistry Letters, 13(17):3840–3849, 2022\.
  * [13] J. Chen, R. Zhao, Y. Tong, and G.-W. Wei.  Evolutionary de Rham-Hodge method.  Discrete and Continuous Dynamical Systems. Series B, 26(7):3785, 2021.
  * [14] H. Edelsbrunner, D. Letscher, and A. Zomorodian.  Topological persistence and simplification.  In Proceedings 41st Annual Symposium on Foundations of Computer Science, pages 454–463. IEEE, 2000.
  * [15] P. Frosini.  Measuring shapes by size functions.  In Intelligent Robots and Computer Vision X: Algorithms and Techniques, volume 1607, pages 122–133. International Society for Optics and Photonics, 1992.
  * [16] S. M.-C. Gobeil, R. Henderson, V. Stalls, K. Janowska, X. Huang, A. May, M. Speakman, E. Beaudoin, K. Manne, D. Li, et al.  Structural diversity of the SARS-CoV-2 omicron spike.  Molecular Cell, 2022.
  * [17] S. M.-C. Gobeil, K. Janowska, S. McDowell, K. Mansouri, R. Parks, V. Stalls, M. F. Kopp, K. Manne, D. Li, K. Wiehe, et al.  Effect of natural mutations of SARS-CoV-2 on spike structure, conformation, and antigenicity.  Science, 373(6555):eabi6226, 2021.
  * [18] J. Grbic, J. Wu, K. Xia, and G. Wei.  Aspects of topological approaches for data science.  Foundations of Data Science, 2022.
  * [19] J. Hansen and R. Ghrist.  Toward a spectral theory of cellular sheaves.  Journal of Applied and Computational Topology, 3(4):315–358, 2019\.
  * [20] M. Hoffmann, H. Kleine-Weber, S. Schroeder, N. Krüger, T. Herrler, S. Erichsen, T. S. Schiergens, G. Herrler, N.-H. Wu, A. Nitsche, et al.  SARS-CoV-2 cell entry depends on ACE2 and TMPRSS2 and is blocked by a clinically proven protease inhibitor.  Cell, 181(2):271–280, 2020.
  * [21] Jurrus, Elizabeth and Engel, Dave and Star, Keith and Monson, Kyle and Brandi, Juan and Felberg, Lisa E and Brookes, David H and Wilson, Leighton and Chen, Jiahui and Liles, Karina and others  Improvements to the APBS biomolecular solvation software suite.  Protein Science, 27(1):112–128, 2018.
  * [22] V. Kovacev-Nikolic, P. Bubenik, D. Nikolić, and G. Heo.  Using persistent homology and dynamical distances to analyze protein binding.  Statistical Applications in Genetics and Molecular Biology, 15(1):19–38, 2016.
  * [23] J. Lan, J. Ge, J. Yu, S. Shan, H. Zhou, S. Fan, Q. Zhang, X. Shi, Q. Wang, L. Zhang, et al.  Structure of the SARS-CoV-2 spike receptor-binding domain bound to the ACE2 receptor.  Nature, 581(7807):215–220, 2020.
  * [24] L. Li, H. Liao, Y. Meng, W. Li, P. Han, K. Liu, Q. Wang, D. Li, Y. Zhang, L. Wang, et al.  Structural basis of human ACE2 higher binding affinity to currently circulating Omicron SARS-CoV-2 sub-variants BA. 2 and BA. 1.1.  Cell, 185(16):2952–2960, 2022.
  * [25] J. Liu, D. Chen, F. Pan, and J. Wu.  Neighborhood Path Complex for the Quantitative Analysis of the Structure and Stability of Carboranes.  Journal of Computational Biophysics and Chemistry, https://doi.org/10.1142/S2737416523500229, 2023\.
  * [26] J. Liu, K.-L. Xia, J. Wu, S. S.-T. Yau, and G.-W. Wei.  Biomolecular topology: Modelling and analysis.  Acta Mathematica Sinica, English Series, 38(10):1901–1938, 2022\.
  * [27] X. Liu, H. Feng, J. Wu, and K. Xia.  Hom-complex-based machine learning (HCMl) for the prediction of protein–protein binding affinity changes upon mutation.  Journal of Chemical Information and Modeling, 62(17):3961–3969, 2022.
  * [28] D. Mannar, J. W. Saville, Z. Sun, X. Zhu, M. M. Marti, S. S. Srivastava, A. M. Berezuk, S. Zhou, K. S. Tuttle, M. D. Sobolewski, et al.  SARS-CoV-2 variants of concern: spike protein mutational analysis and epitope for broad neutralization.  Nature Communications, 13(1):1–12, 2022.
  * [29] D. Mannar, J. W. Saville, X. Zhu, S. S. Srivastava, A. M. Berezuk, K. S. Tuttle, A. C. Marquez, I. Sekirov, and S. Subramaniam.  SARS-CoV-2 Omicron variant: Antibody evasion and cryo-EM structure of spike protein–ACE2 complex.  Science, 375(6582):760–764, 2022.
  * [30] F. Mémoli, Z. Wan, and Y. Wang.  Persistent Laplacians: Properties, algorithms and implications.  SIAM Journal on Mathematics of Data Science, 4(2):858–884, 2022\.
  * [31] Z. Meng and K. Xia.  Persistent spectral–based machine learning (perspect ml) for protein-ligand binding affinity prediction.  Science Advances, 7(19):eabc5329, 2021.
  * [32] D. D. Nguyen, Z. Cang, K. Wu, M. Wang, Y. Cao, and G.-W. Wei.  Mathematical deep learning for pose and binding affinity prediction and ranking in D3R grand challenges.  Journal of Computer-Aided Molecular Design, 33(1):71–82, 2019.
  * [33] E. F. Pettersen, T. D. Goddard, C. C. Huang, E. C. Meng, G. S. Couch, T. I. Croll, J. H. Morris, and T. E. Ferrin.  UCSF Chimerax: Structure visualization for researchers, educators, and developers.  Protein Science, 30(1):70–82, 2021.
  * [34] Y. Qiu and G.-W. Wei.  Persistent spectral theory-guided protein engineering.  Nature Computational Science, 2023.
  * [35] S. Sirin, J. R. Apgar, E. M. Bennett, and A. E. Keating.  AB-Bind: antibody binding mutational database for computational affinity predictions.  Protein Science, 25(2):393–409, 2016.
  * [36] T. N. Starr, A. J. Greaney, C. M. Stewart, A. C. Walls, W. W. Hannon, D. Veesler, and J. D. Bloom.  Deep mutational scans for ace2 binding, rbd expression, and antibody escape in the SARS-CoV-2 Omicron BA.1 and BA.2 receptor-binding domains.  PLoS Pathogens, 18(11):e1010951, 2022.
  * [37] J. Townsend, C. P. Micucci, J. H. Hymel, V. Maroulas, and K. D. Vogiatzis.  Representation of molecular structures with persistent homology for machine learning applications in chemistry.  Nature Communications, 11(1):1–9, 2020.
  * [38] A. C. Walls, Y.-J. Park, M. A. Tortorici, A. Wall, A. T. McGuire, and D. Veesler.  Structure, function, and antigenicity of the SARS-CoV-2 spike glycoprotein.  Cell, 181(2):281–292, 2020.
  * [39] M. Wang, Z. Cang, and G.-W. Wei.  A topology-based network tree for the prediction of protein–protein binding affinity changes following mutation.  Nature Machine Intelligence, 2(2):116–123, 2020.
  * [40] R. Wang, J. Chen, K. Gao, and G.-W. Wei.  Vaccine-escape and fast-growing mutations in the United Kingdom, the United States, Singapore, Spain, India, and other COVID-19-devastated countries.  Genomics, 113(4):2158–2170, 2021.
  * [41] R. Wang, J. Chen, and G.-W. Wei.  Mechanisms of SARS-CoV-2 evolution revealing vaccine-resistant mutations in Europe and America.  The Journal of Physical Chemistry Letters, 12:11850–11857, 2021\.
  * [42] R. Wang, D. D. Nguyen, and G.-W. Wei.  Persistent spectral graph.  International Journal for Numerical Methods in Biomedical Engineering, 36(9):e3376, 2020.
  * [43] R. Wang and G.-W. Wei.  Persistent path Laplacian.  Foundation of Data Science, (5):26–55, 2023.
  * [44] R. Wang, R. Zhao, E. Ribando-Gros, J. Chen, Y. Tong, and G.-W. Wei.  HERMES: Persistent spectral graph software.  Foundations of Data Science, 3(1):67–97, 2020.
  * [45] J. Wee and K. Xia.  Persistent spectral based ensemble learning (perspect-el) for protein–protein binding affinity prediction.  Briefings in Bioinformatics, 23(2), 2022.
  * [46] G.-W. Wei.  Topological data analysis hearing the shapes of drums and bells.  arXiv preprint arXiv:2301.05025, 2023.
  * [47] X. Wei and G.-W. Wei.  Homotopy continuation for the spectra of persistent Laplacians.  Foundations of Data Science, 3(4):677, 2021.
  * [48] X. Wei and G.-W. Wei.  Persistent sheaf Laplacians.  arXiv preprint arXiv:2112.10906, 2021.
  * [49] K. L. Xia and G. W. Wei.  Persistent homology analysis of protein structure, flexibility and folding.  International Journal for Numerical Methods in Biomedical Engineering, 30:814–844, 2014.
  * [50] Z. Xiang and B. Honig.  Extending the accuracy limits of prediction for side-chain conformations.  Journal of Molecular Biology, 311(2):421–430, 2001.
  * [51] C. Xu, Y. Wang, C. Liu, C. Zhang, W. Han, X. Hong, Y. Wang, Q. Hong, S. Wang, Q. Zhao, et al.  Conformational dynamics of sars-cov-2 trimeric spike glycoprotein in complex with receptor ACE2 revealed by cryo-EM.  Science Advances, 7(1):eabe5575, 2021.
  * [52] A. Zomorodian and G. Carlsson.  Computing persistent homology.  Discrete & Computational Geometry, 33(2):249–274, 2005.
  * [53] , J. Curry.  Sheaves, cosheaves and applications.  The University of Pennsylvania, 2014.

##  5 Appendix

In the appendix, we provide the information of PDB structures we used and
additional topological analysis using persistent Laplacian and persistent
sheaf Laplacian. The information of PDB structures is given in Table 2. PTL
results are given in Figures 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, and
24.

Specifically, Figures 13, and 14 are element-specific analysis of the wide
type, Alpha, Beta, Gamma, BA.1, and BA.2 using LP and PSL, respectively.
Figure 15 presents carbon specific analysis of the wide type, Alpha, Beta,
Gamma, BA.1, and BA.2. Figures 16, 17, 18, and 19 demonstrate the PL analysis
of Alpha, Beta, BA.2, and BA.2, respectively. The spectral analysis of three
major types of elements, namely carbon atoms, nitrogen atoms, and oxygen
atoms, is presented in these figures. Finally, 20, 21, 22, 23, and 24
illustrate the PSL analysis of the wide type, Alpha, Beta, BA.2, and BA.2,
respectively. These figures display the spectral analysis of three major types
of elements, namely carbon atoms, nitrogen atoms, and oxygen atoms.

PDB ID | Method | Resolution (unit: Å) | Description
---|---|---|---
6M0J[23] | X-ray diffraction | 2.45 | wild type RBD-ACE2
8DLK[28] | Electron microscopy | 3.04 | Alpha RBD-ACE2
8DLN[28] | Electron microscopy | 3.04 | Beta RBD-ACE2
8DLQ[28] | Electron microscopy | 2.77 | Gamma RBD-ACE2
7T9L[29] | Electron microscopy | 2.66 | BA.1 RBD-ACE2
7XB0[24] | X-ray diffraction | 2.90 | BA.2 RBD-ACE2
7DF3[51] | Electron microscopy | 2.70 | wild type spike
7LWS[17] | Electron microscopy | 3.22 | Alpha spike
7LYM[17] | Electron microscopy | 3.57 | Beta spike
7TF8[16] | Electron microscopy | 3.36 | BA.1 spike
7XIX[4] | Electron microscopy | 3.25 | BA.2 spike

Table 2: Information of PDB 3D structures used in this work.

Figure 13: Illustration of persistent Betti numbers of element specific
persistent Laplacian of the residue 501 mutation site at different filtration
values, i.e., radii (unit: Å). The wild type (PDB ID: 6M0J) and Alpha (PDB ID:
8DLK) are given in the first row. The Beta (PDB ID: 8DLN) and Gamma (PDB ID:
8DLQ) are given in the second row. The BA.1 (PDB ID: 7T9L) and BA.2 (PDB ID:
7XB0) are given the third row.  Figure 14: Illustration of persistent Betti
numbers of element specific persistent sheaf Laplacian of the residue 501
mutation site at different filtration values, i.e., radii (unit: Å). The wild
type (PDB ID: 6M0J) and Alpha (PDB ID: 8DLK) are given in the first row. The
Beta (PDB ID: 8DLN) and Gamma (PDB ID: 8DLQ) are given in the second row. The
BA.1 (PDB ID: 7T9L) and BA.2 (PDB ID: 7XB0) are given the third row.  Figure
15: Illustration of persistent Betti numbers (red line) and the first nonzero
eigenvalues (blue line) of persistent Laplacian of carbon atoms of the RBD
binding site in RBD-ACE2 complex of the wild type (PDB ID: 6M0J), Alpha (PDB
ID: 8DLK), Beta (PDB ID: 8DLN), BA.1 (PDB ID: 7T9L), and BA.2 (PDB ID: 7XB0)
at different filtration values, i.e., radii (unit: Å). The graphs from top to
bottom represent the results of the wild type, Alpha, Beta, BA.1, and BA.2
variants, respectively. Figure 16: Illustration of persistent Betti numbers
(red line) and the first nonzero eigenvalues (blue line) of persistent
Laplacian of the RBD binding site of Alpha RBD-ACE2 complex (PDB ID: 8DLK) and
closed state spike protein (PDB ID: 7LWS, Chain ID: A) at different filtration
values, i.e., radii (unit: Å). The graphs from top to bottom represent the
results of carbon atoms, nitrogen atoms, and oxygen atoms, respectively.
Figure 17: Illustration of persistent Betti numbers (red line) and the first
nonzero eigenvalues (blue line) of persistent Laplacian of the RBD binding
site of Beta RBD-ACE2 complex (PDB ID: 8DLN) and closed state spike protein
(PDB ID: 7LYM, Chain ID: A, B, C) at different filtration values, i.e., radii
(unit: Å). The graphs from top to bottom represent the results of carbon
atoms, nitrogen atoms, and oxygen atoms, respectively. Figure 18: Illustration
of persistent Betti numbers (red line) and the first nonzero eigenvalues (blue
line) of persistent Laplacian of the RBD binding site of BA.1 RBD-ACE2 complex
(PDB ID: 7T9L) and closed state spike protein (PDB ID: 7TF8, Chain ID: A, B,
C) at different filtration values, i.e., radii (unit: Å). The graphs from top
to bottom represent the results of carbon atoms, nitrogen atoms, and oxygen
atoms, respectively. Figure 19: Illustration of persistent Betti numbers (red
line) and the first nonzero eigenvalues (blue line) of persistent Laplacian of
the RBD binding site of BA.2 RBD-ACE2 complex (PDB ID: 7XB0) and closed state
spike protein (PDB ID: 7XIX, Chain ID: A) at different filtration values,
i.e., radii (unit: Å). The graphs from top to bottom represent the results of
carbon atoms, nitrogen atoms, and oxygen atoms, respectively. Figure 20:
Illustration of persistent sheaf Betti numbers (red line) and the first
nonzero eigenvalues (blue line) of persistent sheaf Laplacian of the RBD
binding site of the wild type RBD-ACE2 complex (PDB ID: 6M0J) and closed state
spike protein (PDB ID: 7DF3, Chain ID: A) at different filtration values,
i.e., radii (unit: Å). The graphs from top to bottom represent the results of
carbon atoms, nitrogen atoms, and oxygen atoms respectively. Figure 21:
Illustration of persistent sheaf Betti numbers (red line) and the first
nonzero eigenvalues (blue line) of persistent sheaf Laplacian of the RBD
binding site of Alpha RBD-ACE2 complex (PDB ID: 8DLK) and closed state spike
protein (PDB ID: 7LWS, Chain ID: A) at different filtration values, i.e.,
radii (unit: Å). The graphs from top to bottom represent the results of carbon
atoms, nitrogen atoms, and oxygen atoms, respectively. Figure 22: Illustration
of persistent sheaf Betti numbers (red line) and the first nonzero eigenvalues
(blue line) of persistent sheaf Laplacian of the RBD binding site of Beta RBD-
ACE2 complex (PDB ID: 8DLN) and closed state spike protein (PDB ID: 7LYM,
Chain ID: A, B, C) at different filtration values, i.e., radii (unit: Å). The
graphs from top to bottom represent the results of carbon atoms, nitrogen
atoms, and oxygen atoms, respectively. Figure 23: Illustration of persistent
sheaf Betti numbers (red line) and the first nonzero eigenvalues (blue line)
of persistent sheaf Laplacian of the RBD binding site of BA.1 RBD-ACE2 complex
(PDB ID: 7T9L) and closed state spike protein (PDB ID: 7TF8, Chain ID: A, B,
C) at different filtration values, i.e., radii (unit: Å). The graphs from top
to bottom represent the results of carbon atoms, nitrogen atoms, and oxygen
atoms, respectively. Figure 24: Illustration of persistent sheaf Betti numbers
(red line) and the first nonzero eigenvalues (blue line) of persistent sheaf
Laplacian of the RBD binding site of BA.2 RBD-ACE2 complex (PDB ID: 7XB0) and
closed state spike protein (PDB ID: 7XIX, Chain ID: A) at different filtration
values, i.e., radii (unit: Å). The graphs from top to bottom represent the
results of carbon atoms, nitrogen atoms, and oxygen atoms, respectively.


# Model-Agnostic Human Preference Inversion in Diffusion Models

Jeeyung Kim, Ze Wang, Qiang Qiu
Purdue University
{jkim17, wang5026<EMAIL_ADDRESS>
###### Abstract

Efficient text-to-image generation remains a challenging task due to the high
computational costs associated with the multi-step sampling in diffusion
models. Although distillation of pre-trained diffusion models has been
successful in reducing sampling steps, low-step image generation often falls
short in terms of quality. In this study, we propose a novel sampling design
to achieve high-quality one-step image generation aligning with human
preferences, particularly focusing on exploring the impact of the prior noise
distribution. Our approach, Prompt Adaptive Human Preference Inversion (PAHI),
optimizes the noise distributions for each prompt based on human preferences
without the need for fine-tuning diffusion models. Our experiments showcase
that the tailored noise distributions significantly improve image quality with
only a marginal increase in computational cost. Our findings underscore the
importance of noise optimization and pave the way for efficient and high-
quality text-to-image synthesis.

##  1 Introduction

Recent advances in diffusion models (DMs) have unlocked unprecedented
capabilities in text-to-image generation [25]. Despite their promise, the
widespread adoption of DMs in practical applications may be hindered by the
high inference costs associated with the multi-step sampling. To tackle this
challenge, considerable efforts have been focused on reducing the number of
sampling steps while maintaining image quality. In particular, distilling pre-
trained DMs using adversarial approaches [27, 33] and consistency
regularization on ordinary differential equation (ODE) trajectories [30, 15]
emerged as effective strategies. These methods enable high-fidelity text-to-
image generation in a few steps. However, images produced in fewer steps often
exhibit inferior quality compared to those generated with more steps [27].

Prior studies [9, 28, 14] modify the sampling process to improve the quality
of image generation without altering the training procedure. They propose
methods to enhance the multi-step sampling in DMs, focusing on modifications
to solve the probability flow (PF), such as adjustments to noise schedules [9,
14] and control of stochasticity levels [28]. Nevertheless, for distilled
models with extremely limited sampling steps, alternative sampling
designs—orthogonal to direct manipulation of the PF solver—can be essential to
further advance image quality, particularly considering that the
characteristics of DMs can diminish in distilled models. One potential
approach involves developing a sampling process combined with prompt
optimization. [7, 16] introduce a prompt optimization framework that tailors
user input to model-preferred prompts.

Another promising approach is optimizing noise prior, an additional input
alongside prompts in DMs. Notably, the prior noise distribution, typically
fixed as a standard Gaussian, has been overlooked in optimization to enhance
sampling quality. Especially when the sampling process in DMs is
deterministic, the prior noise directly shapes and influences the resulting
images, underscoring the potential for adjusting the noise distribution for
superior images. Therefore, our study aims to discover a noise distribution
that surpasses the standard Gaussian in generating higher-quality images. We
focus on one-step generation because it clearly reflects the impact of prior
noise, unlike multi-step generation where the effect of prior noise can be
gradually attenuated during sampling.

To discover the (sub-)optimal noise distribution, we employ a DM as an image
generator and a scoring model [12] as an evaluator to assess the generated
image quality. This scoring model was trained on human preferences regarding
image-prompt pairs and plays a pivotal role in providing feedback on the
alignment of generated images with human preferences, emulating a Human-in-
the-Loop framework. Inspired by optimization-based inversion techniques [1,
37] commonly used in Generative Adversarial Networks (GANs) [5], we directly
optimize the parameters of noise distribution based on human preference
scores, leading to enhancements in image quality while keeping DMs intact.

Taking our investigation a step further, we show that (sub-)optimal noise
distributions can vary depending on the text prompts provided. We additionally
introduce a crucial component: a noise predicting model. The light-weight
model, comprising a pre-trained text encoder and shallow layers, processes
text prompts and predicts parameters of a Gaussian distribution. This
distribution is then employed as a prior noise for the image generator. We
train the noise predicting model to produce noises that help DM generate
images with high scores. We demonstrate that the noise Gaussian distribution
with parameters tailored to specific text prompts generates superior images in
one step, aligning with human preference.

Our method involves inverting human preferred images back into the noise
space, termed as Prompt Adaptive Human Preference Inversion (PAHI). PAHI
serves as a model-agnostic image enhancement approach by adjusting the noise
distribution using a lightweight noise-predicting model. Notably, our method
is a general framework applicable in multi-step deterministic sampling
scenarios, extending its potential impact beyond one-step sampling. To the
best of our knowledge, we are the first to investigate the impact of noise
optimization on text-to-image synthesis. By bridging the efficiency of low-
step generation with enhanced quality, we unlock the potential of diffusion
models for real-world applications.

##  2 Related Work

Distilling diffusion models for low-step sampling. Numerous studies [17, 30,
15, 13, 27, 33, 26, 33] focused on accelerating sampling process of DMs. Among
them, Song et al. [30] introduces consistency models, where points on the same
trajectory of ODE map to the same initial point, enabling low-step generation.
However, this method has not yet been applied to text-to-image synthesis. In
the realm of text-to-image generation, Luo et al. [15] applies the consistency
model to latent space and facilitates text-to-image generation by distilling
Stable Diffusion [25]. Another notable approach to regulating trajectories of
ODEs is Liu et al. [13] introducing the concept of Rectified Flow, aimed at
straightening the trajectories of ODEs. On the other hand, Sauer et al. [27]
employs adversarial loss and score distillation sampling loss [20] to generate
images, showcasing superior performance compared to other distillation
methods. However, low-step (1 or 2) generation often display inferior quality
compared to multiple-step image generation.

Prompt adaptive sampling design. Hao et al. [7] introduce prompt adaptation,
adjusting user input to match model-preferred prompts without DM fine-tuning.
During sampling, text inputs undergo prompt adaptation processing before being
fed into diffusion models. Zhang et al. [34] emphasize the significance of
employing diverse sampling steps relying on prompts and propose a framework of
sampling with instance-specific steps to reduce sampling cost without
compromising image quality. However, none of these studies explore the
potential of optimizing noise distribution contingent upon prompts.

Fine-tuning diffusion models on human feedback. [31, 3, 2] align DMs with
human preferences by directly optimizing them based on human comparison data
using reinforcement learning. However, such approaches differ from our model-
agnostic approach, as it requires fine-tuning DMs, which can be resource-
intensive.

Inversion. In prior studies, inversion has been crucial for image
manipulation, aiming to find a latent representation corresponding to a given
image. In the GAN literature, [1, 37, 6, 24] use optimization-based
techniques, directly optimizing latent vectors, while [23, 36, 19] employ
trained encoders to map images to their latent representations. In DMs,
inversion is adapted to accommodate their properties like stochastic, multi-
step sampling and conditioned generation. Textual Inversion [4] represents
user-provided concepts as pseudo-words in the text embedding space for
versatile editing. [28] proposes a deterministic sampling which enables
inversion in a closed-form manner, facilitating image manipulation in DMs [29,
22, 18, 10, 35].

In contrast to prior works, our study does not use inversion technique for the
purpose of image manipulation. Instead, we invert human preferences into noise
space, where samples from this space lead to improved image quality.

##  3 Method

###  3.1 Preliminaries

In this study, we use the distilled diffusion model (Stable Diffusion 2.1
backbone [25]) trained with adversarial loss [27], which exhibits superior
performance in low-step image generation. Similar to general diffusion models,
the generative process of the distilled model progressively denoises a noisy
observation starting from the standard Gaussian where
$p(\bm{x}_{T})=\mathcal{N}(\bm{x}_{T};\bm{0},\bm{I})$ and $T$ represents the
total number of timesteps during the training of the diffusion model. The
denoised observation is predicted with the distilled diffusion model
($\epsilon_{\theta}$) [8] as followed:

$\text{SD}_{\theta}(\bm{x}_{\tau_{l}},c):=\frac{\bm{x}_{\tau_{l}}-\sqrt{1-\alpha_{\tau_{l}}}\cdot\epsilon_{\theta}(\bm{x}_{\tau_{l}},c)}{\sqrt{\alpha_{\tau_{l}}}},$
(1)

where $\text{SD}_{\theta}(\cdot,\cdot)$ represents a denoised image,
$l\in\\{1,\cdots,L\\}$, $\tau_{L}=T$ and $L$ is set to values less than 4 in
[27]. $c$ denotes conditions and $\alpha_{\tau_{l}}$ denotes a variance
schedule at $\tau_{l}$. Note that our study primarily concentrates on one-step
generation ($L=1$).

On the other hand, Kirstain et al. [12] create an open dataset of text prompts
and real users’ preferences over generated images. Based on this dataset, they
train a CLIP-based [21] scoring model, named PickScore, which shows superior
alignment with human preferences on generated images. We use PickScore as our
scoring model in the following sections. Note that alternative models can also
be employed as a scoring model.

###  3.2 The Proposed Method (PAHI)

In this section, we introduce our proposed method which enhances image quality
while preserving intact diffusion models. We first propose a human preference
inversion method in which a single noise distribution is optimized for all
prompts. Following this, we propose Prompt Adaptive Human Preference Inversion
(PAHI) that predicts customized noise distributions for individual prompts.

Optimizing noise distribution across all prompts. We employ a distilled
diffusion model for image generation and a scoring model to evaluate the
generated images. The output of scoring model reflects predicted human
preference for the generated images. The score of the generated image given
the text prompt $c_{i}$ is defined as followed:

$s(\bm{x}_{T}^{m},c_{i})=\text{SC}_{\phi}(\text{SD}_{\theta}(\bm{x}_{T}^{m},c_{i}),c_{i}),$
(2)

where $\text{SC}_{\phi}$ is a scoring model, $\bm{x}_{T}^{m}$ denotes the
$m$-th sample from $p(\bm{x}_{T})$, $c_{i}\in\\{c_{1},c_{2},\dots,c_{n}\\}$
and $n$ denotes the number training text prompts. We interchangeably use
$s(\bm{x}_{T}^{m},c_{i})$ and $s(\bm{x}_{T},c_{i})$.

We posit the existence of a potentially superior Gaussian distribution
($p(\bm{x}_{T}^{\prime})$) compared to the standard Gaussian, which satisfies

$p(\bm{x}_{T}^{\prime})=\mathcal{N}(\bm{x}_{T};\bm{\mu},\text{diag}(\bm{\sigma})),\;s(\bm{x}_{T}^{\prime},c_{i})>s(\bm{x}_{T},c_{i}).$
(3)

Thus, we aim to optimize $\bm{\mu}$ and $\bm{\sigma}$ of the prior noise
($X_{T}^{\prime}$) to maximize scores (align with human preference), where
$\bm{\mu},\bm{\sigma}\in\mathcal{R}^{4k^{2}}$ and $k$ denotes the size of
latent variable in the pre-trained latent diffusion model [25].

We find a superior Gaussian distribution ($p(\bm{x}_{T}^{\prime})$) by
minimizing the objective function defined as followed:

$\displaystyle\mathcal{L}=-\sum^{n}(0\cdot\log$ $\displaystyle
f(\bm{\mu},\bm{\sigma})+1\cdot\log f^{\prime}(\bm{\mu},\bm{\sigma})),$ (4)
$\displaystyle\bm{\mu^{*}},\bm{\sigma^{*}}$
$\displaystyle=\underset{\bm{\mu},\bm{\sigma}}{\mathrm{argmin}}\;\mathcal{L},$
(5)

where $f(\bm{\mu},\bm{\sigma})=\frac{e^{s}}{e^{s}+e^{s^{\prime}}}$ and
$f^{\prime}(\bm{\mu},\bm{\sigma})=\frac{e^{s^{\prime}}}{e^{s}+e^{s^{\prime}}}$.
We use $s^{\prime}$ instead of $s(\bm{x}_{T}^{\prime},c_{i})$ and $s$ instead
of $s(\bm{x}_{T},c_{i})$ for brevity.

To optimize parameters of Gaussian distribution, we use the reparameterization
trick [11] as followed:

$\displaystyle\bm{x}_{T}^{\prime
j}=\bm{\epsilon}^{\prime}\odot\bm{\sigma}^{2}+\bm{\mu},\;\text{where}\;\bm{\epsilon}^{\prime}\sim
N(\textbf{0},\textbf{I}),$ (6)

where $\bm{x}_{T}^{\prime j}$ denotes $j$-th sample from
$p(\bm{x}^{\prime}_{T})$. The sample is used as input to $\text{SD}_{\theta}$
alongside the text prompt $c_{i}$. We use the identified $\bm{\mu^{*}}$ and
$\bm{\sigma^{*}}$ for all prompts during inference.

Prompt-adaptive noise distribution. We take one step further to tailor
$\bm{\mu}(c_{i})$ and $\bm{\sigma}(c_{i})$ for an individual prompt $c_{i}$.
Built upon the previous framework, we further construct a conditional noise
prediction model consisting of a pre-trained text encoder ($E$) and respective
2 MLPs ($g_{\psi}$) for predicting $\bm{\mu}(c_{i})$ and $\bm{\sigma}(c_{i})$.
The $g_{\psi}$ takes text embedding as input and outputs predictions of
parameters of a Gaussian distribution.

The entire procedure is as followed:

$\displaystyle g_{\psi}(E(c_{i}))=$
$\displaystyle(\bm{\mu}(c_{i}),\bm{\sigma}(c_{i})),$ (7)
$\displaystyle\bm{x}_{T}^{\prime
j}=\bm{\epsilon}^{\prime}\odot\bm{\sigma}(c_{i})^{2}+$
$\displaystyle\bm{\mu}(c_{i}),\;\text{where}\;\bm{\epsilon}^{\prime}\sim
N(\textbf{0},\textbf{I}),$ (8) $\displaystyle
s(\bm{x}^{\prime}_{T},c_{i})=\text{SC}_{\phi}$
$\displaystyle(\text{SD}_{\theta}(\bm{x}_{T}^{\prime j},c_{i}),c_{i}),$ (9)

where we use the text encoder of the diffusion model ($\epsilon_{\theta}$) for
$E(\cdot)$ not to impose additional computation. We find $\psi^{*}$ that
minimizes objective function defined in Eq. 4. During inference, we initially
identify the optimal $\bm{\mu}^{*}(c_{i})$ and $\bm{\sigma}^{*}(c_{i})$ for
prompt $c_{i}$ using the noise predicting model and then employ the Gaussian
as a prior noise for the distilled diffusion models to generate images.

When $\psi$ are initialized randomly, $\bm{\mu}(c_{i})$ and
$\bm{\sigma}(c_{i})$ may significantly deviate from their appropriate values,
$\bm{0}\;\text{and}\;\bm{I}$. To stabilize the initial training phase, we opt
to pre-train $g_{\psi}$ to produce parameters close to a standard Gaussian
distribution while retaining the input text embedding information. This is
achieved by minimizing the Kullback-Leibler (KL) divergence between the
standard Gaussian and the Gaussian with the predicted parameters as well as a
reconstruction loss of the text embeddings. We reconstruct text embedding
using a decoder ($h_{\omega}$), 2-layer MLP, taking as input a sample of
$\bm{x}^{\prime}_{T}$ from $g_{\psi}$ and producing reconstructed text
embeddings as output.

The loss function used for pretraining is structured as followed:

$\displaystyle\mathcal{L^{\prime}}=\sum_{i}^{n}\text{KL}(N(\bm{\mu}(c_{i}),\bm{\sigma}(c_{i})),\;$
$\displaystyle N(\textbf{0},\textbf{I}))$
$\displaystyle+\;\text{MSE}(E(c_{i})$
$\displaystyle,h_{\omega}(g_{\psi}(E(c_{i}))).$ (10)

We optimize $\psi$ and $\omega$ by minimizing the loss function. Subsequently,
the updated parameters $\psi$ serve as the initialization of our noise
prediction model for further training.

##  4 Experiment

We validate our framework (PAHI) by demonstrating that the images generated by
our method exhibit enhanced quality, based on one-step generation.

Experiment setups. We conduct a comparison between images generated by our
method and ones generated using the standard Gaussian distribution as a prior.
We employ the dataset proposed by [12], which consists of 35,000 distinct
prompts. We randomly select 500 prompts for validation and another 500 prompts
for the test set, while the remaining prompts are used for training. The
scoring model used for training is PickScore [12]. During evaluation, we
employ both the PickScore and ImageReward scoring models [32] to assess image
quality and determine whether images optimized with PickScore align
effectively with the criteria of ImageReward. As an evaluation metric, we
employ the win rate, determined by comparing the scores of two images
generated from the same prompt: one using a predicted noise distribution and
the other using the standard Gaussian distribution. We denote our prompt-
adaptive inversion method as PAHI, while using a single inversion across all
prompts is referred to as HI.

Implementation details. Our implementation is built upon the Huggingface
Diffuser framework111https://huggingface.co/docs/diffusers/en/index. We
integrate ADD-M [27], where we reduce the computation and memory costs by
replacing the VAE with a tiny
autoencoder222https://github.com/madebyollin/taesd. For PAHI, we use the text
encoder of the employed diffusion model as $E(\cdot)$ as it removes the need
for additional computation. We generate images with a size of 5122. We
configure the batch size to 72 and implement a learning rate warm-up for
10,000 steps, gradually increasing from $1\times 10^{-5}$. We employ the Adam
optimizer. We use early stopping based on the average evaluation loss from
Pickscore and Imagereward scoring models. If this loss does not drop for five
consecutive evaluations, conducted every 1000 steps, training stops. The
inference time is measured using a Nvidia RTX 3090 GPU.

###  4.1 Results

Table 1: The win rate against images generated from the standard Gaussian using different scoring models. The numbers represent the average (std) win rates of 5 runs with different seeds. | PickScore [12] | ImageReward [32]
---|---|---
PAHI |  94.0% (0.2) |  75.5% (2.1)
HI | 64.7% (1.6) | 64.1% (1.8)
Table 2: Comparison of inference times between low step generation and our method: The numerical values represent the average time (scores) taken for an image sampling across 500 text prompts. We use batch size of 1. | Time ($s$) ($\downarrow$) | Scores (PickScore) ($\uparrow$)
---|---|---
PAHI | 0.067 | 0.228
one step [27] | 0.062 | 0.212
two steps [27] | 0.088 | 0.212

Human preference scores comparison. Our approach showcases superior
performance compared to the baseline, as illustrated in Table 1. PAHI
significantly outperforms a standard Gaussian, with a remarkable win rate of
98.4%. HI also shows effectiveness, achieving a win rate of around 65%.
However, its improvement does not match the efficacy of PAHI. Interestingly,
even though trained with PickScore as the scoring model, PAHI (HI) excels in
achieving higher ImageReward scores as well, boasting a win rate of 70.7%
(64.1%). These results demonstrate the effectiveness of optimizing noise
distribution in enhancing image quality. Moreover, it underscores the
importance of tailoring the noise distribution based on each prompt
specifically.

(a) Head shot of a dragon, digital art style

(b) Minnie Mouse in a superman outfit bodybuilding, book illustration

(c) a velociraptor and an MGb in the jungle river, waterfall mist, Chrome
Detailing

Figure 1: The prompts generated by users and the corresponding images sampled
in one-step from the standard Gaussian (left) and the predicted noise
distributions (right).

Sampling computation cost. We investigate the trade-off between image
generation quality and sampling cost during inference. Typically, higher image
quality is achieved with more sampling steps, which incur additional
computation costs. We compare our approach with the standard Gaussian,
employing 1 and 2 sampling steps [27]. We present the time required for
sampling to assess computational efficiency as well as the resulting scores to
identify their quality, as depicted in Table 2. Despite no significant
difference in scores between one-step and two-step generation, two-step
generation incurs additional time for sampling. In contrast, our approach
demonstrates remarkable efficiency, requiring only a minimal increase in
inference time compared to one-step generation (+0.005s), while showcasing
higher scores aligned with human preferences. When it comes to the number of
parameters, PAHI adds 5 million more parameters (two MLP layers) in total,
which are negligible compared to the 983 million parameters of Stable
Diffusion 2.1 [27]. Note that we utilize the text encoder from the employed
diffusion model, thus eliminating the necessity to introduce additional
parameters to encode prompts.

Generated images. Figure 1 illustrates the generated images from our noise
prediction and a standard Gaussian. We underscore the images from the
predicted noise (right) demonstrate improved image quality, aligning better
with human preference.

##  5 Conclusion

Our study investigated improving the quality of text-to-image one-step
generation. We proposed a light-weight noise predicting model, optimizing
noise distributions based on human preferences without fine-tuning diffusion
models. Our experiments exhibited the tailored noise distributions improve
image quality with marginal rise in computing cost. We highlight the efficacy
of noise optimization, promising efficient and high-quality text-to-image
synthesis.

## References

  * Abdal et al. [2019] Rameen Abdal, Yipeng Qin, and Peter Wonka.  Image2stylegan: How to embed images into the stylegan latent space?  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 4432–4441, 2019.
  * Deng et al. [2024] Fei Deng, Qifei Wang, Wei Wei, Matthias Grundmann, and Tingbo Hou.  Prdp: Proximal reward difference prediction for large-scale reward finetuning of diffusion models.  _arXiv preprint arXiv:2402.08714_ , 2024.
  * Fan et al. [2024] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee.  Reinforcement learning for fine-tuning text-to-image diffusion models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ , 2022.
  * Goodfellow et al. [2014] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Gu et al. [2020] Jinjin Gu, Yujun Shen, and Bolei Zhou.  Image processing using multi-code gan prior.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3012–3021, 2020.
  * Hao et al. [2023] Yaru Hao, Zewen Chi, Li Dong, and Furu Wei.  Optimizing prompts for text-to-image generation.  _Advances in Neural Information Processing Systems_ , 36, 2023.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Karras et al. [2022] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _Advances in Neural Information Processing Systems_ , 35:26565–26577, 2022.
  * Kim et al. [2022] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye.  Diffusionclip: Text-guided diffusion models for robust image manipulation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2426–2435, 2022.
  * Kingma and Welling [2013] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ , 2013.
  * Kirstain et al. [2024] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy.  Pick-a-pic: An open dataset of user preferences for text-to-image generation.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Liu et al. [2023] Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al.  Instaflow: One step is enough for high-quality diffusion-based text-to-image generation.  In _The Twelfth International Conference on Learning Representations_ , 2023.
  * Lu et al. [2022] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _Advances in Neural Information Processing Systems_ , 35:5775–5787, 2022.
  * Luo et al. [2023] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao.  Latent consistency models: Synthesizing high-resolution images with few-step inference.  _arXiv preprint arXiv:2310.04378_ , 2023.
  * Mañas et al. [2024] Oscar Mañas, Pietro Astolfi, Melissa Hall, Candace Ross, Jack Urbanek, Adina Williams, Aishwarya Agrawal, Adriana Romero-Soriano, and Michal Drozdzal.  Improving text-to-image consistency via automatic prompt optimization.  _arXiv preprint arXiv:2403.17804_ , 2024.
  * Meng et al. [2023] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans.  On distillation of guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14297–14306, 2023.
  * Mokady et al. [2023] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6038–6047, 2023.
  * Pidhorskyi et al. [2020] Stanislav Pidhorskyi, Donald A Adjeroh, and Gianfranco Doretto.  Adversarial latent autoencoders.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14104–14113, 2020.
  * Poole et al. [2022] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall.  Dreamfusion: Text-to-3d using 2d diffusion.  _arXiv preprint arXiv:2209.14988_ , 2022.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Richardson et al. [2021] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro, and Daniel Cohen-Or.  Encoding in style: a stylegan encoder for image-to-image translation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2287–2296, 2021.
  * Roich et al. [2022] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or.  Pivotal tuning for latent-based editing of real images.  _ACM Transactions on graphics (TOG)_ , 42(1):1–13, 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Salimans and Ho [2022] Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  _arXiv preprint arXiv:2202.00512_ , 2022.
  * Sauer et al. [2023] Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach.  Adversarial diffusion distillation.  _arXiv preprint arXiv:2311.17042_ , 2023.
  * Song et al. [2020a] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ , 2020a.
  * Song et al. [2020b] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020b.
  * Song et al. [2023] Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever.  Consistency models.  _arXiv preprint arXiv:2303.01469_ , 2023.
  * Wallace et al. [2023] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik.  Diffusion model alignment using direct preference optimization.  _arXiv preprint arXiv:2311.12908_ , 2023.
  * Xu et al. [2024] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong.  Imagereward: Learning and evaluating human preferences for text-to-image generation.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Yin et al. [2023] Tianwei Yin, Michaël Gharbi, Richard Zhang, Eli Shechtman, Fredo Durand, William T Freeman, and Taesung Park.  One-step diffusion with distribution matching distillation.  _arXiv preprint arXiv:2311.18828_ , 2023.
  * Zhang et al. [2023] Hui Zhang, Zuxuan Wu, Zhen Xing, Jie Shao, and Yu-Gang Jiang.  Adadiff: Adaptive step selection for fast diffusion.  _arXiv preprint arXiv:2311.14768_ , 2023.
  * Zhang et al. [2024] Yuechen Zhang, Jinbo Xing, Eric Lo, and Jiaya Jia.  Real-world image variation by aligning diffusion inversion chain.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Zhu et al. [2020a] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou.  In-domain gan inversion for real image editing.  In _European conference on computer vision_ , pages 592–608. Springer, 2020a.
  * Zhu et al. [2020b] Peihao Zhu, Rameen Abdal, Yipeng Qin, John Femiani, and Peter Wonka.  Improved stylegan embedding: Where are the good latents?  _arXiv preprint arXiv:2012.09036_ , 2020b.