#  Canalization of the evolutionary trajectory of the human influenza virus

Trevor Bedford  Department of Ecology and Evolutionary Biology, University of
Michigan, Ann Arbor, MI, USA.  Howard Hughes Medical Institute, University of
Michigan, Ann Arbor, MI, USA.  Andrew Rambaut  Institute of Evolutionary
Biology, University of Edinburgh, Edinburgh, UK.  Fogarty International
Center, National Institutes of Health, Bethesda, MD, USA.  Mercedes Pascual
Department of Ecology and Evolutionary Biology, University of Michigan, Ann
Arbor, MI, USA.  Howard Hughes Medical Institute, University of Michigan, Ann
Arbor, MI, USA.

Abstract

Since its emergence in 1968, influenza A (H3N2) has evolved extensively in
genotype and antigenic phenotype. Antigenic evolution occurs in the context of
a two-dimensional ‘antigenic map’, while genetic evolution shows a
characteristic ladder-like genealogical tree. Here, we use a large-scale
individual-based model to show that evolution in a Euclidean antigenic space
provides a remarkable correspondence between model behavior and the
epidemiological, antigenic, genealogical and geographic patterns observed in
influenza virus. We find that evolution away from existing human immunity
results in rapid population turnover in the influenza virus and that this
population turnover occurs primarily along a single antigenic axis. Thus,
selective dynamics induce a canalized evolutionary trajectory, in which the
evolutionary fate of the influenza population is surprisingly repeatable and
hence, in theory, predictable.

## Author Summary

Each year, hundreds of millions become sick with the influenza virus and
hundreds of thousands die from the infection. After recovery from the flu, a
person gains permanent immunity specific to the infecting strain. However,
owing to its RNA makeup, mutations occur rapidly to the virus genome. Some of
these mutations change the shape of proteins visible to the human immune
system, and thus alter the virus’s antigenic phenotype. These mutations allow
the virus to re-infect those who have previously recovered from an earlier
strain, and thus will quickly spread through the virus population. It is
because of influenza’s rapid antigenic evolution that the flu vaccine needs
frequent updating. However, despite strong pressure to evolve away from human
immunity and to diversify in antigenic phenotype, influenza, and especially
influenza A (H3N2), shows paradoxically limited genetic and antigenic
diversity present at any one time. Here, we propose a simple model of
influenza that displays rapid evolution but low standing diversity and
simultaneously accounts for the epidemiological, genetic, antigenic and
geographical patterns displayed by the virus. We find that in evolving
directly away from existing human immunity, the virus severely limits its
future evolutionary potential.

## Introduction

Epidemic influenza is responsible for between 250,000 and 500,000 global
deaths annually, with influenza A (and in particular, A/H3N2) having caused
the bulk of human mortality and morbidity [1]. Influenza A (H3N2) has
continually circulated within the human population since its introduction in
1968, exhibiting recurrent seasonal epidemics in temperate regions and less
periodic transmission in the tropics. During this time, H3N2 influenza has
continually evolved both genetically and antigenically. Most antigenic drift
is thought to be driven by changes to epitopes in the hemagglutinin (HA)
protein [2]. Phylogenetic analysis of the genetic relationships among HA
sequences has revealed a distinctive genealogical tree showing a single
predominant trunk lineage and side branches that persist for only 1–5 years
before going extinct [3]. This tree shape is indicative of serial replacement
of strains over time; H3N2 influenza shows rapid evolution, but low standing
genetic diversity.

This observation has remained puzzling from an epidemiological standpoint.
Antigenic evolution occurs rapidly and strong diversifying selection exists to
escape from human immunity; why then do we see serial replacement of strains
rather than continual genetic and antigenic diversification? Indeed, simple
epidemiological models show explosive diversity of genotype and phenotype over
time [4, 5]. Previous work has sought model-based explanations of the limited
diversity of influenza, relying on short-lived strain-transcending immunity
[4, 5], complex genotype-to-phenotype maps [6] or a limited repertoire of
antigenic phenotypes [7].

Experimental characterization of antigenic phenotype is possible through the
hemagglutination inhibition (HI) assay, which measures the cross-reactivity of
HA from one virus strain to serum raised against another strain [8]. The
results of many HI assays can be combined to yield a two-dimensional map,
representing antigenic similarity and distance between strains as an easily
visualized and quantified measure [9]. The path traced across this map by
influenza A (H3N2) from 1968 until present is largely linear, showing serial
replacement of one strain by another; there are no major bifurcations of
antigenic phenotype [9].

Herein, we seek to simultaneously model the genetic and antigenic evolution of
the influenza virus. We represent antigenic phenotypes as points in a
$N$-dimensional Euclidean space. Based on the finding that a two-dimensional
map adequately explains observed antigenic distance between strains [9], we
begin with antigenic phenotypes as points on a plane, but relax this
assumption later on in the analysis. After exposure to a virus, a host’s risk
of infection is proportional to the Euclidean distance between the infecting
phenotype and the closest phenotype in the host’s immune history. Mutations
perturb antigenic phenotype, moving phenotype in a random radial direction and
for a randomly distributed distance. We implemented this geometrical model in
a large-scale individual-based simulation intended to directly model the
antigenic map and genealogical tree of the global influenza population. The
simulation includes multiple host populations with different seasonal forcing,
hosts with complete immune histories of infection, and viruses with antigenic
phenotypes. As the simulation proceeds, infections are tracked and a complete
genealogy connecting virus samples is constructed. Results shown here are for
a single representative simulation of 40 years of virus evolution in a
population of 90 million hosts.

## Results

The virus persists over the course of the 40-year simulation, infecting a
significant fraction of the host population through annual winter epidemics in
temperate regions and through less periodic epidemics in the tropics (Figure
1A). Across replicate simulations, we observe average yearly attack rates of
6.8% in temperate regions and rates of 7.1% in the tropics, comparable with
estimated attack rates of influenza A (H3N2) of 3–8% per year [10, 11]. Over
the course of the simulation, the virus population evolves in antigenic
phenotype exhibiting, at any point, a handful of highly abundant phenotypes
sampled repeatedly and a large number of phenotypes appearing at low abundance
(Figure 1B). By including measurement noise on antigenic locations, we
approximate an experimental antigenic map of H3N2 influenza (Figure 1D). The
appearance of clusters in the antigenic map comes from the regular spacing of
high abundance phenotypes combined with measurement noise. Over time, clusters
of antigenically similar strains are replaced by novel clusters of more
advanced strains (Figure S1A). Across replicate simulations, clusters persist
for an average of 5.0 years measured as the time it takes for a new cluster to
reach 10% frequency, peak and decline to 10% frequency. The transition between
clusters occurs quickly, taking an average of 1.8 years.

Remarkably, although antigenic phenotype is free to mutate in any direction in
the two-dimensional space, selection pressures force the virus population to
move in nearly a straight line in antigenic space (Figure 1B). Across
replicate simulations, 94% of the variance of antigenic phenotype can be
explained by a single dimension of variation. This mirrors the empirical
results showing a largely linear antigenic map for H3N2 influenza isolates
from 1968 to 2003 [9]. Because of the primarily one-dimensional movement,
antigenic distance from the original phenotype increases nearly linearly with
time (Figure S1B). Antigenic evolution occurs in a punctuated fashion; periods
of relative stasis are interspersed with more rapid antigenic change (Figure
S1B). Antigenic and epidemiological dynamics show a fundamental linkage, so
that large jumps of antigenic phenotype result in increased rates of infection
(Figure 1, Figure S2). In general, evolution via many smaller mutations
results in a smoother antigenic map and less variation in yearly epidemics
(Figure S3), while evolution via rare mutations of large effect exhibits a
more clustered antigenic map and wider variation in seasonal incidence (Figure
S4).

The genealogical tree connecting the evolving virus population appears
characteristically sparse with pronounced trunk and short-lived side branches
(Figure 1C). This tree shape is reflected in low levels of standing diversity;
across replicates, an average of 5.68 years of evolution separate two randomly
sampled viruses in the population. This level of diversity matches what is
observed in phylogenies of influenza A (H3N2) [12]. A spindly genealogical
tree is indicative of population turnover, wherein novel antigenic phenotypes
continually replace more primitive ‘spent’ phenotypes, purging their
genealogical diversity. In general, natural selection reduces effective
population size and genealogical diversity [13]. Here, by comparing mutations
occurring on trunk branches vs. mutations occurring on side branches, we find
evidence for pervasive positive selection for antigenic change (Table 1).
Trunk mutations tend to push antigenic phenotype forward along the line of
primary antigenic variation (Figure S5). We find a roughly linear relationship
between the antigenic effect of a mutation and the likelihood that this
mutation becomes incorporated into the trunk (Figure S6). Additionally, we
find that trunk mutations occur at strikingly regular intervals, with less
variation of waiting times than expected under a simple random process (Figure
S7). There is a relative scarcity of mutation events occurring in intervals
under 1 year and a relative excess of a mutation events occurring in 2–3 year
intervals (Figure S7).

The genealogical tree also contains detailed information on the history of
migration between regions. We find that, consistent with empirical estimates
[14, 15], the trunk resides primarily within the tropics, where seasonal
dynamics are less prevalent (Figure 2A). Across replicate simulations, we
observe 72% of the trunk’s history within the tropics and 28% within temperate
regions. With symmetric host contact rates and equal host population sizes,
and without seasonal forcing, we would expect trunk proportions of one third
for each region. We calculated rates of migration based on observed event
counts across replicate simulations, separating region-specific rates on side
branches from region-specific rates on trunk branches. We find that migration
patterns on side branches are close to symmetric, with similar rates between
all regions, while migration patterns on trunk branches are highly asymmetric,
with high rates of movement between temperate regions and from temperate
regions into the tropics (Figure 2B). Extrapolating from these rates, we
arrive at an expected stationary distribution of 76% tropics and 24% temperate
regions, in line with the observed residency patterns of the trunk. It may at
first seem counter-intuitive to see higher rates of movement from the
temperate regions into the tropics along trunk branches, but it makes sense
when thought of in terms of conditional probability. Only those lineages that
migrate into the tropics or those lineages which rapidly migrate between the
north and south have a chance at becoming the trunk lineage, while lineages
that remain within the temperate regions are doomed to extinction.

These findings suggest that persistence and migration are fundamentally
connected and have important implications for future phylogeographic analyses.
Russell et al. [14] emphasize a source-sink model of movement of the HA
protein of influenza A (H3N2) based on their finding of a trunk lineage
residing within China and the Southeast Asian tropics. Whereas, Bedford et al.
[15] emphasize a global metapopulation model based on phylogenetic inference
of migration rates across the entire tree. Our results suggest that both
scenarios are simultaneously possible; side branches may be highly volatile
moving rapidly and symmetrically between regions, while the trunk lineage may
be more stable remaining within a region (or within a highly connected network
of regions) that has more persistent transmission. In light of these results,
we suggest that future work on the phylogeography of influenza take into
account trunk vs. side branch differences in migration patterns.

## Discussion

### Correspondence between model and data

Although multiple epidemiological/evolutionary mechanisms have been proposed
to explain the restricted genetic diversity and rapid population turnover of
influenza A (H3N2) [4, 5, 6, 7], our results show that a simple model coupling
antigenic and genealogical evolution exhibits broad explanatory power. We find
a strong correspondence between the antigenic and genealogical patterns
generated by our model (Figure 1) and patterns of genetic and antigenic
evolution exhibited by influenza A (H3N2) [3, 9]. Our model suggests that
punctuated antigenic evolution need only be explained by a lack of mutational
opportunity and predicts that more detailed classification of influenza
strains will support a relatively small number of predominant phenotypes
(Figure 1B). We suggest that a large proportion of intra-cluster variation in
the observed antigenic map is due to experimental noise, rather than each
strain possessing a unique antigenic location. Additionally, our model
accurately predicts the contrasting dynamics of other types/subtypes of
influenza. We find that lowering mutation size/effect or lowering intrinsic
$R_{0}$ results in decreased incidence, slower antigenic movement and greater
genealogical diversity, all distinguishing characteristics of H1N1 influenza
and influenza B (Figure S8, Figure S9).

In our model, when antigenic phenotype remains static, there may be multiple
consecutive seasons without appreciable incidence (Figure 1A), a pattern
apparently absent from H3N2 influenza [16]. We suggest that any model
exhibiting punctuated evolution broadly consistent with the punctuated change
seen in the experimental antigenic map will show similar patterns of
incidence. We can ‘fix’ the incidence patterns, but at the cost of too smooth
an antigenic map (Figure S3). Evolutionary patterns of the neuraminidase (NA)
protein may provide an explanation. Epitopes in the HA and NA proteins are
jointly responsible for determining antigenicity [2], and it is now clear that
levels of adaptive evolution are similar between HA and NA [17]. Thus, changes
in NA may be driving incidence patterns as well, resulting in an observed
timeseries of incidence partially divorced from the antigenic map of HA.

It remains a central question as to the extent that short-lived strain-
transcending immunity is responsible for influenza’s limited diversity and
spindly genealogical tree [4, 5]. Our findings suggest a possible resolution.
Although lacking short-lived immunity, our model shows a detailed
correspondence to both the antigenic map and genealogical tree of H3N2
influenza. If an antigenic map were to show a deep bifurcation, where two
viral lineages move in different antigenic directions, then we would expect
the same bifurcation to be evident in the genealogical tree. Short-lived
strain-transcending immunity provides a mechanism by which lineages may
diverge in antigenic phenotype, but still show epidemiological interference.
This mechanism would explain a situation where bifurcations emerge in the
antigenic map, but competition results in the extinction of divergent
antigenic lineages. The empirical antigenic map [9] suggests that this is not
the case; one cluster leads to another cluster in orderly succession and there
is never competition between antigenically distant clusters. This supports the
hypothesis that antigenic evolution is primarily limited by a lack of
mutational availability. This is not to say that short-lived strain-
transcending immunity is not present; observed interference between subtypes
[4, 18] and evolution at CTL epitopes [19] provides substantial evidence for
its existence. Instead, we suggest that short-lived strain-transcending
immunity does not automatically generate antigenic maps and genealogical trees
consistent with empirical evidence.

### Linear antigenic movement

It would seem possible for one viral lineage to move in one antigenic
direction, while another lineage moves tangentially, eventually resulting in
two non-interacting viral lineages. Instead, we find that only movement in a
single antigenic direction is favored. The origins of this pattern can be seen
in the interaction between virus evolution and host immunity (Figure 3). As
the virus population evolves forward it leaves a wake of immunity in the host
population, and evolution away from this immunity results in the canalization
of the antigenic phenotype; mutations that continue along the line of primary
antigenic variation will show a transmission advantage compared to more
tangential mutations.

Following the work of Smith et al. [9], it remained an open question of why a
two-dimensional map should explain the antigenic variation of H3N2 influenza.
Although the authors astutely speculated that “there is a selective advantage
for clusters that move away linearly from previous clusters as they most
effectively escape existing population-level immunity, and this is a plausible
explanation for the somewhat linear antigenic evolution in regions of the
antigenic map.” This hypothesis remained to be tested. Here, we show from a
simple model of epidemiology and evolution that a linear trajectory of
antigenic evolution dynamically emerges due to basic selective pressures. This
result simultaneously explains the linear pattern of antigenic drift [9] and
the characteristically spindly genealogical tree [3] exhibited by influenza A
(H3N2).

To consider to what extent these results were contingent on the dimensionality
of the underlying antigenic model, we further implemented our model in a
10-dimensional antigenic space. Here, mutations occur as 10-spheres, but the
distance moved by a mutation is the same as in the previous two-dimensional
formulation. We arrive at nearly the same results with this model; principal
components analysis shows that the first and second dimensions of variation
account for 87% and 7%, respectively, of the total variance (Figure S10).
Thus, our model predicts that future work probing mutational effects will
support an underlying high-dimensional antigenic space, even though a two-
dimensional map is sufficient to explain observed antigenic relationships
among strains.

### Winding back the tape

It seems clear that, in our model, selection reduces the degrees of freedom of
antigenic evolution. In light of this, we wanted to examine the degree of
stochasticity in replicated evolutionary trajectories, and thereby test what
happens when we “wind back the tape” [20] on the evolution of the virus. We
ran 100 replicate simulations, each starting from the endpoint of the original
40-year simulation (Figure 4, Figure 5). Initially, we find a great detail of
repeatability; during the first year of evolution, every replicate virus
population undergoes a similar antigenic transition (Figure 4), resulting in a
repeatable peak in northern hemisphere incidence (Figure 5). After three
years, repeatability has mostly disappeared, with antigenic phenotype and
incidence appearing highly variable across replicates (Figure 4, Figure 5).
The 1–2 year timescale of repeatability can be explained by the presence of
standing antigenic variation. In the initial virus population, there are
several novel antigenic variants present at low frequency (Figure 3), one of
which, without fail, comes to predominate the virus population.

We see that the initial evolutionary trajectory, during which time standing
variation plays out, is highly repeatable, and thus predictable given enough
information and the right methods of analysis. However, prediction of longer-
term evolutionary scenarios will necessarily be difficult or impossible except
in a vague sense. Through careful surveillance efforts and genetic and
antigenic characterization of influenza strains, the World Health Organization
makes twice-yearly vaccine strain recommendations [21]. It should be possible
to combine these sorts of modeling approaches with surveillance data to gauge
the likelihood that a sampled variant will spread through the population.

Recent work on empirical fitness landscapes has shown that natural selection
follows few mutational paths [22]. The spindly genealogical tree and the
almost linear serial replacement of influenza strains has remained a puzzling
phenomenon. We suggest that the evolutionary and epidemiological dynamics
displayed by the influenza virus may simply be explained as an outgrowth of
selection to avoid host immunity. Natural selection can only ‘see’ one step
ahead, and so sacrifices long-term gains for short-term advantages. The result
is a canalized evolutionary trajectory lacking antigenic diversification.

## Materials and Methods

### Transmission model

To characterize the joint epidemiological, genealogical, antigenic and spatial
patterns of influenza, we implemented a large-scale individual-based model.
This model consists of daily time steps, in which the states of hosts and
viruses are updated. Hosts may be born, may die, may contact other hosts
allowing viral transmission, or may recover from infection. Viruses may mutate
in antigenic phenotype. Each simulation ran for 40 years of model time.

Hosts in this model are divided between three regions: North, South and
Tropics. There are 30 million hosts within each of the three regions, giving
$N=9\times 10^{7}$ hosts. Host population size remains fixed at this number,
but vital dynamics cause births and deaths of hosts at a rate of $1/30$ years
$=9.1\times 10^{-5}$ per host per day. Within each region, transmission
proceeds through mass-action with contacts between hosts occurring at a rate
of $\beta=0.36$ per host per day. Regional transmission rates in temperate
regions vary according to sinusoidal seasonal forcing with amplitude
$\epsilon=0.15$ and opposite phase in the North and in the South. Transmission
rate does not vary over time in the Tropics. Transmission between region $i$
and region $j$ occurs at rate $m\,\beta_{i}$, where $m=0.001$ and is the same
between each pair of regions and $\beta_{i}$ is the within-region contact
rate. Hosts recover from infection at rate $\nu=0.2$ per host per day, so that
$R_{0}$ in a naive host population is 1.8. There is no super-infection in the
model.

Each virus possesses an antigenic phenotype, represented as a location in
Euclidean space. Here, we primarily use a two-dimensional antigenic location.
After recovery, a host ‘remembers’ the phenotype of its infecting virus as
part of its immune history. When a contact event occurs and a virus attempts
to infect a host, the Euclidean distance from infecting phenotype $\phi_{v}$
is calculated to each of the phenotypes in the host immune history
$\phi_{h_{1}},\dots,\phi_{h_{n}}$. Here, one unit of antigenic distance is
designed to correspond to a twofold dilution of antiserum in a
hemagglutination inhibition (HI) assay [9]. The probability that infection
occurs after exposure is proportional to the distance $d$ to the closest
phenotype in the host immune history. Risk of infection follows the form
$\rho=\textrm{min}\\{d\,s,1\\}$, where $s=0.07$. Cross-immunity $\sigma$
equals $1-\rho$. The initial host population begins with enough immunity to
slow down the initial virus upswing and place the dynamics closer to their
equilibrium state; initial $R$ was 1.28.

Our model follows Gog and Grenfell [23] in representing antigenic distance as
distance between points in a geometric space. By forcing one-dimension to
directly modulate $\beta$, Gog and Grenfell find an orderly replacement of
strains. Kryazhimskiy et al. [24] use a two-dimensional strain-space, but
enforce a cross-immunity kernel that directly favors moving along a diagonal
line away from previous strains. Our model does not ‘build in’ the one-
dimensional direction of antigenic drift, which instead emerges dynamically
from competition among strains.

The initial virus population consisted of 10 infections each with the
identical antigenic phenotype of $\\{0,0\\}$. Over time viruses evolve in
antigenic phenotype. Each day there is a chance $\mu=10^{-4}$ that an
infection mutates to a new phenotype. This mutation rate represents a
phenotypic rate, rather than genetic mutation rate, and can be thought of as
arising from multiple genetic sources. When a mutation occurs, the virus’s
phenotype is moved in a completely random direction
$\sim\textrm{Uniform}(0,360)$ degrees. Mutation size is sampled from the
distribution $\sim\textrm{Gamma}(\alpha,\beta)$, where $\alpha$ and $\beta$
are chosen to give a mean mutation size of 0.6 units and a standard deviation
of 0.4 units. This distribution is parameterized so that mutation usually has
little effect on antigenic phenotype, but occasionally has a large effect.
This is similar to the neutral networks implemented by Koelle et al. [6],
wherein most amino acid changes result in little decrease to cross-immunity
between strains, but some changes result in large jumps in cross-immunity.

### Model output

Daily incidence and prevalence are recorded for each region. During the course
of the simulation, samples of current infections are taken from the evolving
virus population at a rate proportional to prevalence. Each viral infection is
assigned a unique ID, and in addition, infections have their phenotypes,
locations and dates of infection recorded. In this model, viruses lack
sequences and so standard phylogenetic inference of the evolutionary
relationships among strains is impossible. Instead, the viral genealogy is
directly recorded. This is made possible by tracking transmission events
connecting infections during the simulation; infections record the ID of their
‘parent’ infection. Proceeding from a sample of infections, their genealogical
history can be reconstructed by following consecutive links to parental
infections. During this procedure, lineages coalesce to the ancestral lineages
shared by the sampled infections, eventually arriving at the initial infection
introduced at the beginning of the simulation. Commonly, phylodynamic
simulations generate sequences that are subsequently analyzed with
phylogenetic software to produce an estimated genealogy [4, 6, 25]. This step
of phylogenetic inference is imperfect and computationally intensive, and by
side-stepping phylogenetic reconstruction we arrive at genealogies quickly and
accurately. Other authors have implemented similar tracking of infection trees
[26, 27]. This genealogy-centric approach makes many otherwise difficult
calculations transparent, such as calculating lineage-specific region-specific
migration rates (Figure 2) and lineage-specific mutation effects (Figure S5).

Infections are sampled at a rate designed to give approximately 6000 samples
over the course of the simulation, with genealogies constructed from a
subsample of approximately 300 samples. The results presented in Figure 1
represent a single representative model output; one hundred replicate
simulations were conducted to arrive at statistical estimates.

### Parameter selection and sensitivity analysis

Estimating what the basic reproductive number $R_{0}$ for seasonal influenza
would be in a naive population is notoriously difficult. Season-to-season
estimates of effective reproductive number $R$ for the USA and France gathered
from mortality timeseries display an interquartile range of 0.9–1.8 [28].
Geographic spread within the USA suggests an average seasonal $R$ of 1.35
[29]. These estimates of $R$ will be lower than the $R_{0}$ of influenza due
to the effects of human immunity. We assumed $R_{0}$ of 1.8, consistent with
the upper range of seasonal estimates. Duration of infection was chosen based
on patterns of viral shedding shown during challenge studies [30]. The linear
form of the risk of infection and its increase as a function of antigenic
distance $s$ was chosen as 0.07 based on experimental work on equine influenza
[31] and from studies of vaccine effectiveness [32]. Between-region contact
rate $m$ was chosen to yield a trunk lineage that resides predominantly in the
tropics. With much higher rates of mixing, the trunk lineage ceases to show a
preference the tropics, and with much lower rates of mixing, particular
seasons in the north and the south will often be skipped. The amplitude of
seasonal forcing $\epsilon$ was chosen to be just large enough to get
consistent fade-outs in the summer months and is consistent with empirical
estimates [33].

Mutational parameters were based, in part, on model behavior. We assumed 10
amino acid sites involved in antigenicity, each mutating at a rate of
$10^{-5}$ [12] to give a phenotypic mutation rate $\mu=10^{-4}$ per infection
per day. We chose mutational effect parameters ($\textrm{mean}=0.6$,
$\textrm{sd}=0.4$) that would give suitably fast rates of antigenic evolution
corresponding to approximately 1.2 units of antigenic change per year, while
simultaneously giving clustered patterns of antigenic evolution [9]. Similar
outcomes are possible under a variety of parameterizations. If mutations are
more common ($\mu=3\times 10^{-4}$) and show less variation in effect size
($\textrm{mean}=0.6$, $\textrm{sd}=0.2$), then antigenic drift occurs in a
more continuous fashion, resulting in less variation in seasonal incidence and
a smoother distribution of antigenic phenotypes (Figure S3). If mutations are
less common ($\mu=5\times 10^{-5}$) and show more variance in effect
($\textrm{mean}=0.7$, $\textrm{sd}=0.5$), then antigenic change occurs in a
more punctuated fashion (Figure S4). Basic reproductive number $R_{0}$ can be
traded off with mutational parameters to some extent. Less mutational input
and higher $R_{0}$ will give similar patterns of antigenic drift and seasonal
incidence. Similarly, Kucharski and Gog [34] find that increasing $R_{0}$
results in increased rates of emergence of antigenically novel strains.

In 20 out of the 100 replicate simulations, we observed a major bifurcation of
antigenic phenotype and a consequent increase in incidence and genealogical
diversity. These simulations were removed from the analysis. Similar to Koelle
et al. [35], we assume that although the historical evolution of H3N2
influenza followed the path of a single lineage, it could have included a
major bifurcation. Further work in these directions will help to determine the
likelihoods of single lineage vs. bifurcating scenarios.

### Antigenic map

Antigenic phenotypes are modeled as discrete entities on the Euclidean plane;
multiple samples have the same antigenic location. However, in the empirical
antigenic map of influenza A (H3N2), each strain appears in a unique location
[9]. We would argue that some of this pattern comes from experimental noise.
Indeed, Smith et al. [9] find that observed measurements and measurements
predicted from the map differ by an average of 0.83 antigenic units with a
standard deviation of 0.67 antigenic units. We take this as a proxy for
experimental noise and add jitter to each sampled antigenic phenotype by
moving it in a random direction for an exponentially distributed distance with
mean of 0.53 antigenic units. If two samples with the same underlying
antigenic phenotype are jittered in this fashion, the distance between them
averages 0.83 antigenic units with a standard deviation of 0.64 units.

We added noise to each of the 5943 sampled viruses in this fashion resulting
in an approximated antigenic map (Figure 1D). Virus samples were then
clustering following standard clustering algorithms. We tried clustering by
the $k$-means algorithm and also agglomerative hierarchical clustering with a
variety of linkage criterion. We found that clustering by Ward’s criterion
consistently outperformed other methods, when measured in terms of within-
cluster and between-cluster variances. However, the exact clustering algorithm
had little effect on our overall results.

### Acknowledgments

We would like to thank Sarah Cobey, Aaron King, Pejman Rohani and the
attendees of the 2011 RAPIDD Workshop on Phylodynamics for helpful discussion.
We would also like to thank Ed Baskerville and Daniel Zinder for programming
advice. The term ‘canalization’ was originally suggested by Micaela Martinez-
Bakker.

### Funding

TB is supported by the Howard Hughes Medical Institute and by the European
Molecular Biology Organization. AR works as a part of the Interdisciplinary
Centre for Human and Avian Influenza Research (ICHAIR) and the University of
Edinburgh’s Centre for Immunity, Infection and Evolution (CIIE). MP is an
investigator of the Howard Hughes Medical Institute.

## References

  * 1. WHO (2009) Influenza Fact sheet.  Available at http://www.who.int/mediacentre/factsheets/fs211/en/.
  * 2. Nelson MI, Holmes EC (2007) The evolution of epidemic influenza.  Nat Rev Genet 8: 196–205.
  * 3. Fitch WM, Bush RM, Bender CA, Cox NJ (1997) Long term trends in the evolution of H(3) HA1 human influenza type A.  Proc Natl Acad Sci USA 94: 7712-8.
  * 4. Ferguson NM, Galvani AP, Bush RM (2003) Ecological and immunological determinants of influenza evolution.  Nature 422: 428–433.
  * 5. Tria F, Lässig M, Peliti L, S F (2005) A minimal stochastic model for influenza evolution.  J Stat Mech .
  * 6. Koelle K, Cobey S, Grenfell B, Pascual M (2006) Epochal evolution shapes the phylodynamics of interpandemic influenza A (H3N2) in humans.  Science 314: 1898–1903.
  * 7. Recker M, Pybus O, Nee S, Gupta S (2007) The generation of influenza outbreaks by a network of host immune responses against a limited set of antigenic types.  Proc Natl Acad Sci USA 104: 7711–7716.
  * 8. Hirst G (1943) Studies of antigenic differences among strains of influenza A by means of red cell agglutination.  J Exp Med 78: 407–423.
  * 9. Smith DJ, Lapedes AS, de Jong JC, Bestebroer TM, Rimmelzwaan GF, et al. (2004) Mapping the antigenic and genetic evolution of influenza virus.  Science 305: 371–376.
  * 10. Monto A, Sullivan K (1993) Acute respiratory illness in the community. Frequency of illness and the agents involved.  Epidemiol Infect 110: 145–160.
  * 11. Koelle K, Kamradt M, Pascual M (2009) Understanding the dynamics of rapidly evolving pathogens through modeling the tempo of antigenic change: influenza as a case study.  Epidemics 1: 129–137.
  * 12. Rambaut A, Pybus OG, Nelson MI, Viboud C, Taubenberger JK, et al. (2008) The genomic and epidemiological dynamics of human influenza A virus.  Nature 453: 615–619.
  * 13. Bedford T, Cobey S, Pascual M (2011) Strength and temp of selection revealed in viral gene genealogies.  BMC Evol Biol 11: 220.
  * 14. Russell CA, Jones TC, Barr IG, Cox NJ, Garten RJ, et al. (2008) The global circulation of seasonal influenza A (H3N2) viruses.  Science 320: 340–346.
  * 15. Bedford T, Cobey S, Beerli P, Pascual M (2010) Global migration dynamics underlie evolution and persistence in human influenza A (H3N2).  PLoS Pathog 6: e1000918.
  * 16. Finkelman B, Viboud C, Koelle K, Ferrari M, Bharti N, et al. (2007) Global patterns in seasonal activity of influenza A/H3N2, A/H1N1, and B from 1997 to 2005: viral coexistence and latitudinal gradients.  PLoS One 2: e1296.
  * 17. Bhatt S, Holmes E, Pybus O (2011) The genomic rate of molecular adaptation of the human influenza A virus.  Mol Biol Evol Advance access.
  * 18. Goldstein E, Cobey S, Takahashi S, Miller J, Lipsitch M (2011) Predicting the epidemic sizes of influenza A/H1N1, A/H3N2, and B: a statistical method.  PLoS Med In press.
  * 19. Voeten J, Bestebroer T, Nieuwkoop N, Fouchier R, Osterhaus A, et al. (2000) Antigenic drift in the influenza A virus (H3N2) nucleoprotein and escape from recognition by cytotoxic T lymphocytes.  J Virol 74: 6800–6807.
  * 20. Gould S (1989) Wonderful life.  Norton New York.
  * 21. Barr I, McCauley J, Cox N, Daniels R, Engelhardt O, et al. (2010) Epidemiological, antigenic and genetic characteristics of seasonal influenza A (H1N1), A (H3N2) and B influenza viruses: basis for the WHO recommendation on the composition of influenza vaccines for use in the 2009-2010 Northern Hemisphere season.  Vaccine 28: 1156–1167.
  * 22. Weinreich D, Delaney N, DePristo M, Hartl D (2006) Darwinian evolution can follow only very few mutational paths to fitter proteins.  Science 312: 111–114.
  * 23. Gog J, Grenfell B (2002) Dynamics and selection of many-strain pathogens.  Proc Natl Acad Sci USA 99: 17209–17214.
  * 24. Kryazhimskiy S, Dieckmann U, Levin S, Dushoff J (2007) On state-space reduction in multi-strain pathogen models, with an application to antigenic drift in influenza A.  PLoS Comp Biol 3: e159.
  * 25. Koelle K, Khatri P, Kamradt M, Kepler T (2010) A two-tiered model for simulating the ecological and evolutionary dynamics of rapidly evolving viruses, with an application to influenza.  J R Soc Interface 7: 1257–1274.
  * 26. Volz E, Kosakovsky Pond S, Ward M, Leigh Brown A, Frost S (2009) Phylodynamics of infectious disease epidemics.  Genetics 183: 1421–1430.
  * 27. O’Dea E, Wilke C (2011) Contact heterogeneity and phylodynamics: how contact networks shape parasite evolutionary trees.  Interdiscip Perspect Infect Dis 2011: 238743.
  * 28. Chowell G, Miller M, Viboud C (2008) Seasonal influenza in the United States, France, and Australia: transmission and prospects for control.  Epidemiol Infect 136: 852–864.
  * 29. Viboud C, Bjørnstad O, Smith D, Simonsen L, Miller M, et al. (2006) Synchrony, waves, and spatial hierarchies in the spread of influenza.  Science 312: 447–451.
  * 30. Carrat F, Vergu E, Ferguson NM, Lemaitre M, Cauchemez S, et al. (2008) Time lines of infection and disease in human influenza: a review of volunteer challenge studies.  Am J Epidemiol 167: 775–785.
  * 31. Park A, Daly J, Lewis N, Smith D, Wood J, et al. (2009) Quantifying the impact of immune escape on transmission dynamics of influenza.  Science 326: 726–728.
  * 32. Gupta V, Earl D, Deem M (2006) Quantifying influenza vaccine efficacy and antigenic distance.  Vaccine 24: 3881–3888.
  * 33. Truscott J, Fraser C, Cauchemez S, Meeyai A, Hinsley W, et al. (2011) Essential epidemiological mechanisms underpinning the transmission dynamics of seasonal influenza.  J R Soc Interface In press.
  * 34. Kucharski A, Gog JR (2011) Influenza emergence in the face of evolutionary constraints.  Proc R Soc Lond B Biol Sci Advance access.
  * 35. Koelle K, Ratmann O, Rasmussen D, Pasour V, Mattingly J (2011) A dimensionless number for understanding the evolutionary dynamics of antigenically variable rna viruses.  Proc R Soc Lond B Biol Sci Advance access.

## Tables

Table Table 1: Rates of mutation and phenotypic change on trunk and side branches and mutational expectation. | Baseline | Side branch | Trunk | Trunk / side branch
---|---|---|---|---
Mutation size (AG units) | 0.60 | 0.79 | 1.58 | 1.99$\times$
Mutation rate (mut per year) | 0.04 | 0.06 | 0.81 | 13.23$\times$
Antigenic flux (AG units per year) | 0.02 | 0.05 | 1.27 | 26.25$\times$

## Figures

Figure Figure 1: Simulation results showing epidemiological, antigenic and
genealogical dynamics. (A) Weekly timeseries of incidence of viral infection
in north and tropics regions. (B) Two-dimensional antigenic phenotypes of 5943
viruses sampled over the course of the simulation. Each discrete virus
phenotype is shown as a bubble, with bubble area proportional to the number of
times this phenotype was sampled. (C) Genealogical tree depicting the
infection history of 376 samples from the virus population. Parent/offspring
relationships were tracked over the course of the simulation, giving a direct
observation of the genealogy rather than a phylogenetic inference. (D)
Antigenic map depicting phenotypes of 5943 viruses sampled over the course of
the simulation. To construct the map, noise was added to each sample and the
resulting observations grouped into 11 clusters and colored accordingly. Grid
lines show single units of antigenic distance. Cluster assignments were used
to color all panels in a consistent fashion.

Figure Figure 2: Patterns of geographic movement of virus lineages. (A)
Evolutionary relationships among 376 viruses sampled evenly through time
colored by geographic location. Lineages residing in the north (N), south (S)
and tropics (T) are colored yellow, red and blue respectively. (B) Observed
migration rates between regions on side branch lineages (left) and on trunk
lineages (right). Arrows denote movement of lineages and arrow width is
proportional to migration rate. Circle area is proportional to the expected
stationary frequency of a region given the observed migration rates. In both
cases, migration rates are calculated across 80 replicate simulations.

Figure Figure 3: Host immunity and antigenic history of the virus population.
Contour lines represent the state of host immunity at the end of the 40-year
simulation. They show the mean risk of infection (as a percentage) after a
random host in the population encounters a virus bearing a particular
antigenic phenotype. Contour lines are spaced in intervals of 2.5%. Bubbles
represent a sample of antigenic phenotypes present at the end of the 40-year
simulation. The area of each bubble is proportional to the number of samples
with this phenotype. Lines leading into these bubbles show past antigenic
history. The current phenotypes rapidly coalesce to a trunk phenotype. The
movement of the virus population from the left to the center of the figure can
be seen from the antigenic history. At the end of the simulation several virus
phenotypes exist with similar antigenic locations; all of these phenotypes lie
significantly ahead of the peak of host immunity.

Figure Figure 4: Antigenic phenotypes over the course of 4 years of evolution
across 100 replicate simulations starting from identical initial conditions.
Replicate simulations were initialized with the end state of the 40-year
simulation shown in Figure 1. Each panel shows an additional year of
evolution, with black points representing the mean antigenic phenotypes of the
100 replicate simulations and gray lines representing the history of each mean
antigenic phenotype.

Figure Figure 5: Timeseries of incidence across 100 replicate simulations with
identical initial conditions. Panels show incidence in the North, Tropics and
South regions over the course of 6 years. Solid black lines represent the
median weekly incidence across the 100 replicate simulations, while gray
intervals represent the interquartile range across simulations. There is
little variability for the first year of replicate simulations. Replicate
simulations were initialized with the end state of the 40-year simulation
shown in Figure 1.

## Supporting Information

Figure Figure S1: Antigenic evolution over the course of the 40-year
simulation. (A) Proportion of virus population comprised of each antigenic
cluster through time. (B) Antigenic distance from initial phenotype ($x=0$,
$y=0$) for each of 5943 virus samples relative to time of virus sampling.
Viruses were sampled at a constant rate proportional to prevalence and
coloring was determined from the antigenic map in Figure 1D.

Figure Figure S2: Correlation between antigenic drift and attack rate.
Antigenic drift is measured as the distance between the centroid of phenotypes
of one year and the centroid of phenotypes of the following year. Measurements
were taken across 80 replicate simulations. Individual pairs of measurements
are shown as gray points and a locally-linear regression (LOESS) is shown as a
black dashed line.

Figure Figure S3: Simulation results showing epidemiological, antigenic and
genealogical dynamics for ‘smoother’ mutation model. (A) Weekly timeseries of
incidence of viral infection in north and tropics regions. (B) Antigenic map
depicting phenotypes of viruses sampled over the course of the simulation.
Grid lines show single units of antigenic distance. (C) Genealogical tree
depicting the infection history of samples from the virus population. Cluster
assignments were used to color panels (A), (B) and (C) in a consistent
fashion. Alternative mutational parameters are $\mu=3\times 10^{-4}$, mean
mutation size of 0.6 units and standard deviation of mutation size of 0.2
units.

Figure Figure S4: Simulation results showing epidemiological, antigenic and
genealogical dynamics for ‘rougher’ mutation model. (A) Weekly timeseries of
incidence of viral infection in north and tropics regions. (B) Antigenic map
depicting phenotypes of viruses sampled over the course of the simulation.
Grid lines show single units of antigenic distance. (C) Genealogical tree
depicting the infection history of samples from the virus population. Cluster
assignments were used to color panels (A), (B) and (C) in a consistent
fashion. Alternative mutational parameters are $\mu=5\times 10^{-5}$, mean
mutation size of 0.7 units and standard deviation of mutation size of 0.5
units.

Figure Figure S5: Mutation spectrum in two-dimensional antigenic space of side
branch mutations and trunk mutations. (A) Histogram of mutation effects along
the axis of primary antigenic variation across 80 replicate simulations. The
left panel shows the distribution of effects of side branch mutations and the
right panel shows the distribution of effects of trunk mutations. (B) Smoothed
two-dimensional histogram of mutation effects along the primary and secondary
axes of antigenic variation across 80 replicate simulations. Histograms were
constructed from 21,405 side branch mutations and 1584 trunk mutations.

Figure Figure S6: Relationship between a mutation’s phenotypic effect and its
likelihood of being part of the trunk. The $x$-axis represents the effect of a
mutation along the line of primary antigenic variation, and the $y$-axis
represents the probability that the mutation is part of the trunk. Mutations
of large effect are increasingly rare, but when they do occur are increasingly
likely to be part of the trunk.

Figure Figure S7: Observed vs. expected distributions of waiting times between
phenotypic mutations along genealogy trunk. (A) Histogram bins show the
observed distribution of waiting times in years across 80 replicate
simulations representing 1584 mutations. The mean of this distribution is 1.76
years. The dashed line shows the Poisson process expectation of exponentially
distributed waiting times. (B) The density distribution of waiting times is
transformed into a hazard function, representing the rate of trunk mutation
after a specific waiting time. The dashed line shows the memoryless hazard
function of the Poisson process expectation.

Figure Figure S8: Simulation results showing epidemiological, antigenic and
genealogical dynamics with weaker mutation. (A) Weekly timeseries of incidence
of viral infection in north and tropics regions. (B) Antigenic map depicting
phenotypes of viruses sampled over the course of the simulation. Grid lines
show single units of antigenic distance. (C) Genealogical tree depicting the
infection history of samples from the virus population. Cluster assignments
were used to color panels (A), (B) and (C) in a consistent fashion. Here,
$\mu=5\times 10^{-5}$, mean mutation size is 0.42 units and standard deviation
of mutation size is 0.28 units.

Figure Figure S9: Simulation results showing epidemiological, antigenic and
genealogical dynamics with lower intrinsic $R_{0}$. (A) Weekly timeseries of
incidence of viral infection in north and tropics regions. (B) Antigenic map
depicting phenotypes of viruses sampled over the course of the simulation.
Grid lines show single units of antigenic distance. (C) Genealogical tree
depicting the infection history of samples from the virus population. Cluster
assignments were used to color panels (A), (B) and (C) in a consistent
fashion. Here, $\beta=0.3$, giving $R_{0}=1.5$.

Figure Figure S10: Principal components of antigenic variation under a
10-sphere mutation model. Each panel shows 5991 samples of antigenic phenotype
over the course of a 40-year simulation. Each phenotype is represented as a
bubble, with bubble area proportional to the number of samples with this
phenotype. Bubbles are colored based on clustering the 10-dimensional
antigenic phenotypes. The original 10-dimensional space was rotated using
principal components analysis to give orthogonal axes in the order of their
contribution to antigenic variation. Each panel shows a two-dimensional slice
of the this rotated space. Principal components 7–10 were left out of the
figure for clarity.


# Object-Driven One-Shot Fine-tuning of Text-to-Image Diffusion with
Prototypical Embedding

Jianxiang Lu  TencentChina<EMAIL_ADDRESS>,  Cong Xie  TencentChina
<EMAIL_ADDRESS>and  Hui Guo  TencentChina<EMAIL_ADDRESS>
###### Abstract.

As large-scale text-to-image generation models have made remarkable progress
in the field of text-to-image generation, many fine-tuning methods have been
proposed. However, these models often struggle with novel objects, especially
with one-shot scenarios. Our proposed method aims to address the challenges of
generalizability and fidelity in an object-driven way, using only a single
input image and the object-specific regions of interest. To improve
generalizability and mitigate overfitting, in our paradigm, a prototypical
embedding is initialized based on the object’s appearance and its class,
before fine-tuning the diffusion model. And during fine-tuning, we propose a
class-characterizing regularization to preserve prior knowledge of object
classes. To further improve fidelity, we introduce object-specific loss, which
can also use to implant multiple objects. Overall, our proposed object-driven
method for implanting new objects can integrate seamlessly with existing
concepts as well as with high fidelity and generalization. Our method
outperforms several existing works. The code will be released.

object-driven, one-shot, diffusion model

††ccs: Computing methodologies Image-based rendering††ccs: Computing
methodologies Non-photorealistic rendering Figure 1. Given an image and its
corresponding mask of an object, our method generates diverse images of the
same object in different contexts. Moreover, our method allows for the
implantation of multiple objects, as demonstrated in the lower right corner of
the figure.

##  1\. Introduction

Recently, text-to-image generation using deep learning models have significant
development (Ramesh et al., 2022; Rombach et al., 2022; Saharia et al., 2022;
Yu et al., 2022; Sauer et al., 2023; Chen et al., 2022). With intricate
designs and vast quantities of training data, these models are capable of
generating images of unparalleled quality from unrestricted natural language
inputs, provided by the user. These systems can create diverse objects,
styles, and scenes, as well as generating different object compositions,
resulting in aesthetically pleasing outputs.

Apart from general image generation, users are also concerned with whether it
is possible to generate images of a specific object in various text
descriptions while providing limited samples, such as a single image. This is
a one-shot generation task that requires accurate understanding of the
specific object and effective model fine-tuning while ensuring the object
similarity in generated images and the generation controllability of the
resultant model. However, this task poses significant challenges due to
limited training samples and a general lack of specific objects in the model
training set.

In order to improve the effectiveness of object fine-tuning, current methods
(Gal et al., 2022; Ruiz et al., 2022; Brooks et al., 2022; Kumari et al.,
2022; Mokady et al., 2022) employ a paradigm that involves optimizing a pre-
trained synthesis model, or a text prompt (Gal et al., 2022; Mokady et al.,
2022) representing the object using multiple training examples. The optimized
prompt is then integrated with user prompts to create images with diverse
styles and content. However, these methods usually rely on the availability of
multiple images of the same object, which exacerbates the problem of object
accuracy and model overfitting in one-shot scenarios.

Focusing on the task of accurate inplantation of a user specified object into
a generative model using only one sample, while maintaining the model’s
generalization, we propose a novel fine-tuning framework. Unlike existing
works (Ruiz et al., 2022; Kumari et al., 2022; Mokady et al., 2022; Wen et
al., 2023) that start from random initialization and iterative optimize a
unique prompt representation of a target object, we propose a novel fine-
tuning method. Based on stable diffusion, a prototypical embedding for the
object is firstly initialized and then trained in the generation model with a
class-characteristic regularization to balance object identity and
generalization variations. And an object-specific loss function supervised by
the object in the given image is proposed, to achieve both fidelity and
generalization in later image synthesis.

While seemingly simple, one-shot generation fine-tuning remains a non-trivial
task that poses several challenges, including: 1) adapting text-to-image
models with conditioned object embeddings, 2) designing a training process to
improve object fidelity using only one image, and 3) enhancing editing
capabilities to generate different object compositions using diverse text
prompts. Given a target image and its class, the text embedding of the
specific object is initialized by finding the best prototypical embedding
between multi-modal representations of the object’s class and its
characteristics. For the fine-tuning quality, we insert additional attention
modules(Hu et al., 2021) to the text-to-image network, and an object-specific
mask loss is used to preserve fidelity in downstream image synthesis. However,
directly fine-tuning the network with the object embedding results in model
overfitting with deteriorated editing capability, i.e. high Kernel Inception
Distance (KID) (Bińkowski et al., 2018) in generation results, where the model
generates solely with the user specific object but not the text instruction.
We therefore propose a class-characterizing regularization to the object
embedding for the generalization of the model. Our method essentially
increases object fidelity and model generalization, leading to performance
gain in terms of output quality, appearance diversity, and object fidelity.

Generating personalized content based on one-shot samples requires an
efficient algorithm to improve its applicability. In this work, we take a step
forward and demonstrate how the fine-tuning process can help the model
maintain object identity as long as generalization editability. Our method
outperforms several existing methods, which generates high-fidelity
photorealistic images and supports content-rich synthesis in one-shot
generation. What’s more, our method can also used in multi-object fine-tuning,
as shown in Figure 2, which achieve better composing results of multi user
specific objects.

Figure 2. Methodology overview. Our method takes an input image along with its
corresponding masks and relevant class names as input, generating object-
specific text embeddings and personalized LoRA weights. During inference, the
text embedding and LoRA weight is combined with other features to generate a
wide range of variations for the object.

##  2\. Related Work

###  2.1. Text-to-Image Synthesis

In recent years, there has been significant progress in the development of
text-to-image generative models that synthesize images based on unconditioned
text prompts. Such models include Generative Adversarial Networks (GANs)
(Brock et al., 2018; Goodfellow et al., 2020; Karras et al., 2021; Sauer et
al., 2023), Vector Quantization (VQs) approaches (Razavi et al., 2019; Esser
et al., 2021; Yu et al., 2022; Chang et al., 2023; Ding et al., 2022), and
diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al.,
2020; Dhariwal and Nichol, 2021; Rombach et al., 2022; Saharia et al., 2022;
Sheynin et al., 2022; Nichol et al., 2021). With the rise of large-scale text-
image datasets, these models have been scaled up to produce high-quality and
semantically rich image synthesis results. Examples of such large text-to-
image models include StyleGAN-T (Sauer et al., 2023), CogView2 (Ding et al.,
2022), Imagen (Saharia et al., 2022), DALL-E2 (Ramesh et al., 2022), and
Stable Diffusion (Rombach et al., 2022). The introduction of text conditioning
to the StyleGAN architecture in StyleGAN-T has showcased the effectiveness of
GANs in text-to-image synthesis. In contrast, VQs utilize an autoencoder to
learn a quantization codebook and predict text-to-image tokens through
transformers for generating the final image. Likewise, diffusion models
leverage a UNet module to iteratively denoise a noise image in a latent space,
conditioned on the text prompt injected to the model via transformer layers,
to synthesize images.

While text-to-image models trained on a specific dataset can generate high-
quality images, they often struggle to generate novel instances or user-
specified objects with identity consistency (Kumari et al., 2022). This can be
attributed to their training set limitation in representing new objects. Our
work relates to the Stable Diffusion model and aims to enhance its ability to
generate high-fidelity images of user-specified objects, thereby expanding the
model’s applicability to content creation tasks that require personalized
text-to-image synthesis.

###  2.2. Personalized image Synthesis

Fine-tuning method with a pre-trained synthesis model for personalized text-
to-image generation purposs mainly includes the following four methods: parts
of the network parameter tuning (Ruiz et al., 2022; Kumari et al., 2022),
image augmentation based method (Esser et al., 2021; Brooks et al., 2022),
prompt tuning or text encoder adapting (Gal et al., 2022; Mokady et al., 2022)
and injecting additional module which adapts to the new objects, i.e., Low-
Rank Adaptation (LoRA) (Hu et al., 2021). Adapting the whole model or part of
the parameters (Ruiz et al., 2022) using a one-shot image can easily lead to
overfitting, as it modifies well-trained parameters and can cause catastrophic
forgetting (French, 1999; Kirkpatrick et al., 2017; Li et al., 2022; Li and
Hoiem, 2017; Ramasesh et al., 2022). While augmentation (Esser et al., 2021)
or prompt tuning (Gal et al., 2022) based methods provide a wide variety of
samples for regularization or text encoder fine-tuning in training process,
they not only struggle to preserve object identity but also require additional
training information. Methods by adding an extra module can ease the
overfitting problem while sharing the same low fidelity problem. In this
paper, a small number of parameters, which relates to text encoder and
transformer cross attention module, is adapted for the new object and the
model is fine-tuned with object-driven method to preserve fidelity and
generalization ability of composing new object with the existing ones.

Figure 3. Fine-tuning details. Given one image with single or multiple
objects, our method fine-tunes a text-to-image diffusion model. Taking single
object as an example, our method utilizes prototypical embedding for
initialization and employs class-characterizing regularization to enhance
generation diversity, along with a class-specific loss function to ensure
fidelity of the synthesized images.

###  2.3. One-shot Synthesis

Fine-tuning a diffusion model with only one sample is an extremely challenging
task. While existing fine-tuning methods (Ruiz et al., 2022) or prompt-tuning
approaches (Gal et al., 2022) suffer from overfitting issues in one-shot
scenarios, additional training strategies need to be developed. To maintain
generation controllability, Dong et al. (Dong et al., 2022) use positive and
negative prompts to contrastively fine-tune the representation of a pseudo
prompt, which had generation failures in multi-object compositions. Another
method (Esser et al., 2021) considers the image background of the one-shot
sample while implanting the object and uses a background-masked object
embedding for fine-tuning. It trains the entire network and the text encoder
with side information such as original training samples for regularization,
which faces the problem of generation defects of the specific object.

In contrast to the aforementioned paradigms, our method only requires one
image and the object’s region of interest, and focusing on the specific
fidelity and generalization variations with specific object synthesis with .
It differs from existing methods in several aspects. Firstly, we introduce an
object-driven prototypical embedding initialization for the new object, which
alleviates the difficulty of representing the object with only one image and
improve the efficiency of object inplantation. Secondly, we introduce an
object-driven specific loss for precise object appearance learning, where
(Kirillov et al., 2023) is used for the object mask. Thirdly, a LoRA (Hu et
al., 2021) module where the main denoising UNet is maintained and a class-
characteristic regularization to protect class prior information for semantic
generalization with other objects and preventing catastrophic forgetting.
Moreover, our method is capable of multiple object fine-tuning, which is
challenging in existing methods (Ruiz et al., 2022; Kumari et al., 2022).

##  3\. Method

###  3.1. Overview

Our proposed method focuses on object-driven fine-tuning of single or multiple
objects specified by the user in one image, as shown in Figure 3. To overcome
the limitations of existing fine-tuning methods, we use prototypical embedding
as the initialization embedding and propose a regularized loss function to
increase the diversity of the generated images and effectively preserve the
prior knowledge of the pre-trained model. In addition, we introduce an object-
specific mask loss function to synthesize high-fidelity images, which can also
be used for multi-object implantation. In this section, we explain the
proposed method in detail.

Figure 4. Qualitative comparison. For one-shot tasks, existing methods face
challenges in achieving both fidelity and generalizability with the given
text. Our method generates images that better match the reference image and
are consistent with the text semantics under multiple cue words. Note that the
* symbol represents a unique identifier.

###  3.2. Latent Diffusion Model

In this work, we adopt the stable diffusion model, a well-established text-to-
image diffusion model, as our synthetic network. Stable diffusion is a Latent
Diffusion Model (LDM) (Rombach et al., 2022), which performs denoising
operations in the latent space instead of the image space. In simple terms,
first, the RGB image $x$ is encoded into the latent representation
$z=\varepsilon(x)$ by the encoder $\varepsilon$ of an Variable Auto Encoder
(VAE). In the latent space, the denoising U-Net of LDM employs the cross-
attention mechanism for text-conditional image generation. The training loss
of the conditional LDM is formulated as:

(1)
$L_{LDM}=\mathbb{E}_{\mathcal{E}(x),y,\epsilon\sim\mathcal{N}(0,1),t}\left[\left\|\epsilon-\epsilon_{\theta}\left(z_{t},t,c\right)\right\|_{2}^{2}\right]$

where $c$ is the text embedding, $\epsilon$ denotes standard Gaussian, and
$\epsilon_{\theta}$ is the model prediction. $z_{t}$ is the noisy version of
input $z$ in timestep $t$.

###  3.3. Prototypical Embedding

When fine-tuning a diffusion model, the text embedding of the object is
usually trained. However, when the training data is only one image, it
sometimes causes overfitting, leading the network to generate output based
only on the text embeddings of the objects, while other textual conditions are
ignored. In practice, proper initialization of text embeddings can enable
faster fitting of the network and alleviate overfitting, such as Textual
Inversion (TI) (Gal et al., 2022) initializing text embeddings based on the
class of objects. In this work, in order to achieve more efficient
initialization, we find prototype embeddings based on the embedding of the
input image and the text embedding of the class name (e.g., dog). Prior to
commencing the training of the diffusion model, we compute prototypical
embedding via:

(2)
$L_{PE}=1-\frac{\mathcal{T}(c_{p})\theta_{m}(\mathcal{I}(x),\mathcal{I}(x_{m}),\mathcal{T}(c_{c}))}{\left\|\mathcal{T}(c_{p})\right\|\left\|\theta_{m}(\mathcal{I}(x),\mathcal{I}(x_{m}),\mathcal{T}(c_{c}))\right\|}$

where $x$ is the training image, the image encoder $\mathcal{I}$ and the text
encoder $\mathcal{T}$ of CLIP (Radford et al., 2021) are used to obtain the
whole image embedding $\mathcal{I}(x)$ , the object mask image embedding
$\mathcal{I}(x_{m})$, $\mathcal{T}(c_{c})$ is the class name text embedding of
the object and $\theta_{m}$ is the way of embedding fusion e.g. averaging. We
aim to obtain a prototype text embedding $\mathcal{T}(c_{p})$ similar to the
target image embedding and the class text embedding as initialization by this
loss function.

###  3.4. Class-characterizing Regularization

Additionally, in order to preserve the synthesis ability of the class of
objects in the pre-trained model, we adjust the text embedding using class-
characterizing regularization during the training process. Class-
characterizing loss is formulated as:

(3)
$L_{CL}=\begin{cases}1-\alpha_{cl}\frac{\mathcal{T}(c_{p})\mathcal{T}(c_{c})}{\left\|\mathcal{T}(c_{p})\right\|\left\|\mathcal{T}(c_{c})\right\|}&\text{if
}p<p_{cl}\\\ 0&\text{otherwise}\par\end{cases}$

where $\mathcal{T}(c_{c})$ is the class name text embedding of the object,
$\alpha_{cl}$ represents the weight of the cosine loss, $p\sim Uni(0,1)$, and
$p_{cl}$ is an adjustable threshold. In this context, it is necessary to
predetermine the class name of each object. Further experiments indicate that
the introduction of this loss function leads to improved generalizability in
synthesis.

###  3.5. Object-specific Loss

Our task is to implant the selected objects into the output domain of the
model and bind them with a unique identifier. Noted that the selected objects
are often parts of the training image rather than the whole image, for this
reason we propose object-specific loss for implantation of selected objects
with improved fidelity. First, we use an image segmentation algorithm such as
SAM(Kirillov et al., 2023) to obtain the mask images $m$ of the objects. The
mask images are introduced into the latent space and the training process. For
single-object implantation is trained as follow:

(4)
$L_{SP}=\left\|\tilde{\epsilon}-\epsilon_{\theta}\left(\tilde{z_{t}},t,c_{m}\right)\right\|^{2}_{2}+\left\|\epsilon-\epsilon_{\theta}\left(z_{t},t,c\right)\right\|^{2}_{2}$

where $c_{m}$ is the text condition of this object with mask, object target
noise $\tilde{\epsilon}=\epsilon\otimes m+\epsilon_{\theta}\otimes(1-m)$, and
mask latent representation $\tilde{z}=z\otimes m$. Our goal is to focus on the
mask region when performing the loss calculation. Further, for multiple object
implantation we make object-specific loss function combinations, assuming that
there are a set of $r$ objects to be implanted and a subset $S$ of $k$
distinct objects are taken at a time, the number of $k$-combinations is
$C_{n}^{k}$. So, in one step of training, the overall object-specific loss is:

(5) $L_{SP}=\sum_{i\in
S}\left\|\tilde{\epsilon}_{i}-\epsilon_{\theta}\left(\tilde{z}_{t,i},t,c_{m,i}\right)\right\|^{2}_{2}+\left\|\epsilon-\epsilon_{\theta}\left(z_{t},t,c\right)\right\|^{2}_{2}$

Note that the text condition $c_{m,i}$ is different for each mask, and the
global text condition $c$ is based on unique identifiers for all objects.

Figure 5. Prototypical embedding initialization. Our proposed method,
utilizing prototypical embedding as the initialization, ensures the generation
of images that are more contextually relevant.

##  4\. Experiments

###  4.1. Settings

We use the pre-trained stable diffusion model as the text-to-image network.
The CLIP image and text encoders are used to compute prototypical embeddings
and take LoRA as the fine-tuned model. We use the images on unsplash as fine-
tuning data and the mask images of the objects are obtained using SAM. In the
object-driven fine-tuning, the learning rate is $10^{-4}$ and the training is
100 steps with a batchsize of 1, on a v100 GPU. We did not over-tune the
superparameter, $\alpha_{cl}$, $p_{cl}$ are set to 1, and $k$ is set to 2 for
object-specific loss combinations. We use Dreambooth, TI and LoRA as three
state-of-the-art methods for comparison, and use their publicly released
implementation code. In addition, their learning rates and training steps are
consistent in training.

###  4.2. Comparison

As shown in Figure 4, we compare our results with existing fine-tuning
methods. Because of the different fine-tuning strategies, the existing methods
may overfit, or not be fidelity, when only one image is given as input. For
action generation, the results of TI and LoRA align the actions in the prompt,
but the generated objects differ significantly from the reference image. And
DreamBooth overfits, resulting in the inability to generate actions. In
contrast, our method is able to generate actions with higher fidelity. For
human faces, all methods except ours cannot generate expressions and preserve
identities at the same time. The same observation is made for animals and
style transformation.

Further, we compare other methods using quantitative metrics. We
quantitatively evaluated 5 categories and 750 images, and the results are
tabulated in Table 1. we used Text Alignment (TA) (Hessel et al., 2021), Image
Alignment (IA) (Gal et al., 2022) and kernel inception distance as metrics,
where TA is the generalizability of the alignment ability to characterize the
method on the new prompt, and IA characterizes the generalizability to the
image similarity. Thus, it is a trade-off, as shown in Figure 6. It can be
observed that our proposed method performs well in this trade-off, with both
fidelity and generalization.

Overall, our proposed method has both fidelity and generalizability, which
indicates that it effectively mitigates the overfitting and learns the
characterization of the object.

Table 1. Quantitative comparison. We use three metrics to evaluate the generalization and fidelity of the fine-tuning method. Methods | TI | DreamBooth | LoRA | Ours
---|---|---|---|---
IA $\uparrow$ | 0.6084 | 0.6216 | 0.6284 | 0.6431
TA $\uparrow$ | 0.2609 | 0.2434 | 0.2774 | 0.2800
KID $\downarrow$ | 0.1322 | 0.2630 | 0.1222 | 0.1882
Figure 6. Quantitative assessment. We visualize the metrics for each method,
the point towards the lower right, the better performance of the method.

###  4.3. Ablation Study

Figure 7. Class-characterizing regularization. Class-characterizing
regularization preserves the prior representation of object classes during
fine-tuning, resulting in a more natural and diverse synthesis of objects.

####  4.3.1. Prototypical Embedding Initialization

As indicated in Section 3.3, we propose prototypical embedding to mitigate the
problem of overfitting for fine-tuning, and to demonstrate its importance we
compare the results of random initialization embedding. Note that in all
comparisons the initialization of the text embedding is based on four token
vectors. As shown in Figure 5, without the prototypical embedding, the
synthesized image has only the reference object and ignores other information
in the prompt. On the contrary, when prototypical embedding is adopted, the
model is able to generate elements other than objects (e.g., buckets). It
demonstrates that prototypical embedding is effective in overcoming the
limitations of overfitting and improving the diversity of image generation.

Figure 8. Object-specific loss. Our object-specific loss function enhances
attention to the specific object region, thereby effectively preserving the
fidelity of the generated objects.

####  4.3.2. Class-characterizing Regularization

Since prototypical embedding is only used as a way to initialize without
involving the training process, the prior representations of object classes
are sometimes lost during fine-tuning. As shown in Figure 7, when without
class-characterizing regularization, the generated hats are not well
integrated with the person, and customized styles cannot be generated. We
observe that with class-characterizing regularization, the prior knowledge of
the object’s class (e.g., hat) is preserved in fine-tuning, and the generated
images possess greater diversity and naturalness.

####  4.3.3. Object-specific Loss

In object-driven training, we adopt an object-specific loss function that
focuses on the object region to enhance object fidelity. To demonstrate the
effectiveness of the loss, we compare the results without the loss function.
In Figure 8, we observe that our results trained with the object-specific loss
function have greater fidelity in the color, shape, and details of the
objects. For example, the color of the object ”flower” does not match the
reference image without this loss function, and the object ”toy duck” exhibits
alterations in its shape and details.

####  4.3.4. Multi-object Implantation

As discussed above, the proposed one-shot method supports multiple objects
implantation, which can derive many interesting applications. We first obtain
the mask images of the objects separately using the mature image segmentation
algorithm and then introduce the object-specific loss combinations mentioned
in Section 3.5 for training. As shown in Figure 9, the implanted objects do
not affect each other when generating images using their unique identifiers
and can generate output related to the prompt. In addition, we generate images
using multiple object identifiers at the same time (e.g., [strawberry*] in the
[cup*]), it can be observed that the output resulting from the combination of
concepts is both natural and faithful.

Figure 9. Multi-object implantation. After implanting multiple objects, our
method can generate multi objects simultaneously.

##  5\. Conclusion

The main goal of our work is to overcome the challenges of one-shot fine-
tuning of text-to-image diffusion, and only one image is required for single
or multiple object implantation. First we propose the prototypical embedding
as an initialization for fine-tuning, and experiments show that it can
effectively mitigate overfitting.During training, we introduce class-
characterizing regularization to protect the prior knowledge of the pre-
trained model and further enhance the synthetic diversity. In addition, the
object-specific Loss effective for improving the fidelity of the generated
images and can handle multiple objects. Our approach is object-driven and
competitive with other existing approaches. We anticipate that this work will
greatly enhance the quality and efficiency of personalized image generation.

At the same time, we also found that our method has some limitations, such as
errors in the mask region for objects with complex edges, which sometimes
cause degradation in the quality of the generated image edges. In addition,
the fidelity of the generated images is slightly degraded when implanting
smaller objects. In order to solve the above problems, future work will be
devoted to improving the way of acquiring mask images and adding multi-scale
perception mechanism for objects.

## References

  * (1)
  * Bińkowski et al. (2018) Mikołaj Bińkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. 2018.  Demystifying mmd gans.  _arXiv preprint arXiv:1801.01401_ (2018).
  * Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018.  Large scale GAN training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ (2018).
  * Brooks et al. (2022) Tim Brooks, Aleksander Holynski, and Alexei A Efros. 2022.  Instructpix2pix: Learning to follow image editing instructions.  _arXiv preprint arXiv:2211.09800_ (2022).
  * Chang et al. (2023) Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. 2023\.  Muse: Text-To-Image Generation via Masked Generative Transformers.  _arXiv preprint arXiv:2301.00704_ (2023).
  * Chen et al. (2022) Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. 2022.  Re-imagen: Retrieval-augmented text-to-image generator.  _arXiv preprint arXiv:2209.14491_ (2022).
  * Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Nichol. 2021.  Diffusion models beat gans on image synthesis.  _Advances in Neural Information Processing Systems_ 34 (2021), 8780–8794.
  * Ding et al. (2022) Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang. 2022.  Cogview2: Faster and better text-to-image generation via hierarchical transformers.  _arXiv preprint arXiv:2204.14217_ (2022).
  * Dong et al. (2022) Ziyi Dong, Pengxu Wei, and Liang Lin. 2022.  Dreamartist: Towards controllable one-shot text-to-image generation via contrastive prompt-tuning.  _arXiv preprint arXiv:2211.11337_ (2022).
  * Esser et al. (2021) Patrick Esser, Robin Rombach, and Bjorn Ommer. 2021.  Taming transformers for high-resolution image synthesis. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 12873–12883.
  * French (1999) Robert M French. 1999.  Catastrophic forgetting in connectionist networks.  _Trends in cognitive sciences_ 3, 4 (1999), 128–135.
  * Gal et al. (2022) Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2022.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ (2022).
  * Goodfellow et al. (2020) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020.  Generative adversarial networks.  _Commun. ACM_ 63, 11 (2020), 139–144.
  * Hessel et al. (2021) Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021.  Clipscore: A reference-free evaluation metric for image captioning.  _arXiv preprint arXiv:2104.08718_ (2021).
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Advances in Neural Information Processing Systems_ 33 (2020), 6840–6851.
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ (2021).
  * Karras et al. (2021) Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2021.  Alias-free generative adversarial networks.  _Advances in Neural Information Processing Systems_ 34 (2021), 852–863.
  * Kirillov et al. (2023) Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. 2023\.  Segment anything.  _arXiv preprint arXiv:2304.02643_ (2023).
  * Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017\.  Overcoming catastrophic forgetting in neural networks.  _Proceedings of the national academy of sciences_ 114, 13 (2017), 3521–3526.
  * Kumari et al. (2022) Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. 2022.  Multi-Concept Customization of Text-to-Image Diffusion.  _arXiv preprint arXiv:2212.04488_ (2022).
  * Li et al. (2022) Dingcheng Li, Zheng Chen, Eunah Cho, Jie Hao, Xiaohu Liu, Fan Xing, Chenlei Guo, and Yang Liu. 2022.  Overcoming catastrophic forgetting during domain adaptation of seq2seq language generation. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 5441–5454.
  * Li and Hoiem (2017) Zhizhong Li and Derek Hoiem. 2017.  Learning without forgetting.  _IEEE transactions on pattern analysis and machine intelligence_ 40, 12 (2017), 2935–2947.
  * Mokady et al. (2022) Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2022.  Null-text Inversion for Editing Real Images using Guided Diffusion Models.  _arXiv preprint arXiv:2211.09794_ (2022).
  * Nichol et al. (2021) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2021.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ (2021).
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International conference on machine learning_. PMLR, 8748–8763.
  * Ramasesh et al. (2022) Vinay Venkatesh Ramasesh, Aitor Lewkowycz, and Ethan Dyer. 2022.  Effect of scale on catastrophic forgetting in neural networks. In _International Conference on Learning Representations_.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ (2022).
  * Razavi et al. (2019) Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. 2019.  Generating diverse high-fidelity images with vq-vae-2.  _Advances in neural information processing systems_ 32 (2019).
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10684–10695.
  * Ruiz et al. (2022) Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2022.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  _arXiv preprint arXiv:2208.12242_ (2022).
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. 2022\.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ 35 (2022), 36479–36494.
  * Sauer et al. (2023) Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. 2023.  Stylegan-t: Unlocking the power of gans for fast large-scale text-to-image synthesis.  _arXiv preprint arXiv:2301.09515_ (2023).
  * Sheynin et al. (2022) Shelly Sheynin, Oron Ashual, Adam Polyak, Uriel Singer, Oran Gafni, Eliya Nachmani, and Yaniv Taigman. 2022.  Knn-diffusion: Image generation via large-scale retrieval.  _arXiv preprint arXiv:2204.02849_ (2022).
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015.  Deep unsupervised learning using nonequilibrium thermodynamics. In _International Conference on Machine Learning_. PMLR, 2256–2265.
  * Song et al. (2020) Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ (2020).
  * Wen et al. (2023) Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2023.  Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery.  _arXiv preprint arXiv:2302.03668_ (2023).
  * Yu et al. (2022) Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. 2022\.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv preprint arXiv:2206.10789_ (2022).