# Freezing delay of a drop impacting on a monolayer of cold grains

Yudong Li  Song-Chuan Zhao  State Key Laboratory for Strength and Vibration of
Mechanical Structures, School of Aerospace Engineering, Xi’an Jiaotong
University, Xi’an 710049, China<EMAIL_ADDRESS>
###### Abstract

We investigate a subfreezing droplet impact scenario in a low-humidity
environment, where the target is a cold granular monolayer. When the
undercooling degree of targets passes a threshold, such a granular layer
effectively postpones the bulk freezing time of the droplet in comparison with
the impact on the bare substrate underneath. In this case, the retraction of
the droplet after impact reduces the contact area with the cold substrate,
even though both the grains and the substrate are wettable to the liquid. We
find that the significant changes in the dynamic behavior are triggered by
freezing the liquid that wets the pores. Owing to the small dimension of the
pores, the freezing process is rapid enough to match the dynamics over the
droplet dimension. In certain circumstances, the rapid freezing may even stop
liquid penetration and shed icing from the underneath surface.

##  1 Introduction

Subfreezing droplet impact is relevant to various artificial and natural
processes, such as aircraft icing, freezing rain hazards, and spray coating
technology. In recent years, there has been a growing focus on anti-icing
surface design 1, 2. The superhydrophobic microstructure is commonly
implemented to reduce the contact time of the droplet with the surface and to
postpone its bulk freezing. Altogether, the impacting droplet is expected to
be removed from the surface readily, for instance, by gravity. While there
have been reports of freezing delay on the repellent coating 3, 4, 5, the
potential for failures arises from liquid penetration through the
microstructure 6 and enhanced adhesion of the solidified residual 7, 8, 9.
Another challenge is the unavoidable random nucleation at high subfreezing
degrees 10, which impedes the superhydrophobicity and results in partial or
complete adhesion 11, 12. Considering the complexity and variety of icing
conditions, a unifying strategy of passive icephobicity is challenging.

Recently, the shielding of icing was reported for droplet impact on a
subfreezing sand pile 13. We suggest the same mechanism might apply to a
monolayer of cold particles. This paper explores the monolayer configuration
in experiments, shedding light on the relative significance between the
granular layer and the underlying substrate. Our findings confirm that a
granular monolayer delays bulk freezing and, under specific conditions, can
even detach the icing formed by an impacting drop onto the underlying
substrate. In our experiments, a hexadecane drop of constant diameter
($D_{0}=2\pm 0.1$\mathrm{m}\mathrm{m}$$) at room temperature ($\approx
24$\mathrm{\SIUnitSymbolCelsius}$$) impacts on a surface covered by a granular
layer in a nitrogen chamber. The granular layer and the underneath silicone
wafer are maintained at an undercooling degree $\Delta T$ below the melting
point of the liquid before the impact. We use polydisperse ceramic (Yttrium
stabilized Zirconium Oxide) spheres of mean diameters
$d=104,\,172$\mathrm{\SIUnitSymbolMicro m}$$ to prepare nearly hexagonal
packings in two-dimension (see SI text). We will first show that the granular
monolayer effectively postpones the bulk freezing time of the droplet.

##  2 Freezing delay

Figure 1:  The freezing time $\tau_{F}$ of the entire droplet for impacts with
the impact velocity $U=1.6$\mathrm{m}\mathrm{/}\mathrm{s}$$ on the subfreezing
plate ($\times$) and a monolayer of grains ($\triangle$ for
$d=104$\mathrm{\SIUnitSymbolMicro m}$$ and $\LEFTcircle$ for
$d=172$\mathrm{\SIUnitSymbolMicro m}$$). For each $\Delta T$, the value of
$\tau_{F}$ is averaged over three measurements. The standard deviation is
plotted as error bars for $d=104$\mathrm{\SIUnitSymbolMicro m}$$. The
uncertainty for $d=172$\mathrm{\SIUnitSymbolMicro m}$$ is comparable, and that
for the cold plate is significantly lower (<0.1$\tau_{F}$). The solid line is
a fit of Eq. 1. The two points highlighted by red squares correspond to the
solidified morphologies in the insets. The delayed branch remains for higher
impact velocities up to $2.75\text{\,}\mathrm{m}\mathrm{/}\mathrm{s}$.

Consider putting a droplet with a volume of $V=D_{0}^{3}\pi/6$ into contact
with a substrate at the undercooling degree $\Delta T$. The solution to the
one-dimensional Stefan problem provides an estimation for the bulk freezing
time:

$\tau_{F}=\left(\frac{V}{A}\right)^{2}\frac{1}{2\alpha_{i}}\frac{1}{\mathrm{St}}.$
(1)

In Equation 1, $\alpha_{i}$ signifies the thermal diffusivity of the
solidified phase. The bulk freezing time, $\tau_{F}$, is inversely related to
contact area $A$ and the Stefan number $\mathrm{St}=c_{p}\Delta T/L$, with $L$
representing the latent heat of the phase transition.

In Fig. 1, we plot $\tau_{F}$ of the droplet impacting on the bare silicon
wafer and the grain layers, respectively, for the impact velocity
$U=1.6$\mathrm{m}\mathrm{/}\mathrm{s}$$. For impact on the cold silicone
wafer, $\tau_{F}$ follows Eq. 1. For impact on the grain layer, $\tau_{F}$
resembles the same relation for low $\Delta T$. However, it begins to rise for
$\Delta T>10$\mathrm{K}$$ and transitions to a separate branch with delayed
freezing, increased by an order of magnitude. As all quantities except for $A$
are experimental parameters, we use the measured $\tau_{F}$ and Eq. 1 to
estimate the contact area for these two branches. Compared to impacts on the
silicon wafer, $A$ is cut to roughly a quarter in the delayed branch,
consistent with the observed changes in solidified morphology (Fig. 1 insets).
Given the high wettability of the silicone wafer towards hexadecane, the
impacting liquid would wet the substrate upon contact. Therefore, the observed
dewetting behavior for $\Delta T>10$\mathrm{K}$$ can be attributed to the
shielding effect of the grain layer, even though it exhibits wettability
towards the impacting liquid as well. This shielding effect must occur at
early impact and be sustained during droplet spreading-retraction. The overall
spreading-retraction process is measured by the capillary time
$\tau_{\gamma}=\sqrt{\rho_{l}D_{0}^{3}/\sigma}\approx
15$\mathrm{m}\mathrm{s}$$, where $\rho_{l}$ and $\sigma$ are the mass density
and surface tension coefficient of the liquid. The freezing delay at seconds
is thus an outcome of changes in dynamics at milliseconds.

Figure 2:  (a-c): The impact events of $U=1.6$\mathrm{m}\mathrm{/}\mathrm{s}$$
on grain layer of $d=172$\mathrm{\SIUnitSymbolMicro m}$$ with various
undercooling degrees, $\Delta T$. Snapshots are captured at inertia time
$\tau_{i}$, maximum spreading time $\tau_{s}$ and retraction time $\tau_{c}$.
In (c), $D_{m}$ and $D_{c}$ measure the droplet diameter at the corresponding
moments. Panels (d-e) display the particle number ratio $\phi$ and the
retraction degree $\beta$ versus $\Delta T$ for
$U=1.6$\mathrm{m}\mathrm{/}\mathrm{s}$$ and $d=172$\mathrm{\SIUnitSymbolMicro
m}$$. Panels (f-g) characterize the variations in $\phi(\Delta T)$ and
$\beta(\Delta T)$ with $U$ and $d$, showcasing their lower and upper bounds
($\phi_{l,h}$ and $\beta_{h}$) and transition regimes ($\Delta T_{\phi}$ and
$\Delta T_{\beta}$). The same color and symbols as in Fig. 1 are used for the
two grain sizes. In (g), the dashed line represents the theoretical prediction
on the upper bound of $\Delta T_{\beta}$ for the small ZrO2 grains. The
diamond point indicates the critical undercooling temperatures in theory
corresponding to Fig. 3d. See text for details.

##  3 Short-time dynamics

Figure 2a-c show impact events of $U=1.6$\mathrm{m}\mathrm{/}\mathrm{s}$$ at
three distinct time scales: inertia time $\tau_{i}$, spreading time
$\tau_{s}$, and retraction time $\tau_{c}$. The inertia time
$\tau_{i}=D_{0}/U$ defines the time scale of momentum transfer of the droplet.
$\tau_{s}\approx\tau_{\gamma}/6$ and $\tau_{c}\approx 1.5\tau_{\gamma}$
respectively marks the endpoint of spreading and retraction (see SI text).
Notable differences in dynamics are observable between high and low
undercooling degrees. The discrimination occurs earlier as $\Delta T$
increases. Using Fig. 2a as a reference, the dark center at $t=\tau_{i}$
indicates that the liquid permeates through the grain layer and wets the
silicon wafer underneath. As time progresses to $t=\tau_{s}$, the wetted area
expands, and the central grains are pushed toward the rim of the spreading
liquid film. The droplet then remains in its expanded state with little
retraction ($t=\tau_{c}$) and freezes on the substrate. The bulk freezing time
and solidified morphology resemble impacts on a bare silicon wafer substrate
[cf Fig. 1]. By increasing the undercooling degree to $\Delta
T=10$\mathrm{K}$$, the droplet still wets the central area but without any
expansion of the wetting area or movement of the grains within it. For
$t>\tau_{s}$, the liquid-grain mixture exhibits no retraction. However, the
liquid in the residual seeps through the inter-grain pores, and the wetting
area expands and fully covers the underneath substrate at $t\sim
30$\mathrm{m}\mathrm{s}$\approx 2\tau_{\gamma}$. For even higher $\Delta T$,
the short-time liquid penetration diminishes ($t\leq\tau_{i}$), and the
droplet largely spreads atop the grain layer. Subsequently, pronounced
retraction occurs during $\tau_{s}<t<\tau_{c}$, and the reduced contact area
postpones the bulk freezing [cf. Fig. 1].

The distinction of impact dynamics depicted in Fig. 2a-c is induced by certain
freezing processes of the impacting liquid, manifested as the dependence of
$\Delta T$. Two dimensionless numbers are introduced to quantify this
dependence: $\phi$, the ratio of grain numbers between $t=\tau_{s}$ and the
initial state, and $\beta=(D_{m}-D_{c})/(D_{m}-D_{0})$, representing the
degree of retraction. In this context, $D_{m}$ and $D_{c}$ denote the droplet
diameters at $\tau_{s}$ and $\tau_{c}$, respectively (see Fig. 2c and SI text
for additional details). While calculating $\phi$, grains within the impact
center, a square region with a diagonal length of $D_{0}$, are counted 111The
dimension $D_{0}$ indicates the span of intense impact stresses that causes
most significant particle motion. Other choices, such as the shape of the
counting area, does not alter the transition temperature $\Delta T_{\phi}$.
Figure 2d plots $\phi$ for impacts of $U=1.6$\mathrm{m}\mathrm{/}\mathrm{s}$$
and $d=172$\mathrm{\SIUnitSymbolMicro m}$$. The number ratio transitions from
$\phi_{l}$, at low $\Delta T$, to a plateau, $\phi_{h}$, at high undercooling
degrees. The transition regime is marked as $\Delta T_{\phi}$. The degree of
retraction, $\beta(\Delta T)$, follows a similar pattern (see Fig. 2e),
undergoing an increase from 0 to its plateau value $\beta_{h}$ when $\Delta T$
exceeds $\Delta T_{\beta}$.

Given the minor contact area between the spherical grains and the silicon
wafer and the significant difference in their thermal conductivities, the
grains and substrate can be viewed as two separate heat sinks. The freezing
processes induced by these two types of heat sinks are referred to as grain
freezing and substrate freezing henceforth. The subsequent analysis will focus
on the magnitude and variants of $\Delta T_{\phi,\,\beta}$, $\phi_{l,h}$ and
$\beta_{h}$ to clarify the underlying freezing process.

###  3.1 Substrate freezing

Grain motion is propelled by the impact stresses of the droplet, measured by
the inertia pressure $\rho_{l}U^{2}$. Consequently, the lower bound $\phi_{l}$
decreases with increased $U$ as displayed in Fig. 2f. In contrast,
$\phi_{h}\approx 0.9$ remains unaffected by $U$, suggesting the formation of a
frozen grain patch for $\Delta T>\Delta T_{\phi}$. The frozen patch needs to
form early to counterbalance the impact stresses, preferably within a
timeframe comparable to $\tau_{i}$. As the liquid wets the grain layer and the
underneath substrate throughout the transition regime $\Delta T_{\phi}$ [cf.
Fig. 2a-b], both grain freezing and substrate freezing could contribute to the
rise of $\phi$. However, their relative significance may yield different
outcomes. The former causes the wetted grains to freeze together, while the
latter results in the grains freezing onto the substrate (see Fig. 3a and b).
In the former case, the grain freezing process covers a dimension proportional
to $d$. Therefore, to match the freezing timescale with $\tau_{i}$, a
dependence on $d$ is expected for $\Delta T_{\phi}$. However, the observed
consistency of $\Delta T_{\phi}$ in Fig. 2f contradicts this expected
relationship, indicating the dominance of substrate freezing for $\phi$
saturation.

To initiate substrate freezing, the droplet must penetrate through the grain
layer. Liquid penetration is driven by the inertia pressure of the droplet and
is resisted by the viscous drag across a pore dimension $d_{p}$, $\sim\mu
U/d_{p}$, with $\mu$ as the dynamic viscosity of the liquid. Let $\tau_{p}$
represent the time required to reach the penetration depth $d$. It can be
shown that $\tau_{p}/\tau_{i}\sim(d/d_{p})^{2}/\mathrm{Re}$, where
$\mathrm{Re}=\rho_{l}UD_{0}/\mu$ represents the Reynolds number. By measuring
the critical impact velocity for $\tau_{p}/\tau_{i}=1$, we can estimate
$\tau_{p}$ (see SI text). The duration for the substrate freezing to take
effect, $\tau_{i}-\tau_{p}$, decreases with $U$ for the parameter range under
investigation. Nevertheless, $\Delta T_{\phi}$ in Fig. 2f is insensitive to
$U$, which indicates that liquid freezes instantaneously upon contact with the
cold substrate. Indeed, the observed upper bound of $\Delta T_{\phi}$,
approximately $9$\mathrm{K}$$, aligns with the undercooling degree of
immediate nucleation for hexadecane reported in Ref. 14. The area of momentum
transfer, equivalent to the area directly beneath the droplet $\pi
D_{0}^{2}/4$ (Fig. 2a-b), is the initial penetration site and largely
determines the dimension of frozen patch 222The dimension of the frozen patch
is always larger than $D_{0}$ by approximately 20% and slightly decreases with
$\Delta T$..

###  3.2 Dewetting initiation

Beyond $\Delta T_{\phi}$, the droplet body impacts upon the immediately formed
frozen layer on the substrate at $t<\tau_{i}$. The liquid ejecting from the
frozen region primarily possesses horizontal momentum and is isolated from the
substrate by the frozen layer, which prompts the dewetting of the spreading
lamella and its retraction afterward. As a result, the rise of $\beta$, marked
by $\Delta T_{\beta}$, closely follows the saturation of $\phi$. The lower
bound of $\Delta T_{\beta}$ aligns with the upper bound of $\Delta T_{\phi}$
within the measurement uncertainty, demonstrating robustness against $U$ and
$d$ as well [Fig. 2g].

Note that the initiation and sustainability of dewetting in this study are
largely independent of ambient air 15, 13. Instead, the resistance from grains
on the spreading lamella redirects its edge upwards 16, similar to droplet
impact on a liquid layer 17, 18. The retraction degree, $\beta$, exhibits
significant fluctuations within the transition regime [Fig. 2e], primarily due
to the lamella recontacting/rewetting the substrate via capillary action. For
$\Delta T>\Delta T_{\beta}$, rewetting diminishes, and the droplet-grain
mixture retracts to its smallest feasible size, the frozen patch formed early
($\sim D_{0}$). This defines the saturation value $\beta_{h}$. The transition
regime, $\Delta T_{\beta}$, aligns with the increase of $\tau_{F}$ in Fig. 1.
This confirms the connections between short-time dynamics and the long-term
overall freezing.

We assume that a certain separation distance from the substrate, ideally $\sim
d/2$, is necessary to ensure the consistent dewetting and the saturation of
$\beta$. According to Eq. 1, substrate freezing contributes a frozen dimension
of 10$\mathrm{\SIUnitSymbolMicro m}$ during $\tau_{i}$ for
$U=1.6$\mathrm{m}\mathrm{/}\mathrm{s}$$ and $\Delta T=30$\mathrm{K}$$,
approximately one tenth of $d$ or less. Moreover, this frozen dimension
remains constant with $d$. Hence, it does not explain the $d$ dependence of
$\beta_{h}$ and the upper bound of $\Delta T_{\beta}$ in Fig. 2g. Therefore,
substrate freezing alone cannot saturate $\beta$, necessitating the
consideration of grain freezing.

###  3.3 Grain freezing

The process of grain freezing refers to the freezing of liquid that wets the
pores in the granular layer. The most pronounced effect of freezing occurs at
the smallest cross-section of the pore, as illustrated in Fig. 3c. The grain
freezing process is of particular interest beyond $\Delta T_{\phi}$, where
little motion of grains is seen [Fig. 2]. A constant pore dimension
$d_{p}=0.1d$ is thus used in the following analysis 333This value is obtained
by dividing the highlighted area ($(\sqrt{3}-\pi/2)d^{2}/4$) in Fig. 3c by the
arc length of the surrounding grains $d\pi/2$, resulting in a length scale of
0.05$d$. This scale measures the dimension from the grain surface to the
pore’s center, equivalent to $d_{p}/2$.. As freezing occurs from all
directions simultaneously, the relevant frozen dimension is $d_{p}/2$. We
denote the freezing time over $d_{p}/2$ as $\tau_{f}$. The classic Stefan
problem (Eq. 1) provides $\tau_{f}\approx\tau_{i}$ at $\Delta T=8$\mathrm{K}$$
for $U_{0}=1.6$\mathrm{m}\mathrm{/}\mathrm{s}$$ and
$d=172$\mathrm{\SIUnitSymbolMicro m}$$. This underestimates $\tau_{f}$;
otherwise, the significance of grain freezing, such as the variance of $\Delta
T_{\phi}$ with $d$, would have been observable. To obtain a more reasonable
estimation of $\tau_{f}$, it is necessary to consider the thermal properties
of the heat sink (grains). Two additional dimensionless numbers,
$B\equiv\rho_{i}c_{p}^{i}/\rho_{g}c_{p}^{g}$ and $K\equiv k_{i}/k_{g}$, are
introduced, representing the ratio of volumetric heat capacity and thermal
conductivity between the solidified phase and grains. It is worth noting that
the contact area between spherical grains is negligible compared to their
surface area. Thus, in the context of grain freezing, the heat sinks are
individual grains of a finite dimension $d$. The grain freezing process thus
has an inherent length scale and deviates from self-similar solutions like Eq.
1. In Ref. 13, we have proposed an approximation solution for $\tau_{f}$ that
accounts for the finite size effect to the leading order:

$\tau_{f}=\mathcal{F}(K)\frac{\hat{d}_{p}^{2}}{2\mathrm{St}}\left(1+\frac{2}{3}\frac{B}{\mathrm{St}}\hat{d}_{p}\right)\frac{d^{2}}{4\alpha_{i}},\,\text{with
}\hat{d}_{p}=\frac{d_{p}}{d}.$ (2)

$\mathcal{F}(K)\geq 1$ increases with $K$. We identify $\mathcal{F}(K=0.15)=2$
for ZrO2 grains (see SI text). The second term in the bracket on the right-
hand side of Eq. 2 is the correction introduced by the finite size effect on
the diffusive propagation of the freezing front. This term is about 0.3 at
$\Delta T=20$\mathrm{K}$$ for the material under investigation and decreases
for higher $\Delta T$.

Setting $\tau_{f}=\tau_{i}$ in Eq. 2 yields a critical undercooling degree
$\Delta T_{i}$. We compute $\Delta T_{i}$ for the small ZrO2 grain layer
($d=104$\mathrm{\SIUnitSymbolMicro m}$$). The result is approximately
inversely proportional to $U$ and aligns with the upper bound of $\Delta
T_{\beta}$ (see the dashed line in Fig. 2g). When $\Delta T$ exceeds $\Delta
T_{i}$, $\tau_{f}$ decreases further, potentially matching the penetration
timescale $\tau_{p}$. The critical undercooling degree that satisfies
$\tau_{f}=\tau_{p}$ is calculated for $U=1.6$\mathrm{m}\mathrm{/}\mathrm{s}$$,
resulting in $\Delta T_{p}=28.6$\mathrm{K}$$. For $\Delta T>\Delta T_{p}$, the
pores are sealed so rapidly by freezing that liquid infiltration halts
completely. We show the bottom image of the solidified residual for impacts on
the small ZrO2 layer around $\Delta T_{p}$. Indeed, grains protrude from the
frozen bottom uniformly, with a consistent protrusion height of
11$\mathrm{\SIUnitSymbolMicro m}$. Given that $\Delta T_{p}\sim U^{2}$,
complete suppression of liquid penetration for higher $U$ values was not
observed within the studied undercooling range.

The freezing time, $\tau_{f}$ in Eq. 2, is anticipated to exhibit a quadratic
increase with the grain size, $d$. Consequently, the larger grains
($d=172$\mathrm{\SIUnitSymbolMicro m}$$) requires a significantly greater
$\Delta T_{i,p}$ to achieve $\tau_{f}=\tau_{i,p}$, a condition unattainable in
our experiments. In other words, grain freezing alone is insufficient to close
the pores within $\tau_{i}$ for the larger grains under the investigated range
of $\Delta T$. In this case, consistent dewetting and the saturation of
$\beta$ arise from the combined effects of grain freezing and substrate
freezing processes, and $\Delta T_{\beta}$ deviates from the prediction of
$\tau_{f}=\tau_{i}$ by Eq. 2. We refer to this scenario as partial
penetration. In the partial penetration scenario, the penetrating liquid wets
the substrate directly beneath the pore and then spreads. The frozen layer
growing from the substrate intersects with that from the grain surface. In
consequence, a portion of the grain bottom unwetted [Fig. 3b]. Figure 3e
illustrates the bottom of the solidified morphology formed on the large ZrO2
layer under the same impact and temperature parameters as Fig. 3d, where the
impact center is penetrated. The exposure area of grains deviates from the
circular shape and increases further away from the impact center. The
estimated protrusion height ranges from 0 to 19$\mathrm{\SIUnitSymbolMicro
m}$, marking the intersection of the frozen layers from the substrate and the
grain. Substrate freezing at $\Delta T\sim 30$\mathrm{K}$$, as previously
discussed, fosters the development of a frozen layer approximately
10$\mathrm{\SIUnitSymbolMicro m}$ thick within $\tau_{i}$, comparable to the
measured protrusion height. The weaker influence of grain freezing for larger
grains results in a notable decrease of $\beta_{h}$ with $U$ (see Fig. 2g).
The reduced $\beta_{h}$ indicates a larger frozen area. The freezing delay
effect at higher $U$ thus becomes less efficient than that shown in Fig. 1.

An increase in thermal conductivity of grains enhances the grain freezing by
reducing $\mathcal{F}(K)$ and thus $\tau_{f}$ in Eq. 2. The consequence is
demonstrated through an experiment involving a layer of Tin balls with a
diameter of $d=155$\mathrm{\SIUnitSymbolMicro m}$$, the bottom image of which
is presented in Fig. 3f. The high thermal conductivity of Tin ($K\sim
10^{-3}\ll 1$) sets $\mathcal{F}=1$ in Eq. 2. From a quantitative standpoint,
the square of the diameter ratio between the small ZrO2 grains and Tin balls,
2.2, matches their difference in $\mathcal{F}$. In other words, the increase
in $\tau_{f}$ due to the grain size of Tin balls is offset by their enhanced
thermal conductivity. As a result, liquid penetration is significantly
inhibited in the Tin ball layer, similar to Fig. 3d. The partial penetration,
more visibly apparent here, primarily occurs at the ‘defects’ in the hexagonal
structure, i.e., the larger pores.

Figure 3: Sketch of grain freezing (a) and substrate freezing (b). The light
(dark) blue area indicates the liquid (solidified) phase. The dashed box in
(b) illustrates the partial penetration scenario. (c) In a dense monolayer of
spheres, the pore (blue area) is a curved triangle. Images of the bottom of
the solidified liquid-grain mixture are shown for impacts on small
($d=104$\mathrm{\SIUnitSymbolMicro m}$$ in (d)) and large
($d=172$\mathrm{\SIUnitSymbolMicro m}$$ in (e)) ZrO2 layers, and Sn balls
($d=155$\mathrm{\SIUnitSymbolMicro m}$$) layer in (f). The impact parameters
are $U=1.6$\mathrm{m}\mathrm{/}\mathrm{s}$$ and $\Delta T=28$\mathrm{K}$$ for
all cases. The top row gives a zoom-in of the impact center area. Note that
the center of the residual is not necessarily the impact center.

##  4 Conclusion

We examine a scenario in this study where the granular layer acts as a shield
for the underlying substrate against icing during droplet impact. Compared to
superhydrophobic coating, the grain layer configuration investigated here
provides an alternative viewpoint. The grain-freezing process could shed the
impact stresses (Fig. 2d) and the contact of the droplet with the underlying
substrate (Fig. 2e and 3d-f). In consequence, the post-impact residual can be
readily removed. For small grains, the Bond number,
$\mathrm{Bo}=\rho_{g}d^{2}g/\gamma$, is typically smaller than 1. Therefore,
the weight of grains does not affect the droplet retraction. On inclines, weak
adhesion that merely balances the gravity can thus be introduced (by static
electrical charges, for instance) without altering the droplet retraction and
freezing delay effect. After impact, the additional droplet mass overwhelms
the weak adhesion, and the residual may be removed under gravity. Lastly, we
highlight that the non-wettability of the grain layer could further enhance
the shielding effect, as it impedes impact penetration. At the intermediate
$\Delta T$ ($\sim\Delta T_{\beta}$), the non-wettability stabilizes dewetting
and retraction. Beyond $\Delta T_{p}$, however, the dominance of grain
freezing reduces the significance of wettability (see SI video).

The authors thank Mr. Hao-Jie Zhang for building the Nitrogen chamber and Ms.
Yashi Li for her contribution to experimental measurement. This work is
supported by NSFC project number 12172277.

The following files are available free of charge.

  * •

SI.pdf: additional details of the experimental protocol, image processing
method, the calculation of the penetration time $\tau_{p}$, and the grain-
freezing model.

  * •

SI_wettability.mp4: illustrating the influence of wettability of grains.

## References

  * Kreder et al. 2016 Kreder, M. J.; Alvarenga, J.; Kim, P.; Aizenberg, J. Design of Anti-Icing Surfaces: Smooth, Textured or Slippery? _Nature Reviews Materials_ 2016, _1_ , 15003
  * Schutzius et al. 2015 Schutzius, T. M.; Jung, S.; Maitra, T.; Eberle, P.; Antonini, C.; Stamatopoulos, C.; Poulikakos, D. Physics of Icing and Rational Design of Surfaces with Extraordinary Icephobicity. _Langmuir : the ACS journal of surfaces and colloids_ 2015, _31_ , 4807–4821
  * Jung et al. 2011 Jung, S.; Dorrestijn, M.; Raps, D.; Das, A.; Megaridis, C. M.; Poulikakos, D. Are Superhydrophobic Surfaces Best for Icephobicity? _Langmuir_ 2011, _27_ , 3059–3066
  * Guo et al. 2012 Guo, P.; Zheng, Y.; Wen, M.; Song, C.; Lin, Y.; Jiang, L. Icephobic/Anti-Icing Properties of Micro/Nanostructured Surfaces. _Advanced Materials_ 2012, _24_ , 2642–2648
  * Alizadeh et al. 2012 Alizadeh, A.; Yamada, M.; Li, R.; Shang, W.; Otta, S.; Zhong, S.; Ge, L.; Dhinojwala, A.; Conway, K. R.; Bahadur, V.; Vinciquerra, A. J.; Stephens, B.; Blohm, M. L. Dynamics of Ice Nucleation on Water Repellent Surfaces. _Langmuir : the ACS journal of surfaces and colloids_ 2012, _28_ , 3180–3186
  * Maitra et al. 2014 Maitra, T.; Antonini, C.; Tiwari, M. K.; Mularczyk, A.; Imeri, Z.; Schoch, P.; Poulikakos, D. Supercooled Water Drops Impacting Superhydrophobic Textures. _Langmuir : the ACS journal of surfaces and colloids_ 2014, _30_ , 10855–10861
  * Varanasi et al. 2010 Varanasi, K. K.; Deng, T.; Smith, J. D.; Hsu, M.; Bhate, N. Frost Formation and Ice Adhesion on Superhydrophobic Surfaces. _Applied Physics Letters_ 2010, _97_ , 234102
  * Kulinich et al. 2011 Kulinich, S. A.; Farhadi, S.; Nose, K.; Du, X. W. Superhydrophobic Surfaces: Are They Really Ice-Repellent? _Langmuir_ 2011, _27_ , 25–29
  * Chen et al. 2012 Chen, J.; Liu, J.; He, M.; Li, K.; Cui, D.; Zhang, Q.; Zeng, X.; Zhang, Y.; Wang, J.; Song, Y. Superhydrophobic Surfaces Cannot Reduce Ice Adhesion. _Applied Physics Letters_ 2012, _101_ , 111603
  * Wang et al. 2019 Wang, L.; Kong, W.; Wang, F.; Liu, H. Effect of Nucleation Time on Freezing Morphology and Type of a Water Droplet Impacting onto Cold Substrate. _International Journal of Heat and Mass Transfer_ 2019, _130_ , 831–842
  * Schremb et al. 2017 Schremb, M.; Roisman, I. V.; Tropea, C. Transient Effects in Ice Nucleation of a Water Drop Impacting onto a Cold Substrate. _Physical Review E_ 2017, _95_ , 022805
  * Zhang et al. 2020 Zhang, X.; Liu, X.; Wu, X.; Min, J. Impacting-Freezing Dynamics of a Supercooled Water Droplet on a Cold Surface: Rebound and Adhesion. _International Journal of Heat and Mass Transfer_ 2020, _158_ , 119997
  * Zhao et al. 2024 Zhao, S.-C.; Zhang, H.-J.; Li, Y. Cold Granular Targets Slow the Bulk Freezing of an Impacting Droplet. _Proceedings of the National Academy of Sciences_ 2024, _121_ , e2311930121
  * Kant et al. 2020 Kant, P.; Koldeweij, R. B. J.; Harth, K.; van Limbeek, M. A. J.; Lohse, D. Fast-Freezing Kinetics inside a Droplet Impacting on a Cold Surface. _Proceedings of the National Academy of Sciences_ 2020, _117_ , 2788–2794
  * Grivet et al. 2023 Grivet, R.; Huerre, A.; Séon, T.; Josserand, C. Making Superhydrophobic Splashes by Surface Cooling. _Physical Review Fluids_ 2023, _8_ , 063603
  * Esmaili et al. 2021 Esmaili, E.; Chen, Z.-Y.; Pandey, A.; Kim, S.; Lee, S.; Jung, S. Corona Splashing Triggered by a Loose Monolayer of Particles. _Applied Physics Letters_ 2021, _119_ , 174103
  * Peregrine 1981 Peregrine, D. H. The Fascination of Fluid Mechanics. _Journal of Fluid Mechanics_ 1981, _106_ , 59–80
  * Marcotte et al. 2019 Marcotte, F.; Michon, G.-J.; Séon, T.; Josserand, C. Ejecta, Corolla, and Splashes from Drop Impacts on Viscous Fluids. _Physical Review Letters_ 2019, _122_ , 014501


# Infusion: Preventing Customized Text-to-Image Diffusion from Overfitting

Weili Zeng  Shanghai Jiao Tong UniversityShanghaiChina<EMAIL_ADDRESS>,
Yichao Yan  Shanghai Jiao Tong UniversityShanghaiChina<EMAIL_ADDRESS>,
Qi Zhu  Shanghai Jiao Tong UniversityShanghaiChina<EMAIL_ADDRESS>,
Zhuo Chen  Shanghai Jiao Tong UniversityShanghaiChina<EMAIL_ADDRESS>,
Pengzhi Chu  Shanghai Jiao Tong UniversityShanghaiChina<EMAIL_ADDRESS>,
Weiming Zhao  Shanghai Jiao Tong UniversityShanghaiChina
<EMAIL_ADDRESS>and  Xiaokang Yang  Shanghai Jiao Tong
UniversityShanghaiChina<EMAIL_ADDRESS>
(2018)

###### Abstract.

Text-to-image (T2I) customization aims to create images that embody specific
visual concepts delineated in textual descriptions. However, existing works
still face a main challenge, concept overfitting. To tackle this challenge, we
first analyze overfitting, categorizing it into concept-agnostic overfitting,
which undermines non-customized concept knowledge, and concept-specific
overfitting, which is confined to customize on limited modalities, i.e,
backgrounds, layouts, styles. To evaluate the overfitting degree, we further
introduce two metrics, i.e, Latent Fisher divergence and Wasserstein metric to
measure the distribution changes of non-customized and customized concept
respectively. Drawing from the analysis, we propose Infusion, a T2I
customization method that enables the learning of target concepts to avoid
being constrained by limited training modalities, while preserving non-
customized knowledge. Remarkably, Infusion achieves this feat with remarkable
efficiency, requiring a mere 11KB of trained parameters. Extensive experiments
also demonstrate that our approach outperforms state-of-the-art methods in
both single and multi-concept customized generation. Project page:
https://zwl666666.github.io/infusion/.

T2I customization, Overfitting, Concept-agnostic, Concept-specific

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
email; October 28 - November 1, 2024; Melbourne, Australia.††isbn:
978-1-4503-XXXX-X/18/06††ccs: Do Not Use This Code Generate the Correct Terms
for Your Paper††ccs: Do Not Use This Code Generate the Correct Terms for Your
Paper††ccs: Do Not Use This Code Generate the Correct Terms for Your
Paper††ccs: Do Not Use This Code Generate the Correct Terms for Your Paper

##  1\. Introduction

In the era of multi-modality, text-to-image (T2I) generation (Rombach et al.,
2022; Podell et al., 2023; Ramesh et al., 2022; Saharia et al., [n. d.]) has
experienced rapid growth, showcasing highly imaginative generation outcomes.
Meanwhile, T2I customization rises as a potential need in image generation,
aiming to create images that embody specific visual concepts aligned with
textual descriptions. Users only need to provide a set of similar images whose
pertinent visual concepts, e.g, preferred pets, toys, or scene styles, are
subsequently extracted by a T2I model. These identified concepts are then
seamlessly interwoven into the users’ textual descriptions for customized
generation.

Typically, T2I customization methods can be classified into two categories:
inversion-based and fine-tuning. Specifically, the inversion-based methods
(Gal et al., 2022a; Voynov et al., 2023) represent concepts by learning
additional conceptual words. While this approach offers flexibility, it
introduces a significant overfitting challenge, as the injected concepts
permeate every part of the cross-attention module with textual information.
Another approach, fine-tuning method (Ruiz et al., 2023), keeps the original
architecture and optimizes model parameters with a set of customized image
data. However, a large parameter space that is fitted by a small dataset leads
to severe overfitting. Besides, the extensive size of these models often
necessitates substantial storage and computation resources. Although
Parameter-efficient tuning (PET) approaches (Kumari et al., 2023; Hu et al.,
2021) shrink the fine-tuning into only a small subset of parameters, they
still fail to eliminate the overfitting issue.

Comprehensively considering the previous methods, current T2I customization
still faces three challenges: 1) Concept overfitting takes away general
generation capabilities of foundational T2I models from customized models. 2)
Lack of appropriate metrics to quantify the impact of overfitting on
customized models. 3) Cumbersome deployment prevents users from seamlessly
switching between customized mode and regular mode as desired.

Figure 1. Example of concept-agnostic overfitting. The first column on the
left is the target concept, and the right is the non-customized generation
results of different methods. The generated “cat” consistently exhibits black
spotted stripes, while the generated “teddy” consistently presents a doll-like
form, both sharing similar backgrounds.

We deeply explore the overfitting problem and categorize it into concept-
agnostic overfitting and concept-specific overfitting. Concept-agnostic
overfitting is depicted in Figure 1. When learning the concepts of
$\langle\text{cat}\rangle$ and $\langle\text{teddy}\rangle$, customization
undermines original generative ability for non-customized concept. Concept-
specific overfitting is depicted in Figure 2. During the customization process
of T2I model, it confuses target concept with limited training modalities,
resulting in a diminution of generative diversity and consequently impacts the
textual controllability. This situation gets worse, particularly when facing
intricate text.

Based on the analysis of the above two categories of overfitting, we introduce
“Latent Fisher divergence” and “Wasserstein metric” to measure the
distribution changes of non-customized and customized concept respectively.
The latent Fisher divergence quantifies the score difference of the non-
customized concept between the original model and the customized model. The
Wasserstein metric quantifies the deviation of the customized distribution
from its original generative distribution under a super-class concept, serving
as a measure of modality diversity.

To overcome the aforementioned challenges of overfitting, we propose the
Infusion method, a T2I customization method that enables the learning of
target concepts to avoid being constrained by limited training modalities.
Infusion can also flexibly inject customized concepts into original models
while preserving their original knowledge. Our novel insight lies in the
decoupling of the attention map and value feature in each cross-attention
module. As c in Figure 3, we implemented a dual-stream structure that shares
pipeline parameters. The foundational T2I pipeline performs regular T2I
generation. To harness its generative capabilities, we imitate the operations
in P2P (Hertz et al., 2022), and use the attention maps in the foundational
T2I pipeline to replace the attention maps in the customized T2I pipeline. We
introduce target concepts by learning a residual value embedding in the
customized pipeline. All the learnable embeddings for a concept are highly
lightweight, only $11\mathrm{KB}$, enabling seamless integration into the
pipeline in a flexible and plug-and-play manner.

In summary, our contributions can be delineated as follows:

  * •

We analyze concept-agnostic and concept-specific overfitting, and propose
employing “Latent Fisher divergence” and “Wasserstein metric” to quantify
their impact.

  * •

We propose Infusion, a method that decouples attention maps and value features
in each cross-attention module, fully utilizing the capability of the original
T2I model.

  * •

Experiments show that Infusion enables plug-and-play single-concept and multi-
concept generation and achieves superior results compared to state-of-the-art
methods.

Figure 2. Example of concept-specific overfitting. Customization with the
prompt “a photo of a $\langle\text{cat}\rangle$”, we reveal that all prior
methods generate cat images with a similar size, pose, or background to the
training data.

##  2\. Related Work

Figure 3. Infusion pipline. (a) Infusion fully preserves the generative
capacity of the original model, precluding concept-agnostic overfitting. (b)
Infusion decouples the cross-attention module, replacing the attention maps in
the customized pipeline with those from the foundational pipeline, thereby
leveraging the modality diversity of the original model to mitigate concept-
specific overfitting.

###  2.1. Text-to-image Generation

Early Text-to-Image (T2I) models primarily relied on Generative Adversarial
Networks (GANs) (Goodfellow et al., 2014; Brock et al., 2018; Karras et al.,
2021; Tao et al., 2022; Zhang et al., 2021; Zhu et al., 2019), yet often
exhibited limitations in terms of generated diversity. Recently, diffusion
models (Ho et al., 2020; Sohl-Dickstein et al., 2015; Song et al., 2020; Song
and Ermon, 2019), coupled with classifier-free guidance (Ho and Salimans,
2022), have emerged as leaders in this domain. Their remarkable generative
performance is particularly noteworthy when applied to large-scale internet
datasets (Nichol et al., 2021; Ramesh et al., 2021; Rombach et al., 2022;
Ramesh et al., 2022). This fact drives us to fully exploit the prior knowledge
of these foundational models for T2I customization.

###  2.2. Text-based Image Editing

The advent of contrastive multimodal models (Radford et al., 2021; Yuan et
al., 2021), exemplified by CLIP (Radford et al., 2021), has introduced a
transformative paradigm shift in the realm of image editing (Avrahami et al.,
2022, 2023; Bar-Tal et al., 2022; Gal et al., 2022b; Patashnik et al., 2021).
Significantly, these models empower global or localized image manipulation
through the exclusive utilization of textual prompts. This paradigm has
engendered a series of text-based image editing methods (Hertz et al., 2022;
Brooks et al., 2023; Mokady et al., 2023; Kawar et al., 2023; Zhang et al.,
2023), among which the most relevant to our work is the prompt-to-prompt (P2P)
(Hertz et al., 2022). P2P achieves image editing by replacing or modifying the
attention map of the cross-attention module. Similarly, our approach employs
attention map replacement for image customization. The difference is that P2P
operates on virtual generated concepts, while we can operate on real concepts.

###  2.3. Text-to-image Customization

Text-to-Image (T2I) customization aims to generate customized concepts aligned
with text description. Textual Inversion (Gal et al., 2022a) is the pioneering
method in this realm, which introduces an innovative approach to learning new
word embeddings for representing specific concepts without tuning model
parameters. Despite its plug-and-play adaptability, its efficacy remains
constrained. DreamBooth (Ruiz et al., 2023) employs class nouns along with
unique identifiers to represent target concepts but requires the comprehensive
training of model parameters. While achieving high conceptual fidelity, this
approach is susceptible to overfitting. Subsequent methodologies,
incorporating low-rank updates or just tuning a few parameters, aim to address
this limitation (Han et al., 2023; Hu et al., 2021; Kumari et al., 2023).
Notably, Custom Diffusion (Kumari et al., 2023) focuses on updating the
weights of cross-attention and introduces regularization sets to mitigate
overfitting. Perfusion (Tewel et al., 2023) introduces a decoupling of cross-
attention into “what” and “where” pathways. It leverages a Rank-one Model
editing method (Meng et al., 2022) to adjust the weights of key and value
projection matrices, thereby alleviating concerns associated with overfitting.
Another research trajectory involves using extensive customized data for pre-
training encoders, enabling training-free inference (Arar et al., 2023; Chen
et al., 2023; Gal et al., 2023; Wei et al., 2023; Ye et al., 2023; Zhou et
al., 2023).

##  3\. Analysis of Concept Overfitting

###  3.1. Preliminaries of T2I Diffusion Models

We applied our method to the Stable Diffusion (SD) model (SD, 2022). During
the training phase, the encoder, denoted as $\mathcal{E}$, initially maps the
input image $x\in\mathcal{X}$ to its corresponding latent code
$z=\mathcal{E}(x)$. Subsequently, the decoder, denoted as $\mathcal{D}$, is
tasked with reconstructing the input image, aiming for
$\mathcal{D}(\mathcal{E}(x))\approx x$.

Diffusion models are then introduced to capture the latent distribution. The
forward diffusion process perturbs the latent input to a noisy variant,
denoted as $z_{t}$, at each time step $t$. Given a conditional prompt $y$, the
latent diffusion model $\varepsilon_{\theta}$ is trained to minimize the loss
function:

(1)
$\mathcal{L}=\mathbb{E}_{z\sim\mathcal{E}(x),y,\varepsilon\sim\mathcal{N}(0,1),t}\left[\left\|\varepsilon-\varepsilon_{\theta}\left(z_{t},t,\tau(y)\right)\right\|_{2}^{2}\right],$

where $\varepsilon_{\theta}$ is a time-conditional UNet (Saharia et al., 2022)
comprising self-attention layers and cross-attention layers. The term
$\tau(y)$ is randomly replaced by an empty input $\varnothing$. This training
objective can be construed as a form of denoising score matching (Vincent,
2011), with $\varepsilon_{\theta}\left(z_{t},t\right)$ serving as an
approximation to the score $\nabla_{z}\log p(z_{t})$ of the latent
distribution.

During inference, the latent variable $z_{T}$ is initialized by sampling from
$\mathcal{N}(0,1)$, and a sampling strategy (Ho et al., 2020; Song et al.,
2020) is iteratively employed for denoising to generate the latent variable
$z_{0}$. Following this denoising process, the denoised latent representation
is forwarded to the decoder, yielding the reconstructed image
$x^{\prime}=\mathcal{D}\left(z_{0}\right)$.

Stable Diffusion employs a cross-attention mechanism to incorporate textual
modal inputs. As previously outlined, the operation $\tau_{\theta}$ projects
the text condition $y$ into an intermediate representation denoted as
$\tau_{\theta}(y)\in\mathbb{R}^{L\times d_{\tau}}$. Subsequently, the cross-
attention operation of the $i$-th layer network is expressed as:

(2)
$\mathit{Atten}(Q_{i},K_{i},V_{i})=\text{softmax}\left(\frac{Q_{i}K_{i}^{T}}{\sqrt{d}}\right)V_{i}=M_{i}V_{i},$

which is utilized to integrate textual information into the UNet network. The
constituents are defined as:

(3) $\displaystyle Q_{i}=\varphi_{i}\left(f_{i}\right)\cdot
W_{Q}^{i},K_{i}=\tau_{\theta}(y)\cdot W_{K}^{i},V_{i}=\tau_{\theta}(y)\cdot
W_{V}^{i},$

where $Q_{i}\in\mathbb{R}^{N_{i}^{2}\times d}$, $K_{i}\in\mathbb{R}^{L\times
d}$, and $V_{i}\in\mathbb{R}^{L\times d}$ respectively represent the features
of the query, key, and value.
$\varphi_{i}\left(f_{i}\right)\in\mathbb{R}^{N_{i}^{2}\times
d_{\epsilon}^{i}}$ represents the (flattened) latent image representation of
the UNet, and $W_{Q}^{i}\in\mathbb{R}^{d_{\epsilon}^{i}\times d}$,
$W_{K}^{i}\in\mathbb{R}^{d_{\tau}\times d}$, and
$W_{V}^{i}\in\mathbb{R}^{d_{\tau}\times d}$ are learnable matrix parameters
(Rombach et al., 2022).

###  3.2. Analysis of Concept-agnostic Overfitting

Figure 4. Concept-agnostic overfitting. We use a four-peak hybrid Gaussian
distribution to represent the foundational model distribution and illustrate
that customized tuning, as observed in Dreambooth (Ruiz et al., 2023) and
Custom Diffusion (Kumari et al., 2023), undermines non-customized generative
capabilities. Figure 5. Concept-specific overfitting. We use a 25-peak hybrid
Gaussian distribution to represent the modalities in foundational model
distribution under a super-class concept. The confusion training between
customized concepts and limited modalities gradually reduces the number of
original modalities.

In this section, we analyze two types of overfitting in the customized
diffusion model: concept-agnostic overfitting and concept-specific
overfitting. We subsequently define two metrics corresponding to each type of
overfitting.

Concept-agnostic overfitting often stems from alterations in the foundational
model parameters aimed at accommodating customized concepts, thereby
compromising the model’s performance on non-customized concepts. As shown in
Figure 4, our toy foundational model is a four-peak hybrid Gaussian model,
where each peak represents a different concept. The process of customized
tuning can be viewed as the model learning a specialized distribution under a
particular concept. Here, we simplify it as learning a linear distribution
under the Gaussian peak in the top-left corner. It can be observed that when
employing a tuning method similar to Dreambooth (Ruiz et al., 2023), the
generated distributions for non-customized concepts become compact. The
shrinking of these distributions implies a reduction in the diversity of
generated samples under non-customized concepts.

Figure 6. Shrinking of non-customized distribution. The gradient field
described by $\nabla_{z}\log p_{\theta}(z|\overline{c})$, gradually
contracting during training, leading to a more convergent field
$\nabla_{z}\log p_{\theta^{\prime}}(z|\overline{c})$.

Therefore, for T2I models, we infer that concept-agnostic overfitting is
primarily related to the offset of the non-customized score. Specifically, let
$c$ represent the customized concept and $\overline{c}$ represent non-
customized concepts. $\theta$, $\theta^{\prime}$ are the parameters of the
original model and customized model. As depicted in Figure 6, the gradient
field described by $\nabla_{z}\log p_{\theta}(z|\overline{c})$, gradually
contracting during training, leading to a more convergent field
$\nabla_{z}\log p_{\theta^{\prime}}(z|\overline{c})$. To quantify this effect,
we define the latent Fisher divergence as definition 3.1, which measures the
distance between two models’ score under the non-customized concepts.

Definition 3.1 (Latent Fisher divergence). Based on the fact that $c$ is only
a small part of a vast image concept, and the training principles of denoise
score matching (Vincent, 2011), the Fisher divergence between
$p_{\theta}(z|\overline{c})$ and $p_{\theta^{\prime}}(z|\overline{c})$ can be
approximated by:

$\displaystyle
D_{F}\left(p_{\theta}(z|\overline{c})\|p_{\theta^{\prime}}(z|\overline{c})\right)$
$\displaystyle=\frac{1}{2}\mathbb{E}_{z\sim\mathcal{E}(x)}\left[\left\|\nabla_{z}\log
p_{\theta}(z|\overline{c})-\nabla_{z}\log
p_{\theta^{\prime}}(z|\overline{c})\right\|_{2}^{2}\right]$ (4)
$\displaystyle\thickapprox\frac{1}{2}\mathbb{E}_{z\sim\mathcal{E}(x)}\left[\left\|\varepsilon_{\theta(z_{t},t,\bar{c})}-\varepsilon_{\theta^{\prime}(z_{t},t,\bar{c})}\right\|_{2}^{2}\right].$

In essence, this metric assesses the degree to which the model’s inherent
generative capacity is compromised throughout the customization process.

###  3.3. Analysis of Concept-specific Overfitting

Concept-specific overfitting primarily stems from the confusion learning
between customized concepts and limited training modalities. As illustrated in
Figure 5, we also employ a toy Gaussian model for clarification, where each
peak region represents different modalities within the same concept. The
original foundational model is capable of generating 25 different modalities
within a similar concept, each representing variations such as diverse
backgrounds, layouts, and styles. The customized target is also to learning a
linear distribution. Typically, customized training datasets only contain a
subset of these modalities. We represent this with the utilization of only
five Gaussian distributions. It is evident from Figure 5 that, during the
training process, the model gradually reduces the number of modalities and
ultimately converges to only include those present in the training dataset.
However, as shown Figure 5(e), our ultimate goal is to represent linear
concepts as comprehensively as possible across all modalities within the
model.

Therefore, during the learning process of T2I model, we infer that the model
confuses the target concept with other modalities such as background, layout,
style, etc., leading to concept-specific overfitting. Based on the manifold
hypothesis (He et al., 2023), each concept’s generative distribution can be
regarded as a probability manifold embedded within the entire generative
space, with the distributions under different modalities of each concept
considered as distinct sub-manifolds embedded within this manifold. To
quantify this overfitting, it becomes necessary to measure the distance
between these sub-manifolds before and after customization training. To
achieve this, we introduce the Wasserstein metric, as defined in definition
3.2.

Definition 3.2 ($2$-Wasserstein metric (Villani et al., 2009)). Let
$(\mathcal{X},d)$ be a Polish metric space. For any two probability measures
$\mu,\nu$ on $\mathcal{X}$, $\text{ law }(X)=\mu$, $\text{ law }(Y)=\nu$. The
$2$-Wasserstein distance between $\mu$ and $\nu$ is defined by the formula:

$\displaystyle W_{2}(\mu,\nu)$
$\displaystyle=\left(\inf_{\pi\in\Pi(\mu,\nu)}\int_{\mathcal{X}}d(x,y)^{2}d\pi(x,y)\right)^{1/2}$
(5)
$\displaystyle=\inf\left\\{\left[\mathbb{E}d(X,Y)^{2}\right]^{\frac{1}{2}}\right\\}.$

However, directly compute Wasserstein metric in high-dimensional space is
difficult. So in this work, inspired by Fréchet Inception Distance (Heusel et
al., 2017), we fits a Gaussian distribution to the latent of SD model for each
distribution and then computes the $2$-Wasserstein distance, between those
Gaussians.

##  4\. Method

To address the aforementioned two types of overfitting, we aim to retain the
generative capacity of the foundational T2I model and exploit its diversity of
generative modalities for customization. For instance, to generate a
customized image with the prompt “a ${\langle\text{cat}\rangle}$ is reading a
book”. An intuitive strategy is that we generate an initial image by
foundational T2I model first. During this process, we can apply a
transformation to replace the depicted “cat” with the desired customized
“${\langle\text{cat}\rangle}$”. Note that the use of “${\langle\rangle}$” is
merely for written distinction, and we do not undertake additional learning of
unique identifiers. As shown in Figure 3, the “cat” was selectively replaced
with a customized concept, while preserving the constancy of other elements.
In this way, we retain the T2I model’s generative capacity, avoid concept-
agnostic overfitting, and simultaneously harness its modality diversity to
mitigate the risk of concept-specific overfitting.

###  4.1. Cross-attention Affects T2I Generation

Following (Tewel et al., 2023), we can partition the cross-attention module
into two pathways: the “where” pathway for the position and geometric
attributes of objects in the final image, and the “what” pathway dictating the
features incorporated into each spatial region, thus controls the content
appearing in the final image. Specifically, expanding the expression denoted
by Equation 2, we derive the following:

$\displaystyle\mathit{Atten}(Q_{i},K_{i},V_{i})$
$\displaystyle=\left(m^{i}_{1},m^{i}_{2},\cdots,m^{i}_{L}\right)((v^{i}_{1})^{\top},(v^{i}_{2})^{\top},\cdots,(v^{i}_{L})^{\top})^{\top}$
(6) $\displaystyle=\sum_{k=1}^{L}m^{i}_{k}\cdot v^{i}_{k}.$

Here, $m^{i}_{k}\in\mathbb{R}^{N_{i}^{2}\times 1}$ denotes the attention map
corresponding to the $k$-th token of the input prompt and
$v^{i}_{k}\in\mathbb{R}^{1\times d}$ represents the value feature associated
with the same token. As shown in Figure 3, there is usually minimal overlap
between attention maps of the foundational T2I model. Various methods (Chefer
et al., 2023; Tunanyan et al., 2023) exist can be directly combined with
Infusion to further prevent their overlap, here we just consider a simplified
scenario. Thus, in conjunction with Equation 6, we observe that independent
attention maps determine the positional presentation of their corresponding
tokens in the image. The content they present is determined by the respective
values.

Under the condition where the conceptualized generation content remains fixed,
the augmentation of spatial layout and posture in the generated entities is
imperative to amplify the modality diversity of customized generation.
Furthermore, we posit that the foundational T2I model, untainted by customized
training, inherently encapsulates the modality spatial diversity. To fully
harness the diversity, our insight is that its attention maps should be
decoupled for customized generation.

It is somewhat akin to the mechanisms employed in P2P, we extract attention
maps from the foundational T2I model’s generation under identical prompts,
utilizing them to replace the attention maps in the customized generation
process. However, P2P is constrained to the substitution of concepts
exclusively within the generative scope of the model.

###  4.2. Concept Learning with Residual Embedding

The essence of Infusion lies in preserving the cross-attention maps of the
foundational T2I model and subsequently learning and infusing concepts during
each time step $t$ along the “what” pathway.

For example, consider the training image $I$ and its corresponding text
description $y$ (“a photo of a cat”). We feed them into both the foundational
T2I pipeline (F-pipeline) and the customized T2I pipeline (C-pipeline). These
two pipelines share fixed foundational model parameters. Differently, to
facilitate the learning of customized concepts, a trainable embedding is
introduced in the “what” pathway of the C-pipeline. Specifically, a residual
embedding $\Delta_{i}^{\text{cat}}$ is incorporated into the value feature of
the “cat” token. This embedding represents the customized concept and fulfills
the transformation from the original category to a specific object.
Consequently, the attention module within the C-pipeline can be structured as
follows:

$\displaystyle\mathit{\overline{Atten}}(Q_{i},K_{i},\overline{V_{i}})$
$\displaystyle=\sum_{k\neq\text{obj}}m^{i}_{k}\cdot\overline{v^{i}_{k}}+m^{i}_{\text{obj}}\cdot\overline{v^{i}_{\text{obj}}}$
(7)
$\displaystyle=\sum_{k\neq\text{obj}}m^{i}_{k}\cdot{v^{i}_{k}}+m^{i}_{\text{obj}}\cdot({v^{i}_{\text{obj}}}+\Delta^{i}_{\langle\text{obj}\rangle}).$

The subscript $obj$ denotes the index of the concept token in the prompt.
Vectors with “$-$” denote those from the C-pipeline, while vectors without
“$-$” originate from the F-pipeline. It is noteworthy that
$\overline{v^{i}_{k}}=v^{i}_{k}$ when $k\neq\text{obj}$, or in other words,
$\overline{V}$ differs from $V$ only in the value feature corresponding to the
token $\langle\text{obj}\rangle$.

Finally, the trainable parameters $\Delta^{i}_{\langle\text{obj}\rangle}$ are
optimized with the customization loss without additional regularization:

(8)
$\mathcal{L}=\mathbb{E}_{z\sim\mathcal{E}(x),y,\varepsilon\sim\mathcal{N}(0,1),t}\left[\left\|\varepsilon-\varepsilon_{\theta^{{}^{\prime}}}\left(z_{t},t,\tau(y)\right)\right\|_{2}^{2}\right],$

where $x$ is the reference few-shot data of $\langle\text{obj}\rangle$,
$\theta^{{}^{\prime}}$ is the parameters of the customized model.

###  4.3. Single-concept and Multi-concept Inference

Similarly, during the inference stage, attention maps from the F-pipeline are
employed to replace the attention maps within each cross-attention module in
the C-pipeline. The previously obtained residual value embedding can be
effortlessly applied to value features of the corresponding concept token. For
a prompt involving $S$ specific concepts
$\langle\text{obj}_{1}\rangle,\langle\text{obj}_{2}\rangle,\cdots,\langle\text{obj}_{S}\rangle$,
it is sufficient to retrieve the pre-computed
$\Delta^{i}_{\langle\text{obj}_{1}\rangle},\Delta^{i}_{\langle\text{obj}_{2}\rangle},\cdots,\Delta^{i}_{\langle\text{obj}_{S}\rangle}$
and integrate them into the value embedding of the $i$-th layer, as follows:

(9)
$\displaystyle\overline{v^{i}_{\text{obj}_{s}}}=v^{i}_{\text{obj}_{s}}+\Delta^{i}_{\langle\text{obj}_{s}\rangle}.$

Combining Equation 7 with Equation 9, we can finally generate single-concept
or multi-concept customized images.

##  5\. Experiments

###  5.1. Experiment Details

Figure 7. Single-concept generation. The visual comparison involves multiple methods, with Infusion demonstrating robust customization capabilities to align textually and conceptually. Figure 8. Multi-concept generation. We show the results of multi-concept generation. Infusion demonstrates superior performance in generating works that exhibit higher levels of imagination and fidelity to customized concepts compared to Custom Diffusion Table 1. Quantitative Results. We compared different baseline methods across various metrics, and Infusion achieved the highest text alignment score. Method | CLIP-T $\uparrow$ | CLIP-I $\uparrow$ | DINO $\uparrow$ | Storage
---|---|---|---|---
DreamBooth (Ruiz et al., 2023) | 0.255 | $\mathbf{0.838}$ | $\mathbf{0.721}$ | 3.3 GB
DreamBooth-lora (von Platen et al., [n. d.]) | 0.264 | 0.791 | 0.615 | 3.29 MB
Textual Inversion (Gal et al., 2022a) | 0.240 | 0.780 | 0.584 |  $\mathbf{3}$KB
Custom Diffusion (Kumari et al., 2023) | 0.256 | 0.800 | 0.643 | 57.1MB
Perfusion (Tewel et al., 2023) | 0.242 | 0.729 | 0.505 | 100KB
Infusion (Ours) | $\mathbf{0.267}$ | 0.816 | 0.697 | 11KB

Experiment Setup. We utilize the pre-trained StableDiffusion (SD)-v1.5 model
(SD, 2022) for ablation studies and comparisons. In each layer of the Unet’s
cross-attention, we train the residual concept
$\Delta_{\langle\text{obj}\rangle}\in\mathbb{R}^{1\times d}$ for the
corresponding value feature, where $d$ = 768. The optimization of these
embeddings is carried out with a learning rate of 0.01 and a batch size of 4.
During the inference process, we employ a DDIM sampler with a sampling step
size of $T=50$, and leverage classifier-free guidance with a guiding scale of
$s=8$. We utilize datasets from prior works (Ruiz et al., 2023; Kumari et al.,
2023), encompassing diverse subjects such as toys, animals, buildings, and
personal items.

Baseline. We benchmark our method against state-of-the-art competitors,
including Textual Inversion, DreamBooth, DreamBooth-lora, Custom Diffusion,
and Perfusion. For Textual Inversion and DreamBooth(lora), we utilize their
Diffusers versions (von Platen et al., [n. d.]), and for Custom Diffusion and
Perfusion, we employ their official implementations with experimental
parameters configured following official recommendations.

Overfitting Evaluation. Referring to Section 3, we employ the Latent Fisher
divergence and Wasserstein metric to assess the concept-agnostic overfitting
and concept-specific overfitting on the customized model. Latent Fisher
divergence is computed based on a random sample of 50000 examples from
LAION-400M (Schuhmann et al., 2021). We sample time steps uniformly and
measure the distance between the outputs of $\epsilon_{\theta}$ and
$\epsilon_{\theta^{\prime}}$. To evaluate concept-specific overfitting, we
generate 1000 samples under 8 concepts based on 20 prompts and calculate the
2-Wasserstein distance between $p_{\theta}(z|x,c)$ and
$p_{\theta^{\prime}}(z|x,c)$.

Customization Evaluation. Following previous works (Ruiz et al., 2023; Wei et
al., 2023), we employ CLIP-I, CLIP-T, and DINO-I to evaluate the model ability
for customized image generation. CLIP-T is to assess the alignment between
generated images and prompts. CLIP-I evaluates the similarity between
generated samples and training data in the CLIP feature space. DINO-I computes
the cosine similarity between the ViTS/16 DINO (Caron et al., 2021) embeddings
of training images and generated images.

Figure 9. Wasserstein metric of different methods at various training steps.
Figure 10. Latent Fisher Divergence of different methods at various training
steps.

###  5.2. Overfitting Comparison

To ensure fair comparison of the anti-overfitting capabilities among different
customized models, we adjusted the learning rate and batch size to achieve
near-optimal performance for each method at the same number of training steps.
Additionally, we continued training each model after convergence to assess the
extent of overfitting. As depicted in Figures 9 and Figures 10, Infusion
demonstrates robust resistance to concept overfitting compared to other
methods. Remarkably, even with repeated training on limited data, the model
maintains excellent fidelity to textual consistency, transcending the modality
distribution of the training data. In contrast, DB(Dreambooth)-lora,
CD(Customized Diffusion), and TI(Textual Inversion) exhibit increasing
susceptibility to concept-specific overfitting with continued training,
becoming increasingly confined to the training modality, and losing control
over textual consistency. Perfusion performs better in this regard, but its
consistency with customized concepts is notably inferior to Infusion.
Moreover, since Infusion can directly generate non-customized concepts through
the F-pipeline, it remains unaffected by concept-agnostic overfitting.

Figure 11. Customized generation results of different methods at various
training steps.

As illustrated in Figure 11, we present the generated results of Infusion at
training steps 100, 200, 400, 1000 and 2000. It is evident that even at higher
training steps, Infusion keeps producing content highly aligned with the given
text. This sets it apart from other methods that require precise control over
training steps to prevent overfitting.

###  5.3. Qualitative Results

Figure 7 illustrates a series of generated images produced by our approach and
its competitors, including Stable Diffusion (SD), for single-concept
customization. To rigorously assess the models’ generative capabilities, we
deliberately selected a set of imaginative descriptive prompts. Notably,
DreamBooth and Textual Inversion exhibit pronounced susceptibility to
overfitting, emphasizing conceptual consistency limits their imagination.
Perfusion, on the other hand, demonstrates commendable imaginative capacities,
yet tends to overlook the generation of customized concepts. Custom Diffusion,
while achieving performance comparable to our method, still lacks some
diversity.

In contrast, Infusion adeptly balances textual expression and concept
fidelity. Furthermore, when compared with the generated results of SD, it
becomes evident how our method leverages the “where” pathway and “what”
pathway to infuse concepts. Our approach effectively preserves the SD’s
generated background and layout while injecting customized concepts onto the
corresponding objects. This strategy allows us to harness the imaginative
capacity of SD for customization and avoid overfitting.

Figure 8 showcases the performance of our method in multi-concept generation.
In this context, we only present the results of Infusion and Custom Diffusion,
as the performance of other competitors in this task significantly lags behind
these two. We observe that Custom Diffusion tends to overlook some concepts
during multi-concept customization. In comparison, our method excels in
crafting works that are not only more imaginative but also more faithful to
the customized concepts.

###  5.4. Quantitative Results

Our quantitative analysis primarily focuses on general objects customization.
Utilizing 20 concept dataset from (Ruiz et al., 2023; Kumari et al., 2023) and
the same 25 prompts as in (Ruiz et al., 2023), we generated 5 images for each
prompt. Table 1 demonstrates our method’s superiority in text alignment. While
it may not surpass Dreambooth and Custom Diffusion in image alignment and DINO
score, we attribute this discrepancy to inherent limitations in the metrics.
These metrics consider background information similarity during computation,
which may not align with the goals of our customized generation approach. The
simplicity of testing prompts also limits a comprehensive assessment, future
work will include more intricate test texts to evaluate the model’s generative
capabilities thoroughly.

Table 2. Quantitative Results. In each paired comparison, Infusion is preferred (over $50\%$) over the baseline methods in both text- and image-alignment. vs Method | Text Alignment $\uparrow$ | Image Alignment $\uparrow$
---|---|---
DreamBooth | $86.90\%$ | $51.19\%$
Textual Inversion | $82.14\%$ | $82.14\%$
Custom Diffusion | $69.05\%$ | $75.00\%$
Perfusion | $65.48\%$ | $84.52\%$

User Study. We evaluate the average preferences of 21 participants for
Infusion and other baseline methods. Each participant responds to 32 questions
about quality comparison. Table 2 indicates users’ general preference for
Infusion in terms of text alignment and conceptual fidelity.

##  6\. Conclusion and Limitations

In this work, we analyze two types of overfitting and introduce “Latent Fisher
divergence” and “Wasserstein metric” to quantify them. Our proposed Infusion
preserves the inherent generative capacity of the original T2I model while
offering flexible, plug-and-play usage for customization. Infusion
demonstrates excellent performance in overall learning of target concepts.
However, for tasks that require high-fidelity preservation of detailed
textures, Infusion exhibits certain limitations. In such cases, there is a
trade-off between diversity and fidelity, necessitating the sacrifice of some
diversity in exchange for higher fidelity, such as training more residual
embeddings.

## References

  * (1)
  * SD (2022) 2022\.  Stable Diffusion.  https://huggingface.co/runwayml/stable-diffusion-v1-5
  * Arar et al. (2023) Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H. Bermano. 2023.  Domain-agnostic tuning-encoder for fast personalization of text-to-image models. In _SIGGRAPH Asia 2023 Conference Papers_. 1–10.
  * Avrahami et al. (2023) Omri Avrahami, Ohad Fried, and Dani Lischinski. 2023.  Blended latent diffusion.  _ACM Transactions on Graphics (TOG)_ 42, 4 (2023), 1–11.
  * Avrahami et al. (2022) Omri Avrahami, Dani Lischinski, and Ohad Fried. 2022.  Blended diffusion for text-driven editing of natural images. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 18208–18218.
  * Bar-Tal et al. (2022) Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. 2022.  Text2live: Text-driven layered image and video editing. In _European conference on computer vision_. Springer, 707–723.
  * Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan. 2018.  Large scale GAN training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ (2018).
  * Brooks et al. (2023) Tim Brooks, Aleksander Holynski, and Alexei A Efros. 2023.  Instructpix2pix: Learning to follow image editing instructions. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 18392–18402.
  * Caron et al. (2021) Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021.  Emerging properties in self-supervised vision transformers. In _Proceedings of the IEEE/CVF international conference on computer vision_. 9650–9660.
  * Chefer et al. (2023) Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. 2023.  Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.  _ACM Transactions on Graphics (TOG)_ 42, 4 (2023), 1–10.
  * Chen et al. (2023) Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Rui, Xuhui Jia, Ming-Wei Chang, and William W Cohen. 2023.  Subject-driven text-to-image generation via apprenticeship learning.  _arXiv preprint arXiv:2304.00186_ (2023).
  * Gal et al. (2022a) Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2022a.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ (2022).
  * Gal et al. (2023) Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2023.  Encoder-based domain tuning for fast personalization of text-to-image models.  _ACM Transactions on Graphics (TOG)_ 42, 4 (2023), 1–13.
  * Gal et al. (2022b) Rinon Gal, Or Patashnik, Haggai Maron, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. 2022b.  StyleGAN-NADA: CLIP-guided domain adaptation of image generators.  _ACM Transactions on Graphics (TOG)_ 41, 4 (2022), 1–13.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014.  Generative adversarial nets.  _Advances in neural information processing systems_ 27 (2014).
  * Han et al. (2023) Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. 2023.  Svdiff: Compact parameter space for diffusion fine-tuning.  _arXiv preprint arXiv:2303.11305_ (2023).
  * He et al. (2023) Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, J Zico Kolter, Ruslan Salakhutdinov, et al. 2023\.  Manifold preserving guided diffusion.  _arXiv preprint arXiv:2311.16424_ (2023).
  * Hertz et al. (2022) Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2022.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ (2022).
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ 30 (2017).
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ 33 (2020), 6840–6851.
  * Ho and Salimans (2022) Jonathan Ho and Tim Salimans. 2022.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ (2022).
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ (2021).
  * Karras et al. (2021) Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2021.  Alias-free generative adversarial networks.  _Advances in Neural Information Processing Systems_ 34 (2021), 852–863.
  * Kawar et al. (2023) Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. 2023.  Imagic: Text-based real image editing with diffusion models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 6007–6017.
  * Kumari et al. (2023) Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. 2023.  Multi-concept customization of text-to-image diffusion. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 1931–1941.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022.  Locating and editing factual associations in GPT.  _Advances in Neural Information Processing Systems_ 35 (2022), 17359–17372.
  * Mokady et al. (2023) Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2023.  Null-text inversion for editing real images using guided diffusion models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 6038–6047.
  * Nichol et al. (2021) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2021.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ (2021).
  * Patashnik et al. (2021) Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. 2021.  Styleclip: Text-driven manipulation of stylegan imagery. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2085–2094.
  * Podell et al. (2023) Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. 2023.  Sdxl: Improving latent diffusion models for high-resolution image synthesis.  _arXiv preprint arXiv:2307.01952_ (2023).
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International conference on machine learning_. PMLR, 8748–8763.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ 1, 2 (2022), 3.
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021.  Zero-shot text-to-image generation. In _International Conference on Machine Learning_. PMLR, 8821–8831.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 10684–10695.
  * Ruiz et al. (2023) Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2023.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 22500–22510.
  * Saharia et al. ([n. d.]) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. [n. d.].  Photorealistic text-to-image diffusion models with deep language understanding, 2022.  _URL https://arxiv. org/abs/2205.11487_ 4 ([n. d.]).
  * Saharia et al. (2022) Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. 2022.  Image super-resolution via iterative refinement.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 45, 4 (2022), 4713–4726.
  * Schuhmann et al. (2021) Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ (2021).
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. 2015.  Deep unsupervised learning using nonequilibrium thermodynamics. In _International conference on machine learning_. PMLR, 2256–2265.
  * Song et al. (2020) Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020.  Denoising diffusion implicit models.  _arXiv preprint arXiv:2010.02502_ (2020).
  * Song and Ermon (2019) Yang Song and Stefano Ermon. 2019.  Generative modeling by estimating gradients of the data distribution.  _Advances in neural information processing systems_ 32 (2019).
  * Tao et al. (2022) Ming Tao, Hao Tang, Fei Wu, Xiao-Yuan Jing, Bing-Kun Bao, and Changsheng Xu. 2022.  Df-gan: A simple and effective baseline for text-to-image synthesis. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 16515–16525.
  * Tewel et al. (2023) Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. 2023.  Key-locked rank one editing for text-to-image personalization. In _ACM SIGGRAPH 2023 Conference Proceedings_. 1–11.
  * Tunanyan et al. (2023) Hazarapet Tunanyan, Dejia Xu, Shant Navasardyan, Zhangyang Wang, and Humphrey Shi. 2023.  Multi-Concept T2I-Zero: Tweaking Only The Text Embeddings and Nothing Else.  _arXiv preprint arXiv:2310.07419_ (2023).
  * Villani et al. (2009) Cédric Villani et al. 2009\.  _Optimal transport: old and new_. Vol. 338.  Springer.
  * Vincent (2011) Pascal Vincent. 2011.  A connection between score matching and denoising autoencoders.  _Neural computation_ 23, 7 (2011), 1661–1674.
  * von Platen et al. ([n. d.]) Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. [n. d.].  _Diffusers: State-of-the-art diffusion models_.  https://github.com/huggingface/diffusers
  * Voynov et al. (2023) Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. 2023.  $P+$: Extended Textual Conditioning in Text-to-Image Generation.  _arXiv preprint arXiv:2303.09522_ (2023).
  * Wei et al. (2023) Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. 2023.  Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation.  _arXiv preprint arXiv:2302.13848_ (2023).
  * Ye et al. (2023) Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. 2023.  Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.  _arXiv preprint arXiv:2308.06721_ (2023).
  * Yuan et al. (2021) Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. 2021.  Multimodal contrastive training for visual representation learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 6995–7004.
  * Zhang et al. (2021) Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. 2021.  Cross-modal contrastive learning for text-to-image generation. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 833–842.
  * Zhang et al. (2023) Zhixing Zhang, Ligong Han, Arnab Ghosh, Dimitris N Metaxas, and Jian Ren. 2023.  Sine: Single image editing with text-to-image diffusion models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 6027–6037.
  * Zhou et al. (2023) Yufan Zhou, Ruiyi Zhang, Tong Sun, and Jinhui Xu. 2023.  Enhancing Detail Preservation for Customized Text-to-Image Generation: A Regularization-Free Approach.  _arXiv preprint arXiv:2305.13579_ (2023).
  * Zhu et al. (2019) Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. 2019.  Dm-gan: Dynamic memory generative adversarial networks for text-to-image synthesis. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 5802–5810.