# Comparing entanglement and identity effects in tunnelling

Pedro Sancho
Centro de Láseres Pulsados CLPU
Parque Científico, 37085 Villamayor, Salamanca, Spain

###### Abstract

The probability of simultaneous tunnelling of two particles is modified when
the system is in a non-separable state, either entangled or symmetrised. We
compare both effects in the rectangular potential barrier by evaluating the
transmission rates in superposition states. For large momenta, their
simultaneous presence greatly changes the form of the transmission rates. The
joint effects are much larger than the superposition ones. Moreover, there are
significant differences between bosons and fermions. We present an unified
view of the combined effects as a quantum interference phenomenon. The
analysis also illustrates a novel aspect of exclusion in entangled systems,
the existence of superposition states one of whose terms is forbidden by
Pauli’s principle.

##  1 Introduction.

The relation between the two types of non-separability in quantum mechanics,
entanglement and identity, is a subject of increasing interest. On the one
hand, the problem of the definition of entanglement measures for identical
particles has not been completely settled yet [1]. On the other hand, several
schemes have been proposed to convert the non-separability of systems of
identical particles into useful entanglement [2, 3, 4].

A related topic is the comparison of both effects and the interplay between
them when they act at once. In [5, 6] light absorption by identical atoms in
multi-particle superposition states was studied, showing a rich variety of
novel behaviours, such as the inhibition of the dependence on overlapping for
fermions for some states or the cancellation of the entanglement effects for
some values of the parameters. In general, sharp differences between bosons
and fermions were observed.

In this paper we want to go deeper into the problem by studying another
arrangement that can be treated in a first approximation in an analytical way,
the tunnelling of a barrier by pairs of identical particles. We evaluate the
probability of simultaneous tunnelling of the two particles in the archetypal
rectangular potential barrier [7, 8]. The simultaneous transmission rates show
large deviations with respect to those of product states and distinguishable
particles.

The role of entanglement in tunnelling processes has been previously
considered in the literature, specially in the context of Bose-Einstein
condensation and of multi-boson systems in optical lattices. For instance, in
[9] the correlated tunnelling in condensates of two interacting species leads
to entanglement between two macroscopic systems. Similarly, in [10] the
entanglement dynamics in macroscopic tunnelling of condensates has been
analyzed in detail. However, our focus here is completely different, we are
only interested in two-particle systems. Some aspects of our approach are also
connected with other studies in the literature. For instance, in the Hong-Ou-
Mandel arrangement [11], the rate of simultaneous detection in the exit paths
depends on the degree of entanglement between the photons impinging on the
beam splitter [12]. Also, in a related context, interferometry in multi-
particle superposition states have been extensively studied [13].

The comparison of the detection patterns of superposition and mixed states in
multi-particle two-slit interference provides a method to analyze multi-
particle coherence in the problem [14]. The comparison can also be used to
study the role of multi-particle coherence in tunnelling processes. As for
pure states of distinguishable particles, entanglement is equivalent to
coherent multi-particle superposition, this is a good characterization of
entanglement effects. On the other hand, for identical particles, it provides
a measure of the importance of multi-particle superposition in the presence of
exchange effects.

Another manifestation of the interplay between multi-particle superposition
and exchange effects is the existence of excluded states not related to
Pauli’s principle [5, 6]. We shall find here another aspect of exclusion
associated with the presence of entanglement. There are states forbidden by
Pauli’s principle [15] that, however, can be present in a superposition. This
is a multi-particle coherence effect. It highlights in a drastic way how
entanglement can modify the behaviour of multi-fermion systems.

##  2 The arrangement.

Two particles, distinguishable or identical, impinge simultaneously on a
rectangular potential barrier of width $d$ and height $V_{0}$. The energy of
the two particles is less than $V_{o}$. Because of the tunnel effect the
particles have a non-zero probability of penetrating the barrier and to be
detected on the classically forbidden region behind the obstacle. We want
evaluate the probability of simultaneous tunnelling of the two particles.

In order to analyze this probability in the different cases signaled in the
Introduction we must consider different pure and mixed initial states of
distinguishable and identical particles. In the distinguishable case we deal
with the superposition state

$|\Psi>=N(a|\psi>_{A}|\phi>_{B}+b|\varphi>_{A}|\chi>_{B})$ (1)

and the mixed one represented by the density matrix

$\rho_{\Psi}=|a|^{2}|\Psi_{a}><\Psi_{a}|+|b|^{2}|\Psi_{b}><\Psi_{b}|$ (2)

with

$|\Psi_{a}>=|\psi>_{A}|\phi>_{B}\;,\;|\Psi_{b}>=|\varphi>_{A}|\chi>_{B}$ (3)

the two product states entering the superposition. In these expressions the
subscripts $A$ and $B$ denote the two particles, and $a$ and $b$ are the
coefficients of the superposition, obeying the relation $|a|^{2}+|b|^{2}=1$.
The four one-particle states are normalized, and the normalization coefficient
of the superposition is

$N=(1+2Re(a^{*}b<\psi|\varphi><\phi|\chi>))^{-1/2}$ (4)

Note that the one-particle states of $A$ and $B$ must be non-orthogonal in
order to impinge on the same potential barrier.

For identical particles we have

$|\Phi>={\cal
N}\\{a(|\psi>_{1}|\phi>_{2}\pm|\phi>_{1}|\psi>_{2})+b(|\varphi>_{1}|\chi>_{2}\pm|\chi>_{1}|\varphi>_{2})\\}$
(5)

for the superposition, and

$\rho_{\Phi}=|a|^{2}|\Phi_{a}><\Phi_{a}|+|b|^{2}|\Phi_{b}><\Phi_{b}|$ (6)

for the mixture, with

$|\Phi_{a}>={\cal N}_{a}(|\psi>_{1}|\phi>_{2}\pm|\phi>_{1}|\psi>_{2})$ (7)

and

$|\Phi_{b}>={\cal N}_{b}(|\varphi>_{1}|\chi>_{2}\pm|\chi>_{1}|\varphi>_{2})$
(8)

The subscripts have been substituted by $1$ and $2$ to emphasize that now we
are dealing with non-distinguishable particles. In the double sign expressions
$\pm$, the upper one holds for bosons and the lower one for fermions. The
normalization factors are

${\cal N}_{a}=(2\pm 2|<\psi|\phi>|^{2})^{-1/2}\;,\;{\cal N}_{b}=(2\pm
2|<\varphi|\chi>|^{2})^{-1/2}$ (9)

and

$\displaystyle{\cal N}^{-2}=|a|^{2}{\cal N}_{a}^{-2}+|b|^{2}{\cal
N}_{b}^{-2}+$ $\displaystyle
4Re(a^{*}b\\{<\psi|\varphi><\phi|\chi>\pm<\psi|\chi><\phi|\varphi>\\})$ (10)

In order to have exchange effects some of the one-particle states of the two
particles must overlap, that is, $<\psi|\phi>\neq 0$, $<\psi|\chi>\neq 0$,
$<\varphi|\chi>\neq 0$ or $<\varphi|\phi>\neq 0$.

The initial one-particle wave functions are multi-mode packets sharply picked
around a central value [16]:

$\psi(x,t)=(2\pi\hbar)^{-1/2}\int dp_{*}f(p_{*})\exp(i(p_{*}x-Et)/\hbar)$ (11)

with the Gaussian mode distribution given by

$f(p_{*})=\frac{2^{1/4}}{\pi^{1/4}P^{1/2}}\exp(-(p_{*}-p)^{2}/P^{2})$ (12)

with $P$ determining the width of the distribution, $p_{*}$ the momentum
variable, $p$ the central value, $E$ the energy, and $x$ the spatial
coordinate of the one-dimensional arrangement. $f(p_{*})$ is the wave function
in the momentum representation and, consequently, it is normalized to unity.

##  3 Evolution and probabilities.

After interacting with the potential barrier the evolution of the one-particle
states is

$|\psi>\rightarrow R|\psi_{R}>+T|\psi_{T}>$ (13)

where $R$ and $T$ are the reflection and transmission coefficients. Similarly,
the subscripts denote the reflected and transmitted states.

Using Eq. (13) and the similar expressions for the other one-particle states,
the final state of the superposition of distinguishable particles is

$|\Psi_{f}>=N(aT_{A}(p)T_{B}(q)|\psi_{T}>_{A}|\phi_{T}>_{B}+bT_{A}(\bar{p})T_{B}(\bar{q})|\varphi_{T}>_{A}|\chi_{T}>_{B})+\cdots$
(14)

$T_{A}(p)$ denotes the transmission coefficient for particle $A$ with central
momentum $p$. The rest of terms in $\Psi_{f}$ contain reflected states. As we
are only interested into double transmission its explicit form is irrelevant
here and is not included in the above equation.

All the contributions to double absorption are contained in the explicitly
written term of the r. h. s. of the above equation. Consequently, the
probability amplitude of double absorption is given by the matrix element of
the state corresponding to that normalized term with the final state, that is,
$<2|\Psi_{f}>$ with

$|2>=N_{T}(aT_{A}(p)T_{B}(q)|\psi_{T}>_{A}|\phi_{T}>_{B}+bT_{A}(\bar{p})T_{B}(\bar{q})|\varphi_{T}>_{A}|\chi_{T}>_{B})$
(15)

and

$\displaystyle
N_{T}^{-2}=|aT_{A}(p)T_{B}(q)|^{2}+|bT_{A}(\bar{p})T_{B}(\bar{q})|^{2}+$
$\displaystyle
2Re(a^{*}bT_{A}^{*}(p)T_{B}^{*}(q)T_{A}(\bar{p})T_{B}(\bar{q})<\psi_{T}|\varphi_{T}><\phi_{T}|\chi_{T}>)$
(16)

The addition of the two contributions in $|2>$ can also be understood from the
point of view of the physical alternatives for double transmission. The two
alternatives are $|\psi>|\phi>\rightarrow|\psi_{T}>|\phi_{T}>$ and
$|\varphi>|\chi>\rightarrow|\varphi_{T}>|\chi_{T}>$. As these alternatives are
indistinguishable their probability amplitudes add (with the adequate weights,
as done in [5]). This is equivalent to evaluate the total probability
amplitude with the state $|2>$.

Then, the probability of double transmission for the superposition of
distinguishable particles is

$P_{dis}^{sup}=|<2|\Psi_{f}>|^{2}=|<2|N_{T}^{-1}N|2>|^{2}=N^{2}/N_{T}^{2}$
(17)

Similarly, for the initial product states $\Psi_{a}$ and $\Psi_{b}$ we have

$P_{dis}^{a}=|T_{A}(p)T_{B}(q)|^{2}\;,\;P_{dis}^{b}=|T_{A}(\bar{p})T_{B}(\bar{q})|^{2}$
(18)

and for the mixture

$P_{dis}^{mix}=|a|^{2}|T_{A}(p)T_{B}(q)|^{2}+|b|^{2}|T_{A}(\bar{p})T_{B}(\bar{q})|^{2}$
(19)

In the case of identical particles we consider first the non superposed
states. For $\Phi_{a}$ the interaction with the barrier leads to an expression
of the type $T(p)T(q){\cal
N}_{a}(|\psi_{T}>_{1}|\phi_{T}>_{2}\pm|\phi_{T}>_{1}|\psi_{T}>_{2})+\cdots$.
All the contributions to double transmission are now contained in the state
${\cal
N}_{T}^{a}(|\psi_{T}>_{1}|\phi_{T}>_{2}\pm|\phi_{T}>_{1}|\psi_{T}>_{2})$, with
${\cal N}_{T}^{a}=(2\pm 2|<\psi_{T}|\phi_{T}>|^{2})^{-1/2}$. Thus, the
probability becomes

$P_{ide}^{a}=|T(p)T(q){\cal N}_{a}/{\cal
N}_{T}^{a}|^{2}=\frac{1\pm|<\psi_{T}|\phi_{T}>|^{2}}{1\pm|<\psi|\phi>|^{2}}|T(p)T(q)|^{2}$
(20)

and a similar expression for $P_{ide}^{b}$ with obvious modifications. For the
mixture we have

$P_{ide}^{mix}=\frac{1\pm|<\psi_{T}|\phi_{T}>|^{2}}{1\pm|<\psi|\phi>|^{2}}|aT(p)T(q)|^{2}+\frac{1\pm|<\varphi_{T}|\chi_{T}>|^{2}}{1\pm|<\varphi|\chi>|^{2}}|bT(\bar{p})T(\bar{q})|^{2}$
(21)

Finally, following the same steps, the probability for a superposition of
identical particles is

$P_{ide}^{sup}={\cal N}^{2}/{\cal N}_{T}^{2}$ (22)

with

$\displaystyle{\cal N}_{T}^{-2}=|aT_{A}(p)T_{B}(q)|^{2}({\cal
N}_{T}^{a})^{-2}+|bT_{A}(\bar{p})T_{B}(\bar{q})|^{2}({\cal N}_{T}^{b})^{-2}+$
$\displaystyle
4Re(a^{*}bT_{A}^{*}(p)T_{B}^{*}(q)T_{A}(\bar{p})T_{B}(\bar{q})\times$
$\displaystyle\\{<\psi_{T}|\varphi_{T}><\phi_{T}|\chi_{T}>\pm<\psi_{T}|\chi_{T}><\phi_{T}|\varphi_{T}>\\})$
(23)

The probabilities depend on the coefficients of the superposition, the
transmission coefficients and the overlapping between the one-particles states
(both the initial and the transmitted ones). The coefficients of the
superposition and the overlapping of the initial states are free parameters of
the input states. In contrast, the transmission coefficients and the
overlapping of the transmitted states must be evaluated . We do it in the next
section.

##  4 The model.

The transmission coefficient of a single mode or plane wave is well-known and
can be found in many textbooks, for instance in [7]:

$T=\frac{2k_{0}k_{1}e^{-ik_{0}d}}{2k_{0}k_{1}\cosh(k_{1}d)+i(k_{1}^{2}-k_{0}^{2})\sinh(k_{1}d)}$
(24)

with $k_{0}=(2mE/\hbar^{2})^{1/2}$ and $k_{1}=(2m(V_{0}-E)/\hbar^{2})^{1/2}$.
The polar form of the coefficient is $T=|T|\exp(i\Omega_{T})$, with

$|T|^{-2}=1+\left(\frac{k_{1}^{2}-k_{0}^{2}}{2k_{1}k_{0}}\sinh(k_{1}d)\right)^{2}$
(25)

and

$\Omega_{T}=arctg\left(\frac{-(k_{1}^{2}-k_{0}^{2})\cos(k_{0}d)\sinh(k_{1}d)-2k_{0}k_{1}\sin(k_{0}d)\cosh(k_{1}d)}{2k_{0}k_{1}\cos(k_{0}d)\cosh(k_{1}d)-(k_{1}^{2}-k_{0}^{2})\sin(k_{0}d)\sinh(k_{1}d)}\right)$
(26)

As we have assumed multi-mode states sharply picked around the central value
we can take, in a first approximation, the $T$ of the state as the
transmission coefficient of the mode associated with the central value. A more
exact evaluation of the transmission could be obtained via a multiple
reflection expansion taking into account the multiple modes present in the
wave packet.

In order to evaluate the overlap of the transmitted wave packets we recall
that there is a time delay with respect to the incident ones, and that it is
dependent on the energy . The explicit form of this delay is

$\tau=\hbar\frac{d\Omega_{T}}{dE}$ (27)

with the derivative evaluated at the central value of the packet [8]. This
expression is usually denoted as the Wigner-Smith time [17, 18].

The time spent by the pick of the free incident packet travelling a distance
equal to the width of the barrier would be $\delta t_{d}=dm/p$. Because of the
delay the transmission time through the barrier is $\delta t=\delta
t_{d}+\tau$. We can introduce an effective momentum $p_{e}=md/\delta t$ that
represents the momentum of a virtual free packet that would cross the barrier
in the same time that the real particle.

The scalar product of the initial one-particle wave functions is
$<\psi|\varphi>=\exp(-(p-\bar{p})^{2}/2P^{2})$. As our model to pass the
barrier is a virtual packet with the same form of the initial one but with an
effective central momentum, the scalar product after the barrier will be

$<\psi_{T}|\varphi_{T}>=\exp(-(p_{e}-\bar{p}_{e})^{2}/2P^{2})$ (28)

and similar expressions for the other scalar products.

##  5 Results.

We represent graphically the probabilities of double transmission in order to
gain insight into the behaviour of the system.

Figure 1: Probability of double transmission versus the momentum $q$ in
$m.u.$, for distinguishable particles. The inner figure contains a zoom of the
comparison between the mixture and superposition curves, which only differ in
the represented interval.

In the simulations we use an units system with the defining constants $m$,
$\hbar$ and $V_{0}$, which are taken with numerical value one: $m=1$ (for the
matter of simplicity, in the case of two distinguishable particles we take
their mass equal) , $\hbar=1$ and $V_{0}=1$. This units system is a variant of
the standard atomic one with the replacements of the defining length constant
(the Bohr radius) by an energy one ($V_{0}$), and the electron mass by the
mass of the particles present in the arrangement. Equivalently, we could use a
system with a length constant, but in this case the Bohr radius should be
replaced by the width of the barrier $d$, the natural length in the problem.
In our system the length unity, $l.u.$, and the momentum one, $m.u.$, are
$l.u.=\hbar/(mV_{0})^{1/2}$ and $m.u.=(2mV_{0})^{1/2}$. In the representations
we take $p=0.95m.u.$, $\bar{p}=1.05m.u.$ and $\bar{q}=1m.u.$ for the values of
the central momenta, $P=0.05m.u.$ for the parameter of dispersion, and
$d=0.7l.u.$. On the other hand, $q$ is the independent variable of the
problem, which as we are studying the regime $E<V_{0}$ runs in the interval
$[0,\sqrt{2})$. The rest of variables can be evaluated following similar
steps. In particular, $\tau$ is a very lengthy expression that has not been
explicitly included in the paper, but that can be deduced in a simple way via
a standard derivative.

First, we consider in Fig. 1 the case of distinguishable particles, where the
effects of entanglement alone can be studied. We compare the probabilities for
product, mixed and superposition states, that is, $P_{dis}^{a}$,
$P_{dis}^{mix}$ and $P_{dis}^{sup}$. For the superposition coefficient we take
the value $a=1/\sqrt{2}$. We see that the curve of product states largely
differs from those of superposition and mixture ones. For instance, for
$q\approx 0$ the probability of double transmission for the superposition is
not null, because the second term of the superposition contributes
(${\bar{p}}$ and ${\bar{q}}$ are both different from zero). However, these
differences are not related to physical processes. They only reflect the fact
that we are comparing different states, a single multi-particle one with two-
component multi-particle ones (the two components present in the
superposition, either in a coherent or incoherent way). When inquiring about
the physical processes present in the system, the proper comparison must be
between the superposition and the mixture, containing both the two components
of the full state. The comparison is presented in the inner box in the figure.
We see in contrast that the differences are very small (for the rest of values
not represented in the box both curves are almost equal). The difference
between these two curves measures the influence of multi-particle coherence on
the double transmission, which is small. As for pure states of distinguishable
particles entanglement is the coherent superposition of multi-particle terms,
we conclude that the entanglement influence on the transmission rates is
small. From the physical point of view, the entanglement-based modifications
can be understood as the quantum interference effects between the probability
amplitudes associated with the two indistinguishable alternatives
$|\psi>_{A}|\phi>_{B}\rightarrow|\psi_{T}>_{A}|\phi_{T}>_{B}$ and
$|\varphi>_{A}|\chi>_{B}\rightarrow|\varphi_{T}>_{A}|\chi_{T}>_{B}$.

Next, we study the effects of identity alone by comparing $P_{dis}^{a}$ and
$P_{ide}^{a}$ in Fig. 2. The exchange effects only manifest when the overlap
between the two particles is large, that is, for $q$ close to $p$. For the
rest of values of $q$ the three curves are almost identical. The effects for
bosons and fermions are clearly different, being more prominent in the second
case. The Pauli principle is reflected in the figure as the interruption of
the fermion curve for the point $q=p$, where the graphical program is unable
to represent the indeterminacy $0/0$ present at that point in the probability
distribution. The deviations with respect to the distinguishable case are due
to the exchange effects, which, as it is well-known, can be interpreted as
two-particle quantum interference effects between the indistinguishable
alternatives $|\psi>_{1}|\phi>_{2}\rightarrow|\psi_{T}>_{1}|\phi_{T}>_{2}$ and
$|\phi>_{1}|\psi>_{2}\rightarrow|\phi_{T}>_{1}|\psi_{T}>_{2}$. In this
picture, entanglement and exchange effects have a common physical origin,
quantum interference effects between indistinguishable alternatives.

Figure 2: As in Fig. 1, comparing distinguishable particles, bosons and
fermions. Figure 3: As in Fig. 1, for the superposition of fermions, bosons
and distinguishable particles.

Now, we move to the scenario where superposition and identity are present at
once. We compare in Fig. 3 the double transmission for the superposition
states of distinguishable and identical particles. The transmission rates are
larger for identical particles, and for many intervals of the variable
momentum they even double the value of the distinguishable case. In the
interval of large overlap between $p$ and $q$ the symmetrisation effects are
more intense and we can observe sharper variations in the curves of identical
particles. As the curve of distinguishable particles equals those of identical
ones with the exchange interactions switched off, we conclude that the joint
effects lead to rates much larger than those associated with superposition
alone. The combined effects of superposition and symmetrisation can also be
viewed as two-particle quantum interference effects. Now, we have a larger set
of available alternatives for the double transmission, those associated with
the terms $|\psi>_{1}|\phi>_{2}$, $|\phi>_{1}|\psi>_{2}$,
$|\varphi>_{1}|\chi>_{2}$ and $|\chi>_{1}|\varphi>_{2}$ in the initial state.
The strength and form of the interference effects are different from those
found for other sets of alternatives.

Figure 4: As in Fig. 1 for $P_{ide}^{mix}$ and $P_{ide}^{sup}$ of bosons and
fermions. The inner graphic is a zoom of the fermion case, showing a sharp
image of the presence of an interruption of the curve in the case of the
mixture, a feature that is absent in the superposition.

In Fig. 4 we compare $P_{ide}^{mix}$ and $P_{ide}^{sup}$ for both bosons and
fermions in order to see that, as in the case of distinguishable particles,
the differences between the mixture and the superposition are rather small
(although not so small as in Fig. 1). Multi-particle coherence is not the
dominant factor in the joint effects, being played that role by identity. The
differences with a product state of identical particles (not represented here)
are as in Fig. 1 very large, but as discussed above, these differences are not
interesting when exploring the underlying physical mechanisms present in the
problem.

The most interesting aspect of Fig. 4 is the manifestation of fermion
exclusion. The interruption associated with the exclusion is present for
$P_{ide}^{mix}$. The important point is that for the superposition the curve
is continuous and smooth, reflecting the fact that the term that is excluded
when is considered in an isolated way, can be present in a superposition
without leading to any special effect. We shall consider this point in detail
in the Discussion.

The above figures represent the behaviour of the system for the values
detailed at the beginning of the section. Now, we discuss how the curves
change for other choices of the parameters. When the coefficient of the
superposition varies from $a=1/\sqrt{2}$ to other values, we observe a
progressive change of the curves towards the product state limits ($a=0$ and
$a=1$), but without significant changes in the above results. For the
distribution width $P$, we have that the differences between the mixture and
superposition curves (Figs. 1 and 4) and superposition ones (Fig. 3) become
very small for values of $P$ close to zero (in this limit the scalar products
present in the probabilities tend to zero), and increase when the parameter is
larger than the value used here, although with similar patterns. Note that
this parameter must be small because of the approximation of taking the
transmission coefficient equal to that of the central value of the momentum
distribution, but not too small since then the overlaps, and consequently the
exchange effects, would be negligible. The only relevant modifications occur
for low momenta values. When $p$, $\bar{p}$ or $\bar{q}$ are small the
transmission rates largely decrease. Moreover, the separation between the
distinguishable and identical curves of superposition (Fig. 3) are in general
very small, just as between superposition and mixture ones (Figs. 1 and 4).
The regime represented in the above four figures, that of large momentum
values, is the most interesting because it provides the larger manifestations
of the non-separability effects. In addition, the transmission rates increase
and are easier to detect in potential tests of the problem.

##  6 Discussion.

The graphics of the previous section illustrate the four main results derived
in the paper. The first one is that in the regime of large incident momenta,
the most interesting range of values in the problem because the non-
separability effects increase with momentum, the combined effects lead to
simultaneous transmission rates much larger than those associated with multi-
particle superposition alone. The importance of multi-particle coherence in
the problem can be analysed by comparing superposition states with mixtures of
the same component states. The differences observed in that comparison are
small, both for identical and distinguishable particles. The behaviour found
here for tunnelling is by no means universal. In the case of two-particle two-
slit interference of distinguishable systems [14], the differences between the
superposition and the mixed states are large, showing the importance of multi-
particle coherence in that interferometric arrangement.

The second result refers to the comparison of boson and fermion rates. For the
interesting range of the momenta values, the differences between the boson and
fermion cases can be as large as those between identical and distinguishable
particles. The strength of the joint effects strongly depends on the bosonic
or fermionic nature of the system.

The third result reveals a new aspect of exclusion in entangled systems. In
[5, 6] an example of excluded state not related to Pauli’s principle was
presented. Here, we have seen another manifestation of the relation between
exclusion and multi-particle superposition: states that are forbidden by
Pauli’s principle can enter into a superposition state. This again highlights
the fact that in the presence of entanglement only the two-fermion system can
be in a definite pure state, whereas there are not individual pure states of
the particles composing it. Then it is not possible to apply Pauli’s principle
to the component fermions, and there is not exclusion. In order to try an
experimental demonstration of this property, it is clear that it is impossible
to prepare the two terms and then to superpose them. Instead, we must prepare
a two-fermion superposition of allowed states and then to change one of them.
For instance, we can have a superposition of two two-electron states, one of
them with the same spatial state for both electrons but with opposite
orientations of the spin. Using electromagnetic pulses we can flip one of the
spins (the intensity of the pulse must be low enough in order the probability
of double flip to be negligible), evolving the above state to one with equal
values of the spatial and spin variables.

Finally, the fourth result relates to the understanding of the underlying
physical mechanisms present in the problem. Non-separability processes are due
to exchange and multi-particle superposition effects. We have noticed here
that it is possible to have an unified view of these effects in terms of the
quantum interference phenomenon. As it is well-known, both effects can be
described as the interference between the indistinguishable alternatives
available to the system. When both effects act at once the number of
alternatives increases leading to different strengths and forms of the
interference phenomena. Another important conceptual aspect of the problem is
the dependence of the transmission rates on the overlap of the (initial and
transmitted) one-particle states. This property is an immediate consequence of
the dependence of the rates on the scalar products of these states. It is
important to realize that there are two types of overlap. For identical
particles we have overlap between the two particles, that is, an inter-
particle phenomenon. This is a familiar property of exchange effects, which
are only relevant for a large overlay between the two particles. On the other
hand, the scalar products present in the case of distinguishable particles
represent the overlap between different states of the same particle, that is,
they are related to the degree of non-orthogonality of these states. Non-
orthogonality enters into the description of entanglement effects of
distinguishable particles. In the case of identical particles in
superposition, inter-particle ($<\psi|\phi>,<\varphi|\chi>,<\psi|\chi>$ and
$<\phi|\varphi>$) and non-orthogonality ($<\psi|\varphi>$ and $<\phi|\chi>$)
overlaps are simultaneously present.

In summary, we have found that in the regime of large momenta there are large
modifications of the simultaneous transmission rates due to the joint effects
of entanglement and identity. The modifications depart from those associated
with superposition and exchange effects alone. The comparison with other
arrangements [5, 14] reveals that the combined effects of superposition and
symmetrisation, and the influence of multi-particle coherence, do not show an
universal behaviour, but they are dependent on the particular system analysed.

## References

  * [1] F. Benatti, R. Floreanini, F. Franchini, U. Marzolino, Phys. Rep. 878 (2020) 1.
  * [2] D. Cavalcanti, L. M. Malard, F. M. Matinaga, M. O. T. Cunha, M. F. Santos, Phys. Rev. B 76 (2007) 113304.
  * [3] N. Killoran, M. Cramer, M. B. Plenio, Phys. Rev. Lett. 112 (2014) 150501.
  * [4] R. Lo Franco, G. Compagno, Phys. Rev. Lett. 120 (2018) 240403.
  * [5] P. Sancho, Ann. Phys. 421 (2020) 168264.
  * [6] P. Sancho, Eur. Phys. J. Plus 137 (2022) 136.
  * [7] D. J. Griffiths, Introduction to Quantum Mechanics, second ed., Pearson Prentice Hall, Upper Saddle River NJ, 2005.
  * [8] A. Galindo, P. Pascual, Quantum Mechanics I, second ed., Springer Science, 2012.
  * [9] H. T. Ng, C. K. Law, P. T. Leung, Phys. Rev. A 68 (2003) 013604.
  * [10] D. A. Alcala, J. A. Glick, L. D. Carr, Phys. Rev. Lett. 118 (2017) 210403.
  * [11] C. K. Hong, Z. Y. Ou, L. Mandel, Phys. Rev. Lett. 59 (1987) 2044.
  * [12] R. Loudon, The Quantum Theory of Light, Oxford University Press, Oxford, 2000.
  * [13] D. M. Greenberger, M. A. Horne, A. Zeilinger, Phys. Today 46 (1993) 22.
  * [14] P. Sancho, Eur. Phys. J. D 76 (2022) 46.
  * [15] W. Pauli, Z. Phys. 31 (1925) 765.
  * [16] C. S. Adams, M. Siegel, J. Mlynek, Phys. Rep. 240 (1994) 143.
  * [17] E. P. Wigner, Phys. Rev. 98 (1955) 145.
  * [18] F. T. Smith, Phys. Rev. 118 (1960) 349.


# Diffusion Self-Guidance for
Controllable Image Generation

Dave Epstein1,2 Allan Jabri1 Ben Poole2 Alexei A. Efros1 Aleksander
Holynski1,2
1UC Berkeley 2Google Research

###### Abstract

Large-scale generative models are capable of producing high-quality images
from detailed text descriptions. However, many aspects of an image are
difficult or impossible to convey through text. We introduce self-guidance, a
method that provides greater control over generated images by guiding the
internal representations of diffusion models. We demonstrate that properties
such as the shape, location, and appearance of objects can be extracted from
these representations and used to steer the sampling process. Self-guidance
operates similarly to standard classifier guidance, but uses signals present
in the pretrained model itself, requiring no additional models or training. We
show how a simple set of properties can be composed to perform challenging
image manipulations, such as modifying the position or size of specific
objects, merging the appearance of objects in one image with the layout of
another, composing objects from multiple images into one, and more. We also
show that self-guidance can be used for editing real images. See our project
page for results and an interactive demo: https://dave.ml/selfguidance

##  1 Introduction

Generative image models have improved rapidly in recent years with the
adoption of large text-image datasets and scalable architectures [38, 32, 27,
7, 28, 34, 12, 10]. These models are able to create realistic images given a
text prompt describing just about anything. However, despite the incredible
abilities of these systems, discovering the right prompt to generate the exact
image a user has in mind can be surprisingly challenging. A key issue is that
all desired aspects of an image must be communicated through text, even those
that are difficult or even impossible to convey precisely.

To address this limitation, previous work has introduced methods [9, 30, 14,
18] that tune pretrained models to better control details that a user has in
mind. These details are often supplied in the form of reference images along
with a new textual prompt [4, 2] or other forms of conditioning [39, 1, 31].
However, these approaches all either rely on fine-tuning with expensive paired
data (thus limiting the scope of possible edits) or must undergo a costly
optimization process to perform the few manipulations they are designed for.
While some methods [11, 36, 21, 22] can perform zero-shot editing of an input
image using a target caption describing the output, these methods only allow
for limited control, often restricted to structure-preserving appearance
manipulation or uncontrolled image-to-image translation.

‘‘a photo of a giant macaron and a croissant splashing in the Seine with the
Eiffel Tower in the background’’

(a) Original

(b) Swap objects

(c) Enlarge macaron

(d) Replace macaron

(e)

Copy scene appearance

(f) Copy scene layout

‘‘a DSLR photo of a meatball and a donut falling from the clouds onto a
neighborhood’’

(a) Original

(b) Move donut

(c) Shrink donut

(d) Replace donut

(e)

Copy scene appearance

(f) Copy scene layout

Figure 1: Self-guidance is a method for controllable image generation that
guides sampling using the attention and activations of a pretrained diffusion
model. With self-guidance, we can move or resize objects, or even replace them
with items from real images, without changing the rest of the scene (b-d). We
can also borrow the appearance of other images or rearrange scenes into new
layouts (e-f).

By consequence, many simple edits still remain out of reach. For example, how
can we move or resize one object in a scene without changing anything else?
How can we take the appearance of an object in one image and copy it over to
another, or combine the layout of one scene with the appearance of a second
one? How can we generate images with certain items having precise shapes at
specific positions on the canvas? This degree of control has been explored in
the past in smaller scale settings [8, 5, 40, 19, 24, 37], but has not been
convincingly demonstrated with modern large-scale diffusion models [32, 38,
26].

We propose self-guidance, a zero-shot approach which allows for direct control
of the shape, position, and appearance of objects in generated images. Self-
guidance leverages the rich representations learned by pretrained text-to-
image diffusion models – namely, intermediate activations and attention – to
steer attributes of entities and interactions between them. These constraints
can be user-specified or transferred from other images, and rely only on
knowledge internal to the diffusion model. Through a variety of challenging
image manipulations, we demonstrate that self-guidance using only a few simple
properties allows for granular, disentangled manipulation of the contents of
generated images (Figure 1). Further, we show that self-guidance can also be
used to reconstruct and edit real images.

Our key contributions are as follows:

  * •

We introduce self-guidance, which takes advantage of the internal
representations of pretrained text-to-image diffusion models to provide
disentangled, zero-shot control over the generative process without requiring
auxiliary models or supervision.

  * •

We find that properties such as the size, location, shape, and appearance of
objects can be extracted from these representations and used to meaningfully
guide sampling in a zero-shot manner.

  * •

We demonstrate that this small set of properties, when composed, allows for a
wide variety of surprisingly complex image manipulations, including control of
relationships between objects and the way modifiers bind to them.

  * •

Finally, by reconstructing captioned images using their layout and appearance
as computed by self-guidance, we show that we can extend our method to editing
real images.

##  2 Background

###  2.1 Diffusion generative models

Diffusion models learn to transform random noise into high-resolution images
through a sequential sampling process [33, 12, 35]. This sampling process aims
to reverse a fixed time-dependent destructive process that corrupts data by
adding noise. The learned component of a diffusion model is a neural network
$\epsilon_{\theta}$ that tries to estimate the denoised image, or equivalently
the noise $\mathbf{\epsilon}_{t}$ that was added to create the noisy image
$z_{t}=\alpha_{t}x+\sigma_{t}\mathbf{\epsilon}_{t}$. This network is trained
with loss:

$L(\theta)=\mathbb{E}_{t\sim\mathcal{U}(1,T),\mathbf{\epsilon}_{t}\sim\mathcal{N}(0,\mathbf{I})}\Big{[}w(t)||\mathbf{\epsilon}_{t}-\mathbf{\epsilon}_{\theta}(z_{t};t,y)||^{2}\Big{]},$
(1)

where $y$ is an additional conditioning signal like text, and $w(t)$ is a
function weighing the contributions of denoising tasks to the training
objective, commonly set to 1 [12, 15]. A common choice for
$\mathbf{\epsilon}_{\theta}$ is a U-Net architecture with self- and cross-
attention at multiple resolutions to attend to conditioning text in $y$ [29,
32, 28]. Diffusion models are score-based models, where
$\mathbf{\epsilon}_{\theta}$ can be seen as an estimate of the score function
for the noisy marginal distributions:
$\mathbf{\epsilon}_{\theta}(z_{t})\approx-\sigma_{t}\nabla_{z_{t}}\log
p(z_{t})$ [35].

Given a trained model, we can generate samples given conditioning $y$ by
starting from noise $z_{T}\sim\mathcal{N}(0,I)$, and then alternating between
estimating the noise component and updating the noisy image:

$\hat{\mathbf{\epsilon}}_{t}=\mathbf{\epsilon}_{\theta}(z_{t};t,y),\quad
z_{t-1}=\text{update}(z_{t},\hat{\mathbf{\epsilon}}_{t},t,t-1,\mathbf{\epsilon}_{t-1}),$
(2)

where the update could be based on DDPM [12], DDIM [34], or another sampling
method (see Appendix for details). Unfortunately, naïvely sampling from
conditional diffusion models does not produce high-quality images that
correspond well to the conditioning $y$. Instead, additional techniques are
utilized to modify the sampling process by altering the update direction
$\hat{\mathbf{\epsilon}}_{t}$.

###  2.2 Guidance

A key capability of diffusion models is the ability to adapt outputs after
training by _guiding_ the sampling process. From the score-based perspective,
we can think of guidance as composing score functions to sample from richer
distributions or to introduce conditioning on auxiliary information [7, 17,
35]. In practice, using guidance involves altering the update direction
$\hat{\mathbf{\epsilon}}_{t}$ at each iteration.

Classifier guidance can generate conditional samples from an unconditional
model by combining the unconditional score function for $p(z_{t})$ with a
classifier $p(y|z_{t})$ to generate samples from $p(z_{t}|y)\propto
p(y|z_{t})p(z_{t})$ [7, 35]. To use classifier guidance, one needs access to a
labeled dataset and has to learn a noise-dependent classifier $p(y|z_{t})$
that can be differentiated with respect to the noisy image $z_{t}$. While
sampling, we can incorporate classifier guidance by modifying
$\hat{\mathbf{\epsilon}}_{t}$:

$\hat{\mathbf{\epsilon}}_{t}=\mathbf{\epsilon}_{\theta}(z_{t};t,y)-s\sigma_{t}\nabla_{z_{t}}\log
p(y|z_{t}),$ (3)

where $s$ is an additional parameter controlling the guidance strength.
Classifier guidance moves the sampling process towards images that are more
likely according to the classifier [7], achieving a similar effect to
truncation in GANs [3], and can also be applied with pretrained classifiers by
first denoising the intermediate noisy image (though this requires additional
approximations [1]).

In general, we can use any energy function $g(z_{t};t,y)$ to guide the
diffusion sampling process, not just the probabilities from a classifier. $g$
could be the approximate energy from another model [17], a similarity score
from a CLIP model [23], an arbitrary time-independent energy as in universal
guidance [1], bounding box penalties on attention [6], or any attributes of
the noisy images. We can incorporate this additional guidance alongside
classifier-free guidance [13] to obtain high-quality text-to-image samples
that also have low energy according to $g$:

$\hat{\mathbf{\epsilon}}_{t}=(1+s)\mathbf{\epsilon}_{\theta}(z_{t};t,y)-s\mathbf{\epsilon}_{\theta}(z_{t};t,\emptyset)+v\sigma_{t}\nabla_{z_{t}}g(z_{t};t,y),$
(4)

where $s$ is the classifier-free guidance strength and $v$ is an additional
guidance weight for $g$. As with classifier guidance, we scale by $\sigma_{t}$
to convert the score function to a prediction of $\mathbf{\epsilon}_{t}$. The
main contribution of our work is to identify energy functions $g$ useful for
controlling properties of objects and interactions between them.

###  2.3 Where can we find signal for controlling diffusion?

While guidance is a flexible way of controlling the sampling process, energy
functions typically used [39, 1] require auxiliary models (adapted to be
noise-dependent) as well as data annotated with properties we would like to
control. Can we circumvent these costs?

Figure 2: Overview: We leverage representations learned by text-image
diffusion models to steer generation with self-guidance. By constraining
intermediate activations $\Psi_{t}$ and attention interactions
$\mathcal{A}_{t}$, self-guidance can control properties of entities named in
the prompt. For example, we can change the position and shape of the burger,
or copy the appearance of ice cream from a source image.

Recent work [11, 36] has shown that the intermediate outputs of the diffusion
U-Net encode valuable information [16, 25] about the structure and content of
the generated images. In particular, the self and cross-attention maps
$\left\\{\mathcal{A}_{i,t}\in\mathbb{R}^{H_{i}\times W_{i}\times K}\right\\}$
often encode structural information [11] about object position and shape,
while the network activations $\left\\{\Psi_{i,t}\in\mathbb{R}^{H_{i}\times
W_{i}\times D_{i}}\right\\}$ allow for maintaining coarse appearance [36] when
extracted from appropriate layers. While these editing approaches typically
share attention and activations naively between subsequent sampling passes,
drastically limiting the scope of possible manipulations, we ask: what if we
tried to harness model internals in a more nuanced way?

##  3 Self-guidance

Inspired by the rich representations learned by diffusion models, we propose
self-guidance, which places constraints on intermediate activations and
attention maps to steer the sampling process and control entities named in
text prompts (see Fig. 2). These constraints can be user-specified or copied
from existing images, and rely only on knowledge internal to the diffusion
model.

We identify a number of properties useful for meaningfully controlling
generated images, derived from the set of softmax-normalized attention
matrices $\left\\{\mathcal{A}_{i,t}\in\mathbb{R}^{H_{i}\times W_{i}\times
K}\right\\}$ and activations $\left\\{\Psi_{i,t}\in\mathbb{R}^{H_{i}\times
W_{i}\times D_{i}}\right\\}$ extracted from the standard denoising forward
pass $\mathbf{\epsilon}_{\theta}(z_{t};t,y)$. To control an object mentioned
in the text conditioning $y$ at token indices $k$, we can manipulate the
corresponding attention channel(s)
$\mathcal{A}_{i,t,\cdot,\cdot,k}\in\mathbb{R}^{H_{i}\times W_{i}\times|k|}$
and activations $\Psi_{i,t}$ (extracted at timestep $t$ from a noisy image
$z_{t}$ given text conditioning $y$) by adding guidance terms to Eqn. 4.

#### Object position.

To represent the position of an object (omitting attention layer index and
timestep for conciseness), we find the center of mass of each relevant
attention channel:

$\displaystyle\texttt{centroid}\left(k\right)$
$\displaystyle=\frac{1}{{\sum_{h,w}\mathcal{A}_{h,w,k}}}\begin{bmatrix}\sum_{h,w}w\cdot\mathcal{A}_{h,w,k}\\\
\sum_{h,w}h\cdot\mathcal{A}_{h,w,k}\end{bmatrix}$ (5)

We can use this property to guide an object to an absolute target position on
the image. For example, to move “burger” to position $(0.3,0.5)$, we can
minimize $\|(0.3,0.5)-\texttt{centroid}\left(k\right)\|_{1}$. We can also
perform a relative transformation, e.g., move “burger” to the right by
$(0.1,0.0)$ by minimizing
$\|\texttt{centroid}_{\text{orig}}\left(k\right)+(0.1,0.0)-\texttt{centroid}\left(k\right)\|_{1}$.

#### Object size.

To compute an object’s size, we spatially sum its corresponding attention
channel:

$\displaystyle\texttt{size}\left(k\right)$
$\displaystyle=\frac{1}{HW}\sum_{h,w}\mathcal{A}_{h,w,k}$ (6)

In practice, we find it beneficial to differentiably threshold the attention
map $\mathcal{A}_{\text{thresh}}$ before computing its size, to eliminate the
effect of background noise. We do this by taking a soft threshold at the
midpoint of the per-channel minimum and maximum values (see Appendix for
details). As with position, one can guide to an absolute size (e.g. half the
canvas) or a relative one (e.g. $10\%$ larger).

‘‘distant shot of the tokyo tower with a massive sun in the sky’’

‘‘a photo of a fluffy cat sitting on a museum bench looking at an oil painting
of cheese’’

(b) Move up (c) Move down (d) Move left (e) Move right (f) Shrink (g) Enlarge

‘‘a photo of a raccoon in a barrel going down a waterfall’’

(a) Original

Figure 3: Moving and resizing objects. By only changing the properties of one
object (as in Eqn. 9), we can move or resize that object without modifying the
rest of the image. In these examples, we modify  “massive sun” ,  “oil
painting of cheese” , and  “raccoon in a barrel” , respectively.

#### Object shape.

For even more granular control than position and size, we can represent the
object’s exact shape directly through the thresholded attention map itself:

$\displaystyle\texttt{shape}(k)=\mathcal{A}_{k}^{\text{thresh}}$ (7)

This shape can then be guided to match a specified binary mask (either
provided by a user or extracted from the attention from another image) with
$\|\texttt{target\\_shape}-\texttt{shape}\left(k\right)\|_{1}$. Note that we
can apply any arbitrary transformation (scale, rotation, translation) to this
shape before using it as a guidance target, which allows us to manipulate
objects while maintaining their silhouette.

#### Object appearance.

Considering thresholded attention a rough proxy for object extent, and spatial
activation maps as representing local appearance (since they ultimately must
be decoded into an unnoised RGB image), we can reach a notion of object-level
appearance by combining the two:

$\displaystyle\texttt{appearance}(k)=\frac{\sum_{h,w}\texttt{shape}(k)\odot{\Psi}}{\sum_{h,w}\texttt{shape}(k)}$
(8)

##  4 Composing self-guidance properties

The small set of properties introduced in Section 3 can be composed to perform
a wide range of image manipulations, including many that are intractable
through text. We showcase this collection of manipulations and, when possible,
compare to prior work that accomplishes similar effects. All experiments were
performed on Imagen [32], producing $1024\times 1024$ samples. For more
samples and details on the implementation of self-guidance, please see the
Appendix .

#### Adjusting specific properties.

By guiding one property to change and all others to keep their original
values, we can modify single objects in isolation (Fig. 3(b)-3(e)). For a
caption $C=y$ with words at indices $\\{c_{i}\\}$, in which
$O=\\{o_{j}\\}\subseteq C$ are objects, we can move an object $o_{k}$ at time
$t$ with:

$\displaystyle g$ $\displaystyle=w_{0}\overbrace{\frac{1}{|O|-1}\sum_{o\neq
o_{k}\in
O}\frac{1}{|\mathcal{A}|}\sum_{i=0}^{|\mathcal{A}|}\|\texttt{shape}_{i,t,\text{orig}}(o)-\texttt{shape}_{i,t}(o)\|_{1}}^{\text{Fix
all other object shapes}}$ (9)
$\displaystyle+w_{1}\overbrace{\frac{1}{|O|}\sum_{o\in
O}\|\texttt{appearance}_{t,\text{orig}}(o)-\texttt{appearance}_{t}(o)\|_{1}}^{\text{Fix
all appearances}}$
$\displaystyle+w_{2}\overbrace{\frac{1}{|\mathcal{A}|}\sum_{i=0}^{|\mathcal{A}|}\|\mathcal{T}\left(\texttt{shape}_{i,t,\text{orig}}(o_{k})\right)-\texttt{shape}_{i,t}(o_{k})\|_{1}}^{\text{Guide
$o_{k}$'s shape to translated original shape}}$

Where $\texttt{shape}_{\text{orig}}$ and shape are extracted from the
generation of the initial and edited image, respectively. Critically,
$\mathcal{T}$ lets us define whatever transformation of the $H_{i}\times
W_{i}$ spatial attention map we want. To move an object, $\mathcal{T}$
translates the attention mask the desired amount. We can also resize objects
(Fig. 3(f)-3(g)) with Eqn. 9 by changing $\mathcal{T}$ to up- or down-sample
shape matrices.

Constraining per-object layout but not appearance finds new “styles” for the
same scene (Fig. 4):

$\displaystyle g$ $\displaystyle=w_{0}\overbrace{\frac{1}{|O|}\sum_{o\in
O}\frac{1}{|\mathcal{A}|}\sum_{i=0}^{|\mathcal{A}|}\|\texttt{shape}_{i,t,\text{orig}}(o)-\texttt{shape}_{i,t}(o)\|_{1}}^{\text{Fix
all object shapes}}$ (10)

We can alternatively choose to guide all words, not just nouns or objects,
changing summands to $\sum_{c\neq o_{k}\in C}$ instead of $\sum_{c\neq
o_{k}\in O}$. See Appendix for further discussion.

‘‘a photo of a parrot riding a horse down a city street’’

‘‘a photo of a bear wearing a suit eating his birthday cake out of the fridge
in a dark kitchen’’

(a) Original

\begin{overpic}[width=433.62pt,height=433.62pt]{images/02/bearnewapp1.jpg}
\put(-15.0,-21.0){\small{(b) New appearances}} \end{overpic} (b)

(c)

ControlNet

[39]

(d) PtP [11]

Figure 4: Sampling new appearances. By guiding object shapes (Eqn. 7) towards
reconstruction of a given image’s layout (a), we can sample new appearances
for a given scene (b-d).

#### Composition between images.

We can compose properties across multiple images into a cohesive sample, e.g.
the layout of an image $A$ with the appearance of objects in another image $B$
(Fig. 5):

$\displaystyle g$ $\displaystyle=w_{0}\overbrace{\frac{1}{|O|}\sum_{o\in
O}\frac{1}{|\mathcal{A}|}\sum_{i=0}^{|\mathcal{A}|}\|\texttt{shape}_{i,t,A}(o)-\texttt{shape}_{i,t}(o)\|_{1}}^{\text{Copy
object shapes from A}}$ (11)
$\displaystyle+w_{1}\overbrace{\frac{1}{|O|}\sum_{o\in
O}\|\texttt{appearance}_{t,B}(o)-\texttt{appearance}_{t}(o)\|_{1}}^{\text{Copy
object appearance from B}}$

We can also borrow only appearances, dropping the first term to sample new
arrangements for the same objects, as in the last two columns of Figure 5.

Highlighting the compositionality of self-guidance terms, we can further
inherit the appearance and/or shape of objects from several images and combine
them into one (Fig. 6). Say we have $J$ images, where we are interested in
keeping a single object $o_{k_{j}}$ from each one. We can collage these
objects “in-place” – i.e. maintaining their shape, size, position, and
appearance – straightforwardly:

$\displaystyle g$
$\displaystyle=w_{0}\overbrace{\frac{1}{J}\sum_{j}\frac{1}{|\mathcal{A}|}\sum_{i=0}^{|\mathcal{A}|}\|\texttt{shape}_{i,t,j}(o_{k_{j}})-\texttt{shape}_{i,t}(o_{k})\|_{1}}^{\text{Copy
each object's shape, position, and size}}$ (12)
$\displaystyle+w_{1}\overbrace{\frac{1}{J}\sum_{j}\|\texttt{appearance}_{t,j}(o_{k_{j}})-\texttt{appearance}_{t}(o_{k})\|_{1}}^{\text{Copy
each object's appearance}}$

\begin{overpic}[width=416.27809pt,height=416.27809pt]{images/04/suitcase0.jpg}
\put(165.0,-338.0){$\leftarrow$ Layout $\rightarrow$}
\put(-20.0,-175.0){\rotatebox{90.0}{$\leftarrow$ Appearance $\rightarrow$}}
\put(85.0,110.0){\tiny{``{a photo of a suitcase, a bowling ball, and a phone
washed up on a beach after a shipwreck}''}} \end{overpic}

\begin{overpic}[width=433.62pt,height=433.62pt]{images/04/suitcaseA0_new0.jpg}
\put(25.0,-322.0){No layout conditioning} \end{overpic}

Figure 5: Mix-and-match. By guiding samples to take object shapes from one
image and appearance from another (Eqn. 11), we can rearrange images into
layouts from other scenes. Input images are along the diagonal. We can also
sample new layouts of a scene by only guiding appearance (right).

“a photo of a picnic blanket, a fruit tree, and a car by the lake”

“a top-down photo of a tea kettle, a bowl of fruit, and a cup of matcha”

“a photo of a dog wearing a knit sweater and a baseball cap drinking a
cocktail”

(a)

Take blanket

(b)

Take

tree

(c)

Take

car

(d) Result

(e)

\+ Target layout

(f) Final result

(a)

Take

matcha

(b)

Take kettle

(c)

Take fruit

(d) Result

(e)

\+ Target layout

(f) Final result

(a)

Take sweater

(b)

Take cocktail

(c)

Take

cap

(d) Result*

(e)

\+ Target layout

(f) Final result

Figure 6: Compositional generation. A new scene (d) can be created by
collaging objects from multiple images (Eqn. 12). Alternatively – e.g. if
objects cannot be combined at their original locations due to
incompatibilities in these images’ layouts (*as in the bottom row) – we can
borrow only their appearance, and specify layout with a new image (e) to
produce a composition (f) (Eqn. 19).

We can also take only the appearances of the objects from these images and
copy the layout from another image, useful if object positions in the $J$
images are not mutually compatible (Fig. 6(f)).

purple wizard outfit

\begin{overpic}[width=433.62pt,height=433.62pt]{images/07/chowchowwizard.jpg}\put(308.0,30.0){\rotatebox{90.0}{Ours}}\put(308.0,-95.0){\rotatebox{90.0}{DreamBooth}}\end{overpic}

chef outfit

superman outfit

floating in milk

pouring tea

floating in the sea

\begin{overpic}[width=433.62pt,height=433.62pt]{images/07/dreamboothsea.jpg}\put(-560.0,70.0){\framebox{\includegraphics[width=40.00006pt]{images/07/chowchoworig.jpg}}}\put(60.0,70.0){\framebox{\includegraphics[width=40.00006pt]{images/07/teapotorig.jpg}}}\end{overpic}

Figure 7: Appearance transfer from real images. By guiding the appearance of a
generated object to match that of one in a real image (outlined) as in Eqn.
13, we can create scenes depicting an object from real life, similar to
DreamBooth [30], but without any fine-tuning and only using one image.

‘‘a photo of a hot dog, fries, and a soda on a solid background’’

‘‘a photo of an eclair and a shot of espresso’’

(a) Real image

(b)

Reconstruct

(c)

Swap w. fries

(d) Width $\downarrow$

(e)

Width $\downarrow$, height $\uparrow$

(f)

Restyle

(g) Real image

(h)

Reconstruct

(i) Move

(j) Width $\downarrow$

(k)

Width, height $\uparrow$

(l)

Restyle

Figure 8: Real image editing. Our method enables the spatial manipulation of
objects (shown in Figure 3 for generated images) for _real_ images as well.

#### Editing with real images.

Our approach is not limited to only images generated by a model, whose
internals we have access to by definition. By running $T$ noised versions of a
(captioned) existing image through a denoiser – one for each forward process
timestep – we extract a set of intermediates that can be treated as if it came
from a reverse sampling process (see Appendix for more details). In Fig. 8, we
show that, by guiding shape and appearance for all tokens, we generate
faithful reconstructions of real images. More importantly, we can manipulate
these real images just as we can generated ones, successfully controlling
properties such as appearance, position, or size. We can also transfer the
appearance of an object of interest into new contexts (Fig. 7), from only one
source image, and without any fine-tuning:

$g=w_{0}\overbrace{\|\texttt{appearance}_{t,\text{orig}}(o_{k_{\text{orig}}})-\texttt{appearance}_{t}\left(o_{k}\right)\|_{1}}^{\text{Copy
object appearance}}$ (13)

#### Attributes and interactions.

So far we have focused only on the manipulation of objects, but we can apply
our method to any concept in the image, as long as it appears in the caption.
We demonstrate manipulation of verbs and adjectives in Fig. 9, and show an
example where certain self-guidance constraints can help in enforcing
attribute binding in the generation process.

\begin{overpic}[width=433.62pt,height=433.62pt]{images/09/monkey_cat_left.jpg}
\put(20.0,-15.0){\small{Move {laughing} to the right}}
\put(5.0,105.0){\tiny{``{a cat and a monkey laughing on a road}''}}
\end{overpic} (a) Original

(b) Modified

\begin{overpic}[width=433.62pt,height=433.62pt]{images/09/messyleft.jpg}
\put(40.0,-15.0){\small{Change {messy} location}} \put(45.0,105.0){\tiny{``{a
photo of a messy room}''}} \end{overpic} (c) At $\langle 0.3,0.6\rangle$

(d) At $\langle 0.8,0.8\rangle$

\begin{overpic}[width=433.62pt,height=433.62pt]{images/09/colororig.jpg}
\put(-5.0,-15.0){\small{Move {red} to jacket, {yellow} to shoes}}
\put(-15.0,105.0){\tiny{``{green hat, blue book, yellow shoes, red jacket}''}}
\end{overpic} (e) Original

(f) Fixed

Figure 9: Manipulating non-objects. The properties of any word in the input
prompt can be manipulated, not only nouns. Here, we show examples of
relocating adjectives and verbs. The last example shows a case in which
additional self-guidance can correct improper attribute binding.

\begin{overpic}[width=433.62pt,height=433.62pt]{images/10/squirrelsamples11.jpg}
\put(10.0,-15.0){\small{Appearance features leak layout}}
\put(30.0,115.0){\tiny{``{a photo of a squirrel trying}}}
\put(37.0,105.0){\tiny{{ to catch a lime mid-air}''}} \end{overpic} (a)
Unguided

(b) “lime”  guided

\begin{overpic}[width=433.62pt,height=433.62pt]{images/10/rainbowcake.jpeg}
\put(-2.0,-15.0){\small{Multi-token layout leaks appearance}}
\put(45.0,105.0){\tiny{``{a picture of a cake}''}} \end{overpic} (c) Real
image

(d) Layout guided

\begin{overpic}[width=433.62pt,height=433.62pt]{images/10/couchpotatoleft.jpg}
\put(20.0,-15.0){\small{Interacting objects entangled}}
\put(1.0,115.0){\tiny{``{a potato sitting on a couch with a bowl}}}
\put(15.0,105.0){\tiny{{of popcorn watching football on TV}''}} \end{overpic}
(e) Original

(f)

Move potato $\rightarrow$

Figure 10: Limitations. Setting high guidance weights for appearance terms
tends to introduce unwanted leakage of object position (a-b). Similarly, while
heavily guiding the shape of one word simply matches that object’s layout as
expected, high guidance on the shapes of all tokens results in a leak of
appearance information (c-d). Finally, in some cases, objects are entangled in
attention space, making it difficult to control them independently (e-f).

##  5 Discussion

We introduce a method for guiding the diffusion sampling process to satisfy
properties derived from the attention maps and activations within the
denoising model itself. While we propose a number of such properties, many
more certainly exist, as do alternative formulations of those presented in
this paper. Among the proposed collection of properties, a few limitations
stand out.

The reliance on cross-attention maps imposes restrictions by construction,
precluding control over any object that is not described in the conditioning
text prompt and hindering fully disentangled control between interacting
objects due to correlations in attention maps (Fig. 10(e)-10(f)). Selectively
applying attention guidance at certain layers or timesteps may result in more
effective disentanglement.

Our experiments also show that appearance features often contain undesirable
information about spatial layout (Fig. 10(a)-10(b)), perhaps since the model
has access to positional information in its architecture. The reverse is also
sometimes true: guiding the shape of multiple tokens occasionally betrays the
appearance of an object (Fig. 10(c)-10(d)), implying that hidden high-
frequency patterns arising from interaction between attention channels may be
used to encode appearance. These findings suggest that our method could serve
as a window into the inner workings of diffusion models and provide valuable
experimental evidence to inform future research.

## Broader impact

The use-cases showcased in this paper, while transformative for creative uses,
carry the risk of producing harmful content that can negatively impact
society. In particular, self-guidance allows for a level of control over the
generation process that might enable potentially harmful image manipulations,
such as pulling in the appearance or layout from real images into arbitrary
generated content (e.g., as in Fig. 7). One such dangerous manipulation might
be the injection of a public figure into an image containing illicit activity.
In our experiments, we mitigate this risk by deliberately refraining from
generating images containing humans. Additional safeguards against these risks
include methods for embedded watermarking [20] and automated systems for safe
filtering of generated imagery.

## Acknowledgements

We thank Oliver Wang, Jason Baldridge, Lucy Chai, and Minyoung Huh for their
helpful comments. Dave is supported by the PD Soros Fellowship. Dave and Allan
conducted part of this research at Google, with additional funding provided by
DARPA MCS and ONR MURI.

## References

  * [1] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein.  Universal guidance for diffusion models.  arXiv preprint arXiv:2302.07121, 2023.
  * [2] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel.  Text2live: Text-driven layered image and video editing.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV, pages 707–723. Springer, 2022.
  * [3] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  ArXiv, abs/1809.11096, 2018.
  * [4] Tim Brooks, Aleksander Holynski, and Alexei A Efros.  Instructpix2pix: Learning to follow image editing instructions.  arXiv preprint arXiv:2211.09800, 2022.
  * [5] Lucy Chai, Jonas Wulff, and Phillip Isola.  Using latent space regression to analyze and leverage compositionality in gans.  arXiv preprint arXiv:2103.10426, 2021.
  * [6] Minghao Chen, Iro Laina, and Andrea Vedaldi.  Training-free layout control with cross-attention guidance.  arXiv preprint arXiv:2304.03373, 2023.
  * [7] Prafulla Dhariwal and Alex Nichol.  Diffusion models beat gans on image synthesis.  ArXiv, abs/2105.05233, 2021.
  * [8] Dave Epstein, Taesung Park, Richard Zhang, Eli Shechtman, and Alexei A Efros.  Blobgan: Spatially disentangled scene representations.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XV, pages 616–635. Springer, 2022.
  * [9] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  arXiv preprint arXiv:2208.01618, 2022.
  * [10] Alexandros Graikos, Nikolay Malkin, Nebojsa Jojic, and Dimitris Samaras.  Diffusion models as plug-and-play priors.  arXiv:2206.09012, 2022.
  * [11] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [12] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  NeurIPS, 2020.
  * [13] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  arXiv:2207.12598, 2022.
  * [14] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-based real image editing with diffusion models.  arXiv preprint arXiv:2210.09276, 2022.
  * [15] Diederik P Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models.  NeurIPS, 2021.
  * [16] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh.  Diffusion models already have a semantic latent space.  arXiv preprint arXiv:2210.10960, 2022.
  * [17] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum.  Compositional visual generation with composable diffusion models.  In European Conference on Computer Vision, 2022.
  * [18] Vivian Liu and Lydia B Chilton.  Design guidelines for prompt engineering text-to-image generative models.  In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1–23, 2022.
  * [19] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem.  Challenging common assumptions in the unsupervised learning of disentangled representations.  In international conference on machine learning, pages 4114–4124. PMLR, 2019.
  * [20] Xiyang Luo, Michael Goebel, Elnaz Barshan, and Feng Yang.  Leca: A learned approach for efficient cover-agnostic watermarking.  arXiv preprint arXiv:2206.10813, 2022.
  * [21] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.  Sdedit: Guided image synthesis and editing with stochastic differential equations.  In International Conference on Learning Representations, 2021.
  * [22] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  arXiv preprint arXiv:2211.09794, 2022.
  * [23] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  In ICML, 2022.
  * [24] Taesung Park, Jun-Yan Zhu, Oliver Wang, Jingwan Lu, Eli Shechtman, Alexei Efros, and Richard Zhang.  Swapping autoencoder for deep image manipulation.  Advances in Neural Information Processing Systems, 33:7198–7211, 2020.
  * [25] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn.  Diffusion autoencoders: Toward a meaningful and decodable representation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10619–10629, 2022.
  * [26] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents, 2022\.
  * [27] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  ICML, 2021.
  * [28] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [29] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  ArXiv, abs/1505.04597, 2015.
  * [30] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  arXiv preprint arXiv:2208.12242, 2022.
  * [31] Chitwan Saharia, William Chan, Huiwen Chang, Chris A Lee, Jonathan Ho, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Palette: Image-to-image diffusion models.  SIGGRAPH, 2022.
  * [32] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv:2205.11487, 2022.
  * [33] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  ICML, 2015.
  * [34] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  CoRR, abs/2010.02502, 2020.
  * [35] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  ICLR, 2021.
  * [36] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  arXiv preprint arXiv:2211.12572, 2022.
  * [37] Hong-Xing Yu, Leonidas J Guibas, and Jiajun Wu.  Unsupervised discovery of object radiance fields.  arXiv preprint arXiv:2107.07905, 2021.
  * [38] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu.  Scaling autoregressive models for content-rich text-to-image generation.  arXiv:2206.10789, 2022.
  * [39] Lvmin Zhang and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  arXiv preprint arXiv:2302.05543, 2023.
  * [40] Jiapeng Zhu, Yujun Shen, Yinghao Xu, Deli Zhao, and Qifeng Chen.  Region-based semantic factorization in gans.  arXiv preprint arXiv:2202.09649, 2022.

## Appendix

“a photo of a carrot and an onion in a hot tub outdoors”

“a photo of an oak tree and a pineapple outside an arctic igloo”

“a photo of an owl and a pig running at the racetrack”

(a)

Original

\begin{overpic}[width=433.62pt,height=433.62pt]{images/supp/move/owlpig7.jpg}
\put(-365.0,-30.0){\small(b) Edited} \end{overpic}

Figure 11: Moving objects. Non-cherry-picked results for moving objects in
scenes using Eqn. 9. We move onion down and to the right, pineapple to the
right, and owl up and pig down, respectively. All scenes use weights
$w_{0}=1.5$, $w_{1}=0.25$, and $w_{2}=2$.

###  A. Implementation details

We apply our self-guidance term following best practices for classifier-free
guidance on Imagen [32]. Specifically, where $N$ is the number of DDPM steps,
we take the first $\frac{3N}{16}$ steps with self-guidance and the last
$\frac{N}{32}$ without. The remaining $\frac{25N}{32}$ steps are alternated
between using self-guidance and not using it. We use $N=1024$ steps. Our
method works with $256$ and $512$ steps as well, though self-guidance weights
occasionally require adjustment. We set $v=7500$ in Eqn. 5 as an overall scale
for gradients of the functions $g$ defined below — we find that the magnitude
of per-pixel gradients is quite small (often in the range of $10^{-7}$ to
$10^{-6}$, so such a high weight is needed to induce changes.

We apply centroid, size, and shape terms on all cross-attention interactions
in the model we use. In total, there are 36 of these, across the encoder,
bottleneck, and decoder, at $8\times 8$, $16\times 16$, and $32\times 32$
resolutions. We apply the appearance term using the activations of the
penultimate layer in the decoder (two layers before the prediction readout)
and the final cross-attention operation.We experimented with features from
other parts of the U-Net denoiser, namely early in the encoder before
positional information can propagate through the image (to prevent appearance-
layout entanglement), but found these to work significantly worse. To avoid
degenerate solutions, we apply a stop-gradient to the attention in the
appearance term so only information about activations is back-propagated. We
take the mean spatially of all shape terms and across activation dimensions
for all appearance terms, which we omit in all equations for conciseness.

#### Attention mask binarization.

In practice, it is beneficial to differentiably binarize the attention map
(with sharpness controlled by $s$) before computing its size or utilizing its
shape, to eliminate the effect of background noise (this is empirically less
important when guiding centroids, so we do not binarize in that case). We do
this by taking a soft threshold at the midpoint of the per-channel minimum and
maximum values. More specifically, we apply a shifted sigmoid on the attention
normalized to have minimum 0 and maximum 1, followed by another such
normalization to ensure the high value is 1 and the low 0 after applying the
sigmoid. We use $s=10$ and redefine Eqn. 10.

$\displaystyle\texttt{normalize}(\mathbf{X})=\frac{\mathbf{X}-\min_{h,w}\left(\mathbf{X}\right)}{\max_{h,w}\left(\mathbf{X}\right)-\min_{h,w}\left(\mathbf{X}\right)}$
(14)
$\displaystyle\mathcal{A}^{\text{thresh}}=\texttt{normalize}\left(\texttt{sigmoid}\left(s\cdot\left(\texttt{normalize}(\mathcal{A})-0.5\right)\right)\right)$
(15)
$\displaystyle\texttt{size}\left(k\right)=\frac{1}{HW}\sum_{h,w}\mathcal{A}_{h,w,k}^{\text{thresh}}$
(16)

###  B. Using self-guidance

#### Maximizing consistency.

In general, we find that sharing the same sequence of noise in the DDPM
process between an image and its edited version is not necessary to maintain
high levels of consistency, but can help if extreme precision is desired. We
find that maintaining object silhouettes under transformations such as
resizing and repositioning is more effective if applying a transformation
$\mathcal{T}$ to the original shape, rather than expressing the same change
through centroid and size.

#### Guiding “background” words.

To keep all objects of the scene fixed but one (Fig. 2), one can either guide
all other tokens in the prompt (including “a photo of” and other abstract
terms) to keep their shape, or only select the other salient objects and hold
those fixed. In general, since abstract words are often used for message
passing and have attention patterns that are correlated with the layout of the
scene, we prefer not to guide their layouts to maximize compositionality.

Mitigating appearance-layout entanglement. When words or concepts span
multiple tokens, we can mean-pooling attention maps across these tokens before
processing them, though do not find this to improve results. We also find that
corrupting target shapes with Gaussian noise helps mitigate this effect,
providing some evidence for this hypothesis.

“a photo of a kangaroo and a punching bag at the gym”

“a photo of a chicken walking across the street with an Italian sports car
waiting for it”

“a photo of a boombox on a camel near a pond”

(a)

Original

\begin{overpic}[width=433.62pt,height=433.62pt]{images/supp/resize/camelbig7.jpg}
\put(-365.0,-30.0){\small(b) Edited} \end{overpic}

Figure 12: Resizing objects. Non-cherry-picked results for resizing objects in
scenes using Eqn. 9, with $\mathcal{T}$ specified to up- or down-sample
attention maps. We reduce the punching bag’s height $0.5\times$ and enlarge
chicken $2.5\times$ and boombox $2\times$. All scenes use $w_{0}=2$,
$w_{1}=0.25$, and $w_{2}=3$.

#### Moving objects.

We use $w_{0}\in[0.5,2],w_{1}\in[0.03,0.3],w_{2}\in[0.5,5]$ in Eqn. 11.
Alternatively, we can express $o_{k}$’s new location through its centroid,
adding a term to keep size fixed:

$\displaystyle g$ $\displaystyle=w_{0}\overbrace{\frac{1}{|O|-1}\sum_{o\neq
o_{k}}\frac{1}{|\mathcal{A}|}\sum_{i=0}^{|\mathcal{A}|}\|\texttt{shape}_{i,t,\text{orig}}(o)-\texttt{shape}_{i,t}(o)\|_{1}}^{\text{Fix
all other object shapes}}$ (17)
$\displaystyle+w_{1}\overbrace{\frac{1}{|O|}\sum_{o\in
O}\|\texttt{appearance}_{t,\text{orig}}(o)-\texttt{appearance}_{t}(o)\|_{1}}^{\text{Fix
all object appearances}}$
$\displaystyle+w_{2}\overbrace{\frac{1}{|\mathcal{A}|}\sum_{i=0}^{|\mathcal{A}|}\|\texttt{size}_{i,t,\text{orig}}(o_{k})-\texttt{size}_{i,t}(o_{k})\|_{1}}^{\text{Fix
$o_{k}$'s size}}$
$\displaystyle+w_{3}\underbrace{\frac{1}{|\mathcal{A}|}\sum_{i=0}^{|\mathcal{A}|}\|\texttt{target\\_centroid}-\texttt{centroid}_{i,t}(o_{k})\|_{1}}_{\text{Change
$o_{k}$'s position}}$

Where target_centroid can be computed as a shfited version of the timestep-
and-attention-specific $\texttt{centroid}_{\text{orig}}$ if desired, or
selected to be an absolute value on the canvas (repeated across all
timesteps). We generally use weights
$w_{0}\in[0.5,2],w_{1}\in[0.03,0.3],w_{2}\in[0.5,2],w_{3}\in[1,3]$.

#### Resizing objects.

We can follow Eqn. 9 to resize objects as well, by setting $\mathcal{T}$ to
upsample or downsample the original mask. We can similarly use Eqn. 17,
omitting the final term and setting the target size to a desired value, either
computed as a function of $\texttt{size}_{\text{orig}}(o_{k})$ or provided as
an absolute proportion of pixels on the canvas that the object should cover.
We use the same weight range for all weights except we set
$w_{2}\in[1,3],w_{3}=0$ for Eqn. 17.

#### Sampling new appearances.

We set $w_{0}\in[0.1,1]$ in Eqn. 10. Generally, higher values lead to
extremely precise layout preservation at the expense of diversity in
appearance.

#### Sampling new layouts.

Just as we can find new appearances for a scene of a given layout, we can
perform the opposite operation, finding new layouts for scenes where objects
have a given appearance:

$\displaystyle g$ $\displaystyle=w_{0}\overbrace{\frac{1}{|O|}\sum_{o\in
O}\|\texttt{appearance}_{t,\text{orig}}(o)-\texttt{appearance}_{t}(o)\|_{1}}^{\text{Fix
all appearances}}$ (18)

We almost always use $w_{0}\in[0.05,0.25]$.

“a photo of a koala picking flowers next to a mansion”

“a photo of a capybara wearing a robe sitting by the fireplace”

“a photo of a bird drinking coffee at a 1950s style diner”

(a)

Original

\begin{overpic}[width=433.62pt,height=433.62pt]{images/supp/newapp/birdapp7.jpg}
\put(-365.0,-30.0){\small(b) Edited} \end{overpic}

Figure 13: Creating new appearances for scenes. Non-cherry-picked results
sampling different “styles” of appearances given the same layout, using Eqn.
10. We use $w_{0}=0.7$, $0.3$, and $0.3$ respectively for each result, to
preserve greater structure in the background of the first picture.

#### Collaging objects in-place.

Eqn. 12 can be easily generalized to more than one object per image (adding
another sum across all objects) or to the case where prompts vary between
images (mapping from $k_{j}$ to the corresponding indices in the new image).
We set $w_{0}\in[0.5,1],w_{1}\in[0.05,0.3]$.

#### Collaging objects with a new layout.

As shown in Fig. 6(f), we can also collage objects into a new layout specified
by a target image $J+1$, in addition to the $J$ images specifying object
appearance:

$\displaystyle g$
$\displaystyle=w_{0}\overbrace{\frac{1}{|\mathcal{A}|}\sum_{i=0}^{|\mathcal{A}|}\|\texttt{shape}_{i,t,J+1}(o_{k_{J+1}})-\texttt{shape}_{i,t}(o_{k})\|_{1}}^{\text{Copy
all object shapes}}$ (19)
$\displaystyle+w_{1}\overbrace{\frac{1}{J}\sum_{j}\|\texttt{appearance}_{t,j}(o_{k_{j}})-\texttt{appearance}_{t}(o_{k})\|_{1}}^{\text{Copy
each object's appearance}}$

As in Eqn. 12, we set $w_{0}\in[0.5,1],w_{1}\in[0.05,0.3]$.

#### Transferring object appearances to new layouts.

Nothing requires the indices (or in fact, the objects those indices refer to)
to be the same in the image being generated and the original image being used
as a source, as long as there is a mapping specified between the indices in
the old and new images which should correspond. Call this mapping $m$. We can
then take the appearance of an object $o_{k}$ in a source image and transfer
it to an image with any new prompt as follows, as specified in Eqn. 13 (with
typical weights $w_{0}\in[0.01,0.1]$):

$g=w_{0}\overbrace{\|\texttt{appearance}_{t,\text{orig}}(o_{k})-\texttt{appearance}_{t}\left(m(o_{k})\right)\|_{1}}^{\text{Copy
object appearance}}$ (20)

#### Merging layout and appearance.

We use $w_{0}\in[1,2]$ and $w_{1}\in[0.1,0.3]$ in Eqn. 11.

“a photo of a rabbit with a birthday balloon and a party hat”

“a photo of cleats, a bright soccer ball, and a cone”

“a calculator, a toy car, and a pillow on a rug”

(a)

Original

\begin{overpic}[width=433.62pt,height=433.62pt]{images/supp/newlayout/pillownewlay7.jpg}
\put(-365.0,-30.0){\small(b) Edited} \end{overpic}

Figure 14: Creating new layouts for scenes. Non-cherry-picked results sampling
new layouts for the same scenes, using Eqn. 18. We use $w_{0}=0.07$, $0.07$,
and $0.2$ respectively.

“a DSLR photo of a backpack at the grand canyon”

“a DSLR photo of a backpack wet in the water”

“a photo of a pair of sunglasses being worn by a bear”

“a photo of a pair of sunglasses on a pile of snow”

(a)

Original

\begin{overpic}[width=433.62pt,height=433.62pt]{images/supp/realimgapp/snow7.jpg}
\put(-442.0,-30.0){\small(b) Object in new contexts} \end{overpic}

Figure 15: Appearance transfer from real images. Non-cherry picked results
sampling new images with a given object’s appearance specified by a real
images, as in Eqn. 13. We use $w_{0}=0.15$.

#### Editing with real images.

Importantly, our method is not limited to editing generated images to whose
internals it has access by definition. We find that we can also meaningfully
guide generation using the attention and activations extracted from a set of
forward-process denoisings of a real image (given a caption) to “approximate”
the reverse process, despite any mismatch in distributions one might imagine.
Concretely, we generate $T$ corrupted versions of a real image $x$,
$\\{\alpha_{t}x+\sigma_{t}\epsilon_{t}\\}_{1}^{T}$, where
$\epsilon_{t}\sim\mathcal{N}(0,1)$. We then extract the attention
$\mathcal{A}_{t}$ and activations $\Psi_{t}$ from the denoising network at
each of these timesteps in parallel and concatenate them into a length-$T$
sequence. We treat this sequence identically to a sequence of $T$ internals
given by subsequent sampling steps, and can thus transfer the appearance of
objects from real images, output images that look like real images with moved
or resized objects, and so on.

“a photo of a chow chow wearing a superman outfit”

“a dslr photo of a teapot floating in the sea”

(a)

Original

(b)

Ours

\begin{overpic}[width=433.62pt,height=433.62pt]{images/supp/realimgsameprompt/teapot7.jpg}
\put(-470.0,-30.0){\small(b) Random samples without self-guidance}
\end{overpic}

Figure 16: Ablating appearance transfer from real images. To verify the
efficacy of our approach, we compare our results from Fig. 7 in the paper to
random samples from the same prompt without apperance transfer. We can see
that appearance of objects varies significantly without self-guidance.

In Fig. 6, the prompts we use to transfer appearance are “A photo of a Chow
Chow…” and “A DSLR photo of a teapot…”. While our method works on less
specific descriptions as well, it is not as reliable when object appearance is
more out-of-distribution. For context, we show unguided samples under the
prompts from Fig. 6 in Fig. 16, which still deviate significantly from the
desired appearance, showing the efficacy of our approach. A weakness of our
simple approach is that it has no constraints on the shape of the generated
objects, which we leave to future work.

#### Weight selection heuristics.

We find weights that work well to remain more or less consistent across
different images given an edit, but ideal weights do vary somewhat (within
predictable ranges) between different combinations of terms. Our heuristics
for weight selection per term are: the more weights there are, the higher per-
term weights can be without causing artifacts (and indeed, need to be, to
provide ample contribution to the final result); appearance terms should have
weights 1 or 2 orders of magnitude lower than layout terms; layout summary
statistics (centroid and size) should have slightly lower weights than terms
on the per-pixel shape; total weight of terms should not add up to more than
$\sim 5$ to avoid artifacts.

###  C. Additional results

We show further non-cherry-picked results for the edits we show in the main
paper. Our general protocol consists of selecting an interesting prompt
manually, verifying that our model creates compelling samples aligning with
this prompt without self-guidance, beginning with the typical weights we use
for an edit, and trying around 3-5 other weight configurations to find the one
that works best for the prompt – in most cases, this is the starting set of
weights. Then, we use the first 8 images we generate, without further
filtering. We generate all results with different seeds to showcase the
strength of guidance even without shared DDPM noise. We show more results for
moving (Fig. 11) and resizing (Fig. 12) objects, sampling new appearances for
given layouts (Fig. 13) as well as new layouts for a given set of objects
(Fig. 14), transferring the appearance of real objects into new contexts (Fig.
15),