# Topological weak-measurement-induced geometric phases revisited

Graciana Puentes1,2<EMAIL_ADDRESS>1-Departamento de Fsica, Facultad de
Ciencias Exactas y Naturales, Universidad de Buenos Aires, Ciudad
Universitaria, 1428 Buenos Aires, Argentina,
2-CONICET-Universidad de Buenos Aires, Instituto de Fsica de Buenos Aires
(IFIBA), Ciudad Universitaria, 1428 Buenos Aires, Argentina.

###### Abstract

We present an analytical and numerical study of a class of geometric phase
induced by weak measurements. In particular, we analyze the dependence of the
geometric phase on the winding $W$ of the polar angle $\varphi$, upon a
sequence of $N$ weak measurements of increased magnitude ($c$), resulting in
the appearance of a multiplicity of critical measurement-strength parameters
where the geometric phase becomes stochastic. Adding to the novelty of our
approach, we not only analyze the weak-measurement induced geometric phase by
a full analytic derivation, valid in the quasicontinuous limit
($N\rightarrow\infty$), but also we analyze the induced geometric phase
numerically, thus enabling us to unravel the finite-$N$ interplay of the
geometric phase with the measurement strength parameter, and its stability to
perturbations in the measurements protocol.

##  I Introduction

When a quantum system undergoes adiabatic cycling, its state can become
quantifiable based solely on its closed path in parameter space 1 .
Conversely, a cyclic series of quantum observations can create a geometric
phase. As reported in Gebhart _et al._ 2 , for closed trajectories the
geometric phase becomes stochastic upon the application of a series of weak
measurements, and a topological transition may occur in the mapping between
the measurement sequence and the geometric phase, when the measurement
strength is changed.

Despite the fact that overall quantum phases cannot be fully determined, when
the quantum system is driven slowly over a cycle returning to its starting
condition, the accumulated phase becomes gauge invariant and can be measured.
As originally pointed out by Sir. M. V. Berry 3 , this is a geometric phase
($\mathcal{X}$) in that it is dependent on characteristics of the closed
trajectory in parameter space, rather than on process dynamics. Geometric
phases can be held responsible for a number of situations: they can modify
material properties in solids, such as conductivity in graphene Berrygraphene
, they can trigger the emergence of surface edge-states in topological
insulators, whose surface electrons experience a geometric phase
Berrytopoinsul , they can modify the outcome of molecular chemical reactions
Berrychemestry , and they can affect electronic properties of matter
Berryelectronic . Furthermore, understanding various physical phenomena
GeomPhase1 ; GeomPhase2 ; GeomPhase3 , defining fractional statistics anyonic
quasiparticles Anyon1 ; Anyon2 ; Anyon3 , and identifying topological
invariants for quantum Hall phases Hall1 , superconductors TopoInsu1 ;
TopoInsu2 , or quantitative characterizations of topological insulators via
the Zak phase ZakDemler ; ZakLonghi , as well as underpinning holonomic and
topological signatures in photonic systems HolonomicWhite ; PuentesCrystal ;
PuentesJOSAB ; PuentesEntropy ; PuentesQuantumRep , are all made possible by
geometric phases.

Figure 1: Schematic representation of the action of the measurement protocol
described by the proposed POVM set for a generic input state
$a|\uparrow\rangle+b|\downarrow\rangle$. Kraus operators
$M_{\eta}({\bf{n}},+/-)$ can be implemented by means of imperfect polarizers
oriented along (a) vertical direction $r^{+}=|+\rangle$; (b) horizontal
directions $r^{-}=|-\rangle$; (c) depicts the Bloch sphere for the system
qubit $\\{|\uparrow\rangle,|\downarrow\rangle\\}$ setting the initial state
$|\psi_{0}\rangle=\frac{|\uparrow\rangle+|\downarrow\rangle}{\sqrt{2}}$, for
initial parameters $(\theta_{0}=\pi/2,\varphi_{0}=0)$; (d) red curve depicts
the discrete jump in the geometric phase $\mathcal{X}$ from 0 to $-\pi$ for a
critical measurement strength $c_{\mathrm{crit}}=2.1$ (dashed line indicating
the onset of topological phase transition; (e) depicts the topological phase
transition in $\mathcal{X}$ for $\alpha=\pi$, corresponding to a single
winding of $\varphi$ ($W=1$). Special attention was taken to ensure the
continuity of $\mathcal{X}$ at the point of the stochastic phase transition in
order to distinguish transitions from random phase jumps. Figure 2: Stochastic
transition in the geometric phase $\mathcal{X}$ vs. angle parameter $\alpha$
for increased winding number $W$: (a) $(\alpha=2\pi,W=2)$ with
$c_{\mathrm{crit}}=3.4$ and $5.7$, (b) $(\alpha=3\pi,W=3)$ with
$c_{\mathrm{crit}}=4.4$ and $7.6$, (c) ($\alpha=4\pi,W=4$) with
$c_{\mathrm{crit}}=5.2$ and $9.1$ and (d) $(\alpha=5\pi,W=5)$ with
$c_{\mathrm{crit}}=6.0$ and $10.5$. We note the actual critical measurement-
strength values are completely unpredictable, thus the transitions at
different winding numbers $W$ are not topologically equivalent.

A class of geometric phase, resulting from the outcome of a series of intense
(projective) measurements that operate on the system and produce certain
measurement read-outs, is the Pancharatnam phase Pancharatnam . Optical
investigations that monitor the Pancharatnam phase caused by polarizer
sequences have readily been reported BerryStack . Notwithstanding the fact
that incoherent measurement processes are generally involved, such a phase may
be reliably detected. A general series of measurements is by its very nature
stochastic. Thus, based on the sequences of measurement readouts linked to the
relevant probabilities, one may expect a distribution of measurement-induced
geometric phases. The induced evolution is entirely predictable for a quasi-
continuous series of strong measurements ($N\rightarrow\infty$) because of the
dynamical quantum Zeno effect Zeno .

In this paper, we present an analytical and numerical study of a novel type of
geometric phase induced by weak measurements. In contrast to PNAS 2020 2 , the
originality of our approach consists of considering the dependence of the
geometric phase on the winding $W$ of the polar angle $\varphi$, which
quantifies the number of full $2\pi$-turns the trajectory makes until it
closes its path, returning to its original state. This is accomplished by
fixing the azymhutmal angle at $\theta=\pi/2$ and considering the initial
state
$|\psi_{0}\rangle=\frac{|\uparrow\rangle+|\downarrow\rangle}{[\sqrt{2}}$, and
a sequence of weak measurements in the angle $\varphi$ of increasing magnitude
$\varphi_{k}=\epsilon k$, with $k=0,..,N$ the measurement index. We ensure the
trajectory induced by the sequence of weak measurements is closed, and the
geometric phase well defined, by parameterizing the rotation parameter as
$\alpha=2\epsilon/N$, where $N$ is the number of measurements. With this
parameterization, the winding number is simply defined as $W=\alpha/\pi$,
resulting in $W=1,...,M$ with $M$ the total number of windings. We find that
different winding numbers $W$ can give rise to unpredictable critical
measurement-strength parameters, where the geometric phase becomes stochastic
and the system undergoes a topological phase transition, thus confirming that
the transitions for different $W$ are not topologically equivalent.
Furthermore, adding to the novelty of our work, we not only analyze the weak-
measurement induced geometric phase by a full analytic derivation based on the
exponential approximation presented in PNAS 2020 2 , valid for
$N\rightarrow\infty$, but also we analyze the induced geometric phase
numerically, thus enabling us to understand the finite-$N$ interplay of the
geometric phase on the measurement strength parameter $c$, and its stability
to fluctuations in the measurement protocol.

The paper is organized as follows: In Section II we outline the measurement
protocol. Next, in Section III, we present our analytic results. In
particular, we characterize the phase transitions for different windings $W$.
Next, in Section IV, we present our numerical results. Namely, we characterize
the interplay of the geometric phase with the measurement strength $c$ for
finite $N$, and we analyze the impact of phase noise on the geometric phase in
different parameter regions of the $N-c$ landscape. Finally, in Section V, we
outline the conclusions and future perspectives.

##  II Measurement Protocol

Figure 3: Topological phase transition for increased winding numbers
$W=2,3,4$, for $c>c_{\mathrm{crit}}$ ($c<c_{\mathrm{crit}}$) red(blue) curves:
(a) and (b) depicts curves for $W=2$, and phase transition from 0 to $|\pi|$
and $|\pi|$ to 0, respectively; (c) and (d) depicts curves for $W=3$, and
phase transition from 0 to $|\pi|$ and $|\pi|$ to 0, respectively; (e) and (f)
depicts curves for $W=4$, and phase transition from 0 to $|\pi|$ and $|\pi|$
to 0, respectively. The different topological nature of each phase transition
is signalled by the increased numbers of oscillations for increased $W$. This
can be explained by noting that increasing the winding number $W=k\pi$
corresponds to closing the trajectory after $k$-windings of the polar phase
$\phi$. Therefore, it is expected to observe a larger number of oscillations
when for increased $W$. (a), (c) and (e) display $k$ oscillations for
$W=k\pi$, while (b), (d) and (f) present $k-1$ oscillations for $W=k\pi$,
since there is no phase transition from $|\pi|$ to 0 for a single winding
$W=1$. Special attention was taken to ensure the continuity of the geometric
phase at the point of the stochastic transition in order to distinguish
transitions from random phase jumps.

The measurement sequence required to accumulate the intended geometric phase
$\mathcal{X}$ can be mathematically described by a complete set of POVMs
(Positive Operator Valued Measures), implemented via the Krauss operators
$\mathcal{M}_{k}^{(r_{k})}=M_{\eta_{k}}({\bf{n_{k}}},r_{k})$,
$|\psi\rangle\rightarrow\mathcal{M}_{k}^{(r_{k})}|\psi\rangle$, as described
in 2 . Such POVM can be implemented by introducing a detector consisting of a
second qubit whose Hilbert space is spanned by the set
$r=\\{|+\rangle,|-\rangle\\}$. We consider the generic initial state of the
system of the form $|\psi_{0}\rangle=a|\uparrow\rangle+b|\downarrow\rangle$,
and assume the detector is in the initial state $|+\rangle$ and that the
initial state of the system plus detector is separable, of the form
$|\psi_{\mathrm{sep}}\rangle=|\psi_{0}\rangle\otimes|+\rangle$. The
measurement coupling $\lambda(t)$ is then switched on for a finite time
$t\in[0,T]$, to obtain the entangled state:

$|\psi_{\mathrm{ent}}\rangle=M_{\eta}({\bf{n}},+)|\psi_{0}\rangle+M_{\eta}({\bf{n}},-)|\psi_{0}\rangle,$
(1)

here the measurement strength is $\eta=\sin^{2}(g)$, with
$g=\int_{0}^{T}\lambda(t)dt$. The POVM set is defined by the Kraus operators:

$M_{\eta}(\hat{z},+)=\begin{pmatrix}1&0\\\ 0&\sqrt{1-\eta}\end{pmatrix}\hskip
14.22636ptM_{\eta}(\hat{z},-)=\begin{pmatrix}1&0\\\
0&\sqrt{\eta}\end{pmatrix},$ (2)

corresponding to a measurement orientation along the $z$-axis ${\bf
n}=\hat{z}$. Kraus operators along a generic orientation ${\bf n}$ can be
obtained via the change of basis $M_{\eta}({\bf n},r)=R^{-1}({\bf
n})M_{\eta}(\hat{z},r)R({\bf n})$, where the unitary matrix $R({\bf n})$ is
given by:

$R({\bf n})=\begin{pmatrix}\cos\theta/2&e^{-i\varphi}\sin\theta/2\\\
\sin\theta/2&e^{-i\varphi}\cos\theta/2\end{pmatrix},$ (3)

representing a rotation of the measurement orientation along the direction
${\bf n}$. This map implements a null-type weak measurement as proposed in
PNAS2020 2 . An imperfect polarizer oriented along the directions $r^{+,-}$
can implement such null-type weak-measurement protocol. This is depicted in
Figure 1. Fig. 1 (a) describes the action of the POVM $M_{\eta}(\hat{z},+)$ on
a generic input state $|\psi_{0}\rangle=a|\uparrow\rangle+b|\downarrow\rangle$
while Fig. 1 (b) describes the action of the POVM $M_{\eta}(\hat{z},-)$ on the
same generic input state. The Bloch sphere for the system qubit
$s=\\{|\uparrow\rangle,|\downarrow\rangle\\}$ is schematized in Fig. 1 (c).

##  III Analytic Results

Figure 4: Plots of the geometric phase $\mathcal{X}$ vs $\alpha$ for
decreasing values of $N$, for a measurement strength parameter
$c>c_{\mathrm{crit}}$ (red curves) and $c<c_{\mathrm{crit}}$ (blue curves),
where $c_{\mathrm{crit}}\approx 2.1$ for $W=1$. (a), (b), (c) and (d)
correspond to $N=10,8,6,4$, respectively. The deviation of the numerical
result from the analytic prediction is readily apparent for $N=8$. For $N=4$
the stochasticity in $\mathcal{X}$ is fully vanished.  Figure 5: 3D plots of
the geometric phase $\mathcal{X}$ for different regions of the $N-c$
landscape. (a), (b) and (c) correspond to parameter values of $0<N<50$ and
$0<c<10$, for $W=1,2,3$ respectively. The magenta colouring of the 3D density
plots indicate regions where the stability of the geometric phase is
dramatically reduced. (d), (e) and (f) present numerical simulations of
$\mathcal{X}$ considering $100<N<500$ and $0<c<10$, for $W=1,2,3$
respectively. (g), (h), (i), for $100<N<500$ and $100<c<400$, for $W=1,2,3$,
respectively. It is observed that fluctuations in $\mathcal{X}$ arise for
$c>300$, for any value of $N$. Moreover, in the stable region characterized by
$N>>c$, there is no apparent critical measurement-strength parameter where
$\mathcal{X}$ makes a discrete jump, within the resolution of the numerical
simulations.

An analytic expression for the geometric phase $\mathcal{X}$ can be derived in
the quasicontinuous measurement limit $N\rightarrow\infty$. $\mathcal{X}$ is
extracted from the quasicontinuous trajectory postselecting all outcomes
$r_{k}=|+\rangle$ as described in PNAS2020 2 . This result is obtained by
setting the initial state $|\psi_{0}\rangle=R^{-1}({\bf
n_{0}})|\uparrow\rangle$. In our case, we consider the initial state to be
eigenstate of $S_{x}$, of the form
$|\psi_{0}\rangle=\frac{|\uparrow\rangle+|\downarrow\rangle}{[\sqrt{2}}$, thus
selecting the initial rotation along $\hat{x}$, this is equivalent to setting
the initial parameters $(\theta_{0}=\pi/2,\varphi_{0}=0)$, as depicted in Fig.
1 (c). We then sequentially rotate the measurement apparatus, in order to
increment the angle $\varphi$ by a fixed amount $\epsilon=2\pi/N$. This
parameterization ensures that the trajectory is closed and the geometric phase
well defined. By selecting the readouts $r_{k}=|+\rangle$, setting
$\eta=4c/N$, the measurement orientations
$(\theta_{k},\varphi_{k})=(\pi/2,2\pi k/N)$, and using the explicit form of
Kraus operators in Eq. 2, one obtains an expression for the geometric phase
$\mathcal{X}$ of the form 2 :

$\langle\psi_{0}|\mathcal{M}^{(+)}_{N-1}...\mathcal{M}^{(+)}_{1}|\psi_{0}\rangle=\langle\uparrow|\delta
R(M_{4c/N}(\hat{z},+)\delta R)^{N-1}|\uparrow\rangle,$ (4)

where $\delta R$ is given by the $k$-independent matrix:

$\delta R=\begin{pmatrix}1/2(1+e^{-i\epsilon})&1/2(1-e^{-i\epsilon})\\\
1/2(1-e^{-i\epsilon})&1/2(1+e^{-i\epsilon})\end{pmatrix}.$ (5)

The quasicontinuous limit ($N\rightarrow\infty$) is obtained by using the
exponential approximation $(1+\Gamma/N)^{N}\approx e^{\Gamma}$, valid for
$N\rightarrow\infty$. This approximation is explicitly invoked by expressing
the Taylor series of $M_{4c/N}(\hat{z},+)\delta R$, and rewriting
$M_{4c/N}(\hat{z},+)\delta R=V(I+\Gamma/N)V^{-1}$, where $I$ is the $2\times
2$ identity matrix and $V$ is the unitary matrix for the change of basis,
obtained by diagonalization of $M_{4c/N}(\hat{z},+)\delta R$. In this
scenario, the geometric phase can be extracted from the upper-most diagonal
element of the matrix product $Ve^{\Gamma}V^{-1}$, resulting in:

$\mathcal{X}=e^{-i\alpha-c}[\cosh(\tau)+c\sinh(\tau)/\tau),$ (6)

where $\tau=\sqrt{c^{2}-\alpha^{2}}$ and $\alpha=\epsilon N/2$. We recover the
analytic result reported in 2 , by setting $z=c$, and $\alpha=\pi$, as
expected for $\theta=\pi/2$.

We note that our case is complementary to the one analyzed in PNAS2020 2 ,
where the authors analyze the dependence of the geometric phase on the
azymhutmal angle $\theta$ for a single winding ($\alpha=\pi$) of the polar
angle $\varphi$. Here we report the dependence of the geometric phase on the
winding of the polar angle $\varphi$, obtained by increasing the winding
number $W=\alpha/\pi$, while fixing $\theta=\pi/2$. We stress that the
critical measurement-strength parameter $c_{\mathrm{crit}}$ obtained for
different winding numbers $W=1,2,3,...,M$ (with $M\in N$) are completely
random. Therefore, we argue the phase transitions observed for different
windings $W$ are not topologically equivalent.

The onset of the topological phase transition in the geometric phase for a
critical measurement strength can be understood by considering two limiting
cases: For the case of a series of strong projective measurements
($c\rightarrow\infty$), resulting in the well-known Pantcharatnam geometric
phase Pancharatnam , the polarization is expected to rotate along the equator
by an amount $\epsilon=2\pi/N$ with each projective measurement, until it
returns to its original state, accumulating a geometric phase equal to 1
(modulo $2\pi$), while for a series with infinitely weak measurement strength
($c\rightarrow 0$) the polarization should not be affected by the measurement
process. Thus, there exists a critical measurement strength
$c_{\mathrm{crit}}$ where $\mathcal{X}$ makes a discrete jump from 0 to 1,
signalling the onset of a topological phase transition, here $\pi$ corresponds
to a full winding of the geometric phase. A plot of the geometric phase
$\mathcal{X}$, given by the analytic expression in Eq. 6, is given in Fig. 1
(d) and (e). Figure 1 (d) depicts the discrete jump of $\mathcal{X}$ from 0 to
$-\pi$ for a critical measurement strength $c_{\mathrm{crit}}=2.1$ indicating
the onset of topological phase transition, while Fig. 1 (e) depicts the
geometric phase vs. $\alpha$, signalling a topological phase transition for
$\alpha=\pi$, corresponding to a full winding of the polar phase ($W=1$).
Special attention was taken to ensure that the geometric phase is
mathematically continuous in order to distinguish the onset of a topological
transition, from a random $\pi$-jump in the geometric phase.

###  III.1 Winding Number ($W=\alpha/\pi$)

Next, we analyzed the existence of additional critical measurement-strength
values, signalling multiple topological phase transition characterized by
discrete jumps in the geometric phase between 0 and $|\pi|$. This was analyzed
by enabling the parameter $\alpha$ to take multiple values $k\pi$ ($k=1,...,M$
with $M\in N$), corresponding to a full winding of the phase $\varphi$
quantified by the winding number $W=\alpha/\pi=1,...,M$ and a wrapping of the
phase at multiples of $2\pi$, revealing the existence of additional critical
measurement-strength parameters for different values of $W$. This is shown in
Figure 2, displaying additional critical-measurement parameter values where
$\mathcal{X}$ jumps from $0\rightarrow-\pi$ and from $-\pi\rightarrow 0$,
respectively. We note that the critical values $c_{\mathrm{crit}}$ are fully
random, therefore the phase transitions observed for different windings $W$
are not topologically equivalent. Figure 2 corresponds to: (a)
$(\alpha=2\pi,W=2)$ with $c_{\mathrm{crit}}=3.4$ and $5.7$, (b)
$(\alpha=3\pi,W=3)$ with $c_{\mathrm{crit}}=4.4$ and $7.6$, (c)
($\alpha=4\pi,W=4$) with $c_{\mathrm{crit}}=5.2$ and $9.1$ and (d)
$(\alpha=5\pi,W=5)$ with $c_{\mathrm{crit}}=6.0$ and $10.5$. We note the
actual critical measurement-strength values are completely unpredictable, thus
we argue the topological transitions indicated by different winding numbers
$W$ are not equivalent.

Further confirmation of the onset of different topological phase transition
was obtained by plotting the geometric phase vs. the angle parameter $\alpha$
for increased winding number $W$. This is presented in Figure 3, for
trajectories with $c<c_{\mathrm{crit}}$ (red curves) and for trajectories with
$c>c_{\mathrm{crit}}$ (blue curves). Figure 3 (a) and (b) depicts curves for
$W=2$, and phase transition from 0 to $|\pi|$ and $|\pi|$ to $0$ (mod $2\pi$),
respectively; (c) and (d) depicts curves for $W=3$, and phase transition from
0 to $|\pi|$ and $|\pi|$ to 0 (mod $2\pi$), respectively; (e) and (f) depicts
curves for $W=2$, and phase transition from 0 to $|\pi|$ and $|\pi|$ to $0$
(mod $2\pi$), respectively. The different topological nature of each phase
transition is signalled by the increased numbers of oscillations for increased
$W$. This can be explained by noting that increasing the winding number
$W=k\pi$ corresponds to closing the trajectory after $k$-windings of $2\pi$ of
the polar angle $\varphi$. Therefore, it is expected to observe a larger
number of oscillations for increased windings $W$, as displayed in Fig. 3.
Figure 3 (a), (c) and (e) display $k$ oscillations for $W=k\pi$, while Figs. 3
(b), (d) and (f) present $k-1$ oscillations for $W=k\pi$, since there is no
phase transition from $|\pi|$ to 0 (modulo 2$\pi$) for a single winding $W=1$.

Figure 6: Numerical simulations of uncorrelated phase noise in the parameter
region $0<N<50$ and $0<c<10$ of the $N-c$ landscape. (a), (b), (c) correspond
to phase noise with a spread $\Delta\alpha=2\%$ for $W=1,2,3$, respectively;
(d), (e), (f) correspond to phase noise with a spread $\Delta\alpha=5\%$ for
$W=1,2,3$, respectively; (g), (h), (i) correspond to phase noise with a spread
$\Delta\alpha=10\%$ for $W=1,2,3$, respectively. For $\Delta\alpha>2\%$
fluctuations in the phase render the measurement protocol significantly
unstable.  Figure 7: Numerical simulations of uncorrelated phase noise in the
parameter region $100<N<500$ and $0<c<10$ of the $N-c$ landscape. (a), (b),
(c) correspond to phase noise with a spread $\Delta\alpha=2\%$ for $W=1,2,3$,
respectively; (d), (e), (f) correspond to phase noise with a spread
$\Delta\alpha=5\%$ for $W=1,2,3$, respectively; (g), (h), (i) correspond to
phase noise with a spread $\Delta\alpha=10\%$ for $W=1,2,3$, respectively.
This region of the $N-c$ landscape is significantly robust to uncorrelated
phase noise and the quantization of the phase is typically preserved. Figure
8: Numerical simulations of uncorrelated phase noise in the parameter region
$100<N<500$ and $100<c<400$ of the $N-c$ landscape. (a), (b), (c) correspond
to phase noise with a spread $\Delta\alpha=2\%$ for $W=1,2,3$, respectively;
(d), (e), (f) correspond to phase noise with a spread $\Delta\alpha=5\%$ for
$W=1,2,3$, respectively; (g), (h), (i) correspond to phase noise with a spread
$\Delta\alpha=10\%$ for $W=1,2,3$, respectively. In this limit, the only
region of the $N-c$ landscape robust to phase noise if for $N>>c$ and $c<300$,
even though there are no apparent phase transitions within the resolution of
the numerical simulations.

##  IV Numerical Results

###  IV.1 Quantization of the geometric phase

The topological character of the phase transition exhibited at the critical
measurement-strength parameter ($c_{\mathrm{crit}}$), where the geometric
phase ($\mathcal{X}$) becomes stochastic, is clearly revealed by the
quantization of $\mathcal{X}$ between 0 and $|\pi|$ (modulo $2\pi$). This
quantization can be understood as a signature of the so-called “topological
protection” of the phase, which prevents fluctuations in the discrete values
of the phase acquired by the system under weak perturbations in the protocol.
This quantization of the phase, confirming the topological nature of the
transition, is clearly valid in the quasicontinuous limit
($N\rightarrow\infty$), where the analytic result holds.

In this Section, we analyze the robustness of the quantization of
$\mathcal{X}$, with respect to fluctuations in the total number of
measurements $N$, meaning that we derive a numerical expression for the
geometric phase for finite-$N$. Evidently, this analysis cannot be derived
from the analytical result reported in Eq. (6), which is only valid for
$N\rightarrow\infty$ and $N>>\Gamma$, for the exponential approximation
$(1+\Gamma/N)^{N}\approx e^{\Gamma}$ to hold. In order to obtain a numerical
value for $\mathcal{X}$ and analyze its robustness to different type of noise
and perturbations, we follow Eq. (4) to obtain a numerical expression for the
$N$-matrix product $\delta R(M_{4c/N}(\hat{z},+)\delta R)^{N-1}$. Next, we
plot the left upper-most matrix element for different values of $N$ and $c$.
For $N$ sufficiently large ($N>500$) and the measurement-strength parameter
sufficiently small ($c<<N$), the analytic result and numerical results are
indistinguishable. We are interested in understanding at which point the
numerical result departs from the analytic prediction.

Plots of the geometric phase $\mathcal{X}$ vs $\alpha$ for decreasing values
of $N$ are presented in Fig. 4, for a measurement strength parameter
$c>c_{\mathrm{crit}}$ (red curves) and $c<c_{\mathrm{crit}}$ (blue curves),
where $c_{\mathrm{crit}}\approx 2.1$ for $W=1$. For simplicity, we consider a
single winding of the polar angle $W=1$, although the same framework can be
applied to $W>1$. Figures 4 (a), (b), (c) and (d) correspond to $N=10,8,6,4$,
respectively. We note that the deviation of the numerical result from the
analytic prediction is readily apparent for $N=8$, where the quantization of
$\mathcal{X}$ between 0 and $\pi$ vanishes, signalling that the topological
nature of the phase transition vanishes. Furthermore, for $N<5$ the stochastic
response of the geometric phase at the critical measurement strength is
completely washed out. Our numerical findings confirms that the quantization
and stochasticity of ($\mathcal{X}$) not only depends on the measurement
strength $c$, but also on the total number of measurements $N$ and its
interplay with the $c$-parameter, with fully stochastic character only for
$N>4$, when considering $c_{\mathrm{crit}}=2.1$ ($W=1$).

###  IV.2 $N-c$ landscape

In order to analyze the interplay between the quantization of the geometric
phase $\mathcal{X}$, for different values of total number of measurements $N$
and strength parameter $c$, we numerically calculated $\mathcal{X}$ in
different relevant regions of the $N-c$ landscape. This is displayed in the 3D
density plot depicted in Fig. 5. Figure 5 (a), (b) and (c) correspond to
parameter values of $0<N<50$ and $0<c<10$, for $W=1,2,3$ respectively.
Clearly, it is observed that the stability of the geometric phase is
dramatically reduced for $N<20$ (magenta regions), meaning that fluctuations
in the geometric phase are readily apparent for $N$ of the same order as $c$
($N\approx c$). This numerical finding also sets a limit in the actual values
of measurement strength $c$ that can be accepted in the quasicontinuous limit.
Figures 5 (d), (e) and (f) present numerical simulations of $\mathcal{X}$
considering $100<N<500$ and $0<c<10$, for $W=1,2,3$ respectively. We find that
for $N>100$, corresponding to $N>>c$, the fluctuations in the geometric phase
vanish, meaning that in this region of the $N-c$ landscape the analytic result
is fully valid, as reported in 2 . Fluctuations are expected to resurge as $c$
approaches values of the same order of $N$ ($N\approx c$) 19 , or for values
larger than $N$ ($N<c$). This is analyzed in detail in Fig. 5 (g), (h), (i),
for $100<N<500$ and $100<c<400$, for $W=1,2,3$, respectively. We note that
there is no clear critical measurement-strength parameter in this region where
$\mathcal{X}$ makes a discrete jump, as a result of the limited resolution in
$c$ required in order to survey such a large portion of the $N-c$ landscape.
Nevertheless, it is observed that fluctuations in $\mathcal{X}$ arise for
$c>300$, for any value of $N$, meaning that the region of validity of the
analytical result can only be considered if $N>>c$. In other words, $N>c$ is
not acceptable for the analytic result to hold. Evidently, there is no value
of $N$ where the analytic result holds for $c>300$. This upper limit is
relevant when analyzing the strong-measurement limit $c\rightarrow\infty$, as
considered in Snizkho _et al._ 19 , and for feasible experimental realizations
of the scheme 21 ; 22 .

###  IV.3 Phase Noise

In order to characterize the robustness of the measurement protocol to
experimental errors arising as a result of phase fluctuations due to limited
temporal stability of the system, which can result for example as a
consequence of temperature, spatial, frequency, or polarizations drifts, among
other realistic sources of errors. We analyze the effect of uncorrelated phase
noise on the geometric phase $\mathcal{X}$ in the $N-c$ landscape, although
other forms of correlated noise could be considered, for instance noise
linearly or polynomially correlated with the total number of measurements $N$,
and the measurement strength $c$. We model the phase noise $\Delta\alpha$ with
a normal distribution centred around $\alpha_{0}=\pi$, with varying spreads
$\Delta\alpha=2\%,5\%,10\%$, considering different winding numbers for the
polar angle $W=1,2,3$. Plots of the accumulated phase noise for different
regions of the $N-c$ landscape are presented in Figs 6, 7, and 8. Numerical
simulations of uncorrelated phase noise in the region $0<N<50$ and $0<c<10$
are presented in Fig. 6. Figures 6 (a), (b), (c) correspond to phase noise
with a spread $\Delta\alpha=2\%$ for $W=1,2,3$, respectively. Figures 6 (d),
(e), (f) correspond to phase noise with a spread $\Delta\alpha=5\%$ for
$W=1,2,3$, respectively. Figures 6 (g), (h), (i) correspond to phase noise
with a spread $\Delta\alpha=10\%$ for $W=1,2,3$, respectively. Clearly, for
$\Delta\alpha>2\%$ fluctuations in the phase render the measurement protocol
significantly unstable, and the topological quantization of the phase between
$0$ and $\pi$ is blurred.

Next, numerical simulations of uncorrelated phase noise in the region
$100<N<500$ and $0<c<10$ are presented in Fig. 7. Figures 7 (a), (b), (c)
correspond to phase noise with a spread $\Delta\alpha=2\%$ for $W=1,2,3$,
respectively. Figures 7 (d), (e), (f) correspond to phase noise with a spread
$\Delta\alpha=5\%$ for $W=1,2,3$, respectively. Figures 7 (g), (h), (i)
correspond to phase noise with a spread $\Delta\alpha=10\%$ for $W=1,2,3$,
respectively. Clearly, this region of the $N-c$ landscape is significantly
robust to uncorrelated phase noise, and the (topological) quantization of the
phase is preserved.

Finally, numerical simulations of uncorrelated phase noise in the region
$100<N<500$ and $100<c<400$ are presented in Fig. 8. Figures 8 (a), (b), (c)
correspond to phase noise with a spread $\Delta\alpha=2\%$ for $W=1,2,3$,
respectively. Figures 8 (d), (e), (f) correspond to phase noise with a spread
$\Delta\alpha=5\%$ for $W=1,2,3$, respectively. Figures 8 (g), (h), (i)
correspond to phase noise with a spread $\Delta\alpha=10\%$ for $W=1,2,3$,
respectively. Clearly, in this limit the only region of the $N-c$ landscape
robust to phase noise if for $N>>c$ and $c<300$, we note that there is no
clear critical measurement-strength parameter $c$ in this region where
$\mathcal{X}$ makes a discrete jump, within the numerical resolution required
in order to cover such a large portion of the $N-c$ landscape. Numerical
results considering phase noise confirm the findings in Fig. 5, with the
additional insight that the $N-c$ landscape region $0<N<50$ and $0<c<10$ is
not sufficiently robust to phase noise $\Delta\alpha>2\%$. Thus the only
entirely acceptable region, regarding uncorrelated phase noise, is for $N>>c$
and $c<300$, such as the case considered in Fig. 7, and in PNAS2020 2 .

##  V Discussion

We presented an analytical and numerical study of a novel type of geometric
phase induced by weak measurements. In contrast to the case considered by
Gebhart _et al._ 2 , we consider the dependence of the geometric phase on the
winding $W$ of the polar angle $\varphi$, which quantifies the number of full
$2\pi$-turns the trajectory makes until it closes its path, enabling to define
a geometric phase $\mathcal{X}$. This is accomplished by fixing the azymhutmal
angle at $\theta=\pi/2$ for a sequence of weak measurements in the angle
$\varphi$ of increasing magnitude $\varphi_{k}=\epsilon k$, with $k=1,..,N$
the measurement index. We ensure the trajectory induced by the sequence of
weak measurements is closed, and the geometric phase well defined, by
parameterizing the rotation parameter as $\alpha=2\epsilon/N$, where $N$ is
the number of measurements. We find that different winding numbers $W$ can
give rise to unpredictable critical measurement-strength parameters, where the
geometric phase becomes stochastic and the system undergoes a topological
phase transition, thus confirming that the topological phase transitions for
different $W$ are not topologically equivalent. Furthermore, adding to the
novelty of our work, we not only analyze the weak-measurement induced
geometric phase by a full analytic derivation based on the exponential
approximation proposed in 2 , valid for $N\rightarrow\infty$, but also we
analyze the induced geometric phase numerically, thus enabling us to
understand the finite-$N$ interplay of $\mathcal{X}$ with the measurement-
strength parameter $c$, and its stability to fluctuations in the protocol.

In particular, we analyzed the impact of uncorrelated phase noise on the
quantization of the geometric phase. We find that while the parameter region
$N>>c$ of the $N-c$ landscape, characterized by parameter values $100<N<500$
and $0<c<10$, is significantly robust to phase noise up to $\Delta\alpha\leq
10\%$, the parameter region $0<N<50$ and $0<c<10$ can only support phase
fluctuations within $\Delta\alpha<2\%$. For $c$ of the same order as $N$
($N\approx c$), or $c>300$, we find no critical measurement-strength parameter
where the geometric phase becomes stochastic, within the resolution of the
numerical simulations. Other models for correlated noise, for instance phase
noise increasing with the number of measurements $N$ and/or with the
measurement-strength parameter $c$ will be considered in a forthcoming work.

We argue that the exhibited sharp transition in the acquired geometric phase
indicates that our protocol could find relevant applications in measurement-
induced manipulation and control of quantum states, for instance via the
implementation of polarization switches or C-NOT gates, among other potential
methods for quantum information processing. More specific, by tuning the
coupling parameter $\eta$, for instance by increasing or decreasing the
integration time $T$, which determines by the level of coupling between the
detection qubit and the system qubit, it is possible to trigger a sharp
transition in the geometric phase, which can be used as a quantum switch to
manipulate and control quantum systems, with high precision. Our findings also
have repercussions on the understanding of the foundations of quantum
mechanics and quantum measurement theory itself 20 .

##  VI Acknowledgements

The author is grateful to Yuval Gefen, Kyrylo Snizhko and Alessandro Romito
for many insightful discussions, and significant assistance in the development
of the numerical codes. G. P. acknowledges financial support from
PICT2015-0710 grant and Raices Programme.

## References

  * (1) A. Messiah, Quantum Mechanics (Dover Publications, 1961).
  * (2) V. Gebhart, K. Snizhko, T. Wellens, A. Buchleitner, A. Romito, and Y.Gefen, Topological transition in measurement-induced geometric phases, Proc. Nat. Acad. Sci. 117, 5706-5713 (2020).
  * (3) M. V. Berry, Quantal Phase Factors Accompanying Adiabatic Changes, Proc. of the Royal Soc. of London A 392, 45-57 (1984).
  * (4) Y. Zhang, Y.-W. Tan, H. L. Stormer, and P. Kim, Experimental observation of the quantum Hall effect and Berry’s phase in graphene, Nature 438 201 (2005).
  * (5) C. L. Kane, and E. J. Mele, Topological Order and the Quantum Spin Hall Effect, Phys. Rev. Lett. 95, 146802 (2005); B. Bernevig _et al._ , Science 314, 1757 (2006); M. Köning _et al._ , Science 318, 766 (2007).
  * (6) G. Delacretaz, E. Grant, R. Whetten, L. Wöste, and J. Zwanzinger, Phys. Rev. Lett. 56, 259 (1986).
  * (7) D. Xiao, M. C. Chang, Q. Niu, _Berry phase effects on electronic properties_ , Rev. Mod. Phys. 82, 1959-2007 (2010).
  * (8) S.-L. Zhu, P. Zanardi, Geometric quantum gates that are robust against stochastic control errors, Phys. Rev. A 72, 020301 (2005).
  * (9) Y. Aharonov, J. Anandan, Phase change during a cyclic quantum evolution, Phys. Rev. Lett. 58, 1593-1596(1987).
  * (10) J. Samuel, R. Bhandari, General setting for Berry’s phase, Phys. Rev. Lett. 60, 2339- 2342 (1988).
  * (11) . F. Wilczek, Fractional Statistics and Anyon Superconductivity (World Scientific Publishing, Singapore, 1990).
  * (12) K. T. Law, D. E. Feldman, Y. Gefen, Electronic Mach-Zehnder interferometer as a tool to probe statistics, Phys. Rev. B 74, 045319 (2006).
  * (13) C. Nayak, S. H. Simon, A. Stern, M. Freedman, S. Das Sarma, Non-Abelian anyons and topological quantum computation, Rev. Mod. Phys. 80, 1083-1159 (2008).
  * (14) D. J. Thouless, M. Kohmoto, M. P. Nightingale, M. den Nijs, Quantized Hall con- ductance in a two-dimensional periodic potential, Phys. Rev. Lett. 49, 405-408 (1982).
  * (15) B. A. Bernevig, Topological Insulators and Topological Superconductors (Princeton University Press, 2013).
  * (16) J. K. Asboth, L. Oroszlany, A. Palyi, A Short Course on Topological Insulators (Springer International Publishing, 2016).
  * (17) M. Atala, M. Aidelsburger, J. Barreiro, D. Abanin, T. Kitagawa, E. Demler, I. Bloch, Direct measurement of Zak phase in topological Bloch bands, Nature Phys. 9, 795 (2013).
  * (18) S. Longhi, Measuring topological invariants in photonic systems, Opt. Lett. 38, 3716 (2013).
  * (19) J. Loredo, M. Broome, D. Smith, and A. White, Observation of Entanglement-Dependent Two-Particle Holonomic Phase, Phys. Rev. Lett. 112, 143603 (2014).
  * (20) G. Puentes, Spontaneous Parametric Down Conversion and Quantum Walk Topology, JOSA B 33, 0740-3224 (2016).
  * (21) G. Puentes, Topology and Holonomy in Discrete-Time Quantum Walks, Crystals 7, 122 (2017). Selected for Cover Story of Special Issue on Quantum Topology.
  * (22) L. Neves and G. Puentes, Photonic Discrete-Time Quantum Walks using Spatial Light Modulators, Entropy 20, 731 (2018).
  * (23) G. Puentes, 2D Zak Phase Lanscape in Photonic Discrete-Time Quantum Walks, accepted for puliblication in Quantum Rep (2023).
  * (24) S. Pancharatnam, Generalized theory of interference, and its applications, Proc. Indian Acad. Sci. Sect. A 44, 247-262 (1956).
  * (25) M. V. Berry, S. Klein, Geometric phases from stacks of crystal plates, J. Mod. Optic. 43, 165-180 (1996).
  * (26) P. Facchi, A. G. Klein, S. Pascazio, L. S. Schulman, Berry phase from a quantum Zeno effect, Phys. Lett. A 257, 232-240 (1999).
  * (27) K. Snizhko, N. Rao, P. Kumar, and Y. Gefen, Weak- measurement-induced phases and dephasing: Broken sym- metry of the geometric phase, Phys. Rev. Res. 3, 043045 (2021).
  * (28) Y. Wang, K. Snizhko, A. Romito, Y. Gefen, and K. Murch, Observing a topological transition in weak-measurement- induced geometric phases, Phys. Rev. Res. 4, 023179 (2022).
  * (29) M. F. Ferrer-Garcia, K. Snizhko, A. DErrico, A. Romito, Y. Gefen, and E. Karimi, Topological transitions of the generalized Pancharatnam-Berry phase, Sci. Adv. 9, 1 (2023).
  * (30) G. Puentes, A. Datta, A. Feito, J. Eisert, M. Plenio, and I. Walmsley, Entanglement quantification from incomplete measurements: Applications using photon-number resolving detectors, New J. Phys. 12, 033042 (2010) (2010).


# Obtaining Favorable Layouts for Multiple
Object Generation

Barak Battash
Faculty of Engineering, Bar Ilan University Amit Rozner
Faculty of Engineering, Bar Ilan University Lior Wolf
School of Computer Science, Tel Aviv University Ofir Lindenbaum
Faculty of Engineering, Bar Ilan University

###### Abstract

Large-scale text-to-image models that can generate high-quality and diverse
images based on textual prompts have shown remarkable success. These models
aim ultimately to create complex scenes, and addressing the challenge of
multi-subject generation is a critical step towards this goal. However, the
existing state-of-the-art diffusion models face difficulty when generating
images that involve multiple subjects. When presented with a prompt containing
more than one subject, these models may omit some subjects or merge them
together. To address this challenge, we propose a novel approach based on a
guiding principle. We allow the diffusion model to initially propose a layout,
and then we rearrange the layout grid. This is achieved by enforcing cross-
attention maps (XAMs) to adhere to proposed masks and by migrating pixels from
latent maps to new locations determined by us. We introduce new loss terms
aimed at reducing XAM entropy for clearer spatial definition of subjects,
reduce the overlap between XAMs, and ensure that XAMs align with their
respective masks. We contrast our approach with several alternative methods
and show that it more faithfully captures the desired concepts across a
variety of text prompts.

Figure 1: This Figure shows the generation outputs of our method and
competitive methods for multiple prompts, with various amount of subjects and
objects.

##  1 Introduction

In recent years, diffusion probabilistic models have garnered considerable
attention from researchers across both academia and industry due to their
remarkable performance and applicability to a wide range of downstream tasks
related to high-quality image generation. State-of-the-art text-to-image
foundation models, such as Stable Diffusion Rombach et al. (2022), Imagen
Saharia et al. (2022), and DALL-E Parmar et al. (2023), are predominantly
based on diffusion models.

Diffusion models operate by iteratively denoising a noise-perturbed input
image. Notably, the Stable Diffusion model Rombach et al. (2022) has showcased
impressive capabilities in generating diverse and realistic images,
underscoring the efficacy of diffusion-based approaches in image synthesis
tasks.

There are, however, a few eminent semantic challenges within such text-to-
image synthesis, three of which are “subject neglect”, “subject blending” and
“attribute binding”. The first pertains to generated samples where not all
subjects described in the input text are present in the resulting image. The
second is where the subjects are blended into other subjects, for example a
bear with elephant trunk. “Attribute binding” is a compositional issue,
wherein attribute details, such as color or texture, are either misaligned
with their intended objects or mistakenly associated with unrelated objects.

This work aims to address the challenges of subject neglect and subject
blending, which become more complicated as the number of subjects increases,
see Figure 1. Although we do not attempt to solve the problem of attributed
binding, our approach can be combined with any other attribute-binding method
and to show better or compatible results.

Well-structured layout is required to avoid subject neglect and blending. If
the layout is such that the subjects are well-separated and no subject
dominates the scene, the diffusion methods can generate desirable images. Our
research hypothesis is that given an initial noise map $z_{T}$ the diffusion
model has bias towards some favorable layouts. Thus, manipulating the latent
map is important as manipulating the attention maps.

The layout is determined in approximately the initial 15 diffusion steps, but
is not apparent during these steps. To mitigate this, our proposed solution
has three phases. During the first few denoising steps, we apply various loss
terms to encourage the XAMs associated with the tokens of each subject to be
excited but separate from other subjects. Our solution to this is considerably
more elaborate than previous attempts since we rely on the spatial structure
and not just on the maximal activation.

Then, after these initial steps, we extract binary masks per object and revise
these masks to obtain favorable layouts. This is done by shifting some of the
objects. The latent space of the diffusion model is then readjusted to match
the optimized masks.

Finally, diffusion continues, but the process is driven such that the per-
subject XAMs match the set of fixed masks we have previously generated.
Overall, the method provides a comprehensive solution to the challenges of
multi-subject generation across all diffusion steps, all subjects, and the
various spatial locations.

In an extensive set of experiments, we demonstrate that our proposed method
outperforms in all established metrics the many baselines that exist in this
field by a sizable margin. Quantitative results demonstrate that these
qualitative advantages match the obtained visual improvement. Figure 1
presents several examples of images generated using competitive methods.
Widely used text-to-image diffusion based generative models struggle to
generate an image containing all the objects in the prompt. Our model improves
the image layout and leads to generated images that are more faithful to the
prompt.

##  2 Related Work

The groundbreaking potential of generative AI has been unleashed by cutting-
edge foundation models, such as Stable Diffusion Rombach et al. (2022), Imagen
Saharia et al. (2022), Midjourney, and DALL-E. They serve as the backbone for
various generative AI applications, which include image, video, and 3D object
generation. These models typically operate in a latent space, mapping from a
low-dimensional latent representation to high-dimensional images. Diffusion
models have recently gained attention for their ability to generate high-
quality images by iteratively denoising a noise-perturbed input image. Notable
work includes the Stable Diffusion model Rombach et al. (2022), which
demonstrated impressive results in generating diverse and realistic images.
Diffusion models offer advantages, such as stable training dynamics and the
ability to generate high-resolution images, making them a promising choice for
image synthesis tasks. The accuracy of the generated image is determined by
its adherence to the user-provided text prompt. While existing generative
models excel at synthesizing single-object images, generating images with
multiple entities poses a more substantial challenge.

Previous works have attempted to enhance the accuracy of text-to-image
diffusion models by improving their semantic faithfulness. One of the major
challenges in this regard has been aligning the image outputs with the input
prompts, which has been thoroughly discussed in the paper by Tang et al. Tang
et al. (2022). To address this issue, Kim et al. Liu et al. (2022) introduced
ComposableDiffusion, a method that allows users to incorporate conjunction and
negation operators in prompts. This approach improves concept composition
guidance. Further advancements were made with the development of
StructureDiffusion Feng et al. (2022), which suggests segmenting the prompts
into noun phrases for exact attention distribution. Wu et al. Wu et al. (2023)
proposed a method for controlling the XAMs with a spatial layout generation
predictor. Agarwal et al. Agarwal et al. (2023) proposed A-star to minimize
concept overlap and change in attention maps through iterations. Kim et al.
Kim et al. (2023) introduced DenseDiffusion, a method for region-specific
textual feature accumulation.

The recent paper in Chefer et al. (2023) aimed to improve attention to
neglected tokens, while Li et al. (2023a) suggested two distinct customized
objective functions to handle the issues of missing objects and incorrect
attribute binding separately. Although these approaches make an effort to
resolve the mentioned problems, they still fail multiple object generation.
MultiDiffusion Bar-Tal et al. (2023) combines multiple diffusion generation
processes for each token, which helps overcome multi-subject generation
difficulties. While MultiDiffusion Bar-Tal et al. (2023) requires a bounding
box for each object in the prompt, which is a significant advantage over our
approach and other baselines, it might be capable of overcoming hurdles
associated with multi-subject generation. The Attend-and-Excite (A&E) method
Chefer et al. (2023) focuses solely on neglected objects, but struggles to
address the problem effectively when the areas of maximum attention are close.
Conversely, Divide-and-Bind Li et al. (2023a) offers an approach to resolve
the issue of incorrect attribute.

##  3 Method

The stable diffusion process is an iterative procedure with indices from $t=T$
to $t=0$, where $T$ is the total number of steps. At each step, a randomly
sampled latent map $z_{T}$ is provided as input to the UNet, which predicts
the noise estimation and gradually removes it until it produces the clean
version of the latents $z_{0}$.

The latent map $z_{t}\in\mathbb{R}^{4\times h\times w}$ represents the output
at step $t$. The early steps are important for creating the layout, while the
later steps improve the local structure. To condition the text, a prompt
$\mathcal{P}$ consisting of $N$ tokens is used, and the latent image $z_{t}$
is viewed as a grid of $P\times P$ patches. To link each patch to each prompt
token, a XAM $A_{t}\in\mathbb{R}^{P\times P\times N}$ is used. Let $S$ be the
set of all subjects in the prompt, and $s$ be the token index. At step $t$,
$A_{t}^{s}\in\mathbb{R}^{P\times P}$ is the XAM of token $s$ that links each
patch to token $s$.

To synthesize an image that accurately represents a particular subject,
several factors must be considered. First, each subject $s$ should have at
least one patch with a high value in $A_{t}^{s}$, especially during the
initial stages of the generation process. Second, the overlap of the attention
maps of the subject with the other attention maps should be minimal.

Our approach contains three steps:

  1. 1.

Excite and distinguish: Occurs in the first $\tau$ steps: $t\in(T,T-\tau)$,
where $T>\tau$. This step gently compels all tokens $s\in S$ be excited, i.e.,
to have a maximal attention score that is high enough, and to spatially
separate these maps as much as possible.

  2. 2.

Rearrange the generation grid: Occurs once at the end of the first phase. For
each subject, extract for each $s\in S$ spatial mask from the XAMs, and then
spatially rearrange the masks to minimize their overlap, obtaining the desired
masks $M_{s}$.

  3. 3.

Follow the masks: Occurs during all subsequent diffusion steps:
$t\in(T-\tau,0)$ and utilizes the masks from the previous phase to guide the
spatial arrangement in the XAMs.

Figure 2: Illustrated herein is the sequential evolution of XAMs throughout
the generation (backward) process. Commencing on the left with $t=T$, the XAMs
exhibit a high degree of spatial entropy, signifying an unorganized state.
During Phase 1, spanning $t=(T,T-\tau)$, the process strategically
consolidates patches pertaining to identical subjects while concurrently
segregating the XAMs of distinct subjects. The resulting XAMs at $t=\tau$
manifest enhanced organization and concentrated focus, enabling a preliminary
prediction of the subjects’ potential generation loci. Phase 2 involves
optimizing the spatial arrangement and generating masks that will be used in
Phase 3, the masks presented are after Gaussian smoothing. In Phase 3, the
attention maps are subtly coerced to align with predefined masks. The extreme
right column depicts the remainder of the diffusion process, which is
instrumental in mitigating artifacts induced by the optimization process.

Figure 2 illustrates the dynamic interplay of the three phases in shaping the
XAMs, thereby steering the layout of image generation towards superior multi-
subject generation.

Phase I  It is widely recognized Hertz et al. (2022); Voynov et al. (2023)
that the first few diffusion steps, largely determine the layout of the
generated image. Therefore, if the goal is to separate the XAMs of subjects,
it must be done within these initial steps. Unfortunately, during these early
generation steps, it is almost impossible to understand the subject’s location
in the XAMs.

As a starting point, consider the A&E Chefer et al. (2023) objective, which
considers the maximal attention each token $s\in S$ receives and attempts to
minimize their neglect by maximizing the attention of the least attended
token:

$L_{A\&E}=\max_{s\in S}(1-\max(A^{s}_{t}))\,,$ (1)

where the second maximization is over all spatial locations of the XAM.

This objective can lead to subject-blending, since the same spatial location
can have a high cross-attention score to several tokens. In order to prevent
this from occurring, we sort the subject tokens $s$ by their maximal cross-
attention values $\max(A^{s}_{t})$ in a descending manner.

Let $s_{0},s_{1},s_{2},...$ be the list of sorted tokens at time $t$. We
define $B_{t}^{s_{0}}$ to be the map that is zero everywhere except for a
fixed-sized rectangle around the maximal values of $A_{t}^{s_{0}}$. We then
consider the filtered XAM
$\tilde{A}_{t}^{s_{1}}=A_{t}^{s_{1}}\odot(1-B_{t}^{s_{0}})$, where $\odot$
marks an element-wise multiplication. We identify the maximal value in the
filtered XAM, i.e., the maximal value that is at a distance from the maximal
attention value in $A_{t}^{s_{0}}$. Let $B$ be a binary mask around the point
of highest attention in $\tilde{A}_{t}^{s_{1}}$. We define $B_{t}^{s_{1}}$ to
be the union (maximum value) of $B_{t}^{s_{0}}$ and $B$.

The process repeats, each time accumulating more regions in which the next
token’s original attention is masked out, see Figure 3 for an illustration.

The refined A&E loss term we employ includes these spatial considerations and
is given as:

$L_{B\&E}=\max_{s_{i}\in S}(1-\max((1-B^{s_{i-1}}_{t})\odot A^{s}_{t}))\,,$
(2)

where $B_{t}^{s_{-1}}=0$ and the second maximization is across all spatial
locations as above.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 3: The computation of the blocking masks $B_{t}^{s}$. The subject
tokens $s_{i}$ are sorted from largest excitation to smallest. At every step,
the mask $B_{t}^{s_{i}}$ accumulates the masked regions from its predecessors
and adds a rectangle around the location of the maximal value in
$\tilde{A}_{t}^{s}=(1-B_{t}^{s_{i-1}})\odot A_{t}^{s_{i}}$. Note (h) will not
be used.

This loss term $L_{B\&E}$ is effective in having each XAM develop a high value
that is spatially far from the high values of other XAMs. However, it does not
prevent much of the high values of one XAM to overlap those of other XAMs. To
mitigate this, we add a direct loss term per subject token $s$

$L^{s}_{ol}=\frac{1}{|S|-1}\sum_{s^{\prime}\in S\setminus
s}\braket{\bar{A}_{t}^{s},\bar{A}_{t}^{s^{\prime}}}_{F},$ (3)

where $\bar{A}_{t}^{s}$ represents a dilated version (with a 3x3 kernel) of
$A_{t}^{s}$,$\braket{}_{F}$ is the fourbinous inner product the sum over
$p_{i},p_{j}$ sums across both spatial dimensions. The dilation is applied to
prevent the diffusion process from erroneously locating subjects within small
gaps or holes that may be present in the image grid.

Lastly, diffusion models may generate large subjects, leading to multiple
subject generation failure due to lack of grid space. To address this issue,
we employ conditional norm regularization, defined as follows:

$L^{s}_{norm}=[\|A_{t}^{s}\|_{F}>C]\|A_{t}^{s}\|_{F},$ (4)

where $C=\frac{P^{2}}{|S|}$ accounts for the size of the image as well as the
number of subjects. The overall loss term in the first phase is:

$L_{I}=\lambda_{B\&E}L_{B\&E}+\sum_{s\in
S}\lambda_{ol}L^{s}_{ol}+\lambda_{norm}L^{s}_{norm}.$ (5)

Where $\lambda_{B\&E},\lambda_{ol},\lambda_{norm}$ are scaling factors.

Phase II  After the first $\tau$ steps, phase II generates a set of per-
subject masks $M_{s}$. It also rearranges the latent space $z_{\tau}$ to be
compatible with the new masks.

The initial masks $\bar{M}^{s}$ are obtained by thresholding the XAMs at step
$t=\tau$. We use the following equation to define our masks:

$\bar{M}^{s}[i,j]=\begin{cases}1&A_{\tau}^{s}>\gamma\max(A_{\tau}^{s}),\\\
0&\text{otherwise}\end{cases}$ (6)

Where $i,j$ are the spatial indexes. The value of $\gamma$ is set to ensure
that the masks are neither too large nor too small, with the default value
being 0.2. It is then adjusted automatically within the range of
$\gamma\in(0.2,0.8)$, see supplementary material for the details.

We next rearrange the masks. We have learned that it is best to move as few
objects as possible, and that an object should never be moved above its
current location, otherwise, nonphysical scenarios are obtained, which SD
cannot generate faithfully.

Figure 4: This figure illustrates Phase2 of the process. On the left, three
masks represent the three subjects. In the middle column, we observe the
initial masks, which estimate the patches in the attention maps contributing
to each subject. On the right, we observe the final masks, after they have
been shifted to their new locations. These final masks will subsequently guide
the shifting of attention maps $A_{t}^{s}$ towards their new location.

The two objects that are moved are chosen based on the size ratio between the
overlapping regions with other object over their own size is the highest. The
latter is simply the sum of the mask $\bar{M}^{s}$. The former is the sum
(over all image locations) of the intersection of all overlaps of the form
$\bar{M}^{s}\cap\bar{M}^{s^{\prime}}$ between the initial mask of subject $s$
and that of $s^{\prime}$.

To determine the new location of a subject $s$, we shift the mask
$\bar{M}^{s}$ spatially, each time computing the total overlap as above, and
selecting the shift that minimizes this quantity. The final set of masks is
denoted $M^{s}$. Please refer to Figure 4 for more intuition.

To ease subject reallocation, we rearrange the patches in $z_{\tau}$. Since we
know the shift each of the objects we moved has (the result of the above 2D
search), we first shift the corresponding locations in $z_{\tau}$. We note
that the size of the masks is four times smaller in each dimension than the
size of the latent, thus we used an upscaled version of $\bar{M}_{s}$.

Shifting requires copying the representation in a certain spatial location to
a new location. The original location can remain vacant if no other pixel is
copied there. In this case, we use one of two imputation techniques: (1) if
there is no background in the prompt, we impute a random vector from the
normal distribution, else (2) we copy the latent activation of a the top-k
locations from the background token $s_{\text{bkg}}$ XAM
$A_{T-\tau}^{s_{\text{bkg}}}$.

|
---|---
|
Figure 5: This figure displays the output generated by our method in response
to the prompt: "A chicken and a duck with a ball at the beach." and "A dog, a
cat, and a bear at the beach". The three rightmost images depict the attention
maps at step 9, a pivotal moment in the generation process that significantly
influences the layout of the generated image. This step was specifically
chosen to highlight its critical role in determining the spatial arrangement
of the depicted entities. This visualization helps us analyze the importance
of separating the attention maps.

Phase III  In this phase of the generation, we encourage $A_{t}^{s}$ to follow
the masks $M^{s}$. This is done using two objectives. The first objective
keeps the object inside its mask, and it is formulated as:

$L_{\text{inside}}=\frac{1}{|S|}\sum_{s\in
S}(1-\frac{\braket{A_{t}^{s},M^{s}}_{F}}{\|A_{t}^{s}\|_{1}})^{2}.$ (7)

The second objective is to try and have the subject’s XAM fill the object’s
mask:

$\quad L_{fill}=\frac{1}{|S|}\sum_{s\in
S}(1-\frac{\braket{A_{t}^{s},M^{s}}_{F}}{\|M^{s}\|_{1}})^{2}.$ (8)

The loss term of this phase combines both:

$L_{III}=\lambda_{inside}L_{inside}+\lambda_{fill}L_{fill},$ (9)

Where $\lambda_{inside},\lambda_{fill}$ are scaling factors.

##  4 Experiments

Table 1: This table presents the evaluation results on three test sets
comprising only animals. Performance metrics were assessed using the Llava1.5
and Qwen models.

Method Two animals Three animals Four animals c-score l-score q-score c-score
l-score q-score c-score l-score q-score Stable Diffusion Rombach et al. (2022)
0.79 0.83 0.64 0.59 0.63 0.60 0.35 0.55 0.53 CompDiffusion Liu et al. (2022)
0.81 0.84 0.79 0.79 0.75 0.72 0.61 0.61 0.63 Divide & Bind Li et al. (2023a)
0.76 0.92 0.92 0.65 0.84 0.83 0.57 0.76 0.69 Attend & Excite Chefer et al.
(2023) 0.82 0.92 0.90 0.81 0.79 0.73 0.72 0.71 0.64 SDXL Podell et al. (2023)
0.62 0.94 0.90 0.45 0.83 0.77 0.15 0.80 0.65 MultiDiffusion Bar-Tal et al.
(2023) 0.72 0.92 0.92 0.76 0.73 0.70 0.39 0.57 0.50 Ours 0.92 0.97 0.97 0.92
0.90 0.91 0.85 0.86 0.88

###  4.1 Data

Our experiment’s methodology expands the A&E Chefer et al. (2023) benchmark by
examining text-to-image models in more extreme cases. We use prompts with at
least two subjects as templates for the benchmark. These are the templates
that were used for the benchmark: (i) “a [animalA] and a [animalB]”, (ii) “a
[animalA], a [animalB] and a [animalC]”, (iii) “a [animalA], a [animalB], a
[animalC] and a [animalD]”, (iv) “a [animalA], a [animalB] and a [object]”,
and (v) “a [animalA], a [animalB], a [animalC] and a [object]”.

Although this work focuses only on “subject neglection” and “subject
blending", we want to demonstrate that solving other issues in image
synthesis, such as attribute binding, can be easily combined with our method.
We utilized only the “Bind” component from Divide-and-Bind Li et al. (2023a)
and incorporated it to optimize the latents in conjunction with our suggested
loss during phase III. We analyzed two sets of text prompts: (1) “a
[colorA][objectA] and a [colorB ][objectB]”, as presented in A&E Chefer et al.
(2023). (2) “a [colorA][objectA], a [colorB][objectB] and a
[colorC][objectC]”. In total, all methods were evaluated using over 700
prompts, each generated using 5 different seeds.

###  4.2 Implementation details

We use the official Stable Diffusion v2.1 text-to-image model. We use default
hyperparameters for all models. All experiments were conducted using an Nvidia
Tesla v100 32GB GPU.

###  4.3 Baselines

We compared our model to several relevant baselines, including Vanilla SD2.1,
Attend-and-Excite Chefer et al. (2023), Divide-and-Bind Li et al. (2023a),
Composable DiffusionLiu et al. (2022) and MultiDiffusion Bar-Tal et al.
(2023). Evaluating MultiDiffusion Bar-Tal et al. (2023) on a large-scale
benchmark is challenging, as it requires the user to input the bonding box. To
ensure a fair evaluation, we provided the model with three sets of masks that
were of a standard size and could be used for generation in a fair manner.
Finally, we report the best result out of these three sets of masks.

Despite implementing more advanced training methodologies, leveraging larger
neural networks and using larger training data, generating multiple subjects
remains a challenge. Unfortunately, due to hardware constraints, we were
unable to test our method on SDXL Podell et al. (2023). However, we will use
it to demonstrate the persistent challenge of generating multiple subjects. We
used the full pipeline of SDXL Podell et al. (2023), both the base model and
the refiner, all done using DiffusersWolf et al. (2019) package.

Figure 6: This figure shows the outputs of each method examined in this work
in response to an input prompt containing four subjects. Other methods suffer
from subject blending or are unable to generate all four subjects. In
contrast, our method generates an image that is more loyal to the prompt.
Table 2: This table shows the results on the set of prompts that consists of
subjects including objects.

Method Two animals and an object Three animals and an object c-score l-score
q-score c-score l-score q-score Stable Diffusion Rombach et al. (2022) 0.77
0.75 0.71 0.59 0.65 0.64 Composable-Diffusion Liu et al. (2022) 0.78 0.75 0.73
0.75 0.69 0.67 Divide & Bind Li et al. (2023a) 0.70 0.81 0.81 0.62 0.72 0.70
Attend & Excite Chefer et al. (2023) 0.85 0.86 0.82 0.66 0.71 0.70 SDXL Podell
et al. (2023) 0.55 0.88 0.8 0.46 0.81 0.74 MultiDiffusion Bar-Tal et al.
(2023) 0.63 0.61 0.58 0.61 0.58 0.57 Ours 0.92 0.93 0.86 0.85 0.87 0.83

Table 3: This table presents the evaluation results on the five test sets evaluated by BLIP2 image-text-matching metric. Each column depicts a different composition of subjects. A=Animal, I=inanimate. Method | 2$\times$A | 2$\times$A+I | 3$\times$A | 3$\times$A+I | 4$\times$A
---|---|---|---|---|---
Stable Diffusion Rombach et al. (2022) | 0.810 | 0.932 | 0.854 | 0.866 | 0.932
Compos-Diffusion Liu et al. (2022) | 0.969 | 0.972 | 0.955 | 0.976 | 0.951
Divide& Bind Li et al. (2023a) | 0.942 | 0.978 | 0.965 | 0.982 | 0.960
Attend & Excite Chefer et al. (2023) | 0.957 | 0.985 | 0.968 | 0.991 | 0.982
SDXL Podell et al. (2023) | 0.941 | 0.9275 | 0.934 | 0.990 | 0.982
MultiDiffusion Bar-Tal et al. (2023) | 0.926 | 0.734 | 0.903 | 0.863 | 0.846
Ours | 0.971 | 0.991 | 0.997 | 0.991 | 0.998
Table 4: This table presents results for a set of prompts comprising subjects
and their associated attribute. It evaluates the model’s ability to correctly
bind attributes to their respective subjects.

Method Two subjects Three subjects Stable Diffusion Rombach et al. (2022) 0.71
0.68 Divide & Bind Li et al. (2023a) 0.88 0.83 Attend & Excite Chefer et al.
(2023) 0.90 0.77 Ours 0.93 0.87

###  4.4 Evaluation Metrics

: Thanks to the rapid advancements in the field of vision-language models, we
have the capability to assess them automatically. In our study, we utilized
Llava1.5 Liu et al. (2023) alongside the newly introduced QWEN-VL-Chat Bai et
al. (2023) with 9.6B parameters, as well as BLIP2Li et al. (2023b). We used
QWEN Bai et al. (2023) to describe the animals or objects it identifies,
requiring grounding through bounding boxes. This facilitated the creation of a
metric, $q$-score, calculated as the ratio of detected subjects to the total
subjects in the prompt. For Llava1.5 Liu et al. (2023), we prompted the model
to identify subject $s$ in the image if it was clearly visible and not blended
with the background, for each subject in the prompt. This evaluation, which we
term the $l$-score, is comparatively less stringent than the QWEN Bai et al.
(2023) evaluation and is formulated similarly to the $q$-score. The full
prompt appears in the appendix. We utilize Llava1.5 Liu et al. (2023) to
extract an additional metric, which will be employed to monitor the numerical
quantity of subjects or objects generated by a model. If Llava1.5 Liu et al.
(2023) detects a subject or object, we prompt the model once more, inquiring
if there are multiple instances. We refer to this metric as the “$c$-score",
where 1 represents the highest count and 0 indicates the lowest. This score
serves not as a simple count but rather as a metric to ensure that the model
generates the appropriate number of objects. Its discriminatory power is
crucial in distinguishing between models.

Table 5: This table presents the evaluation results of the ablation study on
two test sets, which include three or four subjects. The first part of the
table shows ablation experiments on the objectives in Phase 1. The second part
shows the influence of the enhancements components applied in Phase 2, where
“No restart” means the diffusion process is not restarted after the pixel
reallocation. The third block examines the effect of the objectives in Phase
3.

Method Three subjects Four subjects c-score l-score q-score c-score l-score
q-score No $L_{ol}$ 0.91 0.88 0.85 0.83 0.84 0.83 No $L_{B\&E}$ 0.80 0.82 0.82
0.77 0.78 0.81 No $L_{B\&E}$ & $L_{ol}$ 0.78 0.79 0.81 0.71 0.74 0.76 No pixel
reallocation 0.89 0.88 0.87 0.84 0.83 0.86 No restart 0.91 0.88 0.89 0.84 0.84
0.88 No $L_{fill}$ 0.85 0.85 0.85 0.79 0.78 0.79 No $L_{inside}$ 0.84 0.85
0.85 0.82 0.80 0.83 Full method 0.92 0.90 0.91 0.82 0.85 0.89

Additionally, we employed BLIP2 Li et al. (2023b) for image-text matching
(ITM), which assesses the likelihood that the given image and text correspond.
This process was facilitated using the Lavis package Li et al. (2023c).

We evaluate the attribute binding experiment using Llava1.5 Liu et al. (2023),
we prompt the model with two questions: “Do you see a {color} {obj} in the
image? is it clearly seen?” and the second is a sequential and harder prompt:
“Do you see a {obj} in the image? If {obj} exists what is its color?”.

###  4.5 Results

Table 1 and Table 2 present the results on five datasets with a varying number
of subjects and objects. In the “Two animals” benchmark it is not surprising
to see that our method is better by a small gap. Two subjects are what most of
the methods improving image generation models attempt and succeed in solving.
One can note that as the number of subjects in the prompt increases our method
has the lowest decrease in performance, and the gap between the methods
increases. Further than that our model keeps its numeric generation stability
by keeping a $c-score$ of 0.85 even in the Four subjects benchmark.
MultiDiffusion Bar-Tal et al. (2023) also suffers from a low $c-score$, which
expresses the extent to which the model forcibly generates humans even if they
are not needed. Attend-and-Excite Chefer et al. (2023) and Divide-and-Bind Li
et al. (2023a) show decent performance along all evaluation metrics.

Table 3 shows the results on all five benchmarks, using BLIP2 image-text
matching metric. While these metrics are more crude (even nonmatching image-
prompt pairs often score high if there is some common element), our method is
still showing the best performance.

Finally, Table 4 presents the results of the attribute binding experiment, as
discussed before, this experiment’s goal is to show how well our method can
perform by combining methods that tackle different issues. As can be seen, our
method leveraging only the “bind” outperforms all other methods and improves
Divide-and-Bind Li et al. (2023a) attribute binding abilities.

####  4.5.1 Qualitative Results

In the next experiment, we illustrate visually the challenges in generating
multiple subjects using strong text-to-image diffusion models. Figure 1
displays a comparison of various baseline models, highlighting their common
issues of subject neglection or blending.

For instance, SD2.1 suffers mainly from subject neglection and less from
subject blending. Similarly, in the third row, "A gray dog, a brown cat, and a
pink bow", the bow’s color seems to bleed into the cat, indicating a blending
issue. The D&B suffers from a strong subject blending; for example, in the
first row, there is a dog with a frog head; on the right, there is half dog,
half cat. A&E Chefer et al. (2023) Also suffers from two drawbacks: the first
two show a blend between a goat and a duck, and the last row shows no chicken.
It is interesting to see that SDXL Podell et al. (2023) sometimes collapses to
"cartoon mode" when the prompt needs too many compositions. SDXL Podell et al.
(2023) suffered from subject blending; this can be seen in the second row,
where there are ducks with some frog attributes like greenish color and low
and wide center of gravity. SDXL also tends to neglect objects, for example,
in the first row where the model did not generate a dog and generated some
green undefined subject instead of a frog. Our method can better prevent the
blending of subjects and enhance the overall performance of subject appearance
in the scene. We note that we did not include MultiDiffusion Bar-Tal et al.
(2023) since it was not able to generate reasonable results.

Figure 5 shows the output produced by our method in response to the prompt: "A
chicken and a duck with a ball at the beach." and "A dog, a cat, and a bear at
the beach." The three images on the right display the attention maps at step
9, a crucial point in the generation process that greatly impacts the layout
of the resulting image. We specifically selected this step to emphasize its
vital role in shaping the spatial arrangement of the depicted objects. This
visualization aids in understanding the significance of individual attention
maps.

In Figure 6 we present a qualitative comparison between our method and all
baselines. We generated images using the same prompt containing four subjects.
Each row shows the generated images for different random seeds. As evidenced
by these figures, SD2.1 and MultiDiffusion Bar-Tal et al. (2023) produce
images with blended or incorrect objects. While Attend-and-Excite Chefer et
al. (2023) and Divide-and-Bind Li et al. (2023a) improve image generation
quality, they fail to create all four subjects in one image. SDXL Podell et
al. (2023) generates cartoonish-style images (even though not requested in the
prompt) and also generates images that do not match the set of objects in the
prompt. However, our model can create an improved image layout that
accommodates all four objects.

###  4.6 Searching for a Simpler Solution

Recent advancements in large-scale language models (LLMs) have emphasized the
importance of input prompts in influencing model outputs. Stable Diffusion
online communities consist of users who use and design prompts more frequently
than the researchers who developed the models. We want to embrace their
expertise in order to examine much simpler solutions: prompt engineering. We
examined three prompts that were suggested in "stable diffusion" related
forums for multiple character generation, and we will examine the effect of
each prompt on Our method and $A\&E$. Table 6 presents the results; one can
note that all three prompt variations were not able to create an advantage
over the vanilla prompting results presented in Table1.

Table 6: This table presents the evaluation results of using prompt
engineering for better performance using four and three subject sets.
promptA="{prompt}, cinematic wide establishing shot", promptB="A photo of
{prompt} taken from a distance. ", promptC="A photo of {prompt} taken from a
distance, full body"

Method Four subjects c-score l-score q-score SD2.1 promptA 0.56 0.63 0.59 A&E
promptA 0.64 0.73 0.65 Ours promptA 0.83 0.84 0.86 SD2.1 promptB 0.58 0.63
0.59 A&E promptB 0.67 0.74 0.66 Ours promptB 0.83 0.82 0.85 SD2.1 promptC 0.61
0.65 0.61 A&E promptC 0.65 0.73 0.65 Ours promptC 0.81 0.81 0.85

###  4.7 Ablation Study

An ablation study was conducted to investigate the effect of removing each
component. In our method, we made several design choices which we examined
using an ablation across all phases. In Phase 1, we analyzed $L_{B\&E}$ and
$L_{ol}$. In Phase 2, we examined the pixel repositioning in the latents, as
well as restarting the diffusion process statistics. Finally, in Phase 3, we
checked the impact of No $L_{fill}$ and $L_{inside}$. The results of the
ablation study are presented in Table 5.

The findings suggest that reducing the two losses from phase1 simultaneously
results in a significant decrease in performance. When $L_{ol}$ alone is
reduced, the effect is less than when $L_{B\&E}$ is reduced, but it still has
a substantial impact. The enhancements from Phase 2 have a less pronounced
effect on the results, but they still contribute to improved performance.

The objectives of Phase 3 also have a significant impact on performance.

Figure 7: Limitations: the figure presents a few examples of our method and
stable diffusion 2.1 generating scenes that feature multiple subjects and
objects. Although these cases present multiple objects, their arrangement is
unnatural, which may result from imperfect mask generation. Additionally, the
image quality may be lacking. Improving the mask generation process is crucial
to achieving more successful image generation. To enhance image quality, one
can reduce the number of optimization iterations or use a refiner model, such
as in the case of SDXL.

##  5 Limitations

There are a few limitations to consider when using our proposed framework.
Firstly, it increases the inference time of the synthesis model, although only
by a factor of two. However, since rapid generation times are crucial in this
field, this limitation is worth taking into account.

Another limitation is that forcing a layout on the diffusion model can impact
the image quality negatively. In some cases, the resulting layout may even
appear unnatural, as shown in Figure 7 for some lower-quality samples. While
this effect is usually minor, it does create a tradeoff between staying true
to the input prompt and producing high-quality output images.

Lastly, it is worth noting that the method is not aware of subject
proportions. As a result, it may generate a small mask for a large subject,
and vice versa.

##  6 Conclusions

While diffusion models are extremely powerful, they suffer from the same
shortcut issue that plagues classifiers and other deep networks Geirhos et al.
(2020); Hendrycks et al. (2021). In the case of diffusion models, this issue
manifests itself as the problem of neglecting and blending subjects and
tokens. Naturally, the problem becomes more severe as the prompt becomes
increasingly complex. In this work, we offer a multi-step solution that
handles multiple aspects of the generation process, including cross-attention
and the latent space. The intervention we perform combines pulling the
generation process, in inference time, using various loss terms, and more
direct editing of the latent space. As we show in an extensive set of
experiments, our model enhances the ability of text-to-image diffusion models
to generate images with multiple subjects (objects). Although our method does
not attempt to improve the attribute binding phenomena, it is naturally aid
this issue. The ability to clearly generate multiple subjects and objects is a
critical ability of text-to-image models. This capability enables them to
create complex scenes effectively.

## References

  * Rombach et al. (2022) R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10 684–10 695.
  * Saharia et al. (2022) C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans _et al._ , “Photorealistic text-to-image diffusion models with deep language understanding,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 36 479–36 494, 2022.
  * Parmar et al. (2023) G. Parmar, K. Kumar Singh, R. Zhang, Y. Li, J. Lu, and J.-Y. Zhu, “Zero-shot image-to-image translation,” in _ACM SIGGRAPH 2023 Conference Proceedings_ , 2023, pp. 1–11.
  * Tang et al. (2022) R. Tang, L. Liu, A. Pandey, Z. Jiang, G. Yang, K. Kumar, P. Stenetorp, J. Lin, and F. Ture, “What the daam: Interpreting stable diffusion using cross attention,” _arXiv preprint arXiv:2210.04885_ , 2022.
  * Liu et al. (2022) N. Liu, S. Li, Y. Du, A. Torralba, and J. B. Tenenbaum, “Compositional visual generation with composable diffusion models,” in _European Conference on Computer Vision_. Springer, 2022, pp. 423–439.
  * Feng et al. (2022) W. Feng, X. He, T.-J. Fu, V. Jampani, A. Akula, P. Narayana, S. Basu, X. E. Wang, and W. Y. Wang, “Training-free structured diffusion guidance for compositional text-to-image synthesis,” _arXiv preprint arXiv:2212.05032_ , 2022.
  * Wu et al. (2023) Q. Wu, Y. Liu, H. Zhao, T. Bui, Z. Lin, Y. Zhang, and S. Chang, “Harnessing the spatial-temporal attention of diffusion models for high-fidelity text-to-image synthesis,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 7766–7776.
  * Agarwal et al. (2023) A. Agarwal, S. Karanam, K. Joseph, A. Saxena, K. Goswami, and B. V. Srinivasan, “A-star: Test-time attention segregation and retention for text-to-image synthesis,” _arXiv preprint arXiv:2306.14544_ , 2023.
  * Kim et al. (2023) Y. Kim, J. Lee, J.-H. Kim, J.-W. Ha, and J.-Y. Zhu, “Dense text-to-image generation with attention modulation,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 7701–7711.
  * Chefer et al. (2023) H. Chefer, Y. Alaluf, Y. Vinker, L. Wolf, and D. Cohen-Or, “Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models,” _ACM Transactions on Graphics (TOG)_ , vol. 42, no. 4, pp. 1–10, 2023.
  * Li et al. (2023a) Y. Li, M. Keuper, D. Zhang, and A. Khoreva, “Divide & bind your attention for improved generative semantic nursing,” _arXiv preprint arXiv:2307.10864_ , 2023.
  * Bar-Tal et al. (2023) O. Bar-Tal, L. Yariv, Y. Lipman, and T. Dekel, “Multidiffusion: Fusing diffusion paths for controlled image generation,” 2023.
  * Hertz et al. (2022) A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or, “Prompt-to-prompt image editing with cross attention control,” _arXiv preprint arXiv:2208.01626_ , 2022.
  * Voynov et al. (2023) A. Voynov, K. Aberman, and D. Cohen-Or, “Sketch-guided text-to-image diffusion models,” in _ACM SIGGRAPH 2023 Conference Proceedings_ , 2023, pp. 1–11.
  * Podell et al. (2023) D. Podell, Z. English, K. Lacey, A. Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach, “Sdxl: Improving latent diffusion models for high-resolution image synthesis,” _arXiv preprint arXiv:2307.01952_ , 2023.
  * Wolf et al. (2019) T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz _et al._ , “Huggingface’s transformers: State-of-the-art natural language processing,” _arXiv preprint arXiv:1910.03771_ , 2019.
  * Liu et al. (2023) H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” in _NeurIPS_ , 2023.
  * Bai et al. (2023) J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A frontier large vision-language model with versatile abilities,” _arXiv preprint arXiv:2308.12966_ , 2023.
  * Li et al. (2023b) J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” _arXiv preprint arXiv:2301.12597_ , 2023.
  * Li et al. (2023c) D. Li, J. Li, H. Le, G. Wang, S. Savarese, and S. C. Hoi, “LAVIS: A one-stop library for language-vision intelligence,” in _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)_. Toronto, Canada: Association for Computational Linguistics, Jul. 2023, pp. 31–41. [Online]. Available: https://aclanthology.org/2023.acl-demo.3
  * Geirhos et al. (2020) R. Geirhos, J.-H. Jacobsen, C. Michaelis, R. Zemel, W. Brendel, M. Bethge, and F. A. Wichmann, “Shortcut learning in deep neural networks,” _Nature Machine Intelligence_ , vol. 2, no. 11, pp. 665–673, 2020.
  * Hendrycks et al. (2021) D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song, “Natural adversarial examples,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 15 262–15 271.