# Spin-transfer-assisted parametric pumping of magnons in yttrium iron garnet

Therese Frostad  Hans L. Skarsvåg  Alireza Qaiumzadeh  Arne Brataas  Center
for Quantum Spintronics, Department of Physics, NTNU Norwegian University of
Science and Technology, NO-7491 Trondheim, Norway

###### Abstract

The combination of parametric pumping and spin-transfer torque is a powerful
approach that enables high-level control over magnetic excitations in thin-
film ferromagnets. The excitation parameters, such as pumping power and
external field strength, affect the instabilities of individual magnon modes.
We theoretically explore how the simultaneous effects of parametric pumping
and spin transfer torque influence these magnetic instabilities in a thin-film
ferromagnet. Within the Landau-Lifshitz-Gilbert framework, we perform
micromagnetic simulations of magnon excitations in yttrium iron garnet by
pumping, spin transfer torque, and a combination of the two. We find that
consistent with experimental results, the magnitude and direction of the spin-
transfer torque tune the parametric instability thresholds.

##  I Introduction

Spin waves function as information carriers in spintronics. Magnons, the
quanta of spin waves, may be created in ferromagnetic materials through
numerous methods. Here, we address two common approaches for exciting the
magnetization dynamics of thin-film ferromagnets.

Parametric pumping is a conventional nonlinear excitation method that uses a
microwave field [1, 2]. There is an applied AC magnetic field in the same
direction as the external static magnetic field orienting the magnetization
during parallel parametric pumping. The alternating pumping field induces
oscillations in the ferromagnetic spin system. When the strength of the
pumping field exceeds a certain threshold, an instability occurs that creates
two magnons with opposite momenta. Energy conservation ensures that the two
magnons have a lower frequency than the driving microwave field. Parametric
pumping was introduced in the 1950s [3, 4, 5]. In 2006, parametric pumping
generated Bose-Einstein condensation (BEC) of magnons in yttrium iron garnet
(YIG) at room temperature [6]. This work motivated further discussions on
magnon creation by pumping, thermalization processes of the created magnons,
and the coherence of the emerging condensate [7, 8, 9, 10, 11, 12, 13, 14, 15,
16, 17, 18, 19, 20, 21]. Parametric pumping was further investigated as a tool
for spin wave amplification [22, 23].

Spin-transfer torque (STT) is a more recent approach to create magnons [24,
25]. An external current or voltage induces a magnetization torque. One
possible realization is to pass an electrical current through an adjacent
metal such that the spin-Hall effect creates a spin accumulation in the metal.
The spin accumulation may subsequently generate a torque on the magnetization
in the neighboring ferromagnet. The STT may be employed to either inhibit or
assist magnon creation because the torque has a component that acts damping-
like or anti-damping-like, depending on the sign of the spin accumulation [26,
27, 28, 29, 30]. This control over the damping can create spin-torque
oscillators. In this case, the current or voltage controls the oscillator
frequency. STT can also be used to switch the magnetization configuration [31,
32, 33]. The latter feature enables concepts for magnetic random access
memories. Recently, STT was utilized to create BEC in thin-film BiYIG[34].

Combining parametric pumping and STT achieves high-level control over magnon
creation. It is known that STT tunes the instability thresholds of the
parametric pumping mechanism [26, 27, 35]. Lauer et al. [35] merged parallel
parametric pumping and STT on a thin ($100\text{\,}\mathrm{nm}$) YIG film.
They monitored the resulting magnetization dynamics by Brillouin light
scattering spectroscopy. The experiment revealed that the STT tunes the
effective damping, thereby changing the threshold pumping power required to
excite magnons by parametric pumping. Motivated by this experimental study, we
conduct a large-scale micromagnetic simulation of a similar system. We apply
both a parallel pumping field and STT. We determine the temporal evolution of
the magnetization, from which we determine the stability phase diagram of
spin-transfer-assisted parametric pumping. We also resolve the wavevector
dependence on the nonequilibrium population of magnons.

We organize the remainder of this paper as follows. First, Section II
introduces the theoretical framework. Therein, we present the Landau-Lifshitz-
Gilbert (LLG) equation that describes the magnetization dynamics of the thin
YIG film. Section III presents the setup for our micromagnetic simulations. To
separate the effects of magnon excitation by parametric pumping and STT, we
perform different simulations for the two phenomena. Finally, we simulate the
combination of parametric pumping and STT applied to the YIG film. Section IV
presents the results, and Section V summarizes our findings.

##  II Magnetization dynamics

We calculate the time evolution of the unit vector along the magnetization
direction $\boldsymbol{m}(\boldsymbol{r},t)$ within the LLG framework. The LLG
equation reads

$\dot{\boldsymbol{m}}=-\gamma(\boldsymbol{m}\times\boldsymbol{H}_{\text{eff}})+\alpha(\boldsymbol{m}\times\dot{\boldsymbol{m}})\,.$
(1)

Here,
$\gamma=$1.7595\text{\times}{10}^{11}\text{\,}\mathrm{rad}\text{\,}{\mathrm{T}}^{-1}\text{\,}{\mathrm{s}}^{-1}$$
is the gyromagnetic ratio, and $\alpha$ is the dimensionless Gilbert damping
constant. The effective field $\boldsymbol{H}_{\text{eff}}$ includes
contributions from the exchange field $\boldsymbol{H}_{\text{exch}}$, the
external magnetic field $\boldsymbol{H}_{\text{ext}}$, and the dipole-dipole
field $\boldsymbol{H}_{\text{d-d}}$. In our free energy we disregard the
crystalline anisotropy since it is very small in YIG.

The exchange field arises from the Heisenberg exchange interaction between
neighboring spins, and it promotes a homogeneous magnetization,

$\boldsymbol{H}_{\text{exch}}=2\frac{A_{\text{ex}}}{M_{S}}\nabla^{2}\boldsymbol{m}\,.$
(2)

Here, $A_{\text{ex}}$ is the exchange stiffness, and $M_{S}$ is the saturation
magnetization.

We can orient the magnetization in YIG along a desired direction by applying a
sufficiently high static field of strength $H_{0}$. We define a film in the
($x$,$y$)-plane, and choose to apply the static field in the
$\hat{\mathbf{x}}$-direction,

$\boldsymbol{H}_{\text{ext}}=H_{0}\hat{\mathbf{x}}\,.$ (3)

The external field term may include additional dynamic fields applied to the
film.

The dipole-dipole field consists of the static demagnetization field and
dynamic terms due to the long-range magnon-magnon interactions. In general,
the dipole field at position $\boldsymbol{r}$ can be expressed in terms of
contributions from the magnetization $\boldsymbol{m}(\boldsymbol{r^{\prime}})$
at distance $\boldsymbol{d}=\boldsymbol{r}-\boldsymbol{r^{\prime}}$ integrated
over the film volume $V$,

$\boldsymbol{H}_{\text{d-d}}(\boldsymbol{r})=\frac{\mu_{0}}{4\pi}M_{S}\int_{V}\mathrm{d}\boldsymbol{r^{\prime}}\frac{3(\boldsymbol{m}(\boldsymbol{r^{\prime}})\cdot\boldsymbol{d})\boldsymbol{d}}{|\boldsymbol{d}|^{5}}-\frac{\boldsymbol{m}(\boldsymbol{r^{\prime}})}{|\boldsymbol{d}|^{3}}\,.$
(4)

Here, $\mu_{0}$ is the permeability of free space. The dipole field can be
expressed in terms of a $3\times 3$ tensor
$\hat{G}(\boldsymbol{r},\boldsymbol{r^{\prime}})$ [36, 37] ,

$\boldsymbol{H}_{\text{d-d}}(\boldsymbol{r})=\mu_{0}M_{S}\int_{V}\mathrm{d}\boldsymbol{r^{\prime}}\hat{G}(\boldsymbol{r},\boldsymbol{r^{\prime}})\cdot\boldsymbol{m}(\boldsymbol{r^{\prime}})\,.$
(5)

The tensor elements of $\hat{G}(\boldsymbol{r},\boldsymbol{r^{\prime}})$ are
$G_{\alpha\beta}=-\frac{1}{4\pi}\frac{\partial}{\partial\alpha}\frac{\partial}{\partial\beta}\frac{1}{|\boldsymbol{d}|}$,
for $\alpha,\beta=x,y,z$.

$\mathbf{x}$$\mathbf{y}$$\mathbf{z}$ Figure 1:  The magnetization
$\boldsymbol{m}$ precesses around $\boldsymbol{H}_{\text{eff}}$ due to the
field-like torque $-\boldsymbol{m}\times\boldsymbol{H}_{\text{eff}}$. The
damping
$-\boldsymbol{m}\times(\boldsymbol{m}\times\boldsymbol{H}_{\text{eff}})$ can
be tuned by applying an STT. The torque acts as a damping-like or anti-
damping-like torque depending on the sign of the spin accumulation $\mu_{S}$.
The precession in the ($y$,$z$)-plane is elliptical due to the dipole
contributions from the finite film thickness in the
$\hat{\mathbf{z}}$-direction.

###  II.1 Spin-wave spectra

We now employ the thin-film approximation by assuming that the magnetization
is uniform in the $\hat{\mathbf{z}}$-direction. Therefore, the thin film is
effectively two dimensional. In the expression for the dipole field (5), we
average across the variation in the $\hat{\mathbf{z}}$-direction by
integrating over the film thickness $L_{z}$. The total effective field of the
LLG equation becomes a surface integral over the in-plane coordinate
$\boldsymbol{\rho}=x\hat{\mathbf{x}}+y\hat{\mathbf{y}}$,

$\displaystyle\boldsymbol{H}_{\text{eff}}(\boldsymbol{\rho},t)=H_{0}\hat{\mathbf{x}}+2\frac{A_{\text{ex}}}{M_{S}}\nabla_{\boldsymbol{\rho}}^{2}\boldsymbol{m}(\boldsymbol{\rho},t)$
$\displaystyle+\mu_{0}M_{S}\int_{S}\hat{G}(\boldsymbol{\rho},\boldsymbol{\rho^{\prime}})\boldsymbol{m}(\boldsymbol{\rho^{\prime}},t)\mathrm{d}\boldsymbol{\rho^{\prime}}\,,$
(6)

where the effective two-dimensional dipole tensor becomes

$\hat{G}(\boldsymbol{\rho},\boldsymbol{\rho^{\prime}})=\frac{1}{L_{z}}\int_{-L_{z}/2}^{L_{z}/2}\mathrm{d}z\int_{-L_{z}/2}^{L_{z}/2}\mathrm{d}z^{\prime}\hat{G}(\boldsymbol{r},\boldsymbol{r^{\prime}})\,.$
(7)

We will consider the nonlinear response of the magnetization to the parallel
pumping field in our numerical investigations presented below. However,
relating our results to the linear response regime is also instructive.

In the linear response regime, the magnetization direction is
$\boldsymbol{m}(\boldsymbol{\rho},t)=\hat{\mathbf{x}}+\delta\boldsymbol{m}(\boldsymbol{\rho},t)$,
where the small out-of-equilibrium deviation
$\delta\boldsymbol{m}(\boldsymbol{\rho},t)$ lies in the ($y$,$z$)-plane, as
shown in Figure 1. Additionally, we assume the magnetization is precessing
with frequency $\omega$; thus,
$\delta\boldsymbol{m}(\boldsymbol{\rho},t)=\delta\boldsymbol{m}(\boldsymbol{\rho})e^{i\omega
t}$. We insert the effective field into the LLG Equation 1. In the linear
response regime, we retain only the first-order terms in the deviation
$\delta\boldsymbol{m}(\boldsymbol{\rho},t)$. We proceed to define the spatial
Fourier transforms,

$\boldsymbol{m}(\boldsymbol{k})=\mathcal{F}[\boldsymbol{m}(\boldsymbol{\rho})]=\frac{1}{2\pi}\int\boldsymbol{m}(\boldsymbol{\rho})e^{-i\boldsymbol{k}\cdot\boldsymbol{\rho}}\mathrm{d}\boldsymbol{\rho}\,,$
(8)
$\boldsymbol{m}(\boldsymbol{\rho})=\int\boldsymbol{m}(\boldsymbol{k})e^{-i\boldsymbol{k}\cdot\boldsymbol{\rho}}\mathrm{d}\boldsymbol{k}\,.$
(9)

Here,
$\boldsymbol{k}=|k|(\cos\theta_{k}\hat{\mathbf{x}}+\sin\theta_{k}\hat{\mathbf{y}})$
is the magnon wavevector. We can express the LLG equation as an eigenvalue
problem, where the precession frequencies are the eigenvalues. We may then
obtain the well-known dispersion relation for an extended thin film [36, 38,
39],

$\displaystyle\omega(k,\theta_{k})$
$\displaystyle=\sqrt{\omega_{H}+\omega_{M}l_{\text{ex}}^{2}k^{2}+\omega_{M}(1-f_{k})\sin^{2}\theta_{k}}$
$\displaystyle\times\sqrt{\omega_{H}+\omega_{M}l_{\text{ex}}^{2}k^{2}+\omega_{M}f_{k}}\,.$
(10)

Here, we have defined $\omega_{M}=\gamma\mu_{0}M_{S}$,
$\omega_{H}=\gamma\mu_{0}H_{0}$ and the magnetic exchange length
$l_{\text{ex}}=\sqrt{2A_{\text{ex}}/\mu_{0}M_{S}^{2}}$. The form factor
$f_{k}$ accounts for the film thickness,

$f_{k}=\frac{1-e^{-|k|L_{z}}}{|k|L_{z}}\,.$ (11)

The spin waves traveling in the $\hat{\mathbf{x}}$-direction ($\theta_{k}=0$)
have the lowest energy. This lower magnon branch is shown in Figure 2. It is
symmetric in $k$, and has a double minimum at $k=\pm k_{\text{min}}$. The
exchange energy controls the spin waves at high wavevectors $k\gg
l_{\text{ex}}^{-1}$ so that the dispersion is quadratic in $k$. On the other
hand, the dipole interaction controls the spin waves at small wavevectors
$k<L_{z}^{-1}$.

Figure 2:  The dispersion relation for spin waves traveling in the
$\hat{\mathbf{x}}$-direction of a thin YIG film ($k_{x}$-branch), as given by
Equation 10. Spin waves traveling in the $\hat{\mathbf{y}}$-direction have
higher energies, as indicated by the dashed line ($k_{y}$-branch). During
parametric pumping, two magnons are created at wavevectors of opposite signs.
These magnons both have frequency $\omega_{p}/2$. In the $k_{x}$-branch, these
magnons are found at $\pm k_{p1}$ and $\pm k_{p2}$. In turn, they thermalize
and eventually occupy the dispersion minimum at $k_{\text{min}}$. The
parameters are from Tables 1 and 2, with
$\mu_{0}H_{0}=$190\text{\,}\mathrm{mT}$$.

###  II.2 Magnon excitation

We proceed to examine magnon excitation by parametric pumping. In the parallel
pumping geometry, the magnetization is in-plane and oriented along the
$\hat{\mathbf{x}}$-axis by a static field of magnitude $H_{0}$. There is an
additional alternating microwave field $\boldsymbol{h}_{p}(t)$ parallel to the
static field that functions as a magnon pump. The external field has static
and oscillating contributions,

$\boldsymbol{H}_{\text{ext}}=(H_{0}+h_{p}\sin(\omega_{p}t))\hat{\mathbf{x}}\,.$
(12)

Here, $h_{p}$ is the amplitude and $\omega_{p}$ is the frequency of the
oscillating pumping field. When the strength of the pumping field exceeds a
critical threshold, $h_{p}>h_{p}^{\text{crit}}(H_{0})$, the magnetization
starts to precess around the $\hat{\mathbf{x}}$-axis. For a thin film, this
precession is elliptical due to the dipole interactions from the finite
thickness in the $\hat{\mathbf{z}}$-direction. By energy conservation, the
created magnons have half the frequency of the pumping field,
$\omega_{\boldsymbol{k}}=\omega_{\boldsymbol{-k}}=\omega_{p}/2$, and are
opposite wave vectors, as illustrated in Figure 2. This is the oscillation of
the x-component of the magnetic moments that couple to the magnetic pulse in
the parallel pumping mechanism. Since dipolar-magnons in YIG are elliptically
polarized, the x component of the magnons oscillates, and therefore parallel
pumping is possible. Parametric pumping mainly excites elliptical dipole-
dominated low energy magnon modes. We therefore expect that magnons are pumped
in the $k_{x}$-branch, at points $k_{x}=\pm k_{p1}$, as illustrated in Figure
2. Following this reasoning, the most efficient pumping occurs when
$k_{p1}=k_{p2}=0$ at ferromagnetic resonance (FMR) conditions, since the
ellipticity is highest in that situation. For a fixed pumping frequency
$\omega_{p}$, the external magnetic field strength $H_{\text{FMR}}$
corresponding to the resonance frequency $\omega_{\text{FMR}}$ is approximated
by the Kittel formula [40], as expected from Equation 10 in the limit
$k\rightarrow 0$,

$\omega_{\text{FMR}}=\omega_{p}/2=\gamma\sqrt{H_{\text{FMR}}(H_{\text{FMR}}+\mu_{0}M_{S})}\,.$
(13)

Next, we consider how the STT affects the magnetization dynamics. In thick
films, the effective volume of surface modes is smaller than that of volume
modes. Therefore, exciting surface modes by STT is easier than exciting bulk
modes by STT [41, 42]. However, we consider a thin film without any surface
anisotropy, where the magnetization is uniform along the thickness direction.
Surface modes are thus less important in this case, and we will not discuss
them further.

The STT typically arises from a spin accumulation in an adjacent normal metal.
We assume that a charge current in the adjacent metal layer (e.g., Pt)
produces the spin accumulation polarized along the $\hat{\mathbf{x}}$-axis.
The resulting torque on the YIG interface is expressed in terms of the spin-
mixing conductance per area $g_{\perp}~{}[1/\Omega\text{m}^{2}]$ and spin
accumulation density $\boldsymbol{\mu}_{S}=\mu_{S}\hat{\mathbf{x}}$,

$\displaystyle\dot{\boldsymbol{m}}$
$\displaystyle=-\gamma(\boldsymbol{m}\times\boldsymbol{H}_{\text{eff}})+\alpha(\boldsymbol{m}\times\dot{\boldsymbol{m}})$
$\displaystyle-\frac{\gamma\hbar}{2e^{2}L_{z}M_{S}}g_{\perp}\boldsymbol{m}\times(\boldsymbol{m}\times\boldsymbol{\mu_{S}})\,.$
(14)

Here, $e$ is the elementary charge, and $\hbar$ is the reduced Planck
constant. In our geometry, a positive (negative) spin accumulation results in
a damping-like (anti-damping-like) STT. We proceed to find an expression for
the critical spin accumulation $\mu_{S}^{\text{crit}}$ at which the damping is
overpowered. As before, the effective field in Equation 6 includes the static
field, exchange interaction and dipole interactions. Inserting the effective
field into Equation 14, the imaginary part of the eigenfrequencies determines
the critical spin accumulation required to excite magnons with a specific
wavevector,

$\displaystyle\mu_{S}^{\text{crit}}(k,\theta_{k})$
$\displaystyle=-\frac{2L_{z}e^{2}\alpha M_{S}}{\hbar
g_{\perp}\gamma}\Big{(}\omega_{H}+\omega_{M}l_{\text{ex}}^{2}k^{2}$
$\displaystyle+\frac{1}{2}\omega_{M}(f_{k}+(1-f_{k})\sin^{2}\theta_{k})\Big{)}\,.$
(15)

Equation 15 can be minimized to yield the first magnon wavevector to be
excited by STT. This wave vector denoted by $\boldsymbol{k}_{\text{STT}}$ is
in the $k_{x}$-branch ($\theta_{k}=0$), as illustrated in Figure 3.

Figure 3:  The critical spin accumulation $-\mu_{S}^{\text{crit}}g_{\perp}/e$
for exciting spin waves traveling in the $\hat{\mathbf{x}}$-direction of a
thin YIG film ($k_{x}$-branch), as given by Equation 15. Spin waves traveling
in the $\hat{\mathbf{y}}$-direction require a higher spin accumulation, as
indicated by the dashed line ($k_{y}$-branch). The parameters are from Tables
1 and 2, with $\mu_{0}H_{0}=$100\text{\,}\mathrm{mT}$$. For these parameters,
we find that the first magnons are excited at $\theta_{k}=0$,
$|\boldsymbol{k}_{\text{STT}}|\approx$15.5\text{\,}{\mathrm{\SIUnitSymbolMicro
m}}^{-1}$$, for the critical spin accumulation
$\mu_{S}^{\text{crit}}(\boldsymbol{k}_{\text{STT}})g_{\perp}/e\approx-$6.55\text{\times}{10}^{10}\text{\,}\mathrm{A}\text{\,}{\mathrm{m}}^{-2}$$.

We note that the threshold spin accumulation in Equation 15 is linear in the
external field strength.

##  III Numerical approach

We consider the magnetization dynamics in a thin film of YIG. First, we define
a square film with lateral side lengths $L_{x,y}$ in the ($x$,$y$)-plane. The
film thickness $L_{z}=$100\text{\,}\mathrm{nm}$$ is much smaller than the
lateral side lengths. We use the thin-film approximation in which there is no
magnetization variation in the $\hat{\mathbf{z}}$-direction. Laterally, we
divide the film into $N=n_{x,y}^{2}$ cells, each of size $l_{x,y}^{2}\times
L_{z}$. The unit vector in the magnetization direction of each cell is
$\boldsymbol{m}(\boldsymbol{\rho}_{i},t)$. We find the time evolution of each
magnetization vector by solving the LLG equation at successive time intervals.

The simulations provide full information on the local magnetization within
each cell of the film. This information allows the definition of a relative
magnon density $\eta$ in terms of the longitudinal component of the average
magnetization,

$\eta=1-\langle\boldsymbol{m}_{x}\rangle\,,$ (16)

where
$\langle\boldsymbol{m}_{x}\rangle=\sum_{i=1}^{i=N}\boldsymbol{m}(\boldsymbol{\rho}_{i},t)/N$.
We can find the magnon distribution $\zeta(k_{x},k_{y})$ as a function of the
wavevector evaluating the Fourier transform of the transverse components of
the magnetization,

$\zeta(k_{x},k_{y})=|\mathcal{F}[m_{y}(\boldsymbol{\rho},t)]|^{2}+|\mathcal{F}[m_{z}(\boldsymbol{\rho},t)]|^{2}\,.$
(17)

Here, $\mathcal{F}[~{}]$ denotes a discrete Fourier transform defined similar
to the continuous transform in Equation 8.

Our simulations include magnon excitation by parallel pumping, the STT, and
the combination of the two. To this end, we use the open-source GPU-
accelerated software MUMAX3.[43] The simulations start from an initial
magnetization state $\boldsymbol{m}(\boldsymbol{\rho}_{i},t=0)$, which must
deviate from the uniform state for the magnetization dynamics to start. We
apply the pumping fields and/or the STT over the entire surface of the film,
and we investigate the resulting magnon density as a function of the bias
field $H_{0}$, the pumping field amplitude $h_{p}$ and the spin accumulation
$\mu_{S}$. Section IV provides more details on the initialization and
parameter variation of each simulation. Although the lateral size of the film
is much larger than the film thickness, we expect some finite size effects.
The material parameters for YIG are listed in Table 1. The damping coefficient
$\alpha=10^{-2}$ is set higher than, e.g., Ref. [35] ($\alpha\sim 10^{-4}$),
such that the magnetization may reach steady state at acceptable running
times. We detail the typical numerical parameters of the simulations in Table
2.

Table 1: Material parameters of YIG. Parameter | Symbol | Value in SI
---|---|---
Saturation magnetization [35] | $\mu_{0}M_{S}$ | $173\text{\,}\mathrm{mT}$
Exchange stiffness [44] | $A_{\text{ex}}$ | $3.65\text{\times}{10}^{-12}\text{\,}\mathrm{J}\text{\,}{\mathrm{m}}^{-1}$
Gilbert damping parameter | $\alpha$ | $0.01$
Table 2:  System parameters. Parameter | Symbol | Value in SI
---|---|---
YIG film thickness | $L_{z}$ | $100\text{\,}\mathrm{nm}$
YIG film lateral dimensions | $L_{x,y}$ | $5\text{\,}\mathrm{\SIUnitSymbolMicro m}$
Lateral number of cells | $n_{x,y}$ | $2^{9}$
Lateral cell size | $l_{x,y}$ | $\sim$10\text{\,}\mathrm{nm}$$
Microwave pumping frequency | $\omega_{\text{p}}=2\pi f_{p}$ | $2\pi\times$14\text{\,}\mathrm{GHz}$$

##  IV Numerical results

We discuss the numerical results of three scenarios: (i)) excitations
performed exclusively by parallel pumping, (ii)) excitations performed
exclusively by STT, and (iii)) excitations resulting from the combination of
pumping and STT.

###  IV.1 Magnon excitation by parallel parametric pumping

We first present simulations of magnon excitation by parallel parametric
pumping. The number of created magnons is a measure of the efficiency of the
pumping process. The simulations determine the relative magnon density as a
function of the external field strength $H_{0}$ and pumping power $h_{p}$. In
doing so, the simulations provide the critical threshold strength
$h_{p}^{\text{crit}}(H_{0})$ required to excite magnons. The threshold
excitation strength may show hysteretic behaviour depending on whether we
increase or decrease $H_{0}$ [45, 46]. For each fixed value of $H_{0}$, we run
simulations in which $h_{p}$ is decreased in steps for each time interval of
length $\Delta t$, as illustrated in Figure 4.

Each simulation starts from a chosen initial magnetization state. A noisy
input state is created by randomly pulling each magnetization vector slightly
away from its uniform state. This contributes to the initial magnon density,
as observed at $t=0$ in Figure 4. We proceed to strongly pump the system for
two intervals $I_{-2}$ and $I_{-1}$. The gray shading in Figure 4 shows these
initialization intervals, which are not included in the extracted results in
Figure 7. After the initialization, we pump the systems in intervals $I_{j}$
($j=0,1,2,...$) while decreasing the pumping strength for each interval.

$t[ns]$$\langle m_{x}\rangle$$\mu_{0}h_{p}\;[mT]$ Figure 4:  The temporal
evolution of the spatial average of the magnetization $\langle
m_{x}\rangle(t)$. At $t=0$, the magnetization randomly deviates from the
uniform state. The shaded areas show the initialization intervals $I_{-2}$ and
$I_{-1}$. During these intervals, the pumping is strong and creates an initial
state. Thereafter, the pumping power $h_{p}$ decreases in intervals $I_{j}$
(purple). Within each interval, we compute the temporal average of the
magnetization (orange) resulting in the magnon density $\eta(H_{0},h_{p})$.
The presented data are an excerpt from the simulations at
$\mu_{0}H_{0}=$190\text{\,}\mathrm{mT}$$.

Figure 4 shows how the magnetization reaches a steady state when the pumping
is decreased. In the steady state, the magnetization precession is elliptical.
The spatial average of the magnetization $\langle\boldsymbol{m}_{x}\rangle(t)$
determines the relative density of the magnons, as in Equation 16. We compute
a time-averaged value $\eta(I_{j})$ for the magnon density within a time
window of 50ns at the end of each interval, as illustrated in Figure 4. The
resulting magnon density represents one data point in Figure 7b.

The threshold parametric pumping power $h_{p}^{\text{crit}}(H_{0})$ to excite
magnons is occasionally referred to as a butterfly curve [47, 46, 17]. The
minimum of the threshold curve lies near the external field corresponding to
resonance conditions, $H_{\text{FMR}}$, which can be approximated by the
Kittel formula in Equation 13. Inserting the values of $M_{S}$ and
$\omega_{\text{FMR}}=\omega_{\text{p}}/2$ from Table 2 results in
$H_{\text{FMR}}\approx$178\text{\,}\mathrm{mT}$$. Note that the Kittel formula
is valid for extended thin films, where $L_{z}/L_{x,y}\rightarrow 0$. In
contrast, $L_{z}/L_{x,y}\approx 0.02$ in the simulated film (see Table 2).
This causes a small deviation between the simulated FMR frequency and the
Kittel formula. Furthermore, because the lateral lengths of the simulated
films are finite, the magnon energy levels are discrete. Looking at Figure 7b,
we find that the discrete energy levels result in spikes, where it is
difficult to excite magnons. The threshold curve can be compared to
experimental results by Lauer et al. [35].

###  IV.2 Magnon excitation by STT

We now proceed to discuss how STT generates magnons. The number of magnons
created by STT is a function of the spin accumulation $\mu_{S}$ and the
external field strength $H_{0}$. The STT effectively controls the damping. The
sign of the spin accumulation determines whether the torque acts as a damping-
like or anti-damping-like torque.

We use a range of applied bias field strengths to investigate the threshold
spin accumulation for exciting magnons by STT. We decrease the strength of the
spin accumulation in intervals for each $H_{0}$-series, as illustrated in
Figure 5. The initial magnetization at $t=0$ is set to randomly deviate from
the uniform state. An applied spin accumulation
($\mu_{S}g_{\perp}/e=$-8\text{\times}{10}^{10}\text{\,}\mathrm{A}\text{\,}{\mathrm{m}}^{-2}$$)
causes a strong STT well above the excitation threshold. The torque is present
at the entire surface of the film. Next, we gradually decrease the strength of
the torque at intervals $I_{j}$ of duration $\Delta t_{j}$. The magnetization
dynamics is chaotic, and occasionally the system does not easily find a steady
state at high current strengths, as shown in Figure 5. The magnon densities
$\eta(H_{0},\mu_{S})$ calculated for the time windows are shown in Figure 6.

$t[ns]$$\langle m_{x}\rangle$$-\mu_{S}g_{\perp}/e\;[10^{1}0Am^{-2}]$ Figure 5:
The temporal evolution of the spatial average of the magnetization $\langle
m_{x}\rangle(t)$ during excitation by STT. The simulations start from a
magnetization state with random deviations from the uniform state. Magnons are
created by the application of a large spin accumulation that results in an
anti-damping STT. The spin accumulation decreases in intervals $I_{j}$
(purple). We compute the temporal average of the magnetization (orange)
resulting in the magnon density $\eta(H_{0},\mu_{S})$. The presented data are
an excerpt from the simulations at $\mu_{0}H_{0}=$100\text{\,}\mathrm{mT}$$.

We now consider the spin accumulation found by analytical approximations. The
threshold spin accumulation for exciting magnons at wavevector
$\boldsymbol{k}$ is given in Equation 15. For the chosen parameters in Tables
2 and 1, we find that $\mu_{S}^{\text{crit}}(\boldsymbol{k})$ reaches its
minimum when $\theta_{k_{\text{STT}}}=0$,
$|\boldsymbol{k}_{\text{STT}}|\approx$15.5\text{\,}{\mathrm{\SIUnitSymbolMicro
m}}^{-1}$$.

$\mu_{0}H_{0}[mT]$$-\mu_{S}g_{\perp}/e\;[10^{1}0Am^{-2}]$ Figure 6:  The
relative magnon density $\eta(H_{0},\mu_{S})$ during STT excitation of a YIG
film as a function of the external magnetic field and spin accumulation. The
simulation series for each fixed $H_{0}$ as in Figure 5 represents one column
of the figure, where each pixel represents the relative magnon density
$\eta(I_{j})$. The dashed blue line is the critical spin accumulation
$\mu_{S}^{\text{crit}}(\boldsymbol{k}_{\text{STT}})$ required to excite
magnons, as in Equation 15. For comparison, the solid blue line shows
$\mu_{S}^{\text{crit}}(k=0)$.

###  IV.3 Parallel parametric pumping and weak STT

The STT is dissipative and effectively changes the Gilbert damping parameter.
We therefore expect that the STT changes the threshold of parametric pumping.
We investigate this expectation by applying a weak STT
($\mu_{S}g_{\perp}/e=\pm$1\text{\times}{10}^{10}\text{\,}\mathrm{A}\text{\,}{\mathrm{m}}^{-2}$$)
while performing parametric pumping on the YIG film. The procedure of
initialization and interval pumping is similar to that in Section IV.1.

The torque is applied at the entire surface of the film at all times,
including during the initialization intervals $I_{-2}$ and $I_{-1}$. In our
sign convention, a positive (negative) spin accumulation results in a damping-
like (anti-damping-like) torque. Figure 7a shows the effect of applying a
damping-like torque. The magnon instability threshold moves to higher pumping
powers since the torque results in a higher effective damping. Figure 7c shows
the results of an anti-damping torque, which moves the threshold to lower
pumping powers required to excite magnons. Note that in the high magnon
density limit, which is the case in the presence of an antidamping-like
torque, the magnetization dynamics is highly non-linear.

(a)(b)(c)$\mu_{0}H_{0}[mT]$$\mu_{0}h_{p}\;[mT]$ Figure 7:  The relative magnon
density $\eta(H_{0},h_{p})$ during the parametric pumping of a thin YIG film.
$H_{0}$ and $h_{p}$ are the strengths of the external magnetic field and
pumping field. The simulation series for each fixed $H_{0}$ (similar to Figure
4) represents one column of the figure, where each pixel represents the
relative magnon density $\eta(I_{j})$. In a) and c), an STT due to the weak
spin accumulation $\mu_{S}$ is applied.
a) Damping-like STT,
$\mu_{S}g_{\perp}/e=+$1\text{\times}{10}^{10}\text{\,}\mathrm{A}\text{\,}{\mathrm{m}}^{-2}$$
b) No STT, $\mu_{S}=0$
c) Anti-damping-like STT,
$\mu_{S}g_{\perp}/e=-$1\text{\times}{10}^{10}\text{\,}\mathrm{A}\text{\,}{\mathrm{m}}^{-2}$$

###  IV.4 Excited magnon modes

Parametric pumping favors the excitation of spin waves with elliptical
precession. The dipole interaction dominates the dispersion in the long
wavelength limit when the spins precess with an elliptical character. In
contrast, the exchange-dominated spin waves with a shorter wavelength have a
circular precession. As illustrated in Figure 2, we expect to mainly pump
magnons at relatively long wavelengths, $k=k_{p1}$. To further investigate
this aspect, we perform a Fourier transform of the magnetization during
parametric pumping and present the results in Figure 8a. Our numerical results
confirm that the pumped magnons center around $k=k_{p1}$. Keeping the pumping
frequency fixed, $k_{p1}$ can be moved to higher (lower) values if we increase
(decrease) the strength of the bias field.

(a) (a)

(b) (b)

Figure 8:  The relative distributions of the magnons as a function of
frequency and wavevector $k_{x}$, ($k_{y}=0)$. Magnons are excited in the YIG
film during a) only parametric pumping,
$\mu_{0}H_{0}=$190\text{\,}\mathrm{mT}$$,
$\mu_{0}h_{p}=$18\text{\,}\mathrm{mT}$$ and b) only STT,
$\mu_{0}H_{0}=$100\text{\,}\mathrm{mT}$$,
$\mu_{S}g_{\perp}/e=-$6.5\text{\times}{10}^{10}\text{\,}\mathrm{A}\text{\,}{\mathrm{m}}^{-2}$$.
We show the squareroot of the magnon distribution, $\sqrt{\zeta(k_{x},k_{y})}$
from Equation 17, where we disregard edge effects by taking the Fourier
transform over the film middle (64 cell edges). The data are normalized with
respect to the maximum intensity. In a), the analytical energy dispersion from
Equation 10 is illustrated by a gray dashed line, while the pumping frequency
is drawn with a purple dashed line.

In general, the STT excites spinwaves at a wide range of frequencies and
wavevector numbers [48]. For weak currents, the STT is expected to
predominantly excite magnons at $k_{\text{STT}}$, the magnon wavevector that
minimizes the spin accumulation given in Equation 15. Starting from the lowest
spin accumulation needed to excite magnons by STT, $k_{\text{STT}}$ is the
wavevector of the first magnons that we expect to excite. $k_{\text{STT}}$
does not change with the bias field strength, but it depends on the film
thickness. From Figure 8b, we find that we also excite magnons at neighboring
energies. We find that the parametric pumping mainly creates dipole-dominated
magnons at $k_{x}=k_{p1}\leq k_{\text{min}}$. The exchange-dominated magnons
at $k_{x}\geq k_{\text{min}}$ are more easily excited by using the STT.

##  V Conclusions

We have theoretically investigated the density of magnons excited by
parametric pumping, STT, and a combination of the two. The excitation
processes in a thin YIG film are studied by performing micromagnetic
simulations. Spin waves traveling parallel to the bias field direction
(”$k_{x}$-branch”) have lower energy than perpendicular spin waves. During
parametric pumping, magnons of elliptic precession are predominantly excited
at low wavevector numbers ($k=k_{p1}$) in the $k_{x}$-branch. For STT, we
expect the first magnons to be excited at $k=k_{\text{STT}}$ before being
distributed around the energy minimum of the $k_{x}$-branch.

We found the critical pumping amplitude or spin accumulation for exciting
magnons. The presence of a damping-like or anti-damping-like STT increases or
decreases the threshold power for parametric pumping, depending on the sign of
the STT. Our computed results are consistent with the measurements in recent
experiments [35].

##  VI Acknowledgments

The research leading to these results has received funding from the Research
Council of Norway through its Centres of Excellence funding scheme, project
number 262633, ”QuSpin”. The simulation work was supported with computational
resources from the NTNU Idun infrastructure [49]. A.Q. was supported by the
Norwegian Financial Mechanism 2014-2021 under the Polish–Norwegian Research
Project NCN GRIEG “2Dtronics” No. 2019/34/H/ST3/00515. We thank A. A. Serga
and A. Kapelrud for helpful and stimulating discussions.

## References

  * Morgenthaler [1960] F. R. Morgenthaler, Survey of ferromagnetic resonance in small ferrimagnetic ellipsoids, Journal of Applied Physics 31, S95 (1960).
  * Schlömann  _et al._ [1960] E. Schlömann, J. Green, and u. Milano, Recent developments in ferromagnetic resonance at high power levels, Journal of Applied Physics 31, S386 (1960).
  * Anderson and Suhl [1955] P. Anderson and H. Suhl, Instability in the motion of ferromagnets at high microwave power levels, Physical Review 100, 1788 (1955).
  * Suhl [1956] H. Suhl, The nonlinear behavior of ferrites at high microwave signal levels, Proceedings of the IRE 44, 1270 (1956).
  * Suhl [1957] H. Suhl, The theory of ferromagnetic resonance at high signal powers, Journal of Physics and Chemistry of Solids 1, 209 (1957).
  * Demokritov  _et al._ [2006] S. Demokritov, V. Demidov, O. Dzyapko, G. Melkov, A. Serga, B. Hillebrands, and A. Slavin, Bose–einstein condensation of quasi-equilibrium magnons at room temperature under pumping, Nature 443, 430 (2006).
  * Demokritov  _et al._ [2008] S. Demokritov, V. Demidov, O. Dzyapko, G. Melkov, and A. Slavin, Quantum coherence due to bose-einstein condensation of parametrically driven magnons, New Journal of Physics 10, 045029 (2008).
  * Demidov  _et al._ [2008] V. Demidov, O. Dzyapko, S. Demokritov, G. Melkov, and A. Slavin, Observation of spontaneous coherence in bose-einstein condensate of magnons, Physical review letters 100, 047205 (2008).
  * Tupitsyn  _et al._ [2008] I. Tupitsyn, P. Stamp, and A. Burin, Stability of bose-einstein condensates of hot magnons in yttrium iron garnet films, Physical review letters 100, 257202 (2008).
  * Rezende [2009a] S. M. Rezende, Theory of coherence in bose-einstein condensation phenomena in a microwave-driven interacting magnon gas, Physical Review B 79, 174411 (2009a).
  * Rezende [2009b] S. M. Rezende, Theory of microwave superradiance from a bose-einstein condensate of magnons, Physical Review B 79, 060410 (2009b).
  * Nowik-Boltyk  _et al._ [2012] P. Nowik-Boltyk, O. Dzyapko, V. Demidov, N. Berloff, and S. Demokritov, Spatially non-uniform ground state and quantized vortices in a two-component bose-einstein condensate of magnons, Scientific reports 2, 482 (2012).
  * Serga  _et al._ [2014] A. A. Serga, V. S. Tiberkevich, C. W. Sandweg, V. I. Vasyuchka, D. A. Bozhko, A. V. Chumak, T. Neumann, B. Obry, G. A. Melkov, A. N. Slavin, _et al._ , Bose–einstein condensation in an ultra-hot gas of pumped magnons, Nature communications 5 (2014).
  * Clausen  _et al._ [2015] P. Clausen, D. Bozhko, V. Vasyuchka, B. Hillebrands, G. Melkov, and A. Serga, Stimulated thermalization of a parametrically driven magnon gas as a prerequisite for bose-einstein magnon condensation, Physical Review B 91, 220402 (2015).
  * Dzyapko  _et al._ [2016] O. Dzyapko, P. Nowik-Boltyk, B. Koene, V. E. Demidov, J. Jersch, A. Kirilyuk, T. Rasing, and S. O. Demokritov, High-resolution magneto-optical kerr-effect spectroscopy of magnon bose–einstein condensate, IEEE Magnetics Letters 7, 1 (2016).
  * Hahn and Kopietz [2020] V. Hahn and P. Kopietz, Collisionless kinetic theory for parametrically pumped magnons, The European Physical Journal B 93, 1 (2020).
  * Mohseni  _et al._ [2020] M. Mohseni, M. Kewenig, R. Verba, Q. Wang, M. Schneider, B. Heinz, F. Kohl, C. Dubs, B. Lägel, A. A. Serga, _et al._ , Parametric generation of propagating spin waves in ultrathin yttrium iron garnet waveguides, physica status solidi (RRL)–Rapid Research Letters 14, 2000011 (2020).
  * Hayashi and Ando [2018] H. Hayashi and K. Ando, Spin pumping driven by magnon polarons, Physical review letters 121, 237202 (2018).
  * Heinz  _et al._ [2021] B. Heinz, M. Mohseni, A. Lentfert, R. Verba, M. Schneider, B. Lägel, K. Levchenko, T. Brächer, C. Dubs, A. V. Chumak, _et al._ , Parametric generation of spin waves in nano-scaled magnonic conduits, arXiv preprint arXiv:2106.10727 (2021).
  * Noack  _et al._ [2019] T. B. Noack, V. I. Vasyuchka, D. A. Bozhko, B. Heinz, P. Frey, D. V. Slobodianiuk, O. V. Prokopenko, G. A. Melkov, P. Kopietz, B. Hillebrands, _et al._ , Enhancement of the spin pumping effect by magnon confluence process in yig/pt bilayers, physica status solidi (b) 256, 1900121 (2019).
  * Hahn and Kopietz [2021] V. Hahn and P. Kopietz, Effect of magnon decays on parametrically pumped magnons, Physical Review B 103, 094416 (2021).
  * Verba  _et al._ [2018] R. Verba, M. Carpentieri, G. Finocchio, V. Tiberkevich, and A. Slavin, Amplification and stabilization of large-amplitude propagating spin waves by parametric pumping, Applied Physics Letters 112, 042402 (2018).
  * Brächer  _et al._ [2014] T. Brächer, P. Pirro, T. Meyer, F. Heussner, B. Lägel, A. Serga, and B. Hillebrands, Parallel parametric amplification of coherently excited propagating spin waves in a microscopic ni81fe19 waveguide, Applied Physics Letters 104, 202408 (2014).
  * Ralph and Stiles [2008] D. C. Ralph and M. D. Stiles, Spin transfer torques, Journal of Magnetism and Magnetic Materials 320, 1190 (2008).
  * Brataas  _et al._ [2012] A. Brataas, A. D. Kent, and H. Ohno, Current-induced torques in magnetic materials, Nature materials 11, 372 (2012).
  * Urazhdin  _et al._ [2010] S. Urazhdin, V. Tiberkevich, and A. Slavin, Parametric excitation of a magnetic nanocontact by a microwave field, Physical review letters 105, 237204 (2010).
  * Edwards  _et al._ [2012] E. Edwards, H. Ulrichs, V. Demidov, S. Demokritov, and S. Urazhdin, Parametric excitation of magnetization oscillations controlled by pure spin current, Physical Review B 86, 134420 (2012).
  * Hamadeh  _et al._ [2014] A. Hamadeh, O. d. Kelly, C. Hahn, H. Meley, R. Bernard, A. Molpeceres, V. Naletov, M. Viret, A. Anane, V. Cros, _et al._ , Full control of the spin-wave damping in a magnetic insulator using spin-orbit torque, Physical review letters 113, 197203 (2014).
  * Tserkovnyak  _et al._ [2002] Y. Tserkovnyak, A. Brataas, and G. E. Bauer, Enhanced gilbert damping in thin ferromagnetic films, Physical review letters 88, 117601 (2002).
  * Wang  _et al._ [2011] Z. Wang, Y. Sun, M. Wu, V. Tiberkevich, and A. Slavin, Control of spin waves in a thin film ferromagnetic insulator through interfacial spin scattering, Physical review letters 107, 146602 (2011).
  * Slonczewski [1996] J. C. Slonczewski, Current-driven excitation of magnetic multilayers, Journal of Magnetism and Magnetic Materials 159, L1 (1996).
  * Myers  _et al._ [1999] E. Myers, D. Ralph, J. Katine, R. Louie, and R. Buhrman, Current-induced switching of domains in magnetic multilayer devices, Science 285, 867 (1999).
  * Katine  _et al._ [2000] J. Katine, F. Albert, R. Buhrman, E. Myers, and D. Ralph, Current-driven magnetization reversal and spin-wave excitations in co/cu/co pillars, Physical Review Letters 84, 3149 (2000).
  * Divinskiy  _et al._ [2021] B. Divinskiy, H. Merbouche, V. Demidov, K. Nikolaev, L. Soumah, D. Gouéré, R. Lebrun, V. Cros, J. B. Youssef, P. Bortolotti, _et al._ , Evidence for spin current driven bose-einstein condensation of magnons, Nature Communications 12, 1 (2021).
  * Lauer  _et al._ [2016] V. Lauer, D. Bozhko, T. Brächer, P. Pirro, V. Vasyuchka, A. Serga, M. Jungfleisch, M. Agrawal, Y. V. Kobljanskyj, G. Melkov, _et al._ , Spin-transfer torque based damping control of parametrically excited spin waves in a magnetic insulator, Applied Physics Letters 108, 012402 (2016).
  * Kalinikos and Slavin [1986] B. Kalinikos and A. Slavin, Theory of dipole-exchange spin wave spectrum for ferromagnetic films with mixed exchange boundary conditions, Journal of Physics C: Solid State Physics 19, 7013 (1986).
  * Kalinikos [1981] B. Kalinikos, Spectrum and linear excitation of spin waves in ferromagnetic films, Soviet Physics Journal 24, 718 (1981).
  * Kreisel  _et al._ [2009] A. Kreisel, F. Sauli, L. Bartosch, and P. Kopietz, Microscopic spin-wave theory for yttrium-iron garnet films, The European Physical Journal B-Condensed Matter and Complex Systems 71, 59 (2009).
  * Rückriegel and Kopietz [2015] A. Rückriegel and P. Kopietz, Rayleigh-jeans condensation of pumped magnons in thin-film ferromagnets, Physical review letters 115, 157203 (2015).
  * Kittel [1948] C. Kittel, On the theory of ferromagnetic resonance absorption, Physical Review 73, 155 (1948).
  * Xiao  _et al._ [2012] J. Xiao, G. E. Bauer, _et al._ , Spin-wave excitation in magnetic insulators by spin-transfer torque, Physical review letters 108, 217204 (2012).
  * Kapelrud and Brataas [2013] A. Kapelrud and A. Brataas, Spin pumping and enhanced gilbert damping in thin magnetic insulator films, Physical Review Letters 111, 097602 (2013).
  * Vansteenkiste  _et al._ [2014] A. Vansteenkiste, J. Leliaert, M. Dvornik, M. Helsen, F. Garcia-Sanchez, and B. Van Waeyenberge, The design and verification of mumax3, Aip Advances 4, 107133 (2014).
  * Klingler  _et al._ [2014] S. Klingler, A. V. Chumak, T. Mewes, B. Khodadadi, C. Mewes, C. Dubs, O. Surzhenko, B. Hillebrands, and A. Conca, Measurements of the exchange stiffness of yig films using broadband ferromagnetic resonance techniques, Journal of Physics D: Applied Physics 48, 015001 (2014).
  * Chen  _et al._ [2017] Y.-J. Chen, H. K. Lee, R. Verba, J. A. Katine, I. Barsukov, V. Tiberkevich, J. Q. Xiao, A. N. Slavin, and I. N. Krivorotov, Parametric resonance of magnetization excited by electric field, Nano letters 17, 572 (2017).
  * Guo  _et al._ [2014] F. Guo, L. M. Belova, and R. D. McMichael, Parametric pumping of precession modes in ferromagnetic nanodisks, Physical Review B 89, 104422 (2014).
  * Patton and Jantz [1979] C. E. Patton and W. Jantz, Anomalous subsidiary absorption in single-crystal yig and evaluation of spin-wave linewidth, Journal of Applied Physics 50, 7082 (1979).
  * Demidov  _et al._ [2011] V. E. Demidov, S. Urazhdin, E. Edwards, M. D. Stiles, R. D. McMichael, and S. O. Demokritov, Control of magnetic fluctuations by spin current, Physical review letters 107, 107204 (2011).
  * Själander  _et al._ [2019] M. Själander, M. Jahre, G. Tufte, and N. Reissmann, EPIC: An energy-efficient, high-performance GPGPU computing research infrastructure (2019), arXiv:1912.05848 [cs.DC] .


#  LooseControl: Lifting ControlNet for Generalized Depth Conditioning

Shariq Farooq Bhat
KAUST  Niloy J. Mitra
University College London, Adobe Research
Peter Wonka
KAUST

###### Abstract

We present LooseControl to allow generalized depth conditioning for diffusion-
based image generation. ControlNet, the SOTA for depth-conditioned image
generation, produces remarkable results but relies on having access to
detailed depth maps for guidance. Creating such exact depth maps, in many
scenarios, is challenging. This paper introduces a generalized version of
depth conditioning that enables many new content-creation workflows.
Specifically, we allow (C1) scene boundary control for loosely specifying
scenes with only boundary conditions, and (C2) 3D box control for specifying
layout locations of the target objects rather than the exact shape and
appearance of the objects. Using LooseControl, along with text guidance, users
can create complex environments (e.g., rooms, street views, etc.) by
specifying only scene boundaries and locations of primary objects. Further, we
provide two editing mechanisms to refine the results: (E1) 3D box editing
enables the user to refine images by changing, adding, or removing boxes while
freezing the style of the image. This yields minimal changes apart from
changes induced by the edited boxes. (E2) Attribute editing proposes possible
editing directions to change one particular aspect of the scene, such as the
overall object density or a particular object. Extensive tests and comparisons
with baselines demonstrate the generality of our method. We believe that
LooseControl can become an important design tool for easily creating complex
environments and be extended to other forms of guidance channels. Code and
more information is available at https://shariqfarooq123.github.io/loose-
control/.111The project was supported in part by “NTGC-AI” funding at KAUST.

Figure 1: Our framework LooseControl enables multiple ways to control the
generative image modeling process. Left: (C1) Scene boundary control lets the
user specify the boundary of the scene. We show the control inputs on top, the
ControlNet results in the middle, and our results at the bottom. Middle: (C2)
3D box control can additionally specify object locations with the help of
approximate 3D bounding boxes. We again show the control inputs on top, the
ControlNet results in the middle, and our results on the bottom. Right-Top:
(E1) 3D box editing. Note how the overall style of the scene is preserved
across edits. Right-Bottom: (E2) Attribute editing: an example of changing a
couch to another one and changing the overall furniture density in a room.

##  1 Introduction

Diffusion-based generative models now produce images with a remarkable degree
of photorealism. ControlNet [51], trained on top of StableDiffusion [37], is
the most powerful way to control such a generation process. Specifically,
ControlNet allows guidance in the form of one or more of depth, edges, normal,
or semantic channels. This ability to accurately control the image generation
process enables various creative applications without requiring different
architectures for different applications. However, providing perfect guidance
for ControlNet can itself be challenging.

For example, imagine a scenario where a user wants to create living room
images using depth guidance. She is now expected to supply a depth map
containing information about walls, room furniture, all the way to even
smaller objects and decorations. This is non-trivial. Producing realistic
depth maps, especially for cluttered scenes, is probably as challenging as
solving her original task. On the one hand, providing rough guidance, e.g.,
providing depth information for walls, floor, and ceiling, results in
unsatisfactory images. While one may expect a furnished room, the room will be
empty. Also providing approximate depth using target bounding boxes for
furniture does not yield the desired result, because only boxy objects are
generated. Figure 1 shows some examples using ControlNet [51].

We present LooseControl that allows controlling image generation using
generalized guidance. In this work, we focus on guidance through depth maps.
We consider two types of specifications. (C1) Scene boundary control where a
layout can be given by its boundaries, e.g., walls and floors, but the final
scene can be filled by an arbitrary number of objects that are not part of the
depth conditioning as long as these additional objects are closer to the
camera than the scene boundary. (C2) 3D box control where, in addition to
layout boundaries, users can provide finer-scale guidance in the form of
approximate bounding boxes for target objects. In the final generation,
however, there can be additional secondary objects that do not need to
strictly adhere to the given boxes. We demonstrate that many layout-guided
image generation scenarios (e.g., rooms, streets, underwater) can be addressed
in this framework. Figure 1 (left, middle) shows some of our generations.

We also provide two interactive editing modes to refine the results. (E1) 3D
box editing enables the user to change, add, and remove boxes while freezing
the style of the resulting image. The goal is to obtain minimal changes apart
from the changes induced by the edited boxes. We describe how a notion of
style can be formalized and preserved in our diffusion setup. (E2) 3D
attribute editing enables users to change one particular attribute of the
result, such as the type of one particular piece of furniture. Since diffusion
processes return a distribution of results, we perform local analysis around
any generation to reveal dominant modes of variation. Figure 1 (right) shows a
few editing examples.

The technical realization of these four components is built on ControlNet with
a frozen StableDiffusion backbone. We first propose a Low Rank (LoRA) based
network adaptation. The LoRA-based architecture allows the network to be fine-
tuned in a few steps with only a small amount of training data, preventing the
network from forgetting the original generation weights. A second important
component is automatically synthesizing the necessary training data without
requiring manual annotations. Finally, the edits are realized by manipulating
the “keys” and “values” in attention layers, and a singular value analysis of
the ControlNet Jacobian.

We evaluate a range of use cases. Since we enable a strictly more general
depth control, there are no direct competitors. Hence, we compare strict and
weaker guidance by varying the control weighting in ControlNet, as well as
retraining ControlNet on synthetic data. We evaluate our setup on a variety of
different scenes and also perform a user study that reveals over $95\%$
preference for our method compared to baselines. In summary, we are the first
to allow image generation from loose depth specifications and provide multiple
types of edits to semantically explore variations in generations.

##  2 Related Work

Diffusion models have rapidly evolved as a leading generative approach,
demonstrating remarkable success both in 2D image generation [18, 33, 14, 37,
32, 35, 6, 39] as well as 3D shape generation [20, 40, 12, 52, 50, 15, 21,
11]. While there are many similarities and synergies between 2D and 3D
diffusion, in this paper, we only focus on the problem of adding conditional
control to text-to-image diffusion models. In this context, many existing
methods proposed solutions on direct guidance through well-known condition
inputs like inpainting masks [48, 45], sketches [44], scene graphs [49], color
palletes [43, 30, 19], 2D bounding boxes [27], segmentation maps [37, 46, 13],
composition of multiple text descriptions [28], depth maps [51, 30] or by
fine-tuning these models on a few subject-specific images [16, 29, 38, 42]. We
focus on introducing new types of control. ControlNet [51] stands out as a key
method in spatial control, supporting a diverse array of conditions like edge
maps, depth maps, segmentation masks, normal maps, and OpenPose [9, 41, 8, 47]
under a single framework. It is notably based on the widely-used open-source
model, Stable Diffusion [37], which contributes to its popularity. However,
creating spatial control conditions manually by a user for ControlNet can be
challenging, often resulting in indirect control where conditions are derived
from another image source, which can stifle the creative generation process.
Additionally, this method can be restrictive in terms of the diversity of
generations per condition, posing limitations in various scenarios. Our work
builds on top of ControlNet and introduces novel conditioning mechanisms that
are not only easy to construct manually by a user but also offer enhanced
control and flexibility. In particular, we contribute novel forms of loose
control described in the next section.

Figure 2: Pipeline for extracting boundary depth from an image. From left to
right: Input image and its estimated depth map, back-projected 3D mesh,
orthographic projection of the mesh on a horizontal plane, polygon
approximation of the 2D boundary, extrusion of poly sides, resulting boundary
depth map. For ease of visualization, the ceiling is not shown.

##  3 Problem Setup - LooseControl

We formally introduce the core problem statement proposed in this work - loose
depth control. Ordinary depth control, as implemented by the original
ControlNet can be formally described as follows: Given an input condition
depth map ${D}_{c}$, and access to an off-the-shelf monocular depth estimator
$f_{D}$, generate an image $I_{gen}$ such that the estimated depth
$f_{D}(I_{gen})$ respects the input depth condition _i.e_.:

$\text{Generate $I_{gen}$ such that }f_{D}({I}_{gen})=D_{c}.$ (1)

Thus, by design, the conditioning imposed by ControlNet is strict: as per
training, the equality in Eq. 1 must exactly hold. Our goal is to extend this
notion of control to a more flexible generalized form. We therefore define
generalized control via the following setup: Given an input condition depth
map ${D}_{c}$, and access to an off-the-shelf monocular depth estimator
$f_{D}$, and an arbitrary Boolean condition function $\phi(\cdot,\cdot)$:

$\text{Generate $I_{gen}$ such that }\phi(f_{D}(I_{gen}),D_{c})\text{ is
{true}}.$ (2)

It is easy to observe that Eq. 1 is a special case of Eq. 2. In this work, we
propose to consider two other cases: scene boundary control and 3D box
control, as described next.

(C1) Scene boundary control. In this case, we impose the condition such that
the input depth condition $D_{c}$ only specifies the tight upper bound of
depth at each pixel:

$\phi:f_{D}(I_{gen})\leq D_{c}.$ (3)

(C2) 3D box control. In this case, we let the condition $D_{c}$ control only
the approximate position, orientation, and size of objects by specifying their
approximate 3D bounding boxes. This leads to finer control than (C1) yet still
avoids specifying the exact shape and appearance of the objects. Essentially,
we design a condition function $\phi$ that ensures that the objects
$O_{gen}^{i}$ generated in an image conform to their respective specified 3D
bounding boxes $B_{i}$:

$\phi:B_{i}\sim\text{3DBox}(O_{gen}^{i})\quad\forall i,$ (4)

where $\text{3DBox}(O_{gen}^{i})$ represents the oriented 3D bounding box of
the $i$-th object segment in the generated image. This means that the
position, size, and orientation of each object will be approximately within
the bounds set by its corresponding 3D bounding box. Although our 3D box
control training is strict, we show that boxes can be treated as only
approximate, which proves to be highly beneficial in practice.

Both Eq. 3 and Eq. 4 specify a form of depth condition that is strictly more
general than the ordinary control such as realized by ControlNet. We term such
conditioning as LooseControl.

##  4 Realizing LooseControl

To realize ordinary depth control of text-to-image diffusion models, one needs
access to triplets of the form $(T,D_{c},I)$ for training, where $T$ is the
text description of the image $I$ and $D_{c}$ represents the depth condition.
Generally, one has access to the pairs $(T,I)$ and the ordinary depth
condition $D_{c}$ is obtained by applying an off-the-shelf depth estimator
$f_{D}$ on the given image (i.e., $D_{c}=f_{D}(I)$) to obtain the triplets
$(T,D_{c}=f_{D}(I),I)$. However, in our case, the depth condition $D_{c}$ must
take a more generalized form. For our goal, the form of $D_{c}$ is constrained
by three main requirements: (i) Compatibility with ControlNet, where the depth
condition should resemble a conventional depth map. (ii) Efficient to extract
from a given image without manual annotation. (iii) Easy to construct manually
by a user. We describe below how to obtain the appropriate form of $D_{c}$
such that LooseControl is realized.

###  4.1 How to represent scenes as boundaries?

We begin by outlining the estimation of the depth condition $D_{c}$ from a
given image for implementing scene boundary control as described in Eq. 3. In
this context, we seek a depth condition that acts as an upper depth limit for
individual pixels. We propose to extract the scene boundary surface for upper-
bound training in the following manner. Specifically, we define the boundary
as a set of planar surfaces encompassing the scene that accurately delineates
the scene’s spatial extent. For a given bounded 3D scene and a specific camera
viewpoint, the boundary depth is defined as the z-depth of these boundary
surfaces. In practice, this means that all pixels in the image inherit the
depth value associated with the boundary surface, even if the boundary
surfaces are occluded by objects within the scene’s interior. To provide a
concrete example, consider an indoor scene (refer to Fig. 3). The boundary
typically encompasses the walls, ceiling, and floor planes. Consequently, a
boundary depth map for an indoor scene exclusively reflects the depths of
walls, ceiling, and floor, irrespective of the presence of room furniture.

A naive approach to extracting boundary depth involves leveraging annotated
room layout datasets. However, such datasets often contain ambiguous
annotations in multi-room images, and the room-level annotations directly
violate our “upper” bound condition (see Supplementary Materials).
Additionally, this approach necessitates manual annotation for boundary
extraction and confines the scope to room-centric scenarios. Our objective is
to devise a strategy that is applicable across diverse scene types and readily
applicable to extract boundary depth from any given image. To this end, we
propose the following algorithm:

We use a multi-step approach to efficiently extract boundary depth from a
given image. Refer to Fig. 2 for the outline of our approach. We begin by
estimating the depth map of the given image using an off-the-shelf monocular
depth estimator [7]. We then back-project the image into a 3D triangular mesh
within the world space. Our goal is to extract the planar surfaces that
encompass this mesh. For efficiency, we only make use of vertical planes
during training. This reduces the 3D boundary extraction problem to 2D. We
project the 3D mesh of the scene onto a horizontal plane using orthographic
projection. This projection facilitates the precise delineation of the 2D
boundary that encapsulates the scene. We then approximate the 2D boundary of
projection with a polygon and extrude the sides into planes matching the scene
height. For an indoor scene, for example, these ‘poly’-planes represent a good
approximation of wall planes. These poly planes and optionally the ground and
ceiling plane (if available) together form the boundary surface. We then
render the depth of this boundary surface from the original camera view to get
the boundary depth map $D_{c}$, serving as a proxy for the scene boundary
condition.

Figure 3: Illustration of proxy depth for scene boundary control training
(left) and 3D box control training (right).

###  4.2 How to represent scenes as 3D boxes?

We now describe the estimation of $D_{c}$ from a given image to realize 3D box
control (Eq. 4). For a given image, our goal is to construct $D_{c}$ such that
it contains the information about the 3D bounding boxes of the objects present
in the scene. At the same time, we need to ensure compatibility with depth-
conditioned frameworks, such as ControlNet, where the depth condition should
resemble a conventional depth map. To achieve this, our idea is to obtain 3D
bounding boxes of the objects in the image and render the depth of the boxes.
The resulting depth map can serve as a proxy for specifying 3D bounding boxes
(See Fig. 3). However, this task necessitates a robust monocular 3D object
detector, which, currently, presents challenges. Existing monocular 3D object
detection pipelines tend to be either sparse, targeting specific object
categories, or domain-specific, focusing on scenarios like road scenes,
limiting their versatility. To address these limitations, we introduce a
custom monocular pipeline that approximates the 3D bounding boxes of objects
and is simultaneously dense (capable of recovering any object type) and
generalizable (no domain preference).

For a given image, we first obtain its depth map and segmentation map using an
off-the-shelf monocular depth estimator ZoeDepth [7] and SAM [24],
respectively. The depth map provides 3D spatial information, while the
segmentation map delineates object boundaries, which we use together to
approximate 3D bounding boxes of objects. For each segment, we perform back-
projection, transforming the image segment using its depth map into a point
cloud in 3D world space. We then estimate the corresponding oriented 3D
bounding box with minimal volume for each back-projected segment point cloud
and represent it as a cuboidal mesh. Finally, to obtain $D_{c}$, we render
this scene of boxes using the original camera parameters used in back-
projection and extract the depth map. This process is summarized in Fig. 5.

Figure 4: Training pipelines for ControlNet (Top) and LooseControl (Bottom).
Proxy Estimator represents our proxy depth extraction algorithms. During
inference, we enable an option for a user to design the condition depth
manually via a UI. Figure 5: Pipeline for extracting proxy depth for 3D box
control from an image. Left to right: Input image, its estimated depth and
segmentation maps, back-projected 3D mesh, 3D lifting of segmentation, 3D
bounding boxes for segments, resulting proxy depth map.

###  4.3 How to train for LooseControl?

We build our framework on StableDiffusion [37] and ControlNet [51]. In both
scene boundary control and 3D box control, the constructed depth condition
$D_{c}$ serves as a proxy for specifying a more generalized control - scene
boundary and 3D bounding box, respectively - while retaining compatibility
with ordinary depth-conditioned frameworks such as ControlNet. This ensures a
smooth transition and facilitates the easy adoption of these approaches.
Thanks to this backward compatibility, we can efficiently fine-tune a pre-
trained depth ControlNet to achieve generalized depth control. We prepare the
triplets (T, $D_{c}$, I) from the given (T, I) pairs where $D_{c}$ is
constructed using the algorithms presented in previous sections. We explore
two primary options for fine-tuning: (i) Naive fine-tuning: where we finetune
the entire ControlNet; and (ii) LoRA fine-tuning: where we employ LoRA-based
fine-tuning of ControlNet (see Fig. 4). Specifically, for every attention
block in ControlNet, we learn a low-rank update:

$W^{\prime}\mathbf{x}=W\mathbf{x}+BA\mathbf{x}$ (5)

where $W\in\mathbb{R}^{M\times N}$ is the original frozen projection,
$B\in\mathbb{R}^{M\times r}$ and $A\in\mathbb{R}^{r\times N}$ are trainable
low rank matrices of rank $r\ll min(M,N)$. It is important to note that in
both cases Stable Diffusion U-net remains frozen during fine-tuning. For both
control types (C1 and C2), we use the NYU-Depth-v2 [31] dataset for fine-
tuning and obtain textual image descriptions using BLIPv2 [26] captioning.

Figure 6:  Given scene boundary (left), comparing naive fine-tuning, adjusted
skip weights, and our LoRA-based training, resp.

Inference time adjustments. We observe that naive fine-tuning results in
saturated generations (Fig. 6). Interestingly, the color artifacts are
eliminated if the residuals injected into Stable Diffusion U-net from
ControlNet are reweighted at inference time. We observe that residual to the
bottleneck block is responsible for most of the condition adherence and other
higher resolution skip residuals largely add only local texture information.
On the other hand, LoRA fine-tuning proves to be robust without any color
issues. However, we introduce a controllable scale factor $\gamma$ at
inference time such that the update is given as:
$W^{\prime}\mathbf{x}=W\mathbf{x}+\gamma BA\mathbf{x}$. We observe that
controlling $\gamma$ can lead to higher quality results, especially for
$\gamma>1$. We use $\gamma=1.2$ as the default.

###  4.4 3D Box Editing

Both the scene boundary condition and the box condition are easy to manipulate
by a user. This opens up possibilities where a user can interactively
manipulate the scene and objects, for example, by changing, adding, or
removing the boxes. However, it is desirable to ‘lock in’ and maintain the
style and composition of the scene while manipulating the condition. A naive
way to maintain the overall style would to be fix the seed but we observe that
changing the condition even when fixing the seed can lead to diverging
generations. To this end, we propose Style Preserving Edits, inspired by video
diffusion models [22], which allows users to maintain the desired style of the
scene while conducting sequential edits. This feature is realized by replacing
the “keys” and “values” for the given image in the attention layers of the
Stable Diffusion U-net with those from the source image. However, in our case,
sharing “keys” and “values” through all layers of the U-net leads to
undesirable results, producing images identical to source images. We find that
sharing “keys” and “values” from only the last two decoder layers ensures the
preservation of the desired identity or style while adhering to the new target
condition.

Figure 7: Qualitative results of (C1) Scene boundary control (rows 1-3) and
(C2) 3D Box control (rows 4-6). Top: Input depth condition Middle: ControlNet
generations. Bottom: Ours. Our generations are more realistic and adhere to
the prompt better than ControlNet.

###  4.5 Attribute Editing

LooseControl’s generalized nature compared to ordinary depth control allows
for a wider range of possible scenes under a given condition. To explore this
expansive space, we conduct various forms of latent exploration.

ControlNet produces two types of residuals for a given condition: residuals
added to the SD U-net bottleneck and residuals added to decoder skip
connections. Our focus centers on the bottleneck feature space
($\mathcal{H}$-space), known for its semantic behavior [25, 17] and from our
experiments, its pivotal role in condition adherence.

In our exploration, we examine how the latent input, $\mathbf{x}$, affects
ControlNet’s output for a given fixed condition. We extract the most
influential directions in x-space, which cause substantial changes in
$\Delta\mathbf{h}\in\mathcal{H}$, and find their counterparts in
$\mathcal{H}$-space, using Singular Value Decomposition (SVD) of the
ControlNet Jacobian, $\partial\Delta\mathbf{h}/\partial\mathbf{x}$. To keep
the SVD computation feasible, we only extract the first N directions, labeled
$\\{\mathbf{e}_{i}\\}$, which guide our modification of the bottleneck
residual via:
$\Delta\mathbf{h}^{\prime}=\Delta\mathbf{h}+\beta\mathbf{e}_{i}$, where the
scalar $\beta$ controls the magnitude of the edit.

Per-condition latent exploration enhances user control by revealing various
semantic edit directions, for example, sizes, types, and even the number of
objects in the scene. This gives rise to a continuous control space, allowing
for fine-grained adjustments that were challenging with conventional
conditioning methods. This continuous control space operates orthogonal to the
input depth condition, enabling the maintenance of specified
conditions—whether scene boundary control or 3D box control—while
simultaneously exploring a wide range of generated images that adhere to the
given condition.

Figure 8: Qualitative results of (E1) 3D Box Editing, and (E2) Attribute
Editing. (E1) 3D Box editing enables us to move and re-orient objects in the
scene, change their size, split them, and more. (E2) Attribute Editing lets us
change properties like the density of furniture (e.g., increase or decrease
density) and the material of objects (e.g., fabric to leather or vica versa).

##  5 Applications and Results

We developed an interface to enable users to conveniently design a scene
boundary and 3D boxes, and generate images interactively (see supplementary
material). Here, we present qualitative and quantitative results.

\begin{overpic}[width=433.62pt,percent,tics=5]{figures/ablation_cond_scale-
compressed.pdf} \par\par\end{overpic} Figure 9: Effect of conditioning control
scale $\lambda$. Top: ControlNet. Bottom: Ours. The results were produced for
the prompt “A cozy coastal bedroom with a sea view”.

#### (C1) Scene Boundary Control.

Scene boundary control enables diverse applications across various domains. We
present some examples and comparisons between LooseControl and ordinary
control in Fig. 7. One particularly noteworthy application of scene boundary
control is in indoor scene generation. Users can provide somewhat abstract
specifications for room layouts, generally confined to walls and floor, and
still generate images of fully furnished rooms. This is different from
ordinary depth conditioning, which demands precise depth information.

Another scenario involves (partially) bounded outdoor scenes where the user
can provide the scene boundary depth, for example, in terms of locations of
buildings (See Fig. 1). Unlike the ordinary depth control implemented by the
original ControlNet, which leads to empty scenes, our LooseControl model
excels at generating realistic scenes, inclusive of objects like cars, traffic
lights, and pedestrians, all while adhering to the user-provided scene
boundary condition. This feat underscores the reliability and generalization
capability of LooseControl via proxy fine-tuning.

In Fig. 9, we show the effect of varying the conditioning scale, $\lambda$, in
the context of scene boundary control. We observe that ControlNet still
produces ‘empty’ rooms with lesser prompt adherence for most of the
conditioning scales.

#### (C2) 3D Box Control.

We present generations and comparisons with ControlNet for 3D box control for
a variety of scenes in Fig. 7. ControlNet with ordinary control produces
‘boxy’ generations which are not only unrealistic but also result in lesser
prompt adherence. On the other hand, our method is able to produce more
realistic images and shows better adherence to the given prompt demonstrating
the utility of such control.

#### (E1) 3D Box Editing.

We show qualitative results for 3D Box Editing in Fig. 8. 3D box Editing
allows users to manipulate objects, for example, by changing their position in
3D space, their orientation, and size. Users can perform 3D-aware edits, such
as relocating furniture within a room or converting a small sofa chair to a
big sofa. The Style Preserving Edit mechanism facilitates sequential editing
of object poses and locations, preserving the scene’s essence while
introducing creative modifications. Since the 3D box control implicitly
specifies the rough shape and size of the objects, we can edit the boxes
themselves such as splitting a box into two, resulting in semantic edits on
objects.

(E2) Attribute Editing. Extracting the edit directions for a fixed scene
boundary leads to continuous control directions for attributes such as the
density of furniture and furniture type. For a given 3D box control condition
(see Fig. 8), the edit directions represent attributes like the shape,
thickness, and material properties such as “leather”. We observe that edit
directions are smooth and continuous for the most part and varying the
magnitude $\beta$ directly reflects that magnitude of change in the attribute.

#### User Study.

We conducted a user study to quantify the

improvement in control and quality of generations. 41 users were presented
with the prompt and condition and asked to pick the image they preferred
between ours and ControlNet. A majority of users, more than $95\%$, ranked our
result better. See inset for results.

##  6 Conclusion

We have presented LooseControl to support generalized depth control for image
generation. We introduced two types of control: (C1) Scene boundary control
and (C2) 3D Box control to generate initial images. To refine initial results,
we proposed (E1) 3D box editing and (E2) Attribute editing. Our framework
provides new modes of creative generation and editing allowing users to more
effectively explore the design space with depth guidance. A user study
revealed over $95\%$ preference for LooseControl compared to previous work.

Limitations. Although our method works well for primary objects, control over
secondary objects is harder to achieve – we attribute this to the training
data where secondary objects are less common. We expect a scene-based
generator may lead to improved results. Also, similar to the original
ControlNet we find that providing too many constraints as input reduces the
diversity of the results.

Future work. We would like to use LooseControl in conjunction with masks and
also with MultiControlNet through the generalized specification of other maps.
For example, in the context of sketch-based generation, specifying only a
selection of (dominant) edges is easier for users than drawing all the Canny
edges. Further, we would like to explore temporally consistent generation to
produce output videos under smoothly changing guidance as in an interpolated
6DOF object specification.

## References

  * [1] https://huggingface.co/.
  * [2] https://www.babylonjs.com/.
  * [3] https://huggingface.co/docs/diffusers/index.
  * [4] https://streamlit.io/.
  * Abid et al. [2019] Abubakar Abid, Ali Abdalla, Ali Abid, Dawood Khan, Abdulrahman Alfozan, and James Zou.  Gradio: Hassle-free sharing and testing of ml models in the wild.  _arXiv preprint arXiv:1906.02569_ , 2019.
  * Balaji et al. [2022] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Qinsheng Zhang, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu.  ediff-i: Text-to-image diffusion models with ensemble of expert denoisers.  _arXiv preprint arXiv:2211.01324_ , 2022.
  * Bhat et al. [2023] Shariq Farooq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller.  Zoedepth: Zero-shot transfer by combining relative and metric depth.  _arXiv preprint arXiv:2302.12288_ , 2023.
  * Cao et al. [2017] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In _CVPR_ , 2017.
  * Cao et al. [2019] Z. Cao, G. Hidalgo Martinez, T. Simon, S. Wei, and Y. A. Sheikh.  Openpose: Realtime multi-person 2d pose estimation using part affinity fields.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2019.
  * Chang et al. [2017] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang.  Matterport3d: Learning from rgb-d data in indoor environments.  _arXiv preprint arXiv:1709.06158_ , 2017.
  * Chen et al. [2023] Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su.  Single-stage diffusion nerf: A unified approach to 3d generation and reconstruction.  In _ICCV_ , 2023.
  * Chou et al. [2023] Gene Chou, Yuval Bahat, and Felix Heide.  Diffusion-SDF: Conditional generative modeling of signed distance functions, 2023.
  * Couairon et al. [2022] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.  Diffedit: Diffusion-based semantic image editing with mask guidance.  _arXiv preprint arXiv:2210.11427_ , 2022.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Erkoç et al. [2023] Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai.  Hyperdiffusion: Generating implicit neural fields with weight-space diffusion.  _arXiv preprint arXiv:2303.17015_ , 2023.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ , 2022.
  * Haas et al. [2023] René Haas, Inbar Huberman-Spiegelglas, Rotem Mulayoff, and Tomer Michaeli.  Discovering interpretable directions in the semantic latent space of diffusion models.  _arXiv preprint arXiv:2303.11073_ , 2023.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Huang et al. [2023] Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou.  Composer: Creative and controllable image synthesis with composable conditions.  _arXiv preprint arXiv:2302.09778_ , 2023.
  * Hui et al. [2022] Ka-Hei Hui, Ruihui Li, Jingyu Hu, and Chi-Wing Fu.  Neural wavelet-domain diffusion for 3d shape generation.  In _SIGGRAPH Asia 2022 Conference Papers_ , New York, NY, USA, 2022. Association for Computing Machinery.
  * Karnewar et al. [2023] Animesh Karnewar, Niloy J Mitra, Andrea Vedaldi, and David Novotny.  Holofusion: Towards photo-realistic 3d generative modeling.  _ICCV_ , 2023.
  * Khachatryan et al. [2023] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi.  Text2video-zero: Text-to-image diffusion models are zero-shot video generators.  _arXiv preprint arXiv:2303.13439_ , 2023.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023.
  * Kwon et al. [2022] Mingi Kwon, Jaeseok Jeong, and Youngjung Uh.  Diffusion models already have a semantic latent space.  _arXiv preprint arXiv:2210.10960_ , 2022.
  * Li et al. [2023a] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ , 2023a.
  * Li et al. [2023b] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee.  Gligen: Open-set grounded text-to-image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22511–22521, 2023b.
  * Liu et al. [2022] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum.  Compositional visual generation with composable diffusion models.  In _European Conference on Computer Vision_ , pages 423–439. Springer, 2022.
  * Mokady et al. [2023] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6038–6047, 2023.
  * Mou et al. [2023] Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie.  T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models.  _arXiv preprint arXiv:2302.08453_ , 2023.
  * Nathan Silberman and Fergus [2012] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus.  Indoor segmentation and support inference from rgbd images.  In _ECCV_ , 2012.
  * Nichol et al. [2021] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Nichol and Dhariwal [2021] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pages 8162–8171. PMLR, 2021.
  * Paszke et al. [2019] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32, 2019.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Ravi et al. [2020] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari.  Accelerating 3d deep learning with pytorch3d.  _arXiv preprint arXiv:2007.08501_ , 2020.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 10684–10695, 2022.
  * Ruiz et al. [2023] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22500–22510, 2023.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Shue et al. [2023] J. Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein.  3d neural field generation using triplane diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20875–20886, 2023.
  * Simon et al. [2017] Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh.  Hand keypoint detection in single images using multiview bootstrapping.  In _CVPR_ , 2017.
  * Tewel et al. [2023] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon.  Key-locked rank one editing for text-to-image personalization.  _ACM SIGGRAPH 2023 Conference Proceedings_ , 2023.
  * Vavilala and Forsyth [2023] Vaibhav Vavilala and David Forsyth.  Applying a color palette with local control using diffusion models, 2023.
  * Voynov et al. [2023] Andrey Voynov, Kfir Aberman, and Daniel Cohen-Or.  Sketch-guided text-to-image diffusion models.  In _ACM SIGGRAPH 2023 Conference Proceedings_ , pages 1–11, 2023.
  * Wang et al. [2023] Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al.  Imagen editor and editbench: Advancing and evaluating text-guided image inpainting.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 18359–18369, 2023.
  * Wang et al. [2022] Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen.  Pretraining is all you need for image-to-image translation.  _arXiv preprint arXiv:2205.12952_ , 2022.
  * Wei et al. [2016] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh.  Convolutional pose machines.  In _CVPR_ , 2016.
  * Xie et al. [2023] Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang.  Smartbrush: Text and shape guided object inpainting with diffusion model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 22428–22437, 2023.
  * Yang et al. [2022] Ling Yang, Zhilin Huang, Yang Song, Shenda Hong, Guohao Li, Wentao Zhang, Bin Cui, Bernard Ghanem, and Ming-Hsuan Yang.  Diffusion-based scene graph to image generation with masked contrastive pre-training.  _arXiv preprint arXiv:2211.11138_ , 2022.
  * Zhang et al. [2023a] Biao Zhang, Jiapeng Tang, Matthias Nießner, and Peter Wonka.  3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models.  _ACM Trans. Graph._ , 42(4), 2023a.
  * Zhang et al. [2023b] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023b.
  * Zheng et al. [2023] Xin-Yang Zheng, Hao Pan, Peng-Shuai Wang, Xin Tong, Yang Liu, and Heung-Yeung Shum.  Locally attentional SDF diffusion for controllable 3d shape generation.  _ACM Trans. Graph._ , 42(4):91:1–91:13, 2023.
  * Zhou et al. [2018] Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun.  Open3d: A modern library for 3d data processing.  _arXiv preprint arXiv:1801.09847_ , 2018.

Supplementary Material

Figure 10: Upper-bound violations in Matterport3D [10]. Top: RGB panorama
images. Bottom: Depth rendered using the layout labels from the dataset.

##  7 Implementation details

For all our experiments, we use Stable Diffusion v1.5 [37] and the
corresponding ControlNet [51] checkpoint
“lllyasviel/control_v11f1p_sd15_depth” hosted on HugginFace [1]. We use the
PyTorch [34] framework and the diffusers [3] library as the framework for
diffusion models. We use ZoeDepth [7] and SAM [24] for extracting depth maps
and segmentation maps respectively. For SAM, we use `min_mask_area=1e4`. We
make use of Pytorch3D [36] and Open3D [53] in our 3D framework and PyTorch3D
for rendering the depth maps for obtaining the proxy depths for fine-tuning.
For backprojection, we randomly choose an FOV in the range of 43 and 57
degrees. For both Scene Boundary Control and 3D Box Control, we use LoRA rank
$r=8$ and fine-tune only LoRA layers for $200$ steps with a learning rate of
$0.0001$, and batch size of $12$ with Adam [23] optimizer. We use
`controlnet_conditioning_scale=1.0` and LoRA scale factor $\gamma=1.2$ as
default. We built our 3D editor user interface using Gradio [5] and BabylonJS
[2].

##  8 Issues with Room Layout Datasets

As discussed in the main paper, a possible alternative for extracting the
scene boundary for the preparation of the dataset for Scene Boundary Control
could be room layout datasets. However, we note that these datasets often
contain ambiguous layout labels that directly violate our tight upper-bound
condition required to implement scene boundary control. We provide some
examples of these violations for the popular MatterPort3D dataset as the
representative in Fig. 10

##  9 User study additional details

We created an anonymous user study as a web form using streamlit [4]. Users
were presented with ‘Two-alternative forced choice’ (2-AFC) with two options
as images generated by our baseline (ControlNet) and our result and asked to
respond with their preference for the given text prompt and condition image.
The options were anonymized and did not indicate the name of the method. Each
user was asked to respond to 10 randomized questions in total (5 for Scene
Boundary Control, and 5 for 3D box control). The order of options was also
randomized. As mentioned in the main paper, over 95% of responses were in
favor of our method.