# Quantum transport in dispersionless electronic bands

Alexander Kruchkov  Institute of Physics, Swiss Federal Institute of
Technology (EPFL), Lausanne, CH 1015, Switzerland  Department of Physics,
Harvard University, Cambridge, Massachusetts 02138, USA  Branco Weiss Society
in Science, ETH Zurich, Zurich, CH 8092, Switzerland

###### Abstract

Flat electronic bands are counterintuitive: with the electron velocity
vanishing, our conventional notions of quasiparticle transport are no longer
valid. We here study the quantum transport in the generalized families of
perfectly flat bands [PRB 105, L241102 (2022)], and find that while the
conventional contributions indeed vanish, the quantum-geometric contribution
gives rise to the enhanced electronic transport. This contribution is
connected to the Wannier orbital quantization in the perfectly flat bands, and
is present only for geometrically-nontrivial bands (for example, flat Chern
bands). We find structurally similar expressions for thermal conductance,
thermoelectric response, and superfluid weight in the flat bands. In
particular, we report the anomalous thermopower associated with flat
topological bands reaching values as large as $\frac{k_{B}}{e}\ln
2$$\approx$60 $\mu$V/k, the quantum unit of thermopower, which are not
expected for the conventional dispersive bands.

Dispersionless electronic states (”flat bands”) are counterintuitive since the
effective electronic mass becomes infinite, the quasiparticle velocity
vanishes, and the conventional notions of electron transport fail. The
perfectly flat electronic bands is a change of paradigm in condensed matter
physics, but remained largely a hypothetical object until the discovery of the
twisted bilayer graphene Cao  _et al._ (2018a, b), where the dispersionless
electronic states are emerging at the magic angle $1.05^{\circ}$Trambly de
Laissardière  _et al._ (2010); Bistritzer and MacDonald (2011). The underlying
flat band is not just a lucky engineering of material parameters, but it is of
fundamental origin as it can be tuned to perfectly flat Tarnopolsky  _et al._
(2019); San-Jose  _et al._ (2012). It was further understood that the magic-
angle flat bands are dual to the lowest Landau level, and host a number of
unconventional phases, characterized by strange metallicity Polshyn  _et al._
(2019); Cao  _et al._ (2020); Jaoui  _et al._ (2022), unconventional
superconductivity Cao  _et al._ (2018b); Lu  _et al._ (2019), fractional Hall
conductance Xie  _et al._ (2021), and giant thermopower Paul  _et al._
(2022)—untypical for conventional electronic systems.

A recent interest in condensed matter physics is understanding the role
quantum geometry of electronic states, described by the quantum geometric
tensor Provost and Vallee (1980); Ma  _et al._ (2010)

$\displaystyle\mathfrak{G}^{(n)}_{ij}=\langle\partial_{k_{i}}u_{n\mathbf{k}}|\left[1-|u_{n\mathbf{k}}\rangle\langle
u_{n\mathbf{k}}|\right]|\partial_{k_{j}}u_{n\mathbf{k}}\rangle$ (1)

here $u_{n\mathbf{k}}$ is the associated Bloch state of the $n$th electronic
band, which may or may not be flat. The real part of
$\mathcal{G}_{ij}$$=$$\text{Re}\mathfrak{G}_{ij}$, is the Fubini-Study metrics
describing the geometry of the bands, while the imaginary part
$\mathcal{F}_{ij}$$=$$-2\text{Im}\mathfrak{G}_{ij}$ is Berry curvature
reflecting the topology of the Bloch states. In particular, it has been
understood that the largely overlooked quantum metrics plays important role in
different quantum transport phenomena, ranging from quantum noise, optical
conductivity, anomalous Hall effect, and unconventional superconductivity, and
adjacent topics Neupert  _et al._ (2011); Haldane (2004); Rhim  _et al._
(2020); Peotta and Törmä (2015); Huhtinen  _et al._ (2022); Gao and Xiao
(2019); Lapa and Hughes (2019); Kozii  _et al._ (2021); Mitscherling and
Holder (2022); Ahn  _et al._ (2022); Chen and Huang (2021). The quantum-
geometric superconductivity Peotta and Törmä (2015) has a particular important
role in twisted bilayer graphene, where it has been shown that the quantum-
geometric contribution to the superfluid weight is key at the magic angle Hu
_et al._ (2019); Hazra  _et al._ (2019); Julku  _et al._ (2020); Xie  _et al._
(2020). Moreover, it has recently been argued that the quantum-geometric
contribution to superfluid weight it TBG can be probed in experiment with the
ultraclean samples Tian  _et al._ (2021). However, other anomalies in the
topological flat bands,—such as e.g. giant thermoelectric power at the magic
angle Paul  _et al._ (2022)—remain to be revisited from the quantum geometric
perspective as well.

In this direction, recent classification of perfectly flat bands Kruchkov
(2022) presents a handy framework as it allows to bridge the properties of
Wannier orbitals, their quantum geometry, and band flatness limits. If we
allow perfect band flatness, the quantum metrics of dispersionless electronic
bands saturates the ”trace condition”

$\displaystyle\text{Tr}\,\mathcal{G}_{ij}(\mathbf{k})=|\mathcal{F}_{xy}(\mathbf{k})|.$
(2)

However, the nontrivial quantum geometry comes at cost that the electronic
Wannier orbitals will have a finite and large cross-section Marzari and
Vanderbilt (1997). In case of perfectly flat Chern bands (2), the Wannier
orbital crossection $r_{0}^{2}$ experiences Lifshitz-Onsager-like quantization
Kruchkov (2022), with

$\displaystyle r_{0}^{2}=a^{2}\int
d\mathbf{k}\text{Tr}\mathcal{G}_{ij}(\mathbf{k})=a^{2}\int
d^{2}\mathbf{k}\mathcal{F}_{xy}(\mathbf{k})=Ca^{2}.$ (3)

In other words, the dispersionless electronic states are spread over the
atomic lattice, overlapping with $C$ neighboring electronic orbitals, hence
allowing quantum tunneling even in the absence of kinetic terms (Fig.1). We
show that this phenomenon promotes the unconventional quantum transport in
dispersionless electronic bands.

In this work we bring to the common denominator different quantum transport
properties (electric conductivity, quantized Hall effect, thermal
conductivity, thermoelectric response, and the superfluid weight), which for
the perfectly flat band systems can be expressed through their multiorbital
quantum metrics:

$\displaystyle\mathfrak{L}_{ij}=\sum_{nm,\mathbf{k}}\mathcal{I}_{nm}(\mathbf{k})\,\text{Re}\mathfrak{G}^{nm}_{ij}(\mathbf{k})+\mathcal{J}_{nm}(\mathbf{k})\,\text{Im}\mathfrak{G}^{nm}_{ij}(\mathbf{k}),$

where $\mathfrak{L}_{ij}$ is a quantum transport characteristic, and
$\mathfrak{G}^{nm}_{ij}(\mathbf{k})\equiv\langle\partial_{k_{i}}u_{n\mathbf{k}}|u_{m\mathbf{k}}\rangle\langle
u_{m\mathbf{k}}|\partial_{k_{j}}u_{n\mathbf{k}}\rangle$ is the generalized
quantum-geometric tensor (defined further in text), and
$\mathcal{I}_{nm}(\mathbf{k})$ and $\mathcal{J}_{nm}(\mathbf{k})$ are system-
dependent structural tensors expressed through quasiparticle propagators. Note
that neither the quasiparticle velocities nor bandwidth enter this expression;
this flat-band transport is purely quantum, with its origin in Wannier
functions overlap (Fig.1).

Figure 1: (Top) In trivial flat bands, electrons are strongly localized
(Wannier orbitals sharp) and electronic transport is forbidden, the system is
in the insulating phase. (Bottom) In nontrivial flat bands (e.g. Chern bands),
the Wannier orbitals cannot be exponentially localized; electrons do not move
in the classical sense. Under application of external fields, electrons tunnel
between overlapping Wannier orbitals, resulting into unconventional
conductivity without electron velocities.

Quantum transport formalism. In what follows below we consider a weakly-
dispersive Chern band, and then set the bandwidth (and hence the Fermi
velocity) to exact zero. The main result is illustrated for the perfectly flat
Chern bands, however it also applies to all geometrically-nontrivial flat
bands (Wannier function are not exponentially localized), thus including those
in twisted bilayer graphene and similar materials. For the moment, we omit the
explicit dependence on magnetic fields, however such generalization can be
done. Up to this moment, the derivation of polarization tensor is rather
conventional and can be obtained in several ways. A disciplined way to derive
it is through using the current-current correlators in Matsubara framework
Mahan (2000). The quantum transport properties are computed through imaginary-
time Matsubara correlators

$\displaystyle\mathcal{L}^{\alpha\beta}_{ij}(\tau,\tau^{\prime})=-\frac{1}{\hbar}\langle\mathcal{T}_{\tau}\,J^{\alpha}_{i}(\tau)Q^{\beta}_{j}(\tau^{\prime})\rangle,$
(4)

where $Q^{\alpha}$ is the generalized ”charge” operator (we denote $\alpha=1$
for electric charge, $\alpha^{\prime}=2$ for heat transfer), $i,j=x,y$ and
$\mathbf{J}^{\alpha}$ is the generalized current. For example, for the
electric current of charge $e$ one writes Mahan (2000)

$\displaystyle\mathbf{J}=\frac{e}{\hbar}\sum_{\mathbf{k}}c^{{\dagger}}_{\mathbf{k}}\frac{\partial\mathcal{H}_{\mathbf{k}}}{\partial\mathbf{k}}c_{\mathbf{k}}.$
(5)

To calculate the response functions $\mathcal{L}_{ij}$, we introduced the
auxiliary current-current correlators

$\displaystyle\Pi^{\alpha\beta}_{ij}(\tau)=-\langle\mathcal{T}_{\tau}\,J^{\alpha}_{i}(\tau)J^{\beta}_{j}(0)\rangle,$
(6)

so in the frequency representation one has
$\mathcal{L}_{ij}(i\omega_{n})=\frac{1}{i\omega_{n}}\left[\Pi_{ij}(i\omega_{n})-\Pi_{ij}(0)\right].$
To calculate the transport properties (such as conductance
$\sigma_{ij}=\mathcal{L}^{11}_{ij}$), we further proceed to analytical
continuation of $T_{ij}(i\omega_{n})$ and then take the DC limit:

$\displaystyle\mathcal{L}^{\alpha\beta}_{ij}\equiv\lim_{\omega\to
0}\frac{\Pi^{\alpha\beta}_{ij}(\omega)-\Pi^{\alpha\beta}_{ij}(0)}{i\omega}.$
(7)

The Onsager coefficients $\mathcal{L}^{\alpha\beta}_{ij}$ fully describe the
transport properties of an electronic system. Experimentally, the transport
measurements across the sample are performed by $I_{e}=\mathcal{L}_{11}\Delta
V+\mathcal{L}_{12}\Delta T$ (electric measurement) and
$I_{h}=\mathcal{L}_{21}\Delta V+\mathcal{L}_{22}\Delta T$ (heat measurement)
Onsager (1931a); *OnsagerB; Rammer (2018). We further compute the electric
conductivity $\sigma_{ij}=\mathcal{L}_{11}$, thermal conductivity
$\kappa_{ij}=\beta(\mathcal{L}_{22}-L^{2}_{12}/T\mathcal{L}_{11})$, and
thermoelectric response (Seebeck coefficient
$\Theta=\frac{\beta}{\mathcal{L}}_{12}/\mathcal{L}_{11}$) in the
dispersionless electronic bands (here $\beta$$=$$1/T$; $e$ is included in
definition of Eq.(5)). A similar response structure to vector potential
$\mathbf{A}$ will imply the finite superfluid weight $D_{\text{S}}$ in the
dispersionless bands Huhtinen  _et al._ (2022).

We start from the electric conductivity, for which we calculate the electric
polarization tensor $\Pi_{ij}(\omega)$ (we here drop the superscripts
$\alpha\beta$ for brevity); for convenience, below we use
$\Pi_{ij}(\omega)=\frac{e^{2}}{\hbar^{2}}\tilde{\Pi}_{ij}(\omega)$. Evaluating
the current-current correlator (6) with (5) in Matsubara representation gives
111We introduce here two quantities $\tilde{\Pi}^{\pm}_{ij}$ in order to
evaluate carefully zero-frequency limit in Eq.(7)

$\displaystyle\tilde{\Pi}^{\pm}_{ij}(i\omega_{0})=\frac{1}{\beta}\sum_{\mathbf{k}}\sum_{i\omega^{\prime}_{n}}\text{Tr}\,\mathcal{G}_{\mathbf{k}}(i\omega^{\prime}_{n})\frac{\partial\mathcal{H}_{\mathbf{k}}}{\partial
k_{i}}\mathcal{G}_{\mathbf{k}}(i\omega^{\prime}_{n}\pm
i\omega_{0})\frac{\partial\mathcal{H}_{\mathbf{k}}}{\partial k_{j}},$

where $\mathcal{G}_{\mathbf{k}}(i\omega^{\prime})$ is the Matsubara transform
of the (renormalized) Green function
$\mathcal{G}_{\mathbf{k}}(\tau,\tau^{\prime})=-\langle\mathcal{T}_{\tau}\,c^{{\dagger}}_{\mathbf{k}}(\tau)c){\mathbf{k}}(\tau^{\prime})\rangle$,
where expectation value is taken over the interacting vaccum at temperature
$T$. Here Matsubara frequencies $i\omega^{\prime}_{n}$ are fermionic and
$i\omega_{0}$ is bosonic.

The influence of the quantum-geometric tensor can be demonstrated in the
following way. The position operator in the Bloch basis is Blount (1962)

$\displaystyle\hat{\mathbf{r}}_{mn}=i\partial_{\mathbf{k}}\delta_{nm}+\langle
u_{n\mathbf{k}}|\,i\partial_{\mathbf{k}}\,u_{m\mathbf{k}}\rangle.$ (8)

It follows that the generalized velocity operator in this basis is given by

$\displaystyle\frac{1}{\hbar}\frac{\partial\mathcal{H}_{\mathbf{k}}}{\partial\mathbf{k}}=\dot{\mathbf{r}}=\mathbf{v}_{n\mathbf{k}}\delta_{nm}+\omega_{nm,\mathbf{k}}\,\langle
u_{n\mathbf{k}}|\partial_{\mathbf{k}}u_{m\mathbf{k}}\rangle,$ (9)

where
$\mathbf{v}_{n\mathbf{k}}=\frac{\partial\varepsilon_{n\mathbf{k}}}{\partial\hbar{\mathbf{k}}}$
is the quasiparticle velocity in the band (”Fermi velocity”) and
$\hbar\omega_{nm,\mathbf{k}}=\varepsilon_{n\mathbf{k}}-\varepsilon_{m\mathbf{k}}$
are transition frequencies of the multiorbital system (the second term is
called ”anomalous velocity” Haldane (2004)). With the velocity operator (9),
the polarization tensor (Quantum transport in dispersionless electronic bands)
have two terms: the first proportional to the Fermi velocities
$\mathcal{O}(\mathbf{v}_{n\mathbf{k}}^{2})$, and the second being independent
of the band dispersion itself. For illustrational purpose, it is useful to
write down the first contribution which has a generic form for the $n$-th band
$\sigma_{0}=\frac{e^{2}}{\hbar}\sum_{n}\sum_{\mathbf{k}}\frac{\partial\varepsilon_{n\mathbf{k}}}{\partial
k_{i}}\frac{\partial\varepsilon_{n\mathbf{k}}}{\partial
k_{i}}S_{n}(\mathbf{k})=0$. The conventional longitudinal contribution
vanishes exactly since the electronic band is perfectly flat (witnessed in
zero Fermi velocity
$\partial_{\mathbf{k}}\varepsilon_{n\mathbf{k}}$$\equiv$$0$ in all the
Brillouin zone). This is where our conventional intuition comes to an edge.

In contrast, by using Eqs.(7)-(9), we find that there is a quantum-geometric
contribution to the DC transport even in the case of perfectly flat bands,

$\displaystyle\sigma_{ij}=\frac{e^{2}}{\hbar}\sum_{\mathbf{k}}\sum_{nm}$
$\displaystyle\mathcal{I}_{nm}(\mathbf{k})\,\text{Re}\mathfrak{G}^{nm}_{ij}(\mathbf{k})$
$\displaystyle+$
$\displaystyle\mathcal{J}_{nm}(\mathbf{k})\,\text{Im}\mathfrak{G}^{nm}_{ij}(\mathbf{k}),$
(10)

where we have introduced the generalized geometric tensor for multiorbital
system

$\displaystyle\mathfrak{G}^{nm}_{ij}(\mathbf{k})\equiv\langle\partial_{k_{i}}u_{n\mathbf{k}}|u_{m\mathbf{k}}\rangle\langle
u_{m\mathbf{k}}|\partial_{k_{j}}u_{n\mathbf{k}}\rangle;$ (11)

note that $\sum_{m\neq n}\mathfrak{G}^{nm}_{ij}=\mathfrak{G}^{(n)}_{ij}$, with
$\mathfrak{G}^{(n)}_{ij}$ given by formula (1). The structural tensors
$\mathcal{I}_{nm}(\mathbf{k})$ and $\mathcal{J}_{nm}(\mathbf{k})$ are fully
determined by the quasiparticle propagators; they are defined in the
symmetrized form as (see SM SM (2))

$\displaystyle\mathcal{I}_{nm}(\mathbf{k})$
$\displaystyle=-2\omega^{2}_{nm,\mathbf{k}}\int\limits_{-\infty}^{+\infty}\frac{d\omega}{2\pi}\,f^{\prime}(\omega)\,\text{Im}\mathcal{G}^{R}_{n\mathbf{k}}(\omega)\text{Im}\mathcal{G}^{R}_{m\mathbf{k}}(\omega),$
$\displaystyle\mathcal{J}_{nm}(\mathbf{k})$
$\displaystyle=8\,\omega^{2}_{nm,\mathbf{k}}\iint\limits_{-\infty}^{+\infty}\frac{d\omega
d\omega^{\prime}}{(2\pi)^{2}}\,\frac{f(\omega)\,\text{Im}\mathcal{G}^{R}_{n\mathbf{k}}(\omega^{\prime})\text{Im}\mathcal{G}^{R}_{m\mathbf{k}}(\omega)}{(\omega-\omega^{\prime})^{2}},$

with
$\omega_{nm,\mathbf{k}}=\varepsilon_{n\mathbf{k}}-\varepsilon_{m\mathbf{k}}$,
and $f(w)$ is the Fermi-Dirac distribution function. Generically, the
structural tensors $\mathcal{I}_{nm}$ and $\mathcal{J}_{nm}$ are non-zero, and
depend on the nature of quasiparticles in the system, described by causal
(retarded) propagators $\mathcal{G}^{R}_{m\mathbf{k}}(\omega)$. Formula (10)
is working for the many-body propagators with well-defined quasiparticle
poles; the quasiparticle energy bands $\varepsilon_{n\mathbf{k}}$ may or may
not have dispersion as such.

Absence of semiclassical analogues. The first term in Eq.(10) is a quantum
effect which has no classical analogues; to our knowledge the longitudinal
conductance reminding Eq.(10) was obtained in the clean limit by Resta Resta
(2005). The quasiparticle propagator
$Z_{\mathbf{k}}/(\omega-\Sigma(k,\omega))$ in the clean limit is characterized
by $\Sigma(\mathbf{k},\omega)$$\to$$i\delta$, $\delta$$\to$$0$, hence Eq.(10)
gives $\sigma_{xx}(\omega$$=$$0)\propto\delta$, which vanishes for small
$\delta$ as expected (see Fig. 2b). From another perspective, the flat band
quantum transport (10) is determined by the quantum metrics of the electronic
bands $\mathfrak{G}^{nm}_{ij}(\mathbf{k})$; since the Fubini-Study metrics is
inherently related to the uncertainty principle Braunstein and Caves (1994);
Anandan and Aharonov (1990), the longitudinal ($\sigma_{xx}$) term in (10) has
no classical analogues for the ”Drude” electrons. The transverse ($xy$) term
in (10) neither represents the classical case, as it give rise to the
quantized Hall conductance, and is connected to TKNN invariant Thouless  _et
al._ (1982). To illustrate this, we consider a simple model with flat Chern
bands below.

Figure 2: Longitudinal electronic transport in topological flat bands. (a)
Schematic of a two-level system with two topological flat bands of $C=\pm 1$.
(b) Longitudinal conductance as function of level broadening calculated here
for the band gap $\Delta=0.5$ meV and temperature $T$$=$$\Delta$ ($\approx$ 6
K); here $\delta_{1,2}$$\equiv$$\delta$. (c) Longitudinal conductance as
function of temperature, calculated at $\Delta=0.5$ meV,
$\delta_{1,2}$$=$$0.01$ meV. (d,e) Inverse longitudinal conductance (for the
same parameters) shows T-linear scaling both for high temperatures
($T$$\gg$$\Delta$) and low temperatures ($T$$\ll$$\Delta$); $T$-slopes in
different regimes (d) and (e) are slightly different.

Quantum Hall effect. In this key example we focus on the transverse response
associated with the perfectly flat Chern bands. The transverse (Hall)
conductivity determined through quasiparticle propagators is given by Eq. (10)
at $T=0$ as

$\displaystyle\sigma_{xy}=\frac{8e^{2}}{\hbar}\sum_{\mathbf{k}}\sum_{nm}{}^{{}^{\prime}}\iint\limits_{-\infty}^{+\infty}\frac{d\omega
d\omega^{\prime}}{(2\pi)^{2}}$
$\displaystyle\frac{\,\text{Im}\mathcal{G}^{R}_{n\mathbf{k}}(\omega^{\prime})\text{Im}\mathcal{G}^{R}_{m\mathbf{k}}(\omega)}{(\omega-\omega^{\prime})^{2}}$
$\displaystyle\times$
$\displaystyle\omega^{2}_{nm,\mathbf{k}}\,\text{Im}\mathfrak{G}^{nm}_{xy}(\mathbf{k});$
(12)

the summation index $n$ runs over the occupied bands (below Fermi level
$\varepsilon_{F}$). In the limit when the quasiparticles are well-defined, one
may use
$\text{Im}\mathcal{G}^{R}_{n\mathbf{k}}(\omega)=\delta_{n\mathbf{k}}/[(\omega-\omega_{n\mathbf{k}}+i\delta_{n\mathbf{k}})(\omega-\omega_{n\mathbf{k}}-i\delta_{n\mathbf{k}})]$,
the expression (12) contains integrands with pole singularities and we can
resort to residue theorem to evaluate integrals for aribtrary
$\delta_{n\mathbf{k}}$, $\delta_{m\mathbf{k}}$ (see SM SM (2)). By choosing an
appropriate contour, we obtain
$\iint_{-\infty}^{+\infty}\frac{\,\text{Im}\mathcal{G}^{R}_{n\mathbf{k}}(\omega^{\prime})\text{Im}\mathcal{G}^{R}_{m\mathbf{k}}(\omega)}{(\omega-\omega^{\prime})^{2}}d\omega
d\omega^{\prime}\simeq\pi^{2}[{\omega_{n\mathbf{k}}-\omega_{m\mathbf{k}}}]^{-2}\left[1+\mathcal{O}(\delta^{2})\right]\approx\pi^{2}/\omega_{nm,\mathbf{k}}^{2}$.
Using this expression for formula (12), in the limit of well-resolved energy
bands ($\delta_{n\mathbf{k}},\delta_{m\mathbf{k}}\ll\Delta$, where $\Delta$ is
the band gap), we obtain the established quantum Hall conductance

$\displaystyle\sigma_{xy}=$
$\displaystyle\frac{e^{2}}{\hbar}\sum_{n}{}^{{}^{\prime}}\sum_{\mathbf{k}}\mathcal{F}^{(n)}_{xy}(\mathbf{k})=\frac{e^{2}}{h}\sum_{n}{}^{{}^{\prime}}C_{n}.$
(13)

where $\mathcal{F}^{(n)}_{xy}$ is the Berry curvature of the $n$-th band. Thus
we obtain the quantum Hall conductance in the form of TKNN invariant Thouless
_et al._ (1982). Note that the quantized nature of (12) holds in the presence
of moderate interactions, provided the renormalized propagators have well-
defined quasiparticle poles. Note also, that when the Fermi level is within
the flat band, we obtain anomalous Hall contributions similar to Refs. Haldane
(2004); Jungwirth  _et al._ (2002). Importantly, formula (12) provides means
of calculation for quantum Hall response in perfectly flat bands in the case
when the momentum dispersion of the effective Hamiltonian
$\mathcal{H}(\mathbf{k})$ is not defined and conventional minimal coupling to
magnetic vector potential as in approach Thouless  _et al._ (1982) is not
applicable.

Longitudinal conductance. For storytelling, consider the Haldane model Haldane
(1988) on NN and NNN hoppings ($\Lambda=2$),

$\displaystyle\mathcal{H}_{0}=\sum_{i}t_{0}c^{{\dagger}}_{i}c_{i}+\sum_{\langle
ij\rangle}t^{\text{NN}}_{ij}c^{{\dagger}}_{i}c^{{\dagger}}_{j}+\sum_{\langle\langle
ij\rangle\rangle}t^{\text{NNN}}_{ij}c^{{\dagger}}_{i}c^{{\dagger}}_{j},$ (14)

where we fix $t^{\text{NN}}_{ij}$ as real and
$t^{\text{NNN}}_{ij}$$=$$t^{\prime}e^{i\Phi_{ij}}$. Clearly, for
$\Phi_{ij}$$=$$\pm\pi/2$, the spectrum becomes particle-hole symmetric Haldane
(1988). This yields upon transformation (1) two pefectly flat bands positioned
at $E$$=$$\pm E_{0}$ Kruchkov (2022). The resulting
$\mathcal{H}_{0}^{\textbf{flat}}$ becomes nonlocal
($\Lambda^{\prime}$$=$$\infty$), though it is possible to truncate it and make
the band arbitrary flat by choosing a corresponding truncation
$\Lambda^{\prime}$ and minimizing bandwidth (Refs.Neupert  _et al._ (2011);
Kruchkov (2022)). Hence by allowing further hopping terms to (14), this model
features two dispersionless Chern bands with $C=\pm 1$. By using the flat band
Green’s function matrix $1/(\omega-\mathcal{H}_{0}^{\textbf{flat}})$ and
introducing level broadening $i\delta$, we use Eq. (10) to calculate
$\sigma_{xx}$.

The longitudinal conductance $\sigma_{xx}$ of the flat bands has an
unconventional temperature dependence: We find that quite generically the
$\sigma_{xx}$ conductance has inverse temperature scaling (Fig. 2)

$\displaystyle\sigma_{xx}(T)\propto\frac{\sum_{\mathbf{k}}\text{Tr}\,\mathcal{G}_{ij}(\mathbf{k})}{T}\sim\frac{1}{T},$
(15)

reminiscent of strange metal phases. The quantity in nominator is in order of
$\mathcal{O}(1)$ (for a perfectly flat Chern band, it is proportional to $|C|$
of the flat band). This $T^{-1}$ scaling is quite universal, and while the
particular numbers depend on the values of $\delta$ (Fig. 2a), the
$\sigma^{-1}_{xx}\sim T$ scaling is seen both for high temperatures
($T$$\ll$$\Delta$) and high temperatures ($T$$\gg$$\Delta$), (Figs. 3d-e).
Note however that the overall slope is slightly different for high temperature
and low temperature limits.

Thermal and thermoelectric response. We further report that the thermoelectric
and thermal response also have geometric contributions as well. In particular,
the longitudinal thermal conductance is

$\displaystyle\kappa_{xx}$
$\displaystyle=\frac{1}{T}\sum_{nm,\mathbf{k}}\mathcal{I}^{(2)}_{nm}(\mathbf{k})\,\text{Re}\mathfrak{G}^{nm}_{ij}(\mathbf{k})$
$\displaystyle-\frac{1}{T}\frac{[\sum_{nm,\mathbf{k}}\mathcal{I}^{(1)}_{nm}(\mathbf{k})\,\text{Re}\mathfrak{G}^{nm}_{ij}(\mathbf{k})]^{2}}{[\sum_{nm,\mathbf{k}}\mathcal{I}^{(0)}_{nm}(\mathbf{k})\,\text{Re}\mathfrak{G}^{nm}_{ij}(\mathbf{k})]},$
(16)

and thermoelectric power is

$\displaystyle\Theta_{xx}=\frac{\beta}{e}\frac{\sum_{nm,\mathbf{k}}\mathcal{I}^{(1)}_{nm}(\mathbf{k})\,\text{Re}\mathfrak{G}^{nm}_{ij}(\mathbf{k})}{\sum_{nm,\mathbf{k}}\mathcal{I}^{(0)}_{nm}(\mathbf{k})\,\text{Re}\mathfrak{G}^{nm}_{ij}(\mathbf{k})},$
(17)

where we have introduced transport tensors

$\displaystyle\mathcal{I}^{(\alpha)}_{nm}(\mathbf{k})=-2\omega^{2}_{nm,\mathbf{k}}\int\frac{d\omega}{2\pi}\omega^{\alpha}f^{\prime}(\omega)\,\text{Im}\mathcal{G}_{n\mathbf{k}}(\omega)\text{Im}\mathcal{G}_{m\mathbf{k}}(\omega).$

For numerical purposes, we operate with the same flat band model as discussed
below Eq. (14), containing two flat Chern bands of band gap $\Delta$ and
generic level broadening $\delta_{1},\delta_{2}$ (Fig. 3a). The thermopower is
further calculated via (17) and the dimensional units ($k_{B},\hbar$) are
restored. We plot the temperature dependence of thermopower for representative
parameters parameters $\delta_{1,2}\ll\Delta$ in Fig. 3 (additional plos with
different parameters are listed in SM SM (2)). The first observation is that
the thermopower at the fixed temperature $T\sim\Delta$ is nearly independent
of $\delta$ (Fig. 2b), and thus presents a robust quantum transport observable
(in this regard, see also Kruchkov  _et al._ (2020) for thermopower in the SYK
flat band). This is in contrast to the flat band conductance (Fig. 2), for
which we have found $\sigma_{xx}\propto\delta$, and hence being parameter-
dependent. The second observation, is that the thermopower is non-monotonic:
it starts nearly linear at low T, but develops a large local maximum
$\Theta_{*}$, placed at $T_{*}\sim\Delta$. The local maximum presents a
”giant” thermopower $\sim 50-60$ $\mu$V/K, unexpected for a narrow Bloch band
Paul  _et al._ (2022). Above $T_{*}\sim\Delta$, the thermopower drops to
significantly lower values.

Figure 3: Anomalous thermoelectricity of topological flat bands. (a) Schematic
of a two-level system with two topological flat bands of $C=\pm 1$. (b)
Thermoelectric power $\Theta$ as function of level broadening calculated here
for the band gap $\Delta=0.5$ meV and temperature $T$$=$$\Delta$ ($\approx$ 6
K); here $\delta_{1,2}$$\equiv$$\delta$. (c) Thermoelectric power as a
function of temperature, plotted here for $\Delta$$=$$0.5$ meV and
$\delta_{1,2}$$=$$0.01$. Thermopower develops here a pronounced peak at
$T_{*}$$\approx$$\Delta/2$, reaching for some parameters the quantum unit of
thermopower $\frac{k_{B}}{e}\ln 2$. Similar values have been reported for the
magic-angle twisted bilayer graphene Paul  _et al._ (2022), where the system
features narrow topological bands.

In the limit when the level broadening is much smaller than the bandgap, the
thermopower associated with dispersionless electronic bands with nontrivial
quantum geometry has a bound on local thermopower maximum as

$\displaystyle\Theta_{*}\simeq\frac{k_{B}}{e}\ln 2.$ (18)

Even a deeper analogy stems here to another case of topological flat bands: in
Landau levels Girvin and Jonson (1982); Jonson and Girvin (1984) the value of
$\frac{k_{B}}{e}\ln 2\approx 60\,\mu\text{V}/\text{K}$ sets the quantum unit
for thermopower 222We thank Philip Kim for pointing out this observation.. As
thermopower reflects entropy per quasiparticle, the quantum unit
$\frac{k_{B}}{e}\ln 2$ reflects fermionic entropy of $k_{B}\ln 2$ per electron
Hartman  _et al._ (2018); Yang and Halperin (2009), even that there is
formally no Planck constant in Eq. (18).

Discussion. In the absence of magnetic fields, thermopower in graphene-based
systems remains relatively low ($\ll k_{B}/e$) unless thermally excited to
room temperatures Zuev  _et al._ (2009); Nam  _et al._ (2010). In this regard,
observation of ”giant” thermopower $\sim k_{B}/e$ in the flat bands of magic-
angle twisted bilayer graphene Paul  _et al._ (2022) was unexpected, not
captured by conventional theory, and hence attributed to various interaction
effects. However, we remark that two slightly gapped flat bands in TBG at the
magic angle can be approximated with Landau level wave functions Tarnopolsky
_et al._ (2019). Therefore the estimates drawn in this paper for flat band
thermopower should qualitatively hold for TBG as well. In particular, the
observed large values of thermopower in experiments Paul  _et al._ (2022) are
consistent with flat bands thermopower $\frac{k_{B}}{e}\ln 2$ (and taking into
account the number of flat-band electrons per moiré cell). Moreover, the
temperature dependence of thermopower in TBG (Fig.3c in Paul  _et al._ (2022))
is consistent with temperature dependence stemming from quantum geometry
(Fig.3).

It is interesting that the superconductivity in perfectly flat Chern bands
Peotta and Törmä (2015); Huhtinen  _et al._ (2022) takes origin from the
nontrivial quantum metrics. In fact, the superfluid weight $D_{S}$ (defined as
a response to external vector potential $\mathbf{A}$,
$j_{i}=-(D_{S})_{ij}A_{j}$) has a similar linear response structure through
current-current correlators Huhtinen  _et al._ (2022), with a difference that
$\mathbf{k}\to 0$, $\omega\to 0$ limits should be taken carefully. In this
case, one finds

$\displaystyle
D_{S}\sim\Delta_{S}\sum_{\mathbf{k}}\text{Re}\mathfrak{G}_{ii}(\mathbf{k}),$
(19)

where $\Delta_{S}$ is the superfluid gap associated with Bogoliubov-de-Gennes
model and its variations Peotta and Törmä (2015); Huhtinen  _et al._ (2022);
Wang  _et al._ (2020). After a relevant analysis, the result (19) has been
applied to twisted bilayer graphene, see Refs.Hazra  _et al._ (2019); Hu  _et
al._ (2019); Julku  _et al._ (2020); Xie  _et al._ (2020); Guan  _et al._
(2022) and Ref.Tian  _et al._ (2021), where the quantum geometric contribution
to superfluid weight, and hence the
$T_{\text{BKT}}\sim\frac{\hbar^{2}}{e^{2}}D_{S}(T$$=$$0)$ Nelson and
Kosterlitz (1977), is argued to be significant (typically, this mechanism
gives $T_{\text{BKT}}\sim 1$ K in TBG). As a final remark, there is an
observation that the slope of linear-in-T resistivity in strange metals
happens to correlate with the values of superfluid weight (see e.g. Fig. 4 in
the recent review by Phillips and colleagues Phillips  _et al._ (2022)); in
the quantum transport formalism in flat bands, this feature may take place
from nontrivial $\sum_{\mathbf{k}}\text{Re}\mathfrak{G}_{ij}(\mathbf{k})$.

Conclusion. In this work, we have extended the quantum transport formalism for
the dispersionless electronic bands. Since the Fermi velocities nullify in the
perfectly flat bands, the conventional contribution to transport coefficients
vanishes. Nevertheless, the flat bands develop an unconventional contribution
to transport (10), which sources from nontrivial quantum metrics of these
electronic states (1). This contribution leads to enhanced thermoelectricity
(18) and superconductivity (19) in the flat band. The origin of these quantum
phenomena roots to the Wannier orbital quantization of the topological flat
band states: The Wannier orbital cross-section is quantized in Chern numbers,
$r_{0}^{2}=Ca^{2}$. This leads to the spatial overlap of real-space electronic
orbitals (Fig.1), and allows quantum tunneling under external fields, as
witnessed in the purely quantum contributions to electric conductivity,
thermal conductivity, thermoelectric response, and the superfluid weight. It
will be interesting to extend this formalism towards the topological flat-band
systems with SYK interactions Brzezinska  _et al._ (2022).

Acknowedgments. The author thanks Subir Sachdev, Philip Kim, and Bertrand
Halperin for fruitful discussions. This work was financed by the Branco Weiss
Society in Science, ETH Zurich, through the grant on flat bands, strong
interactions, and the SYK physics.

## References

  * Cao  _et al._ (2018a) Y. Cao, V. Fatemi, A. Demir, S. Fang, S. L. Tomarken, J. Y. Luo, J. D. Sanchez-Yamagishi, K. Watanabe, T. Taniguchi, E. Kaxiras, _et al._ , Nature 556, 80 (2018a).
  * Cao  _et al._ (2018b) Y. Cao, V. Fatemi, S. Fang, K. Watanabe, T. Taniguchi, E. Kaxiras, and P. Jarillo-Herrero, Nature 556, 43 (2018b).
  * Trambly de Laissardière  _et al._ (2010) G. Trambly de Laissardière, D. Mayou, and L. Magaud, Nano Letters 10, 804 (2010).
  * Bistritzer and MacDonald (2011) R. Bistritzer and A. H. MacDonald, Proceedings of the National Academy of Sciences 108, 12233 (2011).
  * Tarnopolsky  _et al._ (2019) G. Tarnopolsky, A. J. Kruchkov, and A. Vishwanath, Physical Review Letters 122, 106405 (2019).
  * San-Jose  _et al._ (2012) P. San-Jose, J. Gonzalez, and F. Guinea, Physical Review Letters 108, 216802 (2012).
  * Polshyn  _et al._ (2019) H. Polshyn, M. Yankowitz, S. Chen, Y. Zhang, K. Watanabe, T. Taniguchi, C. R. Dean, and A. F. Young, Nature Physics 15, 1011 (2019).
  * Cao  _et al._ (2020) Y. Cao, D. Chowdhury, D. Rodan-Legrain, O. Rubies-Bigorda, K. Watanabe, T. Taniguchi, T. Senthil, and P. Jarillo-Herrero, Physical Review Letters 124, 076801 (2020).
  * Jaoui  _et al._ (2022) A. Jaoui, I. Das, G. Di Battista, J. Díez-Mérida, X. Lu, K. Watanabe, T. Taniguchi, H. Ishizuka, L. Levitov, and D. K. Efetov, Nature Physics , 1 (2022).
  * Lu  _et al._ (2019) X. Lu, P. Stepanov, W. Yang, M. Xie, M. A. Aamir, I. Das, C. Urgell, K. Watanabe, T. Taniguchi, G. Zhang, _et al._ , Nature 574, 653 (2019).
  * Xie  _et al._ (2021) Y. Xie, A. T. Pierce, J. M. Park, D. E. Parker, E. Khalaf, P. Ledwith, Y. Cao, S. H. Lee, S. Chen, P. R. Forrester, _et al._ , Nature 600, 439 (2021).
  * Paul  _et al._ (2022) A. K. Paul, A. Ghosh, S. Chakraborty, U. Roy, R. Dutta, K. Watanabe, T. Taniguchi, A. Panda, A. Agarwala, S. Mukerjee, _et al._ , Nature Physics , 1 (2022).
  * Provost and Vallee (1980) J. Provost and G. Vallee, Communications in Mathematical Physics 76, 289 (1980).
  * Ma  _et al._ (2010) Y.-Q. Ma, S. Chen, H. Fan, W.-M. Liu, _et al._ , Physical Review B 81, 245129 (2010).
  * Neupert  _et al._ (2011) T. Neupert, L. Santos, C. Chamon, and C. Mudry, Physical Review Letters 106, 236804 (2011).
  * Haldane (2004) F. Haldane, Physical Review Letters 93, 206602 (2004).
  * Rhim  _et al._ (2020) J.-W. Rhim, K. Kim, and B.-J. Yang, Nature 584, 59 (2020).
  * Peotta and Törmä (2015) S. Peotta and P. Törmä, Nature Communications 6, 8944 (2015).
  * Huhtinen  _et al._ (2022) K.-E. Huhtinen, J. Herzog-Arbeitman, A. Chew, B. A. Bernevig, and P. Törmä, Physical Review B 106, 014518 (2022).
  * Gao and Xiao (2019) Y. Gao and D. Xiao, Physical Review Letters 122, 227402 (2019).
  * Lapa and Hughes (2019) M. F. Lapa and T. L. Hughes, Physical Review B 99, 121111 (2019).
  * Kozii  _et al._ (2021) V. Kozii, A. Avdoshkin, S. Zhong, and J. E. Moore, Physical Review Letters 126, 156602 (2021).
  * Mitscherling and Holder (2022) J. Mitscherling and T. Holder, Physical Review B 105, 085154 (2022).
  * Ahn  _et al._ (2022) J. Ahn, G.-Y. Guo, N. Nagaosa, and A. Vishwanath, Nature Physics 18, 290 (2022).
  * Chen and Huang (2021) W. Chen and W. Huang, Physical Review Research 3, L042018 (2021).
  * Hu  _et al._ (2019) X. Hu, T. Hyart, D. I. Pikulin, and E. Rossi, Physical Review Letters 123, 237002 (2019).
  * Hazra  _et al._ (2019) T. Hazra, N. Verma, and M. Randeria, Physical Review X 9, 031049 (2019).
  * Julku  _et al._ (2020) A. Julku, T. J. Peltonen, L. Liang, T. T. Heikkilä, and P. Törmä, Physical Review B 101, 060505 (2020).
  * Xie  _et al._ (2020) F. Xie, Z. Song, B. Lian, and B. A. Bernevig, Physical Review Letters 124, 167002 (2020).
  * Tian  _et al._ (2021) H. Tian, S. Che, T. Xu, P. Cheung, K. Watanabe, T. Taniguchi, M. Randeria, F. Zhang, C. N. Lau, and M. W. Bockrath, arXiv preprint arXiv:2112.13401 (2021).
  * Kruchkov (2022) A. Kruchkov, Physical Review B 105, L241102 (2022).
  * Marzari and Vanderbilt (1997) N. Marzari and D. Vanderbilt, Physical Review B 56, 12847 (1997).
  * Mahan (2000) G. D. Mahan,  _Many Particle Physics, Third Edition_ (Plenum, New York, 2000).
  * Onsager (1931a) L. Onsager, Physical Review 37, 405 (1931a).
  * Onsager (1931b) L. Onsager, Physical Review 38, 2265 (1931b).
  * Rammer (2018) J. Rammer,  _Quantum transport theory_ (CRC Press, 2018).
  * Note (1) We introduce here two quantities $\tilde{\Pi}^{\pm}_{ij}$ in order to evaluate carefully zero-frequency limit in Eq.(7).
  * Blount (1962) E. Blount, in  _Solid State Physics_ , Vol. 13 (Elsevier, 1962) pp. 305–373.
  * SM (2) Supplemental Materials (SM) are available in the published version of this manuscript .
  * Resta (2005) R. Resta, Physical Review Letters 95, 196805 (2005).
  * Braunstein and Caves (1994) S. L. Braunstein and C. M. Caves, Physical Review Letters 72, 3439 (1994).
  * Anandan and Aharonov (1990) J. Anandan and Y. Aharonov, Physical Review Letters 65, 1697 (1990).
  * Thouless  _et al._ (1982) D. J. Thouless, M. Kohmoto, M. P. Nightingale, and M. den Nijs, Physical Review Letters 49, 405 (1982).
  * Jungwirth  _et al._ (2002) T. Jungwirth, Q. Niu, and A. MacDonald, Physical Review Letters 88, 207208 (2002).
  * Haldane (1988) F. D. M. Haldane, Physical Review Letters 61, 2015 (1988).
  * Kruchkov  _et al._ (2020) A. Kruchkov, A. A. Patel, P. Kim, and S. Sachdev, Physical Review B 101, 205148 (2020).
  * Girvin and Jonson (1982) S. Girvin and M. Jonson, Journal of Physics C: Solid State Physics 15, L1147 (1982).
  * Jonson and Girvin (1984) M. Jonson and S. M. Girvin, Physical Review B 29, 1939 (1984).
  * Note (2) We thank Philip Kim for pointing out this observation.
  * Hartman  _et al._ (2018) N. Hartman, C. Olsen, S. Lüscher, M. Samani, S. Fallahi, G. C. Gardner, M. Manfra, and J. Folk, Nature Physics 14, 1083 (2018).
  * Yang and Halperin (2009) K. Yang and B. I. Halperin, Physical Review B 79, 115317 (2009).
  * Zuev  _et al._ (2009) Y. M. Zuev, W. Chang, and P. Kim, Physical Review Letters 102, 096807 (2009).
  * Nam  _et al._ (2010) S.-G. Nam, D.-K. Ki, and H.-J. Lee, Physical Review B 82, 245416 (2010).
  * Wang  _et al._ (2020) Z. Wang, G. Chaudhary, Q. Chen, and K. Levin, Physical Review B 102, 184504 (2020).
  * Guan  _et al._ (2022) Y. Guan, O. V. Yazyev, and A. Kruchkov, Physical Review B 106, L121115 (2022).
  * Nelson and Kosterlitz (1977) D. R. Nelson and J. M. Kosterlitz, Physical Review Letters 39, 1201 (1977).
  * Phillips  _et al._ (2022) P. W. Phillips, N. E. Hussey, and P. Abbamonte, Science 377, eabh4273 (2022).
  * Brzezinska  _et al._ (2022) M. Brzezinska, Y. Guan, O. V. Yazyev, S. Sachdev, and A. Kruchkov, arXiv preprint arXiv:2208.01032 (2022).


# Orthogonal Adaptation for Modular Customization of Diffusion Models

Ryan Po
Stanford University  Guandao Yang
Stanford University  Kfir Aberman
Snap Research  Gordon Wetzstein
Stanford University

###### Abstract

Customization techniques for text-to-image models have paved the way for a
wide range of previously unattainable applications, enabling the generation of
specific concepts across diverse contexts and styles. While existing methods
facilitate high-fidelity customization for individual concepts or a limited,
pre-defined set of them, they fall short of achieving scalability, where a
single model can seamlessly render countless concepts. In this paper, we
address a new problem called Modular Customization, with the goal of
efficiently merging customized models that were fine-tuned independently for
individual concepts. This allows the merged model to jointly synthesize
concepts in one image without compromising fidelity or incurring any
additional computational costs. To address this problem, we introduce
Orthogonal Adaptation, a method designed to encourage the customized models,
which do not have access to each other during fine-tuning, to have orthogonal
residual weights. This ensures that during inference time, the customized
models can be summed with minimal interference. ††Project page:
http://www.ryanpo.com/ortha Our proposed method is both simple and versatile,
applicable to nearly all optimizable weights in the model architecture.
Through an extensive set of quantitative and qualitative evaluations, our
method consistently outperforms relevant baselines in terms of efficiency and
identity preservation, demonstrating a significant leap toward scalable
customization of diffusion models.

Figure 1: Modular Customization of Diffusion Models. Given a large set of
individual concepts (left), the goal of Modular Customization is to enable
independent customization (fine-tuning) per concept, while efficiently merging
a subset of customized models during inference, so that the corresponding
concepts can be jointly synthesized without compromising fidelity. To tackle
this, we propose Orthogonal Adaptation, which encourages customized weights of
one concept to be orthogonal to the customized weights of others.

##  1 Introduction

Diffusion models (DMs) mark a paradigm shift for computer vision and beyond.
DM-based foundation models for text-to-image, video, or 3D generation enable
users to create and edit content with unprecedented quality and diversity
using intuitive text prompts [31]. Although these foundation models are
trained on a massive amount of data, in order to synthesize user-specific
concepts (such as a pet, an item, or a person) with a high fidelity, they
often need to be fine-tuned.

Several recent approaches to customizing DMs to individual concepts have
demonstrated high-quality results [18, 10, 35, 24, 44]. A multi-concept DM
strategy, however, where several pre-trained concepts are mixed in a single
image, remains challenging. Existing multi-concept methods [24, 12] either
show a degradation in the quality of individual concepts when merged or
require access to multiple concepts during training. The latter makes the
process unscalable and raises privacy concerns when the different concepts
belong to different users. Furthermore, in all cases the mixing process is
computationally inefficient.

Figure 2: Gallery of multi-concept generations. Our method enables efficient
merging of individually fine-tuned concepts for modular, efficient multi-
concept customization of text-to-image diffusion models. Each concept shown
above was fine-tuned individually using orthogonal adaptation. Fine-tuned
weight residuals are then merged via summation, enabling multi-concept
generation.

We introduce orthogonal adaptation as a new approach to enabling instantaneous
multi-concept customization of DMs. The primary insight of our work is that
changing how the DM is fine-tuned for novel concepts can lead to very
efficient mixing of these concepts. Specifically, we represent each new
concept using a basis that is approximately orthogonal to the basis of other
concepts. These bases do not need to be know a priori and different concepts
can be trained independently of each other. A key advantage of our approach is
that our model does not need to be re-trained when mixing several of our
orthogonal concepts together, for example to jointly synthesize different
concepts that were never seen together in any training example. Importantly,
our approach is _modular_ in that it enables individual concepts to be learned
independently and in parallel without knowledge of each other. Moreover, it is
privacy aware in the sense that it never requires access to the training
images of concepts to mix them.

Consider a social media platform where millions of users fine-tune a DM using
their personal concepts and want to mix them with their friends’ concepts on
their phones. Efficiency of the customization and mixing processes as well as
data privacy are key challenges in this scenario. Our method addresses
precisely these issues. A core technical contribution of our work is a modular
customization and scalable multi-concept merging approach that offers better
quality in terms of identity preservation than baselines at similar speeds, or
similar quality to state-of-the-art baselines at significantly lower
processing times.

##  2 Related Work

#### Text-conditioned image synthesis.

The field of text-conditioned image synthesis has experienced significant
advancements, driven by developments in GANs [11, 6, 23, 21, 22] and diffusion
models [17, 42, 8, 16, 34, 29, 28]. Earlier efforts focus on applying GANs to
various conditional synthesis tasks, including class-conditioned image
generation [6, 21, 19] and text-driven editing [2, 5, 9, 26, 46, 30, 33]. More
recently, the focus has shifted to large text-to-image models [34, 37, 48, 32]
trained on large-scale datasets [38]. In this paper, we will utilize the open-
source StableDiffusion [34] architecture and build on its pre-trained
checkpoints by fine-tuning.

Method |  Fidelity (Single-concept) |  Efficient Merging |  Fidelity (Multi-concept)
---|---|---|---
TI [10] | ✗ | ✓ | ✗
DB-LoRA111assuming DB-LoRA fine-tuned models are merged with FedAvg [25] [35] | ✓ | ✓ | ✗
Custom Diffusion [24] | ✗ | ✓ | ✗
Mix-of-Show [12] | ✓ | ✗ | ✓
Ours | ✓ | ✓ | ✓
Table 1: Comparison of Solutions to Modular Customization. Our customization
approach excels in three key areas: (1) preserving the identity of individual
concepts with high fidelity, (2) efficiently merging independently customized
models, and (3) maintaining high concept fidelity for multi-concept image
synthesis using the merged model.

#### Customization.

The task of customization aims at capturing a user-defined concept, to be used
for generation under various contexts. Seminal works such as Textual Inversion
(TI) [10] and DreamBooth [35] tackle the problem of customization by taking a
handful of images of the same concept to produce a representation of the
subject to be used for controlled generation. TI captures new concepts by
optimizing a text embedding to reconstruct target images using the
conventional diffusion loss. Follow-up works, such as $\mathcal{P}+$ [14],
extend Texture Inversion with a more expressive token representation,
improving generation subject alignment/fidelity. DreamBooth [35], on the other
hand, picks an uncommon word token and fine-tunes the network weights to
reconstruct the target concept using diffusion loss [17]. Custom Diffusion
[24] works in a similar way but only fine-tunes a subset of the diffusion
model layers, namely the cross-attention layers. LoRA [18] is a low-rank
matrix decomposition method that enables better parameter efficiency for fine-
tuning methods, and was recently adapted to customization of text-to-image
diffusion models [1] (DB-LoRA). Recent works [41, 36, 20, 43, 47, 45, 40] try
to improve speed by training feed-forward networks to predict adaptation
parameters from data, successfully amortize the time taken to create customize
concepts.

#### Multi-concept Customization.

Certain existing works have taken the task of customization one step further,
aiming to inject multiple novel concepts into a model at the same time. Custom
Diffusion [24] achieves this through a joint optimization loss for all
concepts, while Break-a-scene [3] and SVDiff [13] introduces a masked cross-
attention loss to learn individual concepts in images containing multiple
concepts. However, such methods require access to ground truth data of all
concepts training. In this work, we are interested in the task of modular
customization, where concepts are learned independently, and users can then
mix and match individual concepts during inference for multi-concept image
synthesis (Sec. 3.1).

Prior works have provided implicit solutions to the problem of modular
customization, but each existing method comes with its own set of trade-offs.
TI [10, 27, 44] implicitly addresses the task by representing each concept
through a unique token embedding, enabling multi-concept customization by
simply querying each token. However, TI tends to suffer from low subject
fidelity, as token embeddings alone provide limited expressivity. Federated
Averaging (FedAvg) [25] merges fine-tuned models by simply taking a weighted
average between the weights of each model, although fast and expressive, naive
combination tends to lead to loss of concept identity. Custom Diffusion [24]
supports merging of individually fine-tuned networks through solving a
constrained customization problem. This method also struggles with
expressivity, as only a small subset of the diffusion model weights are being
updated. Concurrent work, Mix-of-Show (MoS) [12] expands on this method by
introducing gradient fusion, enabling merging of multiple separately fine-
tuned models without placing restrictions on parameter expressivity. Though
expressive, gradient fusion is computationally demanding, taking $\sim$15-20
minutes just to combine three custom concepts into a single model, which
becomes intractably expensive when deployed at scale. Table 1 summarizes the
key areas in which our approach differs from previous and concurrent works.

##  3 Method

Figure 3: The three stages of Modular Customization: (a) Independent
Customization, (b) Modular Combination, and (c) Joint Synthesis. Note that
during individual fine-tuning, all processes are private, meaning each user
does not have access to ground truth data for other concepts.

In this section, we first introduce the problem setting of modular
customization (Sec. 3.1). We then take a look at the simple solution of FedAvg
[25], and explore where and why this naive method fails to preserve identity
(Sec. 3.2). Motivated by the limitations of FedAvg, we discuss the conditions
to ensure concept identity preservation (Sec. 3.3), and finally introduce our
solution to modular customization – orthogonal adaption (Sec. 3.4 and Sec.
3.5).

###  3.1 Modular Customization

In this paper, we are interested in customizing text-to-image diffusion models
to generate multiple personal concepts in an efficient, scalable, and
decentralized manner.

Figure 4: Overview of Orthogonal Adaptation. (a) LoRA [18] enables training of
both low-rank decomposed matrices. (b) Orthogonal adaption constrains training
only to $A$, leaving $B$ fixed. (c) For two separate concepts, $i$ and $j$, an
orthogonality constraint is imposed between $B_{i}$ and $B_{j}$. (d) When
concepts $i$ and $j$ are trained independently, approximate orthogonality
between $B_{i}$ and $B_{j}$ can be achieved by sampling random columns from a
shared orthogonal matrix. (e) Without the orthogonality constraint, correlated
concepts suffer from “crosstalk” when merged; with the orthogonality
constraint, orthogonal concepts preserve their identities after merging.

In addition to single-concept text-to-image customization, users are usually
interested in seeing multiple concepts interacting together. This calls for a
text-to-image model that is customized to a set of concepts. Being able to
generate multiple personalized concepts in a single model, however, is
challenging. First, the number of sets containing all possible combinations of
concepts is growing exponentially with respect to the number of concepts – an
intractable number even for a relatively small number of concepts. As a
result, it’s important for personalized concepts to be merged with interactive
speed. Furthermore, users usually have limited compute at their end, which
means any computation done on the users end should ideally be trivial.

These requirements motivate an efficient and scalable fine-tuning setting we
call modular customization, where individual fine-tuned models should act like
independent modules, which can be combined with others in a plug-and-play
manner without additional training. The setting of modular customization
involves three stages: independent customization, modular combination and
joint synthesis. Fig. 3 provides an illustration of this three stage process.

With modular customization in mind, our goal is to design a fine-tuning
scheme, such that individually fine-tuned models can be trivially combined
(e.g. summation) with any other fine-tuned model to enable multi-concept
generation.

###  3.2 Federated Averaging

Perhaps the most straight-forward technique for achieving modular
customization is to take a weighted average of each individually fine-tuned
model. This technique is often referred to as FedAvg [25]. Given a set of
learned weight residuals $\Delta\theta_{i}$ optimized on concept $i$, the
resulting merged model is simply given by

$\theta_{\text{merged}}=\theta+\sum_{i}\lambda_{i}\Delta\theta_{i},$ (1)

where $\theta$ represents the pre-trained parameters of the model used for
fine-tuning, and $\lambda_{i}$ is a scalar representing the relative strength
of each concept. While FedAvg is fast and places no constraints on the
expressivity of each individually fine-tuned model, naively averaging these
weights can lead to loss of subject fidelity due to interference between the
learned weight residuals. This effect is especially severe when training
multiple semantically similar concepts (e.g., human identities), as learned
weight residuals tend to be very similar. We coin this undesirable phenomenon
“crosstalk”. Fig. 7 and Fig. 8(a) provide visualizations of the effect of
crosstalk, as FedAvg causes multi-concept generations to exhibit loss of
identity. Our approach is inspired by FedAvg. We adopt its computational
efficiency but modify the fine-tuning process to ensure minimal interference
between learned weight residuals between different concepts. We want to enable
instant, multi-concept customization from individually trained models without
sacrificing subject fidelity.

###  3.3 Preserving Concept Identity

With the goal of addressing the limitations of FedAvg, we first examine where
this method fails. For simplicity, consider the case of merging two concepts
$i$ and $j$. After fine-tuning on each individual task, we receive a set of
learned weight residuals $\Delta\theta_{i}$ and $\Delta\theta_{j}$. The output
of a particular linear layer in the fine-tuned network is

$O_{i}(X_{i})=(\theta+\Delta\theta_{i})X_{i},$ (2)

where $X_{i}$ represents a particular input to the layer corresponding to the
training data of concept $i$. When merging the two concepts using FedAvg with
$\lambda=1$, the resulting merged model produces

$\hat{O}_{i}(X_{i})=(\theta+\Delta\theta_{i}+\Delta\theta_{j})X_{i}.$ (3)

The goal of concept preservation is to have $\hat{O}_{i}(X_{i})=O_{i}(X_{i})$.
Note that, without enforcing specific constraints, it is likely that
$\Delta\theta_{j}X_{i}\neq 0$ and, subsequently, $\hat{O}_{i}\neq O_{i}$.

It follows that the mapping of data for concept $i$ is preserved when
$\Delta\theta_{j}X_{i}=0$ for $j\neq i$. By symmetry, the mapping of data for
concept $j$ is preserved given $\Delta\theta_{i}X_{j}=0$ for $i\neq j$.
Intuitively, $||\Delta\theta_{j}X_{i}||$ measures the amount of crosstalk
between the customized weights of concepts $i$ and $j$. We would like to keep
this value low to ensure subject identity is preserved even after merging.
However, note that given enough data for training a certain concept $i$,
$X_{i}$ is likely to have full column rank. This makes the orthogonality
condition impossible to satisfy. Instead, we propose a relaxation to this
condition, choosing to minimize the crosstalk term for some projection of
$X_{i}$ onto a subspace $S_{i}$. This projection yields $S_{i}S^{T}_{i}X_{i}$,
and our relaxed objective hopes to achieve
$\hat{O}_{i}(S_{i}S^{T}_{i}X_{i})=O_{i}(S_{i}S^{T}_{i}X_{i})$.

###  3.4 Orthogonal Adaptation

Motivated by the relaxed objective above, we propose orthogonal adaptation.
Similar to low-rank adaptation (LoRA), we represent learned weight residuals
through a low-rank decomposition of the form

$\Delta\theta_{i}=A_{i}B_{i}^{T},\theta_{i}\in\mathbb{R}^{n\times
m}A_{i}\in\mathbb{R}^{n\times r},B_{i}\in\mathbb{R}^{m\times r},$ (4)

where the rank $r<\\!\\!<\min(n,m)$. However, contrary to conventionally fine-
tuning with LoRA, we keep $B_{i}$ constant, and only optimize $A_{i}$.

Consider a matrix $\bar{B}_{j}$, where its columns span the orthogonal
complement of the column space of $B_{j}$. We show that by selecting
$S_{i}=\bar{B}_{j}$, we achieve the conditions for achieving the projected
preservation objective. This can be seen from the fact that,

$\displaystyle\hat{O}_{i}(S_{i}S^{T}_{i}X_{i})$
$\displaystyle=O_{i}(S_{i}S^{T}_{i}X_{i})+\Delta\theta_{j}S_{i}S^{T}_{i}X_{i}$
(5)
$\displaystyle=O_{i}(S_{i}S^{T}_{i}X_{i})+A_{j}\cancelto{0}{B^{T}_{j}S_{i}}S^{T}_{i}X_{i}$
(6) $\displaystyle=O_{i}(S_{i}S^{T}_{i}X_{i}).$ (7)

Since $r<\\!\\!<m$, the orthogonal complement of $B_{j}$ covers most of
$\mathbb{R}^{m}$. It follows that $\bar{B}_{j}\bar{B}^{T}_{j}X_{i}\approx
X_{i}$, making $\bar{B}_{j}$ a reasonable candidate for $S_{i}$.

At the same time, since we expect the learned residuals for a concept to have
meaningful interactions with their data, we would also like to ensure
$||\Delta\theta_{i}X_{i}||$ is non-trivial. By approximating $X_{i}$ with its
projection onto $\bar{B}_{j}$, our objective changes to ensuring
$||A_{i}B^{T}_{i}\bar{B}_{j}\bar{B}^{T}_{j}X_{i}||$ is non-trivial. Examining
this term gives us the additional constraint that $B^{T}_{i}\bar{B}_{j}\neq
0$, meaning the columns of $B_{i}$ should live in the orthogonal complement of
the columns space of $B_{j}$. Therefore, to ensure meaningful fine-tuning
results, we should also enforce orthogonality between the learned residuals,
i.e. $B_{i}^{T}B_{j}=0$.

Fig. 4 provides an overview of our orthogonal adaption method. Intuitively, as
illustrated in Fig. 4(e), our method disentangles custom concepts into
orthogonal directions, ensuring that there is no crosstalk between concepts.
As a result, our merged model can better preserve the identity of each
concept.

Figure 5: Over-parameterization of text-to-image models. Despite the added
constraint on the trained weight residuals, due to the over-paramterized
nature of large text-to-image diffusion models, our method is able to achieve
single-concept customization results with comparable fidelity to the
unconstrained setting.

#### Expressivity of orthogonal adaption.

Expressivity of our method arises as a natural concern as we are optimizing
significantly fewer parameters by freezing $B_{i}$. Fortunately, text-to-image
diffusion models are often over-parameterized, with millions/billion of
parameters. Prior works have shown that even fine-tuning a subspace of such
parameters can be expressive enough to capture a novel concept. We also show
this result empirically in Fig. 5, where our method leads to results with
similar fidelity, even without the need to optimize $B_{i}$ during training.

###  3.5 Designing Orthogonal Matrices $B_{i}$’s

A key challenge of the method described in previous sections is to generate a
set of basis matrices $B_{i}$ that are orthogonal to each other. Note that
this is very difficult especially because when choosing $B_{i}$, the user is
not aware of what basis the other users chose to optimize for the concepts to
be combined in the future. Strictly enforcing such orthogonality might be
infeasible without prior knowledge of other tasks. We instead propose a
relaxation to the constraint, introducing a simple and effective method to
achieve approximate orthogonality.

#### Randomized orthogonal basis.

One method for enforcing approximate orthogonality is to determine a shared
orthogonal basis. For some linear weight $\theta\in\mathbb{R}^{m\times n}$, we
first generate a large orthogonal basis $O\in\mathbb{R}^{n\times n}$. This
orthogonal basis is shared between all users. During training of concept $i$,
$B_{i}$ is formed from taking a random subset of $k$ columns from $O$. Given
$k<\\!\\!<n$, the probability of two randomly chosen $B_{i}$’s to share the
same columns is kept low.

#### Randomized Gaussian.

Another approach is to choose random matrix elements. Specifically, we sample
each entry of $B_{i}$ from a zero-mean Gaussian with standard deviation
$\sigma$: $B_{i}[k]\sim\mathcal{N}(0,\sigma^{2}I)$. When the dimensionality of
$B_{i}$ is high, this simple strategy creates matrices that are orthogonal in
expectation: $\mathbb{E}\left[B_{i}^{T}B_{j}\right]=0$ (see supplement for
discussion). Naturally, this method does not require knowledge of a shared
basis to sample from. In practice, however, we found randomized Gaussians lead
to higher levels of crosstalk in our setting, i.e., $||B_{i}^{T}B_{j}||$ tends
to be larger than for the randomized orthogonal basis.

Figure 6: Identity preservation in single-concept generations from a merged
model. We demonstrate our method’s ability to maintain identity consistency
across different single-concept generations. Each column showcases images from
the same merged model, representing three distinct concept identities. Our
approach showcases better identity alignment with the corresponding input
images, offering a significant improvement over comparable merging methods.
Additionally, our method’s performance parallels that of Mix-of-Show (Gradient
Fusion) but with the advantage of near-instantaneous merging, in contrast to
the approximately 15-minute merging time required.

##  4 Experiments

In this section, we show the results of our method applied to the task of
modular customization. Qualitative and quantitative results indicate that our
method outperforms relevant baselines [12, 1, 24] at similar speeds, and
quality on par with state-of-the-art baselines that require significantly
higher processing times [12].

#### Datasets.

We perform evaluations on a custom dataset of 12 concept identities, each
containing 16 unique images of the target concept in different contexts.

#### Implementation details.

We perform fine-tuning on the Stable Diffusion [34] model, specifically the
ChilloutMix checkpoint for its ability to handle high-fidelity human face
generation. For single-concept fine-tuning, we apply orthogonal adaptation to
all linear layers in the Stable Diffusion architecture. Following prior work
[44, 12], we also apply a layer-wise text embedding and represent each fine-
tuned concept as two separate text tokens. We fine-tune the text embeddings
with a learning rate of $1e-3$, the diffusion model parameters with a learning
rate of $1e-5$ and set $r=20$ for all experiments. Single-concept fine-tuning
takes $\sim$10-15 minutes on two A6000 GPUs. For our method, we enforce the
orthogonality constraint using the randomized orthogonal basis method for all
experiments. Methods using FedAvg (including orthogonal adaption) were merged
using $\lambda=0.6$.

Figure 7: Multi-concept results. Examples of multi-concept generations,
synthesized using sampling techniques from concurrent work [12]. While Mix-of-
Show (FedAvg) maintains high-level features, it struggles with crosstalk,
manifesting overly smooth facial features. Mix-of-Show (Gradient Fusion)
exhibits good identity alignment, albeit with a computationally intensive
merging process. $\mathcal{P}+$ manages to preserve identity after merging,
but struggles to capture identity with high-fidelity due to limited parameter
expressivity. Our method stands out by achieving high identity alignment with
a significantly faster merging procedure.

Method |  Merge Time | Text Alignment $\uparrow$ | Image Alignment $\uparrow$ | Identity Alignment $\uparrow$
---|---|---|---|---
Single | Merged | $\Delta$ | Single | Merged | $\Delta$ | Single | Merged | $\Delta$
P+ [44] |  $<$1 s | .643 $\rightarrow$ | .643 | — | .683 $\rightarrow$ | .683 | — | .515 $\rightarrow$ | .515 | —
Custom Diffusion [24] |  $\sim$2 s | .668 $\rightarrow$ | .673 | +.005 | .648 $\rightarrow$ | .623 | -.025 | .504 $\rightarrow$ | .408 | -.096
DB-LoRA (FedAvg) [1] |  $<$1 s | .613 $\rightarrow$ | .682 | +.069 | .744 $\rightarrow$ | .531 | -.213 | .683 $\rightarrow$ | .098 | -.585
MoS (FedAvg) [12] |  $<$1 s | .625 $\rightarrow$ | .621 | -.004 | .745 $\rightarrow$ | .735 | -.010 | .728 $\rightarrow$ | .706 | -.022
MoS (Grad Fusion) [12] |  $\sim$15 m | .625 $\rightarrow$ | .631 | +.006 | .745 $\rightarrow$ | .729 | -.016 | .728 $\rightarrow$ | .717 | -.011
1-11 Ours |  $<$1 s | .624 $\rightarrow$ | .644 | -.010 | .748 $\rightarrow$ | .741 | -.007 | .740 $\rightarrow$ | .745 | +.005

Table 2: Quantitative results. We provide detailed qualitative comparisons for
each method, evaluated both before and after the merging process. Prior to
merging, our method demonstrates comparable performance in all identity-
related metrics, highlighting its expressivity even with the orthogonality
constraint. Post-merging, our method achieves the highest scores in image and
identity alignment. Our method is also capable of maintaining text alignment
scores comparable to other high-fidelity methods such as P+ and MoS.

#### Baselines.

We compare our method against state-of-the-art baselines on the task of
modular customization, namely: DreamBooth-LoRA [1], $\mathcal{P}+$ [44],
Custom Diffusion [24], and Mix-of-Show [12]. Fine-tuned models are merged
differently depending on the method. DreamBooth-LoRA is merged using FedAvg,
Custom Diffusion is merged using their proposed optimization-based merging
method, and Mix-of-Show is merged using gradient fusion as outlined in their
work. Since $\mathcal{P}+$ does not perform fine-tuning on the weights of the
network, merging is done simply by querying each concept’s token embedding.
For completeness, we also compare against Mix-of-Show merged using FedAvg,
serving as an efficient alternative to the computationally demanding gradient
fusion method.

#### Experimental setup and metrics.

First, we fine-tune each concept individually, without access to data for any
other concept. Each fine-tuned model is then combined with two other concepts
at random using their corresponding method for merging. Following prior work,
we evaluate our method on image alignment, which measures the similarity of
image features between generated images and the input reference image by
measuring their similarity in the CLIP image feature space [10]. Similarly, we
evaluate our method using text alignment, ensuring the output generations
still adhere to the input text-prompts by measuring the text-image similarity
also using CLIP [15]. However, to further illustrate the identity preserving
capabilities of our method, we also evaluate our method using the ArcFace [7]
model. Using the ArcFace model, we measure the rate at which the target human
identity is detected in a set of generated images, we refer to this metric as
identity alignment.

###  4.1 Qualitative Comparisons

#### Merged single-concept results.

We illustrate the identity preserving effect of our method by comparing
single-concept generations of different identities from the same merged model.
As mentioned above, each concept is fine-tuned individually and merged
together during inference. Fig 6 shows generations for three separate concept
identities, each column contains images sampled from the same model. Our
method achieves better identity alignment with the input images in the merged
model compared to methods with comparable merging times. We also achieve
similar results to Mix-of-Show (Gradient Fusion), which requires $\sim$15
minutes to merge three concepts, while our method enables near instant
merging.

#### Merged multi-concept results.

We also show generated images containing all three identities in the merged
model. Leveraging multi-concept sampling techniques from concurrent work [12],
we show examples of multi-concept generations in Fig. 7. Once again, multi-
concept models trained using our method generate images with better identity
alignment than competing baselines. Due to the poor performance of DB-LoRA [1]
and Custom Diffusion [24] for single-concept generations, we omit results for
these methods on multi-concept generation due to space constraints.

$\mathcal{P}+$ [14] suffers from low concept fidelity due to limited
expressivity in their training regime. Although Mix-of-Show [12] (FedAvg)
preserves certain high-level features through the layer-wise text-embedding,
it still suffers from crosstalk due to unconstrained training of weight
residuals. Mix-of-Show (Gradient Fusion) shows impressive identity alignment,
however, this is only enabled by a computationally demanding merging
procedure. Our method achieves high identity alignment while keeping the
merging process at near instant rates.

###  4.2 Quantitative Results

We present quantitative comparisons in Table. 4. Specifically, we show all
three evaluation metrics applied to each method before and after merging. Our
method achieves comparable results in all concept alignment metrics before
merging, illustrating the expressivity of our method despite the orthogonality
constraint. After merging, our method achieves the highest image and identity
alignment scores across all methods, while maintaining comparable text
alignment scores with other high-fidelity methods such as Mix-of-Show and
$\mathcal{P}+$. This illustrates that our method is able to achieve high
identity preservation without sacrificing the ability to generalize for
different contexts.

Note that although Custom Diffusion [24] and DB-LoRA [1] achieves higher text
alignment, this is at the cost of significantly lower concept alignment scores
than that of competing methods.

##  5 Ablations

#### Effect of orthogonality.

In Fig. 8(a), we present generated images from a model created from merging
two separate fine-tuned models (concepts $i$ and $j$). To illustrate the
effect of orthogonality on identity preservation, we manipulate the degree of
orthogonality between $B_{i}$ and $B_{j}$. On the left, we have the worst case
scenario, where $B_{i}=B_{j}$. On the right, we show results where perfect
orthogonality is achieved, i.e. $B_{i}^{T}B_{j}=0$. In between, we construct
$B_{i}$ and $B_{j}$ from a shared orthogonal matrix, but choose half of their
columns to be overlapping. Results in Fig. 8(a) show that orthogonality
contributes significantly to identity preservation even in the extreme case of
merging 2 concepts.

#### Number of merged concepts

Fig. 8(b) shows results generated from models with a range of concepts merged
together. With orthogonality, our model is capable of merging a high number of
concepts with minimal identity loss. In contrast, without orthogonality,
concept fidelity quickly degrades, even with relatively low number of concepts
being combined. Running our model without orthogonality is equivalent to Mix-
of-Show [12] merged using FedAvg [25].

##  6 Discussion

#### Limitations.

Despite showcasing the ability to encode several custom concepts into the same
text-to-image model, generating images with complex compositions/interactions
between multiple custom concepts remains challenging. As concepts, such as
human identities, have the tendency to either be entangled, or even completely
ignored. Existing works [12, 4] have developed certain strategies for
remedying this effect, but such methods are still prone to the aforementioned
failure cases. Another limitation of orthogonal adaption is that it directly
modifies the fine-tuning process. Therefore, existing fine-tuned networks
(e.g. LoRAs [1]) can not be adapted post-hoc to ensure orthogonality.

Figure 8: Ablation studies. (a) Images generated from a model formed by
merging two separately fine-tuned models (concepts i and j), focusing on the
role of orthogonality in preserving identity. (b) Image generations from
models that with a varying number of merged concepts. Without orthogonality,
concept identity is lost even when merging a small number of concepts.

#### Ethics Considerations.

Generative AI could be misused for generating edited imagery of real people
with the intent of spreading disinformation. Such misuse of image synthesis
techniques poses a societal threat, and we do not condone using our work for
such purposes. We also recognize a potential biases in the foundation model we
built upon.

#### Conclusions.

By disentangling customization concepts into orthogonal directions, orthogonal
adaptation streamlines the process of integrating multiple independently fine-
tuned concepts into a single model instantly and with trivial compute, while
also ensuring preservation of each concept. Our work makes a significant step
towards modular customization, where multi-concept customization can be
achieved with individual, privately fine-tuned models.

##  7 Acknowledgements

We thank Sara Fridovich-Keil and Kamyar Salahi for fruitful discussions and
pointers for evaluation metrics. Po is supported by the Stanford Graduate
Fellowship. This project was in part supported by Samsung and Stanford HAI.

## References

  * db_[2022] Low-rank adaptation for fast text-to-image diffusion fine-tuning.  https://github.com/cloneofsimo/lora, 2022.
  * Abdal et al. [2022] Rameen Abdal, Peihao Zhu, John Femiani, Niloy Mitra, and Peter Wonka.  Clip2stylegan: Unsupervised extraction of stylegan edit directions.  In _ACM SIGGRAPH 2022 conference proceedings_ , pages 1–9, 2022.
  * Avrahami et al. [2023] Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski.  Break-a-scene: Extracting multiple concepts from a single image.  _ArXiv_ , abs/2305.16311, 2023.
  * Bar-Tal et al. [2023] Omer Bar-Tal, Lior Yariv, Yaron Lipman, and Tali Dekel.  Multidiffusion: Fusing diffusion paths for controlled image generation.  _arXiv preprint arXiv:2302.08113_ , 2023.
  * Bau et al. [2021] David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba.  Paint by word.  _arXiv preprint arXiv:2103.10951_ , 2021.
  * Brock et al. [2018] Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ , 2018.
  * Deng et al. [2022] Jiankang Deng, Jia Guo, Jing Yang, Niannan Xue, Irene Kotsia, and Stefanos Zafeiriou.  ArcFace: Additive angular margin loss for deep face recognition.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(10):5962–5979, 2022.
  * Dhariwal and Nichol [2021] Prafulla Dhariwal and Alex Nichol.  Diffusion models beat gans on image synthesis.  _ArXiv_ , abs/2105.05233, 2021.
  * Gal et al. [2021] Rinon Gal, Or Patashnik, Haggai Maron, Gal Chechik, and Daniel Cohen-Or.  Stylegan-nada: Clip-guided domain adaptation of image generators.  _arXiv preprint arXiv:2108.00946_ , 2021.
  * Gal et al. [2022] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or.  An image is worth one word: Personalizing text-to-image generation using textual inversion.  _arXiv preprint arXiv:2208.01618_ , 2022.
  * Goodfellow et al. [2020] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * Gu et al. [2023] Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al.  Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models.  In _NeurIPS_ , 2023.
  * Han et al. [2023] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang.  Svdiff: Compact parameter space for diffusion fine-tuning.  _arXiv preprint arXiv:2303.11305_ , 2023.
  * Hertz et al. [2022] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  2022\.
  * Hessel et al. [2022] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.  Clipscore: A reference-free evaluation metric for image captioning, 2022.
  * Ho [2022] Jonathan Ho.  Classifier-free diffusion guidance.  _ArXiv_ , abs/2207.12598, 2022.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  _ArXiv_ , abs/2006.11239, 2020.
  * Hu et al. [2022] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In _ICLR_ , 2022.
  * Huang et al. [2022] Xun Huang, Arun Mallya, Ting-Chun Wang, and Ming-Yu Liu.  Multimodal conditional image synthesis with product-of-experts gans.  In _European Conference on Computer Vision_ , pages 91–109. Springer, 2022.
  * Jia et al. [2023] Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su.  Taming encoder for zero fine-tuning image customization with text-to-image diffusion models.  _arXiv preprint arXiv:2304.02642_ , 2023.
  * Karras et al. [2019] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4401–4410, 2019.
  * Karras et al. [2020] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 8110–8119, 2020.
  * Karras et al. [2021] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  _Advances in Neural Information Processing Systems_ , 34:852–863, 2021.
  * Kumari et al. [2023] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu.  Multi-concept customization of text-to-image diffusion.  In _CVPR_ , 2023.
  * McMahan et al. [2023] H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas.  Communication-efficient learning of deep networks from decentralized data, 2023.
  * Mokady et al. [2022] Ron Mokady, Omer Tov, Michal Yarom, Oran Lang, Inbar Mosseri, Tali Dekel, Daniel Cohen-Or, and Michal Irani.  Self-distilled stylegan: Towards generation from internet photos.  In _ACM SIGGRAPH 2022 Conference Proceedings_ , pages 1–9, 2022.
  * Mokady et al. [2023] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  In _CVPR_ , pages 6038–6047, 2023.
  * Nichol and Dhariwal [2021] Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pages 8162–8171. PMLR, 2021.
  * Pandey et al. [2022] Kushagra Pandey, Avideep Mukherjee, Piyush Rai, and Abhishek Kumar.  Diffusevae: Efficient, controllable and high-fidelity generation from low-dimensional latents.  _Trans. Mach. Learn. Res._ , 2022, 2022.
  * Parmar et al. [2022] Gaurav Parmar, Yijun Li, Jingwan Lu, Richard Zhang, Jun-Yan Zhu, and Krishna Kumar Singh.  Spatially-adaptive multilayer selection for gan inversion and editing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11399–11409, 2022.
  * Po et al. [2023] Ryan Po, Wang Yifan, and Vladislav Golyanik et al.  State of the art on diffusion models for visual computing.  In _arxiv_ , 2023.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Roich et al. [2022] Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or.  Pivotal tuning for latent-based editing of real images.  _ACM Transactions on graphics (TOG)_ , 42(1):1–13, 2022.
  * Rombach et al. [2021] Robin Rombach, A. Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  _2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10674–10685, 2021.
  * Ruiz et al. [2023a] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _CVPR_ , pages 22500–22510, 2023a.
  * Ruiz et al. [2023b] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, Neal Wadhwa, Michael Rubinstein, and Kfir Aberman.  Hyperdreambooth: Hypernetworks for fast personalization of text-to-image models.  _arXiv preprint arXiv:2307.06949_ , 2023b.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Schuhmann et al. [2022] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294, 2022.
  * Serengil and Ozpinar [2020] Sefik Ilkin Serengil and A. Ozpinar.  Lightface: A hybrid deep face recognition framework.  _2020 Innovations in Intelligent Systems and Applications Conference (ASYU)_ , pages 1–5, 2020.
  * Shi et al. [2023] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung.  Instantbooth: Personalized text-to-image generation without test-time finetuning.  _ArXiv_ , abs/2304.03411, 2023.
  * Sohn et al. [2023] Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, Huiwen Chang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al.  Styledrop: Text-to-image generation in any style.  _arXiv preprint arXiv:2306.00983_ , 2023.
  * Song et al. [2020] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _ArXiv_ , abs/2010.02502, 2020.
  * Su et al. [2023] Yu-Chuan Su, Kelvin C. K. Chan, Yandong Li, Yang Zhao, Han-Ying Zhang, Boqing Gong, H. Wang, and Xuhui Jia.  Identity encoder for personalized diffusion.  _ArXiv_ , abs/2304.07429, 2023.
  * Voynov et al. [2023] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman.  $p+$: Extended textual conditioning in text-to-image generation.  _arXiv preprint arXiv:2303.09522_ , 2023.
  * Wang et al. [2023] Zhouxia Wang, Xintao Wang, Liangbin Xie, Zhongang Qi, Ying Shan, Wenping Wang, and Ping Luo.  Styleadapter: A single-pass lora-free model for stylized image generation.  _ArXiv_ , abs/2309.01770, 2023.
  * Xia et al. [2021] Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu.  Tedigan: Text-guided diverse face image generation and manipulation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 2256–2265, 2021.
  * Ye et al. [2023] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang.  Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.  2023\.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  _arXiv preprint arXiv:2206.10789_ , 2(3):5, 2022.

Supplementary Material

##  8 Gaussian random orthogonal matrices

######  Theorem 8.1.

Let $\mathbf{v}\in\mathbb{R}^{d}$ and $\mathbf{u}\in\mathbb{R}^{d}$ be two
random vectors. Let $\mathbf{v}_{i}\sim\mathcal{N}(0,\sigma^{2}I)$ and
$\mathbf{u}_{i}\sim\mathcal{N}(0,\sigma^{2}I)$ for all $i\in[1,d]$
independently, then $\mathbb{E}\left[\mathbf{v}^{T}\mathbf{u}\right]=0$.

###### Proof.

$\displaystyle\mathbb{E}\left[\mathbf{v}^{T}\mathbf{u}\right]$
$\displaystyle=\mathbb{E}\left[\sum_{i=1}^{d}\mathbf{v}_{i}\mathbf{u}_{i}\right]$
$\displaystyle=\sum_{i=1}^{d}\mathbb{E}\left[\mathbf{v}_{i}\mathbf{u}_{i}\right]$
(Linearity of expectation)
$\displaystyle=\sum_{i=1}^{d}\mathbb{E}[\mathbf{v}_{i}]\mathbb{E}[\mathbf{u}_{i}]$
(Independent) $\displaystyle=\sum_{i=1}^{d}0\cdot 0=0.$

∎

######  Corollary 8.1.1.

Let $\mathbf{A}\in\mathbb{R}^{n\times m}$ and
$\mathbf{B}\in\mathbb{R}^{n\times m}$. All entries of these matrices are
independently sampled from $\mathcal{N}(0,\sigma^{2}I)$. Then
$\mathbb{E}[\mathbf{A}^{T}\mathbf{B}]=\mathbf{0}\in\mathbb{R}^{m\times m}$.

###### Proof.

$\displaystyle\mathbb{E}[\mathbf{A}^{T}\mathbf{B}]_{ij}=\mathbb{E}[\mathbf{A}_{i}^{T}\mathbf{B}_{j}]=0.$

∎

##  9 Implementation details

#### Dataset.

We chose to evaluate our method on human datasets due to the robustness of
face recognition algorithms for evaluation purposes. While prior works [24,
35, 12, 13] have employed CLIP-based metrics as a method of evaluating
identity alignment, we found that CLIP features are often poor at identifying
fine details in a custom concept. In Fig. 9, we illustrate that our method
works for non-human objects too.

Figure 9: Identity loss due to crosstalk. We illustrate the effects of
crosstalk by examining the effects of interfering signals between
independently trained LoRAs. Measuring crosstalk through the norm of the
product between two LoRA weights, our method results in lower crosstalk
between independently trained LoRAs. Combined via the same method, our
training regime leads to less crosstalk and therefore better identity
preservation after merging.

#### Evaluation details.

We introduce the identity alignment metric for measuring the ability of our
method (and competing baselines) in capturing the target human identity in
resulting generations. We use the ArcFace [39] facial recognition algorithm
and consider a detection to be recorded when the ArcFace distance between two
detected faces falls below 0.680 [39]. We choose to use detection probability
as a metric rather than the raw distance metric as we found the distance
metric to favor over-fitted models. Past the detection threshold, the distance
metric directly measures the similarity between two faces, which is not ideal
for use-cases such as re-stylization and accessorization.

#### Orthogonal adaptation details.

In our method, we enforce the orthogonality constraint through the LoRA down
projection matrix $B$. This formulation ensures orthogonality in the row-space
of the resulting LoRA matrices. In theory, we can also achieve orthogonality
between trained weight residuals in the column-space, in which case the
orthogonality constraint would have to be enforced on the up-projection matrix
$A$ instead. We choose to enforce orthogonality in the row-space since the
weight residuals interact with the layer inputs through their rows. The
concept preservation formulation presented in Sec. 3 is also reliant on row-
space orthogonality.

Figure 10: Multi-concept failure cases. Multi-concept generation remains as an
open challenge. Despite employing techniques such as regionally controllable
sampling from prior work [12], this method can still suffer from failure cases
such as: (left) ignoring concepts, and (right) leakage of concept attributes
to neighboring identities.

In our results, we chose to use the random orthogonal basis method for
enforcing orthogonality in all our results. Although the Gaussian random
method results in orthogonality on expectation, the orthogonal basis method
led to lower crosstalk emperically. The orthogonal basis method requires a
shared orthogonal matrix to sample from. In practice, using Stable Diffusion
v1.5, there are only four unique input dimensions for all layers in the
diffusion model (320, 640, 768, 1280). Therefore, we only have to store four
unique square matrices from which all sampled $B_{i}$’s can then be sampled
from. These four orthogonal matrices can be downloaded along with the base
model, but they can also be generated on the fly with a fix seed to ensure
they are shared among all users.

#### FedAvg merging coefficient.

Existing work considers FedAvg merging with affine coefficients. However, with
a larger number of concepts, affinely combining each LoRA will lead to
dilution of signal from individual LoRAs. It is also a common practice to
scale individual LoRA weights post-hoc [1] for direct control over the signal
strength from the fine-tuning process. We combine this scaling factor along
with the FedAvg merging factor to obtain a single scale factor $\lambda_{i}$
as shown in Eq. 1. We consider merging coefficients as a hyper-parameter that
can be tuned based on user preferences.

##  10 Additional results

#### Illustration of crosstalk.

Fig. 9 illustrates the importance of minimizing crosstalk for identity
preservation when merging LoRA weights into a single model. We measure
crosstalk formally using the norm of the matrix product between individually
trained LoRA weight residuals. Upper right of Fig. 9 shwos a direct comparison
of the layer-wise normalized matrix product norms between two LoRAs trained
with and without orthogonality constraints. Our method leads to a much lower
levels of crosstalk, which translates to better identity preservation as
observed from the resulting generations.

#### Extended baseline comparisons.

In Fig. 11 We show an extended version of Fig. 6 with generated images of each
identity for each method before they are merged. These results aim to show
that our method is capable of retaining identity alignment with the target
concept before and after merging, while achieving merging of individual LoRAs
instantly without any further fine-tuning or optimization stages.

#### Over-fitting.

Since we are fine-tuning our network over a small custom dataset and we
initialize our custom tokens with a user-defined class label, it may be
susceptible to over-fitting. Prior works such as DreamBooth [35] and Custom
Diffusion [24] alleviate this effect by adding a class preservation loss that
ensures generating images from the class token still produces diverse results.
In our method, we do not employ an explicit loss to prevent over-fitting,
however, we found that our fine-tuned models still preserve the ability to
generate diverse images for the trained class label as shown in Fig. 12

##  11 Limitations and future work

Our method takes an important step towards achieving modular customization.
However, a few important limitations should also be addressed in future work.

Generating multiple custom concepts within the same image remains challenging.
Simply prompting a merged model with multiple custom tokens usually leads to
incoherent hybrids of both objects. Prior works [12] have explored spatial
guidance for better disentangling concepts in a single generation, and we have
also employed similar techniques to generate our results. However, these
methods still lead to failure cases as illustrated in Fig. 10. Concepts are
often ignored, or attributes can leak to neighboring concepts. Future work
should aim to address these struggles to further enable multi-concept
generations.

Storing individual LoRAs, even those trained with our method can also be
expensive. Although LoRAs are already compressive due to their low-ranked
nature, storing a large bank of concepts for modualr customization can still
be expensive. Works such as SVDiff [13] takes steps towards further
compressing LoRAs while maintaining fidelity of generated images. However, our
method does not naturally fit in with the SVDiff method, implying the need for
a tailored compressing methodology.

Figure 11: Extended multi-concept results. We show results for each method
before and after merging the individually trained models into a single, merged
model. Our method is able to capture the target identity with high fidelity
before and after the merging process, while keeping the merging process
instantaneous.

Figure 12: Preservation of class label. Although our method does not enforce
an explicit class preservation loss similar to prior works [35, 24], our
method is able to preserve diversity when generating images of the class label
used for initialization of the custom concept token. We show this across three
different classes, namely: man, woman, and dog.