aainstitutetext: International Centre for Theoretical Physics Asia-Pacific
(ICTP-AP),
University of Chinese Academy of Sciences (UCAS), Beijing, China

# Thermal Corrections to Rényi Entropy in BMS Field Theory

Yuan Zhong<EMAIL_ADDRESS>
###### Abstract

In the study of three-dimensional flat holography, the BMS field theory
manifests the infinite-dimensional BMS3 symmetry, a powerful tool in
elucidating numerous universal phenomena. This paper explores a certain low-
temperature limit of the BMS field theory. The primary focus lies in the
calculation of the thermal correction to the Rényi entropy of the single
interval on the cylinder from the replica trick and the uniformizing map. As a
double check, an alternative method calculating the entanglement entropy is
introduced, with the entanglement first law and the modular Hamiltonian.

##  1 Introduction

In the pursuit of understanding quantum gravity, the holographic principle has
emerged as a profound concept, establishing a deep-seated connection between
$(d+1)$-dimensional quantum gravity and $d$-dimensional quantum field theory
tHooft:1993dmi . A pinnacle realization of this principle is the AdS/CFT
correspondence Maldacena:1997re ; Gubser:1998bc ; Witten:1998qj , which aligns
quantum gravity in $(d+1)$-dimensional asymptotically anti-de Sitter (AdS)
spacetime with a $d$-dimensional conformal field theory (CFT) on the
asymptotic boundary. Central to this correspondence is the notion of symmetry.
The asymptotic symmetry of the bulk theory equates the symmetry of the
boundary theory, providing a powerful framework for deriving universal results
and constraints.

In the study of holographic description of asymptotically flat gravity,
inspired by the success of the role of asymptotic symmetry played in the
AdS/CFT correspondence, the study of the asymptotic symmetry in the
asymptotically flat spacetime, known as the Bondi–van der Burg–Metzner–Sachs
symmetry Bondi:1962px ; Sachs:1962wk , receives much interest in the last few
years. A simpler version of the asymptotically flat gravity is the three-
dimensional BMS3 symmetry. Based on the BMS3 symmetry, the three-dimensional
flat holography was proposed Bagchi:2010zz ; Bagchi:2012cy  that the three-
dimensional asymptotic flat gravity is holographically described by a two-
dimensional quantum field theory governed by the BMS3 symmetry, known as the
BMS field theory (BMSFT) or Carrollian conformal field theory, since the BMS3
algebra is isomorphic to the Carrollian conformal algebra. This is an
infinite-dimensional algebra, and the constraints from it lead to powerful
constraints in the study of the BMS field theories.

In the AdS/CFT framework, an important probe of the duality involves the
holographic entanglement entropy. The Ryu-Takayanagi formula Ryu:2006bv ;
Ryu:2006ef  establishes a correspondence between the entanglement entropy in
the boundary and the area of a minimal surface in the bulk. In the case of the
flat holography, an analogous formula has been proposed Bagchi:2014iea ;
Basu:2015evh ; Hosseini:2015uba ; Jiang:2017ecm ; Hijano:2017eii ;
Godet:2019wje ; Fareghbal:2019czx ; Apolo:2020bld . On the BMS field theory
side, the entanglement entropy for a single interval on the cylinder or plane
in the vacuum state can be obtained through the replica trick Bagchi:2014iea .

While the entanglement entropy serves as a good measurement of entanglement in
pure states, in practice, however, it is always thermally polluted. This paper
addresses the entanglement entropy for a single interval in the thermal state
— a task that is challenging in general given the existence of both the
thermal circle and the spatial circle. However, in the low-temperature limit
$\beta_{\phi}\gg L,{\beta_{u}}/{\beta_{\phi}}\leq O(1)$, where $L$ represents
the circumference of the cylinder coordinated by $\phi$ and $u$, and
$\beta_{\phi}$ and $\beta_{u}$ denote the lengths of the thermal circle along
the $\phi$\- and $u$-directions respectively, the leading thermal correction
to the Rényi entropy becomes computationally feasible. Inspired by universal
results in the low-temperature limit in Conformal Field Theory (CFT)
Cardy:2014jwa ; Herzog:2014fra ; Herzog:2014tfa ; Herzog:2015cxa , we employ
the replica trick to express the leading contribution in the thermal
correction as a correlation function on the branched covering space,
subsequently resolving it with the help of the uniformizing map.

The resulting leading thermal correction to the Rényi entropy takes a
universal form

$\displaystyle\delta S_{n}=\frac{n}{1-n}\left[\left(\frac{\sin\frac{\pi
l_{\phi}}{L}}{n\sin\frac{\pi l_{\phi}}{nL}}\right)^{2\Delta}e^{\frac{2\pi
l_{u}\xi}{L}\left(\cot\frac{\pi l_{\phi}}{L}-\frac{1}{n}\cot\frac{\pi
l_{\phi}}{nL}\right)}-1\right]e^{-\frac{2\pi\beta_{\phi}\Delta}{L}-\frac{2\pi\beta_{u}\xi}{L}},$
(1)

which solely dependent on the scaling dimension $\Delta$ and the boost charge
$\xi$ of the first excited state, together with the geometric configuration of
the entanglement interval including the ranges $l_{\phi}$ and $l_{u}$ of the
interval along the $\phi$\- and $u$-directions. The thermal correction to the
entanglement entropy is obtained through $\delta S_{E}=\delta S_{n\to 1}$.

As a double check, we also employ the entanglement first law, which translates
the calculation of the variation $\delta S_{E}$ of the entanglement entropy
into the variation $\delta\langle K_{{\cal A}}\rangle$ of the expectation
value of the modular Hamiltonian. The latter can be computed directly since
the modular Hamiltonian for a single interval on the cylinder in the thermal
state can be written explicitly. Our calculations demonstrate a consistent
agreement between these two approaches.

This paper is organized as follows. In Sec. 2, we provide a quick review of
BMS field theory. In Sec. 3, we calculate the thermal correction to the Rényi
entropy in a certain low-temperature limit using the replica trick and the
uniformizing map. Additionally, we present an alternative method to compute
the thermal correction to the entanglement entropy through the modular
Hamiltonian and the entanglement first law, serving as a double-check. We
conclude in Sec. 4 with a summary and outline potential future directions.

##  2 Review on the BMS Field Theory

In this section, we give a quick review on some aspects of the BMS field
theory.

####  $\bullet$ BMSFT on the cylinder

A BMSFT on a cylinder $(\phi,u)$ with a circumference

$\displaystyle\phi\sim\phi+L$ (2)

is a two-dimensional quantum field theory that is invariant under the
following BMS transformations

$\displaystyle\phi\to f(\phi),$ (3) $\displaystyle u\to
f^{\prime}(\phi)u+g(\phi).$ (4)

Here, $f(\phi)$ and $g(\phi)$ are periodic functions in $\phi$ with the
periodicity $L$. Then, the infinitesimal BMS transformation generators are
obtained by taking the Fourier modes

$\displaystyle l_{n}$
$\displaystyle=i\frac{L}{2\pi}e^{in\frac{2\pi}{L}\phi}\partial_{\phi}-ne^{in\frac{2\pi}{L}\phi}u\partial_{u},$
(5) $\displaystyle m_{n}$
$\displaystyle=i\frac{L}{2\pi}e^{in\frac{2\pi}{L}\phi}\partial_{u}.$ (6)

####  $\bullet$ BMSFT on the plane

The BMSFT on the $(x,y)$-plane is obtained from the following plane-to-
cylinder transformation Bagchi:2015nca

$\displaystyle x$ $\displaystyle=e^{\frac{2\pi i}{L}\phi},$ (7) $\displaystyle
y$ $\displaystyle=\frac{2\pi i}{L}e^{\frac{2\pi i}{L}\phi}u.$

The infinitesimal symmetry generators on the plane are

$\displaystyle l_{n}$
$\displaystyle=-x^{n+1}\partial_{x}-(n+1)x^{n}y\partial_{y},$ (8)
$\displaystyle m_{n}$ $\displaystyle=-x^{n+1}\partial_{y}.$ (9)

They form the BMS algebra without a central term via the Lie bracket

$\displaystyle[l_{n},l_{m}]$ $\displaystyle=(n-m)l_{m+n},$ (10)
$\displaystyle[l_{n},m_{m}]$ $\displaystyle=(n-m)m_{m+n},$ (11)
$\displaystyle[m_{n},m_{m}]$ $\displaystyle=0.$ (12)

At the quantum level, these symmetry generators $l_{n}$ and $m_{n}$ will
become operators $L_{n}$ and $M_{n}$ which act on the state space. They form
the BMS algebra with central charges $c_{M}$ and $c_{L}$ as

$\displaystyle[L_{n},L_{m}]$
$\displaystyle=(n-m)L_{m+n}+\frac{c_{L}}{12}n(n^{2}-1)\delta_{m+n},$ (13)
$\displaystyle[L_{n},M_{m}]$
$\displaystyle=(n-m)M_{m+n}+\frac{c_{M}}{12}n(n^{2}-1)\delta_{m+n},$ (14)
$\displaystyle[M_{n},M_{m}]$ $\displaystyle=0.$ (15)

A primary operator $\psi$ of the boost charge $\xi$ and the conformal
dimension $\Delta$ is specified by the following conditions

$\displaystyle[L_{0},\psi]$ $\displaystyle=\Delta\psi,$ (16)
$\displaystyle[M_{0},\psi]$ $\displaystyle=\xi\psi,$ (17)
$\displaystyle[L_{n},\psi]$ $\displaystyle=0,~{}n>0,$ (18)
$\displaystyle[M_{n},\psi]$ $\displaystyle=0,~{}n>0.$ (19)

Under a BMS transformation

$\displaystyle\tilde{x}$ $\displaystyle=f(x),$ (20) $\displaystyle\tilde{y}$
$\displaystyle=f^{\prime}(x)y+g(x),$ (21)

a primary operator $\psi$ transforms as

$\displaystyle\tilde{\psi}(\tilde{x},\tilde{y})=(f^{\prime})^{-\Delta}e^{-\xi\frac{yf^{\prime\prime}+g^{\prime}}{f^{\prime}}}\psi(x,y).$
(22)

On the plane, the currents $J(x)$ and $P(x)$ admit the following mode
expansions

$\displaystyle J(x)$ $\displaystyle=\sum_{n}L_{n}x^{-n-2},$ (23)
$\displaystyle P(x)$ $\displaystyle=\sum_{n}M_{n}x^{-n-2}.$ (24)

Under the BMS transformation (20) and (21), the currents $J(x)$ and $P(x)$
transform as Jiang:2017ecm

$\displaystyle\tilde{P}(\tilde{x})$ $\displaystyle=\left(\frac{\partial
f}{\partial x}\right)^{-2}\left(P(x)-\frac{c_{M}}{12}\\{f,x\\}\right),$ (25)
$\displaystyle\tilde{J}(\tilde{x})$ $\displaystyle=\left(\frac{\partial
f}{\partial
x}\right)^{-2}\left(J(x)-\frac{c_{L}}{12}\\{f,x\\}\right)+\left(\frac{\partial
g}{\partial x}\right)^{-2}\left(P(x)-\frac{c_{M}}{12}\\{g,x\\}\right).$

####  $\bullet$ State-operator correspondence

On the $(x,y)$-plane, the in-state corresponds to an operator inserted at
$x=0$ Hao:2021urq . From the plane-to-cylinder map (7), in the cylinder
coordinate, the in-state is inserted at $\phi=i\infty$. Similarly, the out-
state is inserted at $\phi=-i\infty$ in the cylinder coordinate.

####  $\bullet$ Vacuum state

In particular, the above discussion can be specified to the vacuum state
$|0\rangle$ which is the highest weight state with vanishing conformal
dimension $\Delta$ and boost charge $\xi$.

$\displaystyle L_{n}|0\rangle=0,~{}n\geq 0,$ (26) $\displaystyle
M_{n}|0\rangle=0,~{}n\geq 0.$ (27)

In the following of the paper, we will assume that the vacuum state is unique.
Also, to make sense of the first excited state, we will assume that the
spectrum of the primaries is bounded from below. Moreover, for the first
excited state to exist and dominate, we will constrain to theories satisfying
the vacuum gap condition, which means the first excited state is gapped away
from the vacuum state with finitely large conformal dimension and boost
charge.

A prototype of counter examples that does not satisfy the above conditions
could be the BMS free scalar with non-compact scalar field $\Phi$. In this
case, the boost charge $\xi=-\frac{\alpha^{2}}{2}$ of the vertex operator
$e^{i\alpha\Phi}$ could be either positive or negative with
$\alpha\in\mathbb{R}\cup i\mathbb{R}$ hence the spectrum is not bounded from
below. Also, with $\alpha$ small enough, the boost charge could be as close to
the $0$ as one demonstrates, hence the vacuum state is not separable from
other states as well.

However, when one consider a compact free scalar $\Phi\in S^{1}$, the above
assumptions about the vacuum states are satisfied. Also, for a BMS field
theory with finitely many primaries Yu:2022bcp ; Hao:2022xhq , the above
assumptions are satisfied as well. At the cost of mild loss of the
universality, in the following of the paper, we will constrain to BMS filed
theories with a unique separable vacuum state and a spectrum that is bounded
from below.

##  3 Thermal Corrections to the Rényi Entropy

In this section, we use the replica trick and the uniformizing map to
calculate the thermal correction to the Rényi entropy in the BMSFT for a
single interval ${\cal A}$ on the cylinder with circumference $L$.

###  3.1 Thermal Corrections to Rényi Entropy in CFT2

Before we continue our calculation of the thermal correction to the Rényi
entropy for a single interval on the cylinder in the BMSFT, we would like to
first review the similar calculation in the case of CFT2 Cardy:2014jwa  first.

We assume that the theory is put on a cylinder with the circumference $L$,
coordinatized by $w=x-it$, the thermal density matrix written in terms of a
complete set of states is

$\displaystyle\rho=\frac{1}{\text{tr}(e^{-\beta
H})}\sum_{|\phi\rangle}|\phi\rangle\langle\phi|e^{-\beta E_{\phi}}.$ (28)

The Hamiltonian on the cylinder in the CFT is the combination of the left- and
the right-moving zeroth-level Virasoro generators and the central charge,

$\displaystyle H=\frac{2\pi}{L}\left(L_{0}+\bar{L}_{0}-\frac{c}{12}\right).$
(29)

Here, we have assumed that $c_{L}=c_{R}=c$. With the assumptions that there
exists a unique ground state $|0\rangle$, and that the spectrum of conformal
dimensions $\Delta=h+\bar{h}$ is positive and gapped from the smallest
positive value, there should exist an operator $\psi$ of conformal weights
$(h,\bar{h})$ carrying this smallest $\Delta$. This $\psi$ has the smallest
energy $E_{\psi}=\frac{2\pi}{L}(\Delta-\frac{c}{12})$. Then, in the low-
temperature limit $\beta\gg L$, the thermal density matrix admits the
following expansion

$\displaystyle\rho=\frac{|0\rangle\langle
0|+|\psi\rangle\langle\psi|e^{-2\pi\Delta\beta/L}+\cdots}{1+e^{-2\pi\Delta\beta/L}+\cdots}.$
(30)

We consider the entanglement region to be a single interval ${\cal A}$ with
two endpoints

$\displaystyle\partial_{-}{\cal A}:w=\bar{w}=w_{1},~{}\partial_{+}{\cal
A}:w=\bar{w}=w_{2}.$ (31)

For convenience, we also introduce the rescaled endpoints

$\displaystyle\theta_{1,2}=\frac{2\pi w_{1,2}}{L}$ (32)

and their difference

$\displaystyle l=w_{2}-w_{1}.$ (33)

Recall the Renyi entropy is defined by

$\displaystyle S_{n}=\frac{1}{1-n}\log\text{tr}(\rho_{A}^{n}).$ (34)

So we need to calculate the trace of the $n$-th power of the reduced density
matrix $\rho_{\cal A}$. This can be expanded according to the expansion (30)
of the thermal density matrix as

$\displaystyle\text{tr}\rho_{A}^{n}$
$\displaystyle=\frac{\text{tr}\left[\text{tr}_{\bar{{\cal
A}}}(|0\rangle\langle
0|+|\psi\rangle\langle\psi|e^{-2\pi\Delta\beta/L}+\cdots)\right]^{n}}{(1+e^{-2\pi\Delta\beta/L}+\cdots)^{n}}$
(35) $\displaystyle=\text{tr}(\text{tr}_{\bar{{\cal A}}}|0\rangle\langle
0|)^{n}\bigg{[}1+\bigg{(}\frac{\text{tr}(\text{tr}_{\bar{{\cal
A}}}|\psi\rangle\langle\psi|(\text{tr}_{\bar{{\cal A}}}|0\rangle\langle
0|)^{n-1})}{\text{tr}(\text{tr}_{\bar{{\cal A}}}|0\rangle\langle
0|)^{n}}-1\bigg{)}ne^{-2\pi\Delta\beta/L}+\cdots\bigg{]}.$ (36)

The first term $\text{tr}(\text{tr}_{\bar{{\cal A}}}|0\rangle\langle 0|)^{n}$
is just the zero-temperature Rényi entropy. And the expression in the second
term

$\displaystyle\frac{\text{tr}(\text{tr}_{\bar{{\cal
A}}}|\psi\rangle\langle\psi|(\text{tr}_{\bar{{\cal A}}}|0\rangle\langle
0|)^{n-1})}{\text{tr}(\text{tr}_{\bar{{\cal A}}}|0\rangle\langle 0|)^{n}},$
(37)

which determines the leading thermal correction, can be recasted as a 2-point
function of the operator $\psi(w)$ on an $n$-sheeted copy $C_{n}$ of the
cylinder branched over ${\cal A}$ via the state operator correspondence
$|\psi\rangle\sim\lim_{t\to-\infty}\psi(x,t)|0\rangle$ and
$\langle\psi|\sim\lim_{t\to\infty}\langle 0|\psi(x,t)$ as

$\displaystyle\frac{\text{tr}(\text{tr}_{{\cal
A}}|\psi\rangle\langle\psi|(\text{tr}_{{\cal A}}|0\rangle\langle
0|)^{n-1})}{\text{tr}(\text{tr}_{{\cal A}}|0\rangle\langle
0|)^{n}}=\lim_{t_{2}\to\infty,t_{1}\to-\infty}\frac{\langle\psi(w_{2},\bar{w}_{2})\psi(w_{1},\bar{w_{1}})\rangle_{C_{n}}}{\langle\psi(w_{2},\bar{w}_{2})\psi(w_{1},\bar{w_{1}})\rangle_{C_{1}}}.$
(38)

To calculate the 2-point function
$\langle\psi(w_{2},\bar{w}_{2})\psi(w_{1},\bar{w_{1}})\rangle_{C_{n}}$ on the
$n$-sheeted copy $C_{n}$, we can use the following uniformizing map

$\displaystyle\zeta^{(n)}=\bigg{(}\frac{e^{2\pi iw/L}-e^{i\theta_{2}}}{e^{2\pi
iw/L}-e^{i\theta_{1}}}\bigg{)}^{1/n}$ (39)

to send $C_{n}$ to the $\zeta$-plane. The 2-point function on a plane in the
CFT is just

$\displaystyle\langle\psi(\zeta^{(n)}_{2},\bar{\zeta}^{(n)}_{2})\psi(\zeta^{(n)}_{1},\bar{\zeta}^{(n)}_{1})\rangle=\frac{1}{(\zeta^{(n)}_{21})^{2h}(\bar{\zeta}^{(n)}_{21})^{2\bar{h}}}.$
(40)

Mapping it back to the $n$-sheeted copy $C_{n}$ along the uniformizing map
(39), we obtain the expression of the 2-point function on $C_{n}$ as

$\displaystyle\langle\psi(w_{2},\bar{w}_{2})\psi(w_{1},\bar{w_{1}})\rangle_{C_{n}}=\frac{(\frac{d\zeta_{1}}{dw_{1}}\frac{d\zeta_{2}}{dw_{2}})^{h}}{\zeta_{12}^{2h}}\frac{(\frac{d\bar{\zeta}_{1}}{d\bar{w}_{1}}\frac{d\bar{\zeta}_{2}}{d\bar{w}_{2}})^{\bar{h}}}{\bar{\zeta}_{12}^{2\bar{h}}}.$
(41)

Substituting this into (38),we have

$\displaystyle\frac{\langle\psi(w_{2},\bar{w}_{2})\psi(w_{1},\bar{w_{1}})\rangle_{C_{n}}}{\langle\psi(w_{2},\bar{w}_{2})\psi(w_{1},\bar{w_{1}})\rangle_{C_{1}}}=\bigg{[}\frac{1}{n^{2h}}\left(\frac{\zeta_{1}^{(n)}\zeta_{2}^{(n)}}{\zeta_{1}^{(1)}\zeta_{2}^{(1)}}\right)^{h}\left(\frac{\zeta_{2}^{(1)}-\zeta_{1}^{(1)}}{\zeta_{2}^{(n)}-\zeta_{1}^{(n)}}\right)^{2h}\Bigg{]}\cdot[\text{complex
conjugate}].$ (42)

After taking the limit $t_{1}\to-\infty$ and $t_{2}\to\infty$, we have

$\displaystyle\frac{\langle\psi(i\infty)\psi(-i\infty)\rangle_{C_{n}}}{\langle\psi(i\infty)\psi(-i\infty)\rangle_{C_{1}}}\frac{1}{n^{2\Delta}}=\left(\frac{\sin\frac{\theta_{2}-\theta_{1}}{2}}{\sin\frac{\theta_{2}-\theta_{1}}{2n}}\right)^{2\Delta}.$
(43)

Then, from (36) and the definition of the Rényi entropy, we obtain the leading
thermal correction to the Rényi entropy as

$\displaystyle\delta S_{n}=\frac{1}{1-n}\bigg{(}\frac{\sin^{2\Delta}(\pi
l/L)}{n^{2\Delta-1}\sin^{2\Delta}(\pi
l/nL)}-n\bigg{)}e^{-2\pi\Delta\beta/L}+o(e^{-2\pi\Delta\beta/L}).$ (44)

In this calculation, suitable assumptions about the spectrum have been
proposed so that the leading contribution to the thermal correction of the
Rényi entropy is captured by the correlation function of the lightest operator
on the branched covering space. The latter is further worked out with the help
of the uniformizing map that sends this $n$-sheeted copy space to the plane.

###  3.2 Thermal Correction Dominated by the Singlet Primary

Consider a two-dimensional BMS filed theory on the cylinder coordinated by
$(\phi,u)$ with circumference $L$

$\displaystyle\phi\sim\phi+L.$ (45)

To introduce the temperature, we consider the following thermal identification
111Here, we consider the case that $\beta_{u}$ takes the same sign as
$\beta_{\phi}$, because we are going to assume the boost charge $\xi$ is
bounded from below. If $\xi$ is bounded from above instead, then we should
consider $(\phi,u)\sim(\phi+i\beta_{\phi},u-i\beta_{u})$ instead.

$\displaystyle(\phi,u)\sim(\phi+i\beta_{\phi},u+i\beta_{u}).$ (46)

Although it is most natural to consider a thermal circle along the
$u$-direction along since the $u$-direction is the null time direction.
However, as we will discuss in the Sect. 3.4, it is challenging to consider
the low temperature expansion of the thermal state in this case since there
are infinitely many $M_{-\vec{k}}$ descendants of the vacuum sharing the same
boost charge with the vacuum hence they have the same effective energy
conjugate to the $u$-direction. Instead, to fully exploit the power the BMS3
symmetry, we will consider the thermal circle $(\beta_{\phi},\beta_{u})$ along
both $\phi$\- and $u$-directions for the low temperature limit.

The corresponding thermal density matrix is

$\displaystyle\rho=\frac{e^{-\beta_{\phi}L_{0}^{cyl}-\beta_{u}M_{0}^{cyl}}}{\text{tr}\left(e^{-\beta_{\phi}L_{0}^{cyl}-\beta_{u}M_{0}^{cyl}}\right)}.$
(47)

Here, $L_{0}^{cyl}$ and $M_{0}^{cyl}$ are charges generating translations
along $\phi$ and $u$ directions respectively. Under the plane-to-cylinder
transformation of the currents (25), these cylinder translation generators are
related to the canonical BMS generators $L_{0}$ and $M_{0}$ as

$\displaystyle L_{0}^{cyl}=\frac{2\pi}{L}(L_{0}-\frac{c_{L}}{24}),\quad
M_{0}^{cyl}=\frac{2\pi}{L}(M_{0}-\frac{c_{M}}{24}).$ (48)

Substituting this back into (47), the thermal density matrix written in terms
of canonical BMS generators is

$\displaystyle\rho$
$\displaystyle=\frac{e^{-\beta_{\phi}\frac{2\pi}{L}(L_{0}-\frac{c_{L}}{24})-\beta_{u}\frac{2\pi}{L}(M_{0}-\frac{c_{M}}{24})}}{\text{tr}\left(e^{-\beta_{\phi}\frac{2\pi}{L}(L_{0}-\frac{c_{L}}{24})-\beta_{u}\frac{2\pi}{L}(M_{0}-\frac{c_{M}}{24})}\right)}$
$\displaystyle=\frac{e^{-\beta_{\phi}\frac{2\pi}{L}L_{0}-\beta_{u}\frac{2\pi}{L}M_{0}}}{\text{tr}\left(e^{-\beta_{\phi}\frac{2\pi}{L}L_{0}-\beta_{u}\frac{2\pi}{L}M_{0}}\right)}.$
(49)

####  $\bullet$ Low Temperature Expansion

We consider the BMSFT whose spectrum satisfies the following conditions so
that the low-temperature expansion of the thermal density matrix is dominated
by the first excited state.

  * –

There exists a unique ground state $|0\rangle$, around which we can turn on a
small temperature and expand the thermal density matrix.

  * –

In the spectrum both the conformal weight $\Delta$ and the boost charge $\xi$
are bounded from below. This is the analogue of the condition that the energy
is bounded from below in usual Lorentzian field theories. With this
assumption, the effective energy
$-\beta_{\phi}\frac{2\pi}{L}L_{0}-\beta_{u}\frac{2\pi}{L}M_{0}$ conjugate to
the thermal circle is then bounded from below. As a result, it is possible to
expand the thermal density matrix around the vacuum state.

  * –

There exists a gap between the ground state $|0\rangle$ and the lightest state
$|\psi\rangle$ corresponding to the primary operator $\psi$ labelled by
$(\Delta,\xi)$.

The last condition requires more explanation. As we turn on a small
temperature, there might be several candidate lightest states above the ground
state. Depending on the approach to the low-temperature limit, the operator
$\psi$ with the smallest $\Delta+\frac{\beta_{u}}{\beta_{\phi}}\xi$ excites
first.

There are still several difficulties to obtain an expansion dominated by
$\psi$. First, due to the non-unitary nature, although $M_{0}$ is self-
adjoint, it is not diagonalizable. For example, there are two descendants of
$\psi$ at the level 1, $M_{-1}|\psi\rangle$ and $L_{-1}|\psi\rangle$. $M_{0}$
acts on them non-diagonally as a Jordan block

$\displaystyle M_{0}\begin{pmatrix}M_{-1}|\psi\rangle\\\
L_{-1}|\psi\rangle\end{pmatrix}=\begin{pmatrix}\xi&0\\\
1&\xi\end{pmatrix}\begin{pmatrix}M_{-1}|\psi\rangle\\\
L_{-1}|\psi\rangle\end{pmatrix}.$ (50)

As a consequence, the thermal density matrix $\rho$ is also non-
diagonalizable, and it is not possible to expand $\rho$ in terms of
eigenstates $\\{\Phi\\}$ of $L_{0}$ and $M_{0}$ such as

$\displaystyle\rho\propto\sum_{\Phi}e^{-\frac{2\pi}{L}\left(\beta_{\phi}L_{0}^{\Phi}+\beta_{u}M_{0}^{\Phi}\right)}|\Phi\rangle\langle\Phi|.$
(51)

Another problem is that there are infinitely many descendants created by
$M_{-k}$’s with the same boost charge $\xi$ as $\psi$ itself, because $M_{-k}$
all commute with $M_{0}$. So, in a low-temperature limit with
$\beta_{u}\gg\beta_{\phi}$, these descendants will not be suppressed.

At this point, we will not try to answer the interesting question of the
meaning of a non-diagonalizable density matrix. Instead, we restrict to a
particular type of low-temperature limit to avoid the above difficulties.

  * –

Consider the following low-temperature limit

$\displaystyle\beta_{\phi}\gg L,\quad\frac{\beta_{u}}{\beta_{\phi}}\leq O(1).$
(52)

Then, the primary operator $\psi$ dominates the thermal density matrix
expansion.

Under these assumptions, the thermal density matrix is dominated by $\psi$ at
this low temperature as

$\displaystyle\rho$ $\displaystyle=\frac{|0\rangle\langle
0|+|\psi\rangle\langle\psi|e^{-\frac{2\pi\beta_{\phi}}{L}\Delta-\frac{2\pi\beta_{u}}{L}\xi}+\cdots}{1+e^{-\frac{2\pi\beta_{\phi}}{L}\Delta-\frac{2\pi\beta_{u}}{L}\xi}+\cdots}.$
(53)

####  $\bullet$ Entanglement measurements

Consider the entanglement region ${\cal A}$, the reduced density matrix on
${\cal A}$ is

$\displaystyle\rho_{\cal A}=\text{tr}_{\bar{\cal A}}\rho.$ (54)

We are interested in the following entanglement measurements: the Renyi
entropy

$\displaystyle S_{n}=\frac{1}{1-n}\log\text{tr}(\rho_{A}^{n})$ (55)

and the entanglement entropy

$\displaystyle S_{E}=-\text{tr}\rho_{A}\log\rho_{A}=S_{n\to 1}.$ (56)

Concretely, we consider the entanglement region to be a single interval $\cal
A$ specified by its endpoints

$\displaystyle\partial_{-}{\cal A}=(\phi_{-},u_{-}),\quad\partial_{+}{\cal
A}=(\phi_{+},u_{+}).$ (57)

For convenience, let us introduce the range of the interval ${\cal A}$ in the
$\phi$\- and the $u$-directions as

$\displaystyle l_{\phi}=\phi_{+}-\phi_{-},~{}l_{u}=u_{+}-u_{-}.$ (58)

Under the above low-temperature expansion (53), $\text{tr}\rho_{\cal A}^{n}$
can be expanded as

$\displaystyle\text{tr}\rho_{\cal A}^{n}$
$\displaystyle=\frac{\text{tr}[\text{tr}_{\bar{\cal A}}({|0\rangle\langle
0|+|\psi\rangle\langle\psi|e^{-2\pi\beta_{\phi}\Delta/L-2\pi\beta_{u}\xi/L}+\cdots})]^{n}}{(1+e^{-2\pi\beta_{\phi}\Delta/L-2\pi\beta_{u}\xi/L}+\cdots)^{n}}$
(59) $\displaystyle=\text{tr}(\text{tr}_{\bar{\cal A}}|0\rangle\langle
0|)^{n}\bigg{[}1+\bigg{(}\frac{\text{tr}[\text{tr}_{\bar{\cal
A}}|\psi\rangle\langle\psi|(\text{tr}_{\bar{\cal A}}|0\rangle\langle
0|)^{n-1}]}{\text{tr}(\text{tr}_{\bar{\cal A}}|0\rangle\langle
0|)^{n}}-1\bigg{)}ne^{-\frac{2\pi\beta_{\phi}}{L}\Delta-\frac{2\pi\beta_{u}}{L}\xi}+\cdots\bigg{]}.$
(60)

The first term $\text{tr}(\text{tr}_{\bar{\cal A}}|0\rangle\langle 0|)^{n}$
corresponds to the ground-state Rényi entropy. The second term determines the
leading contribution to the low-temperature thermal correction. To calculate
this term, we use the replica trick and the state-operator correspondence to
replace it by a 2-point function of $\psi$ on the $n$-sheeted copy $C_{n}$ of
the original space branched over $\partial{\cal A}$. Use the state-operator
correspondence, the in-state $|\psi\rangle$ corresponds to

$\displaystyle|\psi\rangle=\lim_{\phi\to i\infty}\psi(\phi,u)|0\rangle,$ (61)

and the out-state $\langle\psi|$ corresponds to

$\displaystyle\langle\psi|=\lim_{\phi\to-i\infty}\langle 0|\psi(\phi,u).$ (62)

Together with the replica trick, the coefficient in the thermal correction
term can be written as

$\displaystyle\frac{\text{tr}[\text{tr}_{\bar{\cal
A}}|\psi\rangle\langle\psi|(\text{tr}_{\bar{\cal A}}|0\rangle\langle
0|)^{n-1}]}{\text{tr}(\text{tr}_{\bar{\cal A}}|0\rangle\langle 0|)^{n}}$
$\displaystyle=\lim_{\begin{subarray}{c}\phi_{1}\to+i\infty\\\
\phi_{2}\to-i\infty\end{subarray}}\frac{\text{tr}[\text{tr}_{\bar{\cal
A}}\left(\psi(\phi_{1},u_{1})|0\rangle\langle
0|\psi(\phi_{2},u_{2})\right)(\text{tr}_{\bar{\cal A}}|0\rangle\langle
0|)^{n-1}]}{\text{tr}(\text{tr}_{\bar{\cal A}}|0\rangle\langle 0|)^{n}}$ (63)
$\displaystyle=\lim_{\begin{subarray}{c}\phi_{1}\to+i\infty\\\
\phi_{2}\to-i\infty\end{subarray}}\frac{\langle\psi(\phi_{2},u_{2})\psi(\phi_{1},u_{1})\rangle_{C_{n}}}{\langle\psi(\phi_{2},u_{2})\psi(\phi_{1},u_{1})\rangle_{C_{1}}}.$
(64)

Now, we can use the uniformizing map to calculate this 2-point function of
$\psi$ on $C_{n}$.

####  $\bullet$ Uniformizing Map

To calculate the 2-point function on $C_{n}$, we use the following
uniformizing map from $C_{n}$ to the plane,

$\displaystyle x$ $\displaystyle=\left(\frac{e^{2\pi i\phi/L}-e^{2\pi
i\phi_{-}/L}}{e^{2\pi i\phi/L}-e^{2\pi
i\phi_{+}/L}}\right)^{1/n}=:f^{(n)}(\phi)$ (65) $\displaystyle y$
$\displaystyle=\left(u-\frac{l_{u}}{2\sin\frac{\pi
l_{\phi}}{L}}\sin\frac{\pi(2\phi-\phi_{-}-\phi_{+})}{L}\right)\frac{d}{d\phi}f^{(n)}(\phi).$

This transformation can be decomposed into several steps. In the
$x$-direction, the plane-to-cylinder map $z=e^{2\pi i\phi/L}$ maps the
$S^{1}$-coordinate $\phi$ to the analytically continued complex $z$-plane.
Then, on this complex plane, the $z$-coordinate of $\partial{\cal A}$ becomes
$z_{\pm}=e^{2\pi i\phi_{\pm}/L}$. To introduce the $n$-sheeted copy of this
analytically continued space branched over $z_{\pm}$, we apply an
$SL(2,\mathbb{C})$ transformation $w=\frac{z-z_{-}}{z-z_{+}}$ which sends
$z_{-}$ to $0$ and $z_{+}$ to $\infty$, and take the $n$-th root of it. In the
$y$-direction, the subtraction $\left(u-\frac{l_{u}}{2\sin\frac{\pi
l_{\phi}}{L}}\sin\frac{\pi(2\phi-\phi_{-}-\phi_{+})}{L}\right)$ cancels the
range $l_{u}$ of the interval $\cal A$ in $u$-direction.

The 2-point function of the primary operators on the plane is determined by
the symmetry up to a normalization factor $N$ Bagchi:2009pe ; Chen:2019hbj ,

$\displaystyle\langle\psi(x_{1},y_{1})\psi(x_{2},y_{2})\rangle=Nx_{12}^{-2\Delta}e^{-2\xi\frac{y_{12}}{x_{12}}}.$
(66)

Mapped to the cylinder coordinate along (65), the primary operator $\psi$
transforms as

$\displaystyle\psi(\phi,u)$
$\displaystyle=\left(\frac{dx}{d\phi}\right)^{\Delta}e^{-\xi\frac{{y}\frac{d^{2}\phi}{dx^{2}}}{\frac{d\phi}{dx}}-\xi\frac{d}{d\phi}\left(\frac{l_{u}}{2\sin\frac{\pi
l_{\phi}}{L}}\sin\frac{\pi(2\phi-\phi_{-}-\phi_{+})}{L}\right)}\psi(x,y)$
$\displaystyle=f^{(n)^{\prime}\Delta}(\phi)e^{-\xi\left(uf^{(n)^{\prime}}(\phi)\frac{d(f^{(n)^{\prime}}(\phi)^{-1})}{d\phi}+\frac{\pi
l_{u}}{L\sin\frac{\pi
l_{\phi}}{L}}\cos\frac{\pi(2\phi-\phi_{-}-\phi_{+})}{L}\right)}\psi(x,y).$
(67)

Thus, the correlation function on $C_{n}$ becomes

$\displaystyle\langle\psi(\phi_{2},u_{2})\psi(\phi_{1},u_{1})\rangle_{C_{n}}$
$\displaystyle=$ $\displaystyle
Nf^{(n)^{\prime}\Delta}(\phi_{2})e^{-\xi\left(u_{2}f^{(n)^{\prime}}(\phi)\frac{d(f^{(n)^{\prime}}(\phi_{2})^{-1})}{d\phi_{2}}+\frac{\pi
l_{u}}{L\sin\frac{\pi
l_{\phi}}{L}}\cos\frac{\pi(2\phi_{2}-\phi_{-}-\phi_{+})}{L}\right)}$
$\displaystyle\times
f^{(n)^{\prime}\Delta}(\phi_{1})e^{-\xi\left(uf^{(n)^{\prime}}(\phi_{1})\frac{d(f^{(n)^{\prime}}(\phi_{1})^{-1})}{d\phi_{1}}+\frac{\pi
l_{u}}{L\sin\frac{\pi
l_{\phi}}{L}}\cos\frac{\pi(2\phi_{1}-\phi_{-}-\phi_{+})}{L}\right)}x_{12}^{-2\Delta}e^{-2\xi\frac{y_{12}}{x_{12}}}.$
(68)

Substituting this into (64) and take the limit, we obtain the the correction
term

$\displaystyle\frac{\text{tr}[\text{tr}_{\bar{\cal
A}}|\psi\rangle\langle\psi|(\text{tr}_{\bar{\cal A}}|0\rangle\langle
0|)^{n-1}]}{\text{tr}(\text{tr}_{\bar{\cal A}}|0\rangle\langle
0|)^{n}}=\lim_{\begin{subarray}{c}\phi_{1}\to+i\infty\\\
\phi_{2}\to-i\infty\end{subarray}}\frac{\langle\psi(\phi_{2},u_{2})\psi(\phi_{1},u_{1})\rangle_{C_{n}}}{\langle\psi(\phi_{2},u_{2})\psi(\phi_{1},u_{1})\rangle_{C_{1}}}$
(69) $\displaystyle=$ $\displaystyle\left(\frac{\sin\frac{\pi
l_{\phi}}{L}}{n\sin\frac{\pi l_{\phi}}{nL}}\right)^{2\Delta}e^{\frac{2\pi
l_{u}\xi}{L}\left(\cot\frac{\pi l_{\phi}}{L}-\frac{1}{n}\cot\frac{\pi
l_{\phi}}{nL}\right)}.$ (70)

In the explicit calculation, to take the limit
$\phi_{1}\to+i\infty,\phi_{2}\to-i\infty$, we have set $\phi_{1}=iT_{1}$ and
$\phi_{2}=-iT_{2}$ and expand the above in order of $\epsilon_{1}=e^{-2\pi
T_{1}/L}$ and $\epsilon_{2}=e^{-2\pi T_{2}/L}$.

Substituting this back to the definition (55) of the Rényi entropy, we obtain
the thermal correction to the Rényi entropy

$\displaystyle\delta S_{n}=\frac{n}{1-n}\left[\left(\frac{\sin\frac{\pi
l_{\phi}}{L}}{n\sin\frac{\pi l_{\phi}}{nL}}\right)^{2\Delta}e^{\frac{2\pi
l_{u}\xi}{L}\left(\cot\frac{\pi l_{\phi}}{L}-\frac{1}{n}\cot\frac{\pi
l_{\phi}}{nL}\right)}-1\right]e^{-\frac{2\pi\beta_{\phi}\Delta}{L}-\frac{2\pi\beta_{u}\xi}{L}}.$
(71)

The thermal correction to the entanglement entropy can be obtained by taking
the $n\to 1$ limit,

$\displaystyle\delta S_{E}=\left[2\Delta(1-\frac{\pi l_{\phi}}{L}\cot\frac{\pi
l_{\phi}}{L})+2\xi\left(\frac{\pi^{2}l_{u}l_{\phi}}{L^{2}\sin^{2}\frac{\pi
l_{\phi}}{L}}-\frac{\pi l_{u}}{L}\cot\frac{\pi
l_{\phi}}{L}\right)\right]e^{-\frac{2\pi\beta_{\phi}\Delta}{L}-\frac{2\pi\beta_{u}\xi}{L}}.$
(72)

For a pure state, $S_{n}({\cal A})=S_{n}(\bar{\cal A})$. However, the thermal
correction contribution violates this equality. The compliment of ${\cal A}$
is an interval of range $L-l_{\phi}$ in the $\phi$-direction and $-l_{u}$ in
the $u$-direction. Since $\delta S_{n}(L-l_{\phi},-l_{u})\neq\delta
S_{n}(l_{\phi},l_{u})$, the Rényi entropy is indeed thermally polluted.

###  3.3 Thermal Correction Dominated by the Multiplet Primary

Previously, we obtained the thermal correction to the Rényi entropy and the
entanglement entropy in the case that a singlet primary dominates the thermal
correction. Now, we consider the case that a multiplet primary dominates the
thermal correction. As we will see, the thermal correction to the Rényi
entropy is just that of a singlet multiplied by the rank of the multiplet.
However, this seemingly intuitive result is not that trivial. Actually, the
off-diagonal terms dominate the expansion of the thermal density matrix, but
they just do not contribute to the thermal correction to the Rényi entropy.

The $M_{0}$ acts on a rank-$r$ primary multiplet
$\boldsymbol{O}=(O_{0},O_{1},\cdots,O_{r-1})^{\text{T}}$ as Bagchi:2009ca ;
Chen:2020vvn ; Chen:2022jhx

$\displaystyle M_{0}|O_{a}\rangle$
$\displaystyle=\xi|O_{a}\rangle+|O_{a-1}\rangle,~{}a=1,\cdots,r-1,$ (73)
$\displaystyle M_{0}|O_{0}\rangle$ $\displaystyle=\xi|O_{0}\rangle,~{}a=0.$
(74)

Or in a more compact form,
$M_{0}\boldsymbol{O}=(\xi\boldsymbol{I}_{r}+\boldsymbol{J}_{r})\boldsymbol{O}$.
Here, $\boldsymbol{I}_{r}$ is the rank-$r$ identity matrix, and
$\boldsymbol{J}_{r}$ is the rank-$r$ Jordan cell

$\boldsymbol{J}_{r}=\begin{pmatrix}0&&&\\\ 1&0&&\\\ &\ddots&\ddots&\\\
&&1&0\\\ \end{pmatrix}_{r\times r},$ (75)

which is nilpotent $(\boldsymbol{J}_{r})^{r}=0$. The action of
$e^{-\beta_{\phi}\frac{2\pi}{L}L_{0}-\beta_{u}\frac{2\pi}{L}M_{0}}$ on the
primary part of this multiplet becomes
$e^{-\beta_{u}\frac{2\pi}{L}\boldsymbol{J}_{r}}e^{-\frac{2\pi\beta_{\phi}}{L}\Delta-\frac{2\pi\beta_{u}}{L}\xi}$.
The matrix part $e^{-\beta_{u}\frac{2\pi}{L}\boldsymbol{J}_{r}}$ can be
expanded into finitely many terms as

$\displaystyle
e^{-\beta_{u}\frac{2\pi}{L}\boldsymbol{J}_{r}}=\sum_{k=0}^{r-1}\frac{(-\beta_{u}\frac{2\pi}{L})^{k}}{k!}(\boldsymbol{J}_{r})^{k}.$
(76)

Since $\beta_{u}\gg L$, it seems that the $k=r-1$ term dominates the expansion
(76). However, as we will see later, although this $\boldsymbol{J}^{r-1}$ term
dominates the expansion of the matrix, it does not contribute to the thermal
correction term to the Rényi entropy after taking trace. It is the
$\boldsymbol{J}^{0}$ term that dominates the thermal correction. Explicitly,
the $k=r-1$ term is

$\displaystyle\frac{(-\beta_{u}\frac{2\pi}{L})^{r-1}}{(r-1)!}(\boldsymbol{J}_{r})^{k}=\frac{(-\beta_{u}\frac{2\pi}{L})^{r-1}}{(r-1)!}\begin{pmatrix}0&&&0\\\
\vdots&\ddots\\\ 0&&\ddots&\\\ 1&0&\cdots&0\end{pmatrix}_{r\times
r}=\frac{(-\beta_{u}\frac{2\pi}{L})^{r-1}}{(r-1)!}|O_{0}\rangle\langle
O_{r-1}^{\vee}|.$ (77)

Here, the dual basis $\langle O_{a}^{\vee}|$ is defined by

$\displaystyle\langle O_{a}^{\vee}|O_{b}\rangle=\delta_{a,b}.$ (78)

Putting everything together, the multiplet version of the low-temperature
expansion of density matrix (53) is

$\displaystyle\rho$ $\displaystyle=\frac{|0\rangle\langle
0|+|O_{0}\rangle\langle
O_{r-1}^{\vee}|\frac{(-\frac{2\pi\beta_{u}}{L})^{r-1}}{(r-1)!}e^{-\frac{2\pi\beta_{\phi}}{L}\Delta-\frac{2\pi\beta_{u}}{L}\xi}+\cdots}{1+re^{-\frac{2\pi\beta_{\phi}}{L}\Delta-\frac{2\pi\beta_{u}}{L}\xi}+\cdots}.$
(79)

We can use the inner product between the in-state and the out-state within a
multiplet Chen:2020vvn

$\displaystyle\langle O_{a}|O_{b}\rangle=\delta_{a+b,r-1}$ (80)

to transfrom from the dual basis to the out-states,

$\displaystyle\langle O_{a}^{\vee}|=\langle O_{r-1-a}|.$ (81)

Then, the density matrix can be written as

$\displaystyle\rho$ $\displaystyle=\frac{|0\rangle\langle
0|+|O_{0}\rangle\langle
O_{0}|\frac{(-\frac{2\pi\beta_{u}}{L})^{r-1}}{(r-1)!}e^{-\frac{2\pi\beta_{\phi}}{L}\Delta-\frac{2\pi\beta_{u}}{L}\xi}+\cdots}{1+re^{-\frac{2\pi\beta_{\phi}}{L}\Delta-\frac{2\pi\beta_{u}}{L}\xi}+\cdots}.$
(82)

The correlation function Henkel:2013fja ; Chen:2020vvn  among the rank-$r$
multiplet is

$\langle
O_{a}(x_{x},y_{x})O_{b}(x_{1},y_{1})\rangle=\left\\{\begin{array}[]{ll}0&\mbox{for
$q<0$}\\\
\,\,d_{r}\,x_{12}^{-2\Delta_{i}}e^{-2\xi_{i}\frac{y_{12}}{x_{12}}}\frac{1}{q!}\left(-\frac{2y_{12}}{x_{12}}\right)^{q},&\mbox{otherwise}\end{array}\right.,~{}q=a+b-r+1.$
(83)

In particular, for $r>1$

$\displaystyle\langle O_{0}(x,y)O_{0}(x^{\prime},y^{\prime})\rangle=0.$ (84)

So, we see this $\boldsymbol{J}^{r-1}$ term does not contribute to the thermal
correction term at all. Moreover, it turns out that all the off-diagonal terms
do not contribute to the leading correction to the Rényi entropy. To see this,
consider the $\boldsymbol{J}^{k}$ summand in (76) written in the basis

$\displaystyle\boldsymbol{J}_{r}^{k}=\sum_{a=0}^{r-1-k}|O_{a}\rangle\langle
O_{a+k}^{\vee}|=\sum_{a=0}^{r-1-k}|O_{a}\rangle\langle O_{r-1-a-k}|.$ (85)

Since $q=(a)+(r-1-a-k)=r-1-k\geq r-1$ and the equality holds only if $k=0$,
the correlation function $\langle
O_{a}(x,y)O_{r-1-a-k}(x^{\prime},y^{\prime})\rangle$ vanishes for any $k>0$
because of (83). Only for $k=0$, the correlation function does not vanish,
i.e.,

$\displaystyle\langle
O_{a}(x_{2},y_{2})O_{r-1-a}(x_{1},y_{1})\rangle=Nx_{12}^{-2\Delta}e^{-\xi\frac{y_{12}}{x_{12}}},~{}a=0,\cdots,r-1,$
(86)

which is the same as the correlation function of a singlet (66). So, the
thermal correction to the Rényi entropy is just that of a singlet multiplied
by $r$,

$\displaystyle\delta S_{n}=r\frac{n}{1-n}\left[\left(\frac{\sin\frac{\pi
l_{\phi}}{L}}{n\sin\frac{\pi l_{\phi}}{nL}}\right)^{2\Delta}e^{\frac{2\pi
l_{u}\xi}{L}\left(\cot\frac{\pi l_{\phi}}{L}-\frac{1}{n}\cot\frac{\pi
l_{\phi}}{nL}\right)}-1\right]e^{-\frac{2\pi\beta_{\phi}\Delta}{L}-\frac{2\pi\beta_{u}\xi}{L}}.$
(87)

We see that when a multiplet primary dominates the low-temperature expansion,
although the off-diagonal contributions dominate the correction to the thermal
density matrix, they do not contribute to the correction of the Rényi entropy.
The result is just that of the singlet multiplied by the rank $r$. It will be
interesting to find if there exist any other entanglement measurements to
which the off-diagonal contributions do not vanish. We leave this to future
work.

###  3.4 Comments on Another Limit

So far, we consider the particular low-temperature limit (52), but there is
also a complimentary choice to reach the low-temperature limit so that the
boost charge $\xi$ dominates the first excited state. An extreme case is that
the thermal circle is purely along the $u$-direction. The thermal circle is

$\displaystyle u\sim u+i\beta_{u},~{}\beta_{u}\gg L.$ (88)

Then, the density matrix is proportional to $e^{-\beta_{u}M_{0}}$. In this
case, any primary $\psi$ with boost charge $\xi>0$ is heavier than not only
the vacuum state, but all the descendants of the vacuum (e.g.,
$M_{-\vec{k}}|0\rangle$), because these descendants of the vacuum all have the
boost charge $\xi=0$.

If in the spectrum the boost charge is gapped, then in the $\beta_{u}\gg L$
limit, the density matrix is dominated by the vacuum block, and all the vacuum
descendants are just as heavy as the vacuum, thus a low-temperature expansion
is hardly accessible. However, the result of such thermal correction to the
entanglement entropy might be even more universal than the previous case, as
it depends only on the vacuum block and the algebraic structure, not on the
details of the spectrum.

On the other hand, if there exist any other primary operators with boost
charge $0$, then the density matrix is dominated by these blocks together with
the vacuum block. Since the operator $e^{-\beta_{u}M_{0}}$ does not care about
the conformal weight at all, the results of the thermal correction might be
similar to the case that only the vacuum block dominates.

To summarize, in this type of low-temperature limit, since all descendants of
the vacuum are equally heavy measured by their boost charge, an honest
calculation must include them all. Even it is still possible to expand the
density matrix $e^{-\beta_{u}M_{0}}$ organized by the orders of the Taylor
expansion and the levels of the descendants, it is still hard to trace out
$\bar{{\cal A}}$ and obtain the reduced density matrix on ${\cal A}$ in a
workable way. However, since we expect the result to be universal, once we
work it out in one explicit example, hopefully we might find a solution
according to the answer. Currently, since this type of thermal circle is in
BMSFT is still not well understood, we leave this to future work.

###  3.5 Modular Hamiltonian Approach

In this subsection, we calculate the thermal correction to the entanglement
entropy from the modular Hamiltonian. As a double check, the result agrees
with the previous calculation (72). The modular Hamiltonian for the reduced
density matrix on ${\cal A}$ is defined to be

$\displaystyle K_{{\cal A}}=-\log\rho_{{\cal A}}.$ (89)

From the entanglement first law, for an infinitesimal variation of the state,
the calculation of the variation of the entanglement entropy can be replaced
by the variation of the expectation value of the modular Hamiltonian

$\displaystyle\delta S_{{\cal A}}=\delta\langle K_{{\cal A}}\rangle.$ (90)

In general, the modular Hamiltonian $K_{{\cal A}}$ cannot be written down in
terms of local data. Only in theories with enough symmetry the modular
Hamiltonian has an explicit formula for simple entanglement regions and
special states. In particular, the modular Hamiltonian in BMSFT Apolo:2020qjm
can be written down explicitly for a single interval on the cylinder under the
vacuum state.

For the single interval ${\cal A}$ in the vacuum state on the cylinder with
circumference $L$, the modular Hamiltonian $K_{\cal A}$ can be written as a
local integral of the modular generator $\zeta_{{\cal A}}$ against the
currents $J(\phi)$ and $P(\phi)$ as

$\displaystyle K_{{\cal
A}}=\int_{\phi_{-}}^{\phi_{+}}d\phi\left[\frac{L}{2\pi}\frac{\cos\frac{\pi
l_{\phi}}{L}-\cos\frac{\pi(2\phi-\phi_{+}-\phi_{-})}{L}}{\sin\frac{\pi
l_{\phi}}{L}}J(\phi)+\frac{l_{u}}{2}\frac{\cot\frac{\pi
l_{\phi}}{L}\cos\frac{\pi(2\phi-\phi_{+}-\phi_{-})}{L}-\csc\frac{\pi
l_{\phi}}{L}}{\sin\frac{\pi l_{\phi}}{L}}P(\phi)\right].$ (91)

To calculate the variation of the modular Hamiltonian, we need to calculate
the variation of the currents $J(\phi)$ and $P(\phi)$,

$\displaystyle\delta\langle J\rangle=\langle J\rangle_{\rho}-\langle
J\rangle_{|0\rangle},$ (92) $\displaystyle\delta\langle P\rangle=\langle
J\rangle_{\rho}-\langle P\rangle_{|0\rangle}.$ (93)

Substitute the low-temperature expansion (53) of the thermal density matrix
$\rho$,

$\displaystyle\delta\langle
J(\phi)\rangle=e^{-\frac{2\pi\beta_{\phi}}{L}\Delta-\frac{2\pi\beta_{u}}{L}\xi}\left(\langle
J(\phi)\rangle_{|\psi\rangle}-\langle J(\phi)\rangle_{|0\rangle}\right),$ (94)
$\displaystyle\delta\langle
P(\phi)\rangle=e^{-\frac{2\pi\beta_{\phi}}{L}\Delta-\frac{2\pi\beta_{u}}{L}\xi}\left(\langle
P(\phi)\rangle_{|\psi\rangle}-\langle P(\phi)\rangle_{|0\rangle}\right).$

So, we need to calculate the difference of the expectation values of the
currents between the primary state $|\psi\rangle$ and the vacuum $|0\rangle$.
For this, we apply the plane-to-cylinder transformation (7) and insert the
primary operator $\psi$ at the origin of the $(x,y)$-plane. Recall the mode
expansion of the currents on the plane

$\displaystyle J(x)=\sum_{n}L_{n}x^{-n-2},~{}P(x)=\sum_{n}M_{n}x^{-n-2}.$ (95)

Thus, the expectation values of the currents on the plane under a primary
state are

$\displaystyle\langle J^{pl}(x)\rangle=x^{-2}\Delta,~{}\langle
P^{pl}(x)\rangle=x^{-2}\xi.$ (96)

Applying the transformation of currents (25), the expectation values of
currents on cylinder become

$\displaystyle\langle J(\phi)\rangle$ $\displaystyle=\left(\frac{\partial
x}{\partial\phi}\right)^{2}J^{pl}(x)+\frac{c_{L}}{12}\\{x,\phi\\}=-\frac{4\pi^{2}}{L^{2}}\Delta+\frac{\pi^{2}}{L^{2}}\frac{c_{L}}{6},$
(97) $\displaystyle\langle P(\phi)\rangle$ $\displaystyle=\left(\frac{\partial
x}{\partial\phi}\right)^{2}P^{pl}(x)+\frac{c_{M}}{12}\\{x,\phi\\}=-\frac{4\pi^{2}}{L^{2}}\xi+\frac{\pi^{2}}{L^{2}}\frac{c_{M}}{6}.$
(98)

Thus, the difference of the expectation values of the currents between
$|\psi\rangle$ and $|0\rangle$ are

$\displaystyle\langle J(\phi)\rangle_{|\psi\rangle}-\langle
J(\phi)\rangle_{|0\rangle}$ $\displaystyle=-\frac{4\pi^{2}}{L^{2}}\Delta,$
(99) $\displaystyle\langle P(\phi)\rangle_{|\psi\rangle}-\langle
P(\phi)\rangle_{|0\rangle}$ $\displaystyle=-\frac{4\pi^{2}}{L^{2}}\xi.$ (100)

Substituting this into (94), we obtain the variation of the currents

$\displaystyle\delta\langle J(\phi)\rangle$
$\displaystyle=e^{-\frac{2\pi\beta_{\phi}}{L}\Delta-\frac{2\pi\beta_{u}}{L}\xi}\left(\langle
J(\phi)\rangle_{|\psi\rangle}-\langle
J(\phi)\rangle_{|0\rangle}\right)=-\frac{4\pi^{2}}{L^{2}}\Delta\,e^{-\frac{2\pi\beta_{\phi}}{L}\Delta-\frac{2\pi\beta_{u}}{L}\xi},$
(101) $\displaystyle\delta\langle P(\phi)\rangle$
$\displaystyle=e^{-\frac{2\pi\beta_{\phi}}{L}\Delta-\frac{2\pi\beta_{u}}{L}\xi}\left(\langle
P(\phi)\rangle_{|\psi\rangle}-\langle
P(\phi)\rangle_{|0\rangle}\right)=-\frac{4\pi^{2}}{L^{2}}\xi\,e^{-\frac{2\pi\beta_{\phi}}{L}\Delta-\frac{2\pi\beta_{u}}{L}\xi}.$
(102)

For the modular Hamiltonian (91), the variation of the modular Hamiltonian is

$\displaystyle\delta\langle K_{{\cal A}}\rangle=$
$\displaystyle\int_{\phi_{-}}^{\phi_{+}}d\phi\left[\frac{L}{2\pi}\frac{\cos\frac{\pi
l_{\phi}}{L}-\cos\frac{\pi(2\phi-\phi_{+}-\phi_{-})}{L}}{\sin\frac{\pi
l_{\phi}}{L}}\delta\langle J(\phi)\rangle+\frac{l_{u}}{2}\frac{\cot\frac{\pi
l_{\phi}}{L}\cos\frac{\pi(2\phi-\phi_{+}-\phi_{-})}{L}-\csc\frac{\pi
l_{\phi}}{L}}{\sin\frac{\pi l_{\phi}}{L}}\delta\langle P(\phi)\rangle\right]$
$\displaystyle=$ $\displaystyle\left[2\Delta(1-\frac{\pi
l_{\phi}}{L}\cot\frac{\pi
l_{\phi}}{L})+2\xi\left(\frac{\pi^{2}l_{u}l_{\phi}}{L^{2}\sin^{2}\frac{\pi
l_{\phi}}{L}}-\frac{\pi l_{u}}{L}\cot\frac{\pi
l_{\phi}}{L}\right)\right]e^{-\frac{2\pi\beta_{\phi}}{L}\Delta-\frac{2\pi\beta_{u}}{L}\xi}.$
(103)

This result agrees with the previous calculation (72) of the variation of the
entanglement entropy.

##  4 Discussion

In this paper, we consider the single interval entanglement region on the
cylinder in the BMSFT. We find a suitable low-temperature limit under which an
expansion of the thermal density matrix dominated by the first excited
operator is possible. In this limit, we calculate the thermal correction to
the Rényi entropy by the replica trick and the uniformizing map. As a double
check, for the thermal correction to the entanglement entropy, we also provide
an alternative calculation by the modular Hamiltonian and the entanglement
first law.

Though we provide a double check from another calculation of the entanglement
entropy by modular Hamiltonian, it will be more satisfactory to have a
numerical check in the concrete model. Despite the fact that several concrete
BMSFT models have been found and studied recently, it seems that we still do
not have a satisfactory understanding of their underlying Hilbert space
structure and the correct way to discretize the models in a meaningful way. We
leave this to future work until we have a better understanding of these
concrete models. Also, a concrete model analysis might be helpful to
understand another type of low-temperature limit in Sec. 3.4.

Another interesting thing is to test this thermal correction term in the
holographic entanglement proposals. For the finite temperature, the
calculation on the cylinder is secretly on a torus, and the replica trick
fails as the covering space is of high genus. However, a holographic
calculation with temperature in the bulk is still possible using the geometric
picture. Hence, a comparison between the low-temperature result in the bulk
and boundary is possible.

###### Acknowledgements.

I would like to thank Peng-xiang Hao, Wenxin Lai and Jun Nian for useful
discussions. I would like to specially thank Jun Nian for proofreading the
manuscript. This work was supported in part by the NSFC under grant No.
12147103.

## References

  * (1) G. ’t Hooft, _Dimensional reduction in quantum gravity_ , _Conf. Proc. C_ 930308 (1993) 284 [gr-qc/9310026].
  * (2) J.M. Maldacena, _The Large N limit of superconformal field theories and supergravity_ , _Adv. Theor. Math. Phys._ 2 (1998) 231 [hep-th/9711200].
  * (3) S.S. Gubser, I.R. Klebanov and A.M. Polyakov, _Gauge theory correlators from noncritical string theory_ , _Phys. Lett. B_ 428 (1998) 105 [hep-th/9802109].
  * (4) E. Witten, _Anti-de Sitter space and holography_ , _Adv. Theor. Math. Phys._ 2 (1998) 253 [hep-th/9802150].
  * (5) H. Bondi, M.G.J. van der Burg and A.W.K. Metzner, _Gravitational waves in general relativity. 7. Waves from axisymmetric isolated systems_ , _Proc. Roy. Soc. Lond. A_ 269 (1962) 21.
  * (6) R.K. Sachs, _Gravitational waves in general relativity. 8. Waves in asymptotically flat space-times_ , _Proc. Roy. Soc. Lond. A_ 270 (1962) 103.
  * (7) A. Bagchi, _Correspondence between Asymptotically Flat Spacetimes and Nonrelativistic Conformal Field Theories_ , _Phys. Rev. Lett._ 105 (2010) 171601 [1006.3354].
  * (8) A. Bagchi and R. Fareghbal, _BMS/GCA Redux: Towards Flatspace Holography from Non-Relativistic Symmetries_ , _JHEP_ 10 (2012) 092 [1203.5795].
  * (9) S. Ryu and T. Takayanagi, _Holographic derivation of entanglement entropy from AdS/CFT_ , _Phys. Rev. Lett._ 96 (2006) 181602 [hep-th/0603001].
  * (10) S. Ryu and T. Takayanagi, _Aspects of Holographic Entanglement Entropy_ , _JHEP_ 08 (2006) 045 [hep-th/0605073].
  * (11) A. Bagchi, R. Basu, D. Grumiller and M. Riegler, _Entanglement entropy in Galilean conformal field theories and flat holography_ , _Phys. Rev. Lett._ 114 (2015) 111602 [1410.4089].
  * (12) R. Basu and M. Riegler, _Wilson Lines and Holographic Entanglement Entropy in Galilean Conformal Field Theories_ , _Phys. Rev. D_ 93 (2016) 045003 [1511.08662].
  * (13) S.M. Hosseini and A. Véliz-Osorio, _Gravitational anomalies, entanglement entropy, and flat-space holography_ , _Phys. Rev. D_ 93 (2016) 046005 [1507.06625].
  * (14) H. Jiang, W. Song and Q. Wen, _Entanglement Entropy in Flat Holography_ , _JHEP_ 07 (2017) 142 [1706.07552].
  * (15) E. Hijano and C. Rabideau, _Holographic entanglement and Poincaré blocks in three-dimensional flat space_ , _JHEP_ 05 (2018) 068 [1712.07131].
  * (16) V. Godet and C. Marteau, _Gravitation in flat spacetime from entanglement_ , _JHEP_ 12 (2019) 057 [1908.02044].
  * (17) R. Fareghbal and M. Hakami Shalamzari, _First Law of Entanglement Entropy in Flat-Space Holography_ , _Phys. Rev. D_ 100 (2019) 106006 [1908.02560].
  * (18) L. Apolo, H. Jiang, W. Song and Y. Zhong, _Swing surfaces and holographic entanglement beyond AdS/CFT_ , _JHEP_ 12 (2020) 064 [2006.10740].
  * (19) J. Cardy and C.P. Herzog, _Universal Thermal Corrections to Single Interval Entanglement Entropy for Two Dimensional Conformal Field Theories_ , _Phys. Rev. Lett._ 112 (2014) 171603 [1403.0578].
  * (20) C.P. Herzog, _Universal Thermal Corrections to Entanglement Entropy for Conformal Field Theories on Spheres_ , _JHEP_ 10 (2014) 028 [1407.1358].
  * (21) C.P. Herzog and J. Nian, _Thermal corrections to Rényi entropies for conformal field theories_ , _JHEP_ 06 (2015) 009 [1411.6505].
  * (22) C.P. Herzog and M. Spillane, _Thermal corrections to Rényi entropies for free fermions_ , _JHEP_ 04 (2016) 124 [1506.06757].
  * (23) A. Bagchi, S. Chakrabortty and P. Parekh, _Tensionless Strings from Worldsheet Symmetries_ , _JHEP_ 01 (2016) 158 [1507.04361].
  * (24) P.-x. Hao, W. Song, X. Xie and Y. Zhong, _BMS-invariant free scalar model_ , _Phys. Rev. D_ 105 (2022) 125005 [2111.04701].
  * (25) Z.-f. Yu and B. Chen, _Free field realization of the BMS Ising model_ , _JHEP_ 08 (2023) 116 [2211.06926].
  * (26) P.-X. Hao, W. Song, Z. Xiao and X. Xie, _BMS-invariant free fermion models_ , _Phys. Rev. D_ 109 (2024) 025002 [2211.06927].
  * (27) A. Bagchi, R. Gopakumar, I. Mandal and A. Miwa, _GCA in 2d_ , _JHEP_ 08 (2010) 004 [0912.1090].
  * (28) B. Chen, P.-X. Hao and Z.-F. Yu, _2d Galilean Field Theories with Anisotropic Scaling_ , _Phys. Rev. D_ 101 (2020) 066029 [1906.03102].
  * (29) A. Bagchi and I. Mandal, _On Representations and Correlation Functions of Galilean Conformal Algebras_ , _Phys. Lett. B_ 675 (2009) 393 [0903.4524].
  * (30) B. Chen, P.-X. Hao, R. Liu and Z.-F. Yu, _On Galilean conformal bootstrap_ , _JHEP_ 06 (2021) 112 [2011.11092].
  * (31) B. Chen, P.-x. Hao, R. Liu and Z.-f. Yu, _On Galilean conformal bootstrap. Part II. $\xi$ = 0 sector_, _JHEP_ 12 (2022) 019 [2207.01474].
  * (32) M. Henkel and S. Rouhani, _Logarithmic correlators or responses in non-relativistic analogues of conformal invariance_ , _J. Phys. A_ 46 (2013) 494004 [1302.7136].
  * (33) L. Apolo, H. Jiang, W. Song and Y. Zhong, _Modular Hamiltonians in flat holography and (W)AdS/WCFT_ , _JHEP_ 09 (2020) 033 [2006.10741].


# MagicMix: Semantic Mixing with Diffusion Models

Jun Hao Liew*, Hanshu Yan*, Daquan Zhou & Jiashi Feng
ByteDance Inc.
{junhao.liew, hanshu.yan, daquanzhou<EMAIL_ADDRESS>

###### Abstract

Have you ever imagined what a corgi-alike coffee machine or a tiger-alike
rabbit would look like? In this work, we attempt to answer these questions by
exploring a new task called semantic mixing, aiming at blending two different
semantics to create a new concept (e.g., corgi $+$ coffee machine
$\rightarrow$ corgi-alike coffee machine). Unlike style transfer where an
image is stylized according to the reference style without changing the image
content, semantic blending mixes two different concepts in a semantic manner
to synthesize a novel concept while preserving the spatial layout and
geometry. To this end, we present MagicMix, a simple yet effective solution
based on pre-trained text-conditioned diffusion models. Motivated by the
progressive generation property of diffusion models where layout/shape emerges
at early denoising steps while semantically meaningful details appear at later
steps during the denoising process, our method first obtains a coarse layout
(either by corrupting an image or denoising from a pure Gaussian noise given a
text prompt), followed by injection of conditional prompt for semantic mixing.
Our method does not require any spatial mask or re-training, yet is able to
synthesize novel objects with high fidelity. To improve the mixing quality, we
further devise two simple strategies to provide better control and flexibility
over the synthesized content. With our method, we present our results over
diverse downstream applications, including semantic style transfer, novel
object synthesis, breed mixing, and concept removal, demonstrating the
flexibility of our method. More results can be found on the project page
https://magicmix.github.io/. ††*Equal contribution

Keywords: text-to-image generation, semantic mixing, diffusion model

##  1 Introduction

Figure 1: Task comparison. (a) Style transfer stylizes a content image
according to the given style (e.g., The Starry Night by Vincent van Gogh,
1889) while keeping the image content (e.g., Mona Lisa by Leonardo da Vinci,
1503) unchanged. (b) Compositional generation composes multiple individual
components (e.g., “camel”, “jungle”) to generate a complex scene. While the
composition itself may be novel, each individual component is known. (c)
Differently, semantic mixing aims to blend multiple semantics into one single
object (e.g., “corgi” $+$ “coffee machine” $\rightarrow$ “corgi-alike coffee
machine”). Image credit (input corgi image): Unsplash.

Have you ever imagined what a corgi-alike coffee machine would look like? What
about a rabbit that looks like a tiger? Rendering such imaginary scenes is
extremely challenging due to the non-existence of such objects in the real
world. In this work, we are interested in studying a new problem termed
semantic mixing, whose objective is to blend two different semantics (e.g.,
“corgi” and “coffee machine”) in a semantic manner to create a new concept
(e.g., a corgi-alike coffee machine) while being photo-realistic.

Recently developed large-scale text-conditioned image generation models, such
as DALL-E 2 (Ramesh et al., 2022), Imagen (Saharia et al., 2022), Parti (Yu et
al., 2022), etc., have demonstrated the capabilities in generating astonishing
high-quality images given only text descriptions. Such models can even
generate novel compositions (e.g., an astronaut riding a horse) due to the
strong semantic prior learned from a large collection of image-caption pairs.
Despite the novel combination, each object instance (e.g., “astronaut”,
“horse”) is known given the learned priors. Besides, unlike the compositional
generation (e.g., a corgi sitting beside a coffee machine), we are interested
in synthesizing a novel concept (e.g., a corgi-alike coffee machine or vice-
versa) by semantically mixing two different concepts. Nevertheless, such a
problem is challenging since even a human user might not know how is it
supposed to look like.

To address this, we present a new approach termed MagicMix, which is built
upon existing text-conditioned image diffusion-based generative models. Our
approach is extremely simple, requiring neither re-training nor user-provided
masks. Our method is motivated by the progressive property of diffusion-based
models where layout/shape/color emerges first at early denoising steps while
semantically meaningful contents appear much later during the denoising
process. Given this, we factorize the semantic mixing task into two stages:
(1) layout (e.g., shape and color) semantics and (2) content semantics (e.g.,
the semantic category) generation. Specifically, consider the example of
mixing “corgi” and “coffee machine”, our MagicMix first obtains a coarse
layout semantic either by corrupting a given real photo of corgi or denoising
from a pure Gaussian noise given a text prompt “a photo of corgi”. Then, it
injects a new concept (“coffee machine” in this case) and continues the
denoising process until we obtain the final synthesized results. Such a simple
approach works surprisingly well in general. To improve the blending, we
further devise two simple strategies to provide better control and flexibility
over the generated content.

Semantic mixing is conceptually different from other image editing and
generation tasks, such as the style transfer or compositional generation.
Style transfer stylizes a content image according to the given style (e.g.,
van Gogh’s The Starry Night) while preserving the image content. Compositional
generation, on the other hand, composes multiple individual components to
generate a complex scene (e.g., compositing “camel” and “jungle” leads to an
image of a camel standing in a jungle). While the composition itself may be
novel, each individual component has been already known (i.e., what does a
camel look like). Differently, semantic mixing aims to fuse multiple semantics
into one single novel object/concept (e.g., “corgi” $+$ “coffee machine”
$\rightarrow$ a corgi-alike coffee machine). The differences between these
tasks are illustrated in Figure 1.

Thanks to the strong capability in generating novel concepts, our MagicMix
supports a large variety of creative applications, including semantic style
transfer (e.g., generating a new sign given a reference sign layout and a
certain desired content), novel object synthesis (e.g., generating a lamp that
looks like a watermelon slice), breed mixing (e.g., generating a new species
by mixing “rabbit” and “tiger”) and concept removal (e.g., synthesizing a non-
orange object that looks like an orange). Although the solution is simple, it
paves a new direction in the computational graphics field and provides new
possibilities for AI-aided designs for artists in a wide field, such as
entertainment, cinematography, and CG effects.

In summary, our contributions in this work are:

  * •

A new problem: semantic mixing. The goal is to synthesize a novel concept by
mixing two different semantics while being photo-realistic.

  * •

A new technique: MagicMix. It is built upon large-scale pre-trained text-to-
image diffusion-based generative models and factorizes the semantic mixing
task into the layout and content generation stage.

  * •

We demonstrate several creative applications given our MagicMix, including
semantic style transfer, novel object synthesis, breed mixing, and concept
removal.

##  2 Related Works

###  2.1 Diffusion Probabilistic Models

The Diffusion Probabilistic Models (DPM) family has achieved great success in
both unconditional and conditional generative modeling tasks (Ho et al., 2020;
Song et al., 2022; Ho et al., 2022; Song et al., 2021), including image/video
generation(Ho et al., 2022; Nichol & Dhariwal, 2021), molecular generation(Xu
et al., 2022), and time-series modeling(Rasul et al., 2021). They are not only
able to generate perceptually high-quality samples but also can yield
outstanding log-likelihood scores. However, the computational cost of
diffusion-based models is extremely high due to the iterative sampling
procedure (Song et al., 2022; Lu et al., 2022; Liu et al., 2022a). To
ameliorate this issue, advanced samplers and novel modeling frameworks have
been proposed. For example, Song et al. (2021) proposed the probability-flow-
ODE sampling strategy which inspires the development of DDIM (Song et al.,
2022) and DPM-solver (Lu et al., 2022). Rombach et al. (2022) and Vahdat et
al. (2021) concurrently propose to map data into a lower-dimensional latent
space and use a diffusion model to fit the distribution of latent codes. In
the application of image generation, Ho et al. (2020) demonstrate that DDPM
synthesizes images in a progressive manner, i.e., the layout information in
the intermediate noise (e.g., shape and color) emerges first while the details
are enhanced later. This phenomenon facilitates image editing in the latent
noise space, e.g., image interpolation and inpainting. Our work also exploits
the progressive generation property to achieve semantic mixing in the latent
noise space.

###  2.2 Controllable Image Generation

Generative models can be used to synthesize images conditioned on certain
control signals (Kingma & Welling, 2014; Goodfellow et al., 2020; Oord et al.,
2016; Kobyzev et al., 2021), such as class labels, text descriptions (Saharia
et al., 2022; Yu et al., 2022; Ramesh et al., 2022), and degraded images
(Kawar et al., 2022a). Many approaches have been developed based on auto-
regressive models, variational auto-encoders (VAE), generative adversarial
networks (GAN), and diffusion/score-based models. For example, for text-to-
image generation, Yu et al. (2022) propose to model the probability densities
of image tokens conditioned on text tokens in an auto-regressive manner;
Saharia et al. (2022) directly approximates the conditional probability
densities of images in the RGB space with a diffusion model. To reduce the
computational cost of diffusion-based generation, Rombach et al. (2022)
proposed a latent diffusion model that compresses images into lower-
dimensional codes and models the conditional distribution of latent codes.

###  2.3 Image Editing

Semantic mixing is related to several image editing tasks. The first one is
masked image inpainting which aims to fill in the masked region with
reasonable contents (Lugmayr et al., 2021; Saharia et al., 2021; Peng et al.,
2021; Zhao et al., 2020). Without semantic guidance about the empty area,
generative models tend to synthesize contents such that the entire image
locates in a high-density region. Users cannot interactively control the
synthesized contents to be of interest (Lugmayr et al., 2021). Even though
certain semantic guidance is given, the generated contents may not look
harmonious with other parts of the original image.

The second related task is style transfer which attempts to transfer the
artistic style of one source image to another target one (Gatys et al., 2015;
Karras et al., 2019; Luan et al., 2017; Ulyanov et al., 2016; Zhu et al.,
2020) by modifying the color, shape, and texture of the target image in a
global manner. However, style transfer cannot change the semantic content of
the target image. On the other hand, semantic mixing aims to inject the
content semantics from another object into the layout semantics; it
automatically detects which part of the layout object is to be modified (e.g.,
when mixing the camel sign with “husky” in Fig. LABEL:fig:teaser, only the
camel is replaced by husky while the overall layout remains unchanged). The
resultant image looks natural both entirely and locally.

The third related task is text-driven image editing based on diffusion
generative models. Recent works (Hertz et al., 2022; Gal et al., 2022;
Couairon et al., 2022; Kawar et al., 2022b; Wu & De la Torre, 2022;
Chandramouli & Gandikota, 2022; Kwon & Ye, 2022) explore using diffusion
generative models for text-driven image editing, such as object replacement,
style or color changes, object additions, etc. However, unlike our semantic
mixing, such editing does not lead to synthesis of new unknown object/concept,
which is the main focus of this work. Compositional generation, on the other
hand, composes multiple individual components to generate a complex scene. For
example, Liu et al. (2022b) factorizes the diffusion model conditioned on
multiple prompts as the product of diffusion models conditioned on each prompt
separately. Thus, it can combine scenes described in multiple prompts into one
image. Different from these tasks, semantic mixing aims to fuse multiple
semantics into one single object instead of composing multiple objects within
one image.

Another related task is prompt interpolation, where two different text prompts
are interpolated in the text latent space before being used for content
generation. However, such approach only works well for prompts with similar
semantics (e.g., two dog breeds or two faces). In cases where the two concepts
are extremely dissimilar (e.g., “corgi” and “coffee machine”), the generated
content is usually dominated by one of the concepts (Figure 2). On the
contrary, our semantic mixing can successfully blend two semantics that are
highly dissimilar.

Figure 2: Prompt interpolation fails to yield plausible output when the two
concepts are extremely dissimilar (e.g., “corgi” and “coffee machine”).

##  3 Method

In this section, we first introduce the background of denoising diffusion
probabilistic model (DDPM). Then, we formulate the new problem of semantic
mixing which intends to combine two different semantics to create novel
concepts, and propose an effective diffusion-based framework for implementing
such objective. Besides, we discuss two application instances of the proposed
framework and elucidate the implementation details.

###  3.1 Preliminaries on diffusion models

Deep generative modeling aims to approximate the probability densities of a
set of data via deep neural networks. The deep neural networks are optimized
to mimic the distribution from which the training data are sampled (Ho et al.,
2020; Kingma & Welling, 2014; Goodfellow et al., 2020; Song et al., 2021).
Denoising diffusion probabilistic models (DDPM) are a family of latent
generative models that approximate the probability density of training data
via the reversed processes of Markovian Gaussian diffusion processes (Sohl-
Dickstein et al., 2015; Ho et al., 2020).

Given a set of training data ${\mathbb{D}}=\\{{\bm{x}}^{i}\\}_{i=1}^{N}$, for
all $i=1,\dots,N$, ${\bm{x}}^{i}\in{\mathbb{R}}^{d}$ and are i.i.d. sampled
from certain data distribution $q(\cdot)$, DDPM models the probability density
$q({\bm{x}})$ as the marginal of the joint distribution between ${\bm{x}}$ and
a series of latent variables $x_{1:T}$,

$p_{\theta}({\bm{x}})=\int
p_{\theta}({\bm{x}}_{0:T})d{\bm{x}}_{1:T}\quad\text{with}\quad{\bm{x}}={\bm{x}}_{0}.$

The joint distribution is defined as a Markov chain with learned Gaussian
transitions starting from the standard normal distribution
$\mathcal{N}(\cdot;{\bm{0}},{\bm{I}})$, i.e.,

$p_{\theta}({\bm{x}}_{T})=\mathcal{N}({\bm{x}}_{T};{\bm{0}},{\bm{I}}),\quad
p_{\theta}({\bm{x}}_{t-1}|{\bm{x}}_{t})\equiv\mathcal{N}({\bm{x}}_{t-1};{\bm{\mu}}_{\theta}({\bm{x}}_{t},t),\Sigma_{\theta}({\bm{x}}_{t},t)).$

and thus

$p_{\theta}({\bm{x}}_{0:T})=p_{\theta}({\bm{x}}_{T})\prod_{t=T}^{1}p_{\theta}({\bm{x}}_{t-1}|{\bm{x}}_{t}).$

To perform likelihood maximization of the parameterized marginal
$p_{\theta}(\cdot)$, DDPM uses a fixed Markov Gaussian diffusion process,
$q({\bm{x}}_{1:T}|{\bm{x}}_{0})$, to approximate the posterior
$p_{\theta}({\bm{x}}_{1:T}|{\bm{x}}_{0})$. In specific, two series,
$\alpha_{0:T}$ and $\sigma^{2}_{0:T}$, are defined, where
$1=\alpha_{0}>\alpha_{1}>\dots,>\alpha_{T}\geq 0$ and
$0=\sigma^{2}_{0}<\sigma^{2}_{1}<\dots<\sigma^{2}_{T}$. For any $t>s\geq 0$,

$q({\bm{x}}_{t}|{\bm{x}}_{s})=\mathcal{N}({\bm{x}}_{t};\alpha_{t|s}{\bm{x}}_{s},\sigma^{2}_{t|s}{\bm{I}}),\quad\text{where}\quad\alpha_{t|s}=\frac{\alpha_{t}}{\alpha_{s}},\quad\sigma^{2}_{t|s}=\sigma^{2}_{t}-\alpha^{2}_{t|s}\sigma^{2}_{s}.$

Thus,

$q_{(}{\bm{x}}_{t}|{\bm{x}}_{0})=\mathcal{N}({\bm{x}}_{t}|\alpha_{t}{\bm{x}}_{0},\sigma^{2}_{t}{\bm{I}}).$

The parameterized reversed process $p_{\theta}$ of DDPM is optimized by
maximizing the associated evidence lower bound (ELBO):

$\displaystyle-\log p_{\theta}({\bm{x}}_{0})$ $\displaystyle\leq-\log
p_{\theta}({\bm{x}}_{0})+D_{\mathrm{KL}}(q({\bm{x}}_{1:T}|{\bm{x}}_{0})\|p_{\theta}({\bm{x}}_{1:T}|x_{0}))$
$\displaystyle=D_{\mathrm{KL}}(q({\bm{x}}_{T}|{\bm{x}}_{0})\|p_{\theta}({\bm{x}}_{T}))+\sum_{t=1}^{T}D_{\mathrm{KL}}(q({\bm{x}}_{t-1}|{\bm{x}}_{t},{\bm{x}}_{0})\|p_{\theta}({\bm{x}}_{t-1}|{\bm{x}}_{t}))$

Given a well-trained DDPM, $p_{\theta}(\cdot)$, we can generate novel data via
various types of samplers, including the Langevin ancestral sampling and
probability-flow ODE solvers (Song et al., 2021). During the reversed process
(sampling procedure), a signal with random Gaussian noise will be
progressively transformed into a data point located on the manifold of
training data. In the case of image generation, an image with pure noise will
gradually evolve into a semantically meaningful and perceptually high-quality
image. In each stage, we can estimate the true clean image from the
corresponding noise, and the reconstructions develop from coarse to fine (Ho
et al., 2020). More specifically, it has been shown that the sampling
procedure of DDPMs first crafts the layouts or profiles of the final output
images, and then, synthesizes the details such as the face of a human or the
texture of a flower. Consider a certain intermediate step where the noise has
already contained the information of layout, Ho et al. (2020) demonstrate that
if we fix the noise and run multiple sampling actions starting from this step,
the resultant images will share the common layout. Inspired by this phenomenon
(progressive generation), we will explore how to use diffusion-based models to
perform semantic mixing, i.e., given a certain semantic layout, can we mix it
with any arbitrary contents of our interests?

Figure 3: The overall pipeline of MagicMix. MagicMix is built upon pre-trained
text-to-image diffusion-based generative models. It enables semantic mixing of
two different concepts (e.g., “corgi” and “coffee machine”) by first
synthesizing coarse layout semantics (either by adding Gaussian noise to a
given image or denoising from a random Gaussian noise given a text prompt),
followed by denoising on condition of the desired concept (“coffee machine” in
this example) to obtain a new concept (e.g., corgi-alike coffee machine).
Image credit (input image): Unsplash.

###  3.2 Semantic mixing with diffusion models

The creation of new concepts and objects plays an important part in multimedia
productions, such as creating anthropomorphic animation roles. One paradigm of
concept creation is to mix the semantics of multiple things. For example, many
classical animation characters are designed by mixing animal faces with human
bodies, such as the “Monkey King” and “Puss in Boots”. In this section, we
introduce a novel task of image generation, semantic mixing, which aims to
modify the content in a certain part of a given object while preserving its
layout semantics. The new content is synthesized based on the content
semantics of another object. For example, given the shape and color layout
semantic extracted from one object (e.g., a watermelon slice), one can
generate an object of certain content semantic (e.g., a lamp) in that shape
and color.

Inspired by the progressive generation property of diffusion-based models, we
propose a method, MagicMix, to blend the semantics of two objects. MagicMix
utilizes a pre-trained text-to-image diffusion-based generative model,
$p_{\theta}({\bm{x}}|{\bm{y}})$, to extract and mix two semantics. The overall
framework is illustrated in figure 3. The layout semantic can be extracted
from either a given image or a text prompt, while the content semantic is
determined by a conditioning text prompt. We can generate images of mixed
semantics by denoising the noisy layout images with a conditioning content
prompt. Depending on the input type for layout generation, our MagicMix can
operate in two different modes: (a) image-text mixing and (b) text-text
mixing.

(a) Image-text mixing. In the case where the layout semantic is specified by a
given image ${\bm{x}}$, we first generate its noisy versions corresponding to
the intermediate steps from $K_{\text{min}}$ to $K_{\text{max}}$. Each of the
noisy images $\\{{\bm{x}}_{k}\\}_{K_{\text{min}}}^{K_{\text{max}}}$ consists
of the layout and profile information of the given image ${\bm{x}}$, with
coarse-to-fine layout. Then, we perform the denoising process by conditioning
the text of the content semantic ${\bm{y}}$. The reversed process starts from
the noise of layout semantic
$\hat{{\bm{x}}}_{K_{\text{max}}}={\bm{x}}_{K_{\text{max}}}$. For each step $k$
from $K_{\text{max}}$ to $K_{\text{min}}$, the denoising process utilizes
information from the generative model
$p_{\theta}(\hat{{\bm{x}}}_{k-1}|\hat{{\bm{x}}}_{k},y)$ as well as the
information from the layout noise ${\bm{x}}_{k-1}$. In specific, we first
sample $\hat{{\bm{x}}}^{\prime}_{k-1}$ from
$p_{\theta}(\hat{{\bm{x}}}^{\prime}_{k-1}|\hat{{\bm{x}}}_{k},y)$. Then, we
perform a linear combination of ${\bm{x}}_{k-1}$ and
$\hat{{\bm{x}}}^{\prime}_{k-1}$ with a constant $\nu\in[0,1]$ to craft the
mixed noise
$\hat{{\bm{x}}}_{k-1}=\nu\hat{{\bm{x}}}^{\prime}_{k-1}+(1-\nu){\bm{x}}_{k-1}$.
From step $K_{\text{min}}$ to $0$, the denoising process only depends on the
conditional generative model and no linear interpolation is applied. Figure 4
illustrates the detailed process of image-text mixing.

Figure 4: The detailed pipeline of MagicMix (image-text mixing). Given an
image ${\bm{x}}_{0}$ of layout semantics, we first craft its corresponding
layout noises from step $K_{\text{min}}$ to $K_{\text{max}}$. Starting from
$K_{\text{max}}$, the conditional generation process progressively mixes the
two concepts by denoising given the conditioning content semantics (“coffee
machine” in this example). For each step
$k\in[K_{\text{min}},K_{\text{max}}]$, the generated noise of mixed semantics
is interpolated with the layout noise ${\bm{x}}_{k}$ to preserve more layout
details. For $k\in[0,K_{\text{min}}]$, no interpolation is used.

(b) Text-text mixing. In the other case where the layout semantic is
determined by a text prompt $y_{\text{layout}}$, we first sample a sequence of
layout noise $\\{{\bm{x}}_{k}\\}_{K_{\text{min}}}^{K_{\text{max}}}$ from the
distribution $p_{\theta}({\bm{x}}_{k}|{\bm{y}}_{\text{layout}})$. Then,
similar to the case of image-text mixing, we iteratively denoise the layout
noise to synthesize image of mixed semantics via the generation process
conditioned on ${\bm{y}}_{\text{content}}$. Interpolations are still only
applied from step $K_{\text{max}}$ to $K_{\text{min}}$.

###  3.3 Controlling the mixing ratio

While being able to synthesize images with mixed semantics, it remains unclear
how to control the amount of blending elements, e.g., increasing the element
of “coffee machine” or preserving more elements of “corgi”. Next, we present
several tricks to provide better control and flexibility over the generated
content.

Figure 5: Varying time-step for content injection. Consider
$K=K_{\text{max}}=K_{\text{min}}$. In this example, “corgi” is blended with
“coffee machine” and “husky” in the top and bottom row, respectively. The
time-step to inject the content conditioning affects the preservation of the
layout semantics and the fusion of the content semantics. Image credit (source
image): Unsplash.

####  3.3.1 Time-step for injecting content prompt

As discussed earlier, MagicMix enables mixing of two different concepts by
first crafting noisy images of layout semantic from step $K_{\text{max}}$ to
$K_{\text{min}}$, followed by injection of conditional prompt. We choose
$K_{\text{min}}$ such that the noisy layout image contains rich details from
the given layout image and choose $K_{\text{max}}$ such that the irrelevant
details are destroyed and only a coarse layout is preserved. By integrating
noise across different time steps, the generation process can inject content
semantics to proper regions in the given layout image and preserve more layout
semantics such as shape and color.

Varying time-step for content injection. In Figure 5, let $K=K_{\rm
max}=K_{\rm min}$ and $\nu=1$, we first study the effect of varying the time-
step $K$ for content injection. We first notice that when $K$ is small, the
generation process
$p_{\theta}(\hat{{\bm{x}}}_{0}|{\bm{x}}_{K},{\bm{y}}_{\text{content}})$ can
only modify a small part of image content due to limited number of denoising
steps available. As a result, we can fuse two concepts with similar semantics
(e.g., corgi and husky) but fail to mix two very different objects (e.g.,
corgi and coffee machine). For example, with $K=0.4T$, the eyes and texture of
husky begin to appear on the face of corgi but no elements of “coffee machine”
is found when mixing “corgi” with “husky” and “coffee machine”, respectively.
On the other hand, to enable mixing of two distinct objects, the generation
process conditioned on ${\bm{y}}_{\text{content}}$ requires much larger $K$ to
ensure sufficient steps for mixing. As shown in the top row of Figure 5, given
$K=0.6T$, the conditional generation process successfully synthesizes a coffee
machine in the shape of a corgi.

Preserving more layout details. To preserve more elements of the given layout
object, we perform denoising starting from step ${\bm{x}}_{K_{\text{max}}}$
and propose to interpolate the original layout noise with the synthesized
noise obtained from the conditional generation process earlier. The mixing
constant $\nu$ controls the ratio between layout and content semantics. Once
again, we show an example of mixing the layout of a “corgi” with the content
of a “coffee machine” in Figure 6. When $\nu=1$, the conditional generation
process starts from step ${K_{\text{max}}}$ and uses no information from
$\\{{\bm{x}}_{k}\\}_{K_{\text{min}}}^{K_{\text{max}}}$. We can synthesize an
image of “coffee machine” in a similar color as the “corgi” image but contains
almost no element of “corgi” asides from the shape. Interesting, when $\nu\leq
0.4$, we notice that only a coffee cup is synthesized due to the dominance of
“corgi” elements. In this example, we can obtain an image of corgi-alike
coffee machine when we set $\nu=0.5$ or $0.6$. In practice, we fix $K_{\rm
max}=0.6T$ and $K_{\rm min}=0.3T$, and vary only $\nu$.

Figure 6: Linear interpolation between layout noise and conditionally
generated noise. The constant $\nu$ controls the ratio between the layout
(“corgi”) and content semantics (“coffee machine”). Image credit (source
image): Unsplash.

Optimal value of $\nu$. We also notice that the “optimal” interpolation
constant $\nu$ is determined by the semantic similarity between the two
concepts. For example, when mixing “corgi” and “husky”, diffusion models only
need to modify the eyes and texture. Therefore, we can use a small value of
$\nu$ (e.g., $0.1$). On the contrary, when mixing “corgi” and “coffee
machine”, since the two concepts are extremely dissimilar, the diffusion
models require more denoising steps in order to overwrite the rabbit details.
In this case, we can use a large value of $\nu$ (e.g., $0.9$).

####  3.3.2 Weighted image-text cross-attention

Inspired by Prompt-to-Prompt (Hertz et al., 2022), we also find it effective
to re-weigh the image-text cross-attention to increase or reduce the magnitude
of a concept. Consider the case of mixing “rabbit” and “tiger”. Given text-
image cross-attention maps ${\bm{M}}\in\mathcal{R}^{N_{\rm image}\times N_{\rm
text}}$, where $N_{\rm image}$ and $N_{\rm text}$ denote number of spatial and
text tokens, respectively, and a conditional prompt ${\bm{y}}=$“a photo of
tiger”, we scale the attention map corresponding to the “tiger” tokens with
parameter $s\in[-2,2]$ while keeping the remaining attention maps unchanged.
As shown in Figure 7, the extent of “tiger” content can be adjusted using
different values of positive scale $s$ (e.g., the amount of tiger stripes).

Concept removal. On the other hand, we observe that applying negative $s$
leads to an interesting behaviour: given a hamburger image and a conditioning
prompt ${\bm{y}}=$“a photo of hamburger”, using a negative $s$ amounts to
encouraging the diffusion models to generate an image with a layout similar to
that of hamburger while not being a hamburger. We call this concept removal.
As shown in the right subfigure of Figure 7, when “hamburger” concept is
eliminated, the diffusion model is forced to imagine the most probable non-
hamburger object, such as an airship or crab.

Figure 7: Image-text cross-attention re-weighting. Scaling the attention map
of the desired words enables adjustment of the corresponding elements (e.g.,
observe the change in amount of tiger stripes given different $s$). On the
contrary, multiplying the cross-attention map by a negative scale $s$ removes
the concept, forcing the model to generate a new object that does not belong
to the original class.

###  3.4 Implementation details

In practice, we use Latent Diffusion Models (LDM) for semantic mixing. Since
the auto-encoder in LDM is trained with patch-wise losses, the auto-encoder
preserves the spatial correspondence between the latent space and the original
RGB space. We also observe the progressive generation property in the sampling
procedure of LDMs. Our implementation is developed based on the Stable
Diffusion111Stable-Diffusion: https://huggingface.co/CompVis/stable-diffusion
code base which is an open-source implementation of LDM. One can use Stable
Diffusion to generate high-quality images. It also offers multiple types of
samplers to balance the trade-off between sample quality and computational
efficiency. We use DDIM sampler in our experiments.

##  4 Applications

In this section, we show several applications using our MagicMix, including
(a) semantic style transfer (Section 4.1), (b) novel object synthesis (Section
4.2), (c) breed mixing (Section 4.3) and (d) concept removal (Section 4.4).

###  4.1 Semantic style transfer

We first demonstrate semantic style transfer application by synthesizing signs
with different semantics (e.g., replacing the arrows in two-way sign with
people). Unlike style transfer where the content image is stylized based on
the reference style image without changing the image content, our MagicMix
allows the user to inject new semantics while preserving the spatial layout
and geometry (e.g., the triangular sign). We show some examples in Figure 8.
Note that the background is well-preserved despite the large content change.
Such application could possibly be used to assist design of new logo/sign by
injecting a new concept to a template.

Figure 8: Semantic style transfer. A new sign can be synthesized by injecting
new concept-of-interest (e.g., replacing the skull with raccoon or husky).
Note that the spatial layout and background is well preserved despite the
large content change. Image credit (source images): Unsplash.

###  4.2 Novel Object Synthesis

Our MagicMix also allows the synthesis of novel objects by injecting new
concepts (e.g., coffee machine) into an existing object (e.g., bus). This can
be extremely useful in inspiring creativity when designing new commercial
products. We show some examples in Figure 9. It is worth nothing that the
background context adapts accordingly based on the conditioning prompt. For
example, the road has turned into sea when the “submarine” is mixed with a
pumpkin image. Similarly, when a pagoda is mixed with “chocolate cake”, the
road has become a table to better fit the entire image context. This suggests
that mixing of the two concepts occurs at a semantic level.

Figure 9: Novel object synthesis enables creation of new object by injecting a
new concept into existing object (e.g., bus $+$ coffee machine). Note that the
image background adapts to the conditioning prompt (e.g., road has turned into
sea when “submarine” is injected to the pumpkin image.). Image credit (source
images): Unsplash.

###  4.3 Breed mixing

Next, we demonstrate the possibility of mixing two different breeds given our
method. As depicted in the first two rows of Figure 10, our method can mix two
different animal breeds (e.g., Labrador and bulldog) and generate plausible
results with distinct features (Labrador’s ears and bulldog’s face). More
interestingly, our method can even mix two different species and generate new
unseen animal species as depicted in the third and forth rows. Note that some
of these combinations share almost no commonalities (e.g., rabbit and chicken,
rabbit and tiger), yet we can still obtain photo-realistic results. Similarly,
in the last two rows, we also demonstrate the mixing of two different fruits
(e.g., pineapple and grapes) or flowers (e.g., rose and dandelion).

Figure 10: Breed mixing. (1st-2nd rows) Our method enables mixing of two
different animal breeds (e.g., Labrador and bulldog) while retaining each
species’ distinct features (Labrador’s ears and bulldog’s face). (3rd-4th
rows) In addition, our MagicMix allows mixing of two different animal species
and synthesizes new unseen species (e.g., rabbit + tiger) with high-quality
synthesis and fidelity. (5th-6th rows) Similarly, we can also mix two
different fruits (e.g., pineapple and grapes) or flower species (e.g., rose
and dandelion) to create a new species. Image credit (source images):
Unsplash.

###  4.4 Concept Removal

We have presented various applications by injecting new semantics into
existing ones. Here, we are also interested in generating a new image by
removing its original semantic and let the model to decide what to generate
aside from its original content. This can be easily achieved by multiplying
the image-text cross-attention map by a negative weight (Section 3.3.2). Some
examples are shown in Figure 11. As we can see, the generated images largely
preserve the overall layout while removing the original semantics. For
example, as shown in the last row, given a basket of fruits, by removing the
concept “fruits”, we obtain a basket of flowers instead. On the other hand,
removing the concept “basket” leads to generation of a cake with fruits on
top.

Figure 11: Concept removal enables synthesis of new image without its original
content. Note that the overall layout is largely preserved while the original
semantic is removed. Image credit (source images): Unsplash.

###  4.5 Text-text Semantic Mixing

In the previous sections, we have demonstrated several applications of our
MagicMix using image-text mixing (layout semantics is crafted based on a given
image). Next, we also provide some results of MagicMix using text-text mixing
mode where no image is needed. As shown in Figure 12, our method successfully
mixes two different semantics and generates photo-realistic results. However,
one limitation of text-text semantic mixing is that, since no image is
provided for layout generation, the final synthesis result is unpredictable.

Figure 12: Examples of text-text semantic mixing where no input image is
required.

##  5 Limitations

We identify a failure case of our method where two concepts cannot be mixed if
they do not share any shape similarity (e.g., mixing “van” and “cat” or
“toilet roll” and “corgi”). In this case, the two concepts will be simply
composited (e.g., a cat riding a van or a painting of corgi on the toilet
roll). Some examples can be found in Figure 13. We leave solving these to
future work.

Figure 13: Failure cases. When the two concepts do not share any shape
similarity, MagicMix will instead compose the two (e.g., a cat riding a van or
a painting of corgi on the toilet roll).

##  6 Conclusion

In this work, we present a novel task called semantic mixing, whose objective
is to mix two different semantics to synthesize a new unseen concept. To this
end, we present MagicMix, a simple solution based on pre-trained text-
conditioned diffusion-based image generation models. Our method exploits the
properties of diffusion-based generative models by injecting new concepts
during the denoising process. Our approach does not require any spatial masks
or re-training, while preserving the layout and geometry. Given this, our
MagicMix supports several downstream applications, including semantic style
transfer, novel object synthesis, breed mixing and concept removal.

##  7 Societal Impact

The goal of our work is to synthesize a novel object of mixed concepts.
Similar to other deep learning-based image synthesis and editing algorithms,
our method has both positive and negative societal impacts depending on the
applications and usages. On the positive side, MagicMix could inspire creation
of new commercial products (e.g., corgi-alike coffee machine). On the down
side, it could be used by malicious parties to deceive or mislead humans.
Another issue is that the pre-trained model used in this work, Stable
Diffusion v1.4 (Rombach et al., 2022), was trained on LAION dataset, which is
known to have social and cultural bias.

## References

  * Chandramouli & Gandikota (2022) Paramanand Chandramouli and Kanchana Vaishnavi Gandikota.  LDEdit: Towards Generalized Text Guided Image Manipulation via Latent Diffusion Models, October 2022.  arXiv:2210.02249 [cs].
  * Couairon et al. (2022) Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord.  DiffEdit: Diffusion-based semantic image editing with mask guidance, October 2022.  arXiv:2210.11427 [cs].
  * Gal et al. (2022) Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or.  An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion.  2022\.  Publisher: arXiv Version Number: 1.
  * Gatys et al. (2015) Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge.  A Neural Algorithm of Artistic Style, September 2015.  arXiv:1508.06576 [cs, q-bio] version: 2.
  * Goodfellow et al. (2020) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  _Communications of the ACM_ , 63(11):139–144, 2020.
  * Hertz et al. (2022) Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-Prompt Image Editing with Cross Attention Control, August 2022.  arXiv:2208.01626 [cs].
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising Diffusion Probabilistic Models, December 2020.  arXiv:2006.11239 [cs, stat].
  * Ho et al. (2022) Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet.  Video Diffusion Models.  Technical Report arXiv:2204.03458, arXiv, June 2022.  arXiv:2204.03458 [cs] type: article.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila.  A Style-Based Generator Architecture for Generative Adversarial Networks, March 2019.  arXiv:1812.04948 [cs, stat].
  * Kawar et al. (2022a) Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song.  Denoising Diffusion Restoration Models, February 2022a.  arXiv:2201.11793 [cs, eess].
  * Kawar et al. (2022b) Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani.  Imagic: Text-Based Real Image Editing with Diffusion Models, October 2022b.  arXiv:2210.09276 [cs].
  * Kingma & Welling (2014) Diederik P. Kingma and Max Welling.  Auto-Encoding Variational Bayes, May 2014.  arXiv:1312.6114 [cs, stat].
  * Kobyzev et al. (2021) Ivan Kobyzev, Simon J.D. Prince, and Marcus A. Brubaker.  Normalizing Flows: An Introduction and Review of Current Methods.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 43(11):3964–3979, November 2021.  Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.
  * Kwon & Ye (2022) Gihyun Kwon and Jong Chul Ye.  Diffusion-based Image Translation using Disentangled Style and Content Representation, September 2022.  arXiv:2209.15264 [cs, stat].
  * Liu et al. (2022a) Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao.  Pseudo Numerical Methods for Diffusion Models on Manifolds.  Technical Report arXiv:2202.09778, arXiv, February 2022a.  arXiv:2202.09778 [cs, math, stat] type: article.
  * Liu et al. (2022b) Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum.  Compositional Visual Generation with Composable Diffusion Models, July 2022b.  arXiv:2206.01714 [cs].
  * Lu et al. (2022) Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps, August 2022.  arXiv:2206.00927 [cs, stat].
  * Luan et al. (2017) Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala.  Deep Photo Style Transfer, April 2017.  arXiv:1703.07511 [cs] version: 3.
  * Lugmayr et al. (2021) Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool.  RePaint: Inpainting Using Denoising Diffusion Probabilistic Models.  pp. 11, 2021.
  * Nichol & Dhariwal (2021) Alex Nichol and Prafulla Dhariwal.  Improved Denoising Diffusion Probabilistic Models.  pp. 10, 2021.
  * Oord et al. (2016) Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu.  Pixel Recurrent Neural Networks, August 2016.  arXiv:1601.06759 [cs] version: 3.
  * Peng et al. (2021) Jialun Peng, Dong Liu, Songcen Xu, and Houqiang Li.  Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE, March 2021.  arXiv:2103.10022 [cs].
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical Text-Conditional Image Generation with CLIP Latents, April 2022.  arXiv:2204.06125 [cs].
  * Rasul et al. (2021) Kashif Rasul, Calvin Seward, Ingmar Schuster, and Roland Vollgraf.  Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting.  In _Proceedings of the 38th International Conference on Machine Learning_ , pp. 8857–8868. PMLR, July 2021.  ISSN: 2640-3498.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-Resolution Image Synthesis with Latent Diffusion Models, April 2022.  arXiv:2112.10752 [cs].
  * Saharia et al. (2021) Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi.  Image Super-Resolution via Iterative Refinement, June 2021.  arXiv:2104.07636 [cs, eess].
  * Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi.  Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding.  Technical Report arXiv:2205.11487, arXiv, May 2022.  arXiv:2205.11487 [cs] type: article.
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep Unsupervised Learning using Nonequilibrium Thermodynamics.  pp. 10, 2015.
  * Song et al. (2022) Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising Diffusion Implicit Models, June 2022.  arXiv:2010.02502 [cs].
  * Song et al. (2021) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  SCORE-BASED GENERATIVE MODELING THROUGH STOCHASTIC DIFFERENTIAL EQUATIONS.  pp. 36, 2021.
  * Ulyanov et al. (2016) Dmitry Ulyanov, Vadim Lebedev, Andrea Vedaldi, and Victor Lempitsky.  Texture Networks: Feed-forward Synthesis of Textures and Stylized Images, March 2016.  arXiv:1603.03417 [cs].
  * Vahdat et al. (2021) Arash Vahdat, Karsten Kreis, and Jan Kautz.  Score-based Generative Modeling in Latent Space.  Technical Report arXiv:2106.05931, arXiv, December 2021.  arXiv:2106.05931 [cs, stat] version: 3 type: article.
  * Wu & De la Torre (2022) Chen Henry Wu and Fernando De la Torre.  Unifying Diffusion Models’ Latent Space, with Applications to CycleDiffusion and Guidance, October 2022.  arXiv:2210.05559 [cs].
  * Xu et al. (2022) Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang.  GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation, March 2022.  arXiv:2203.02923 [cs, q-bio].
  * Yu et al. (2022) Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu.  Scaling Autoregressive Models for Content-Rich Text-to-Image Generation, June 2022.  arXiv:2206.10789 [cs].
  * Zhao et al. (2020) Lei Zhao, Qihang Mo, Sihuan Lin, Zhizhong Wang, Zhiwen Zuo, Haibo Chen, Wei Xing, and Dongming Lu.  UCTGAN: Diverse Image Inpainting Based on Unsupervised Cross-Space Translation.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 5740–5749, June 2020.  ISSN: 2575-7075.
  * Zhu et al. (2020) Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.  Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, August 2020.  arXiv:1703.10593 [cs].