# Room-temperature quantum emission from interface excitons in mixed-
dimensional heterostructures

N. Fang<EMAIL_ADDRESS>Nanoscale Quantum Photonics Laboratory, RIKEN
Cluster for Pioneering Research, Saitama 351-0198, Japan  Y. R. Chang
Nanoscale Quantum Photonics Laboratory, RIKEN Cluster for Pioneering Research,
Saitama 351-0198, Japan  S. Fujii  Quantum Optoelectronics Research Team,
RIKEN Center for Advanced Photonics, Saitama 351-0198, Japan  Department of
Physics, Keio University, Yokohama 223-8522, Japan  D. Yamashita  Quantum
Optoelectronics Research Team, RIKEN Center for Advanced Photonics, Saitama
351-0198, Japan  Platform Photonics Research Center, National Institute of
Advanced Industrial Science and Technology (AIST), Ibaraki 305-8568, Japan  M.
Maruyama  Department of Physics, University of Tsukuba, Ibaraki 305-8571,
Japan  Y. Gao  Department of Physics, University of Tsukuba, Ibaraki 305-8571,
Japan  C. F. Fong  Nanoscale Quantum Photonics Laboratory, RIKEN Cluster for
Pioneering Research, Saitama 351-0198, Japan  D. Kozawa  Nanoscale Quantum
Photonics Laboratory, RIKEN Cluster for Pioneering Research, Saitama 351-0198,
Japan  Quantum Optoelectronics Research Team, RIKEN Center for Advanced
Photonics, Saitama 351-0198, Japan  Research Center for Materials, National
Institute for Materials Science, Ibaraki 305-0044, Japan  K. Otsuka  Nanoscale
Quantum Photonics Laboratory, RIKEN Cluster for Pioneering Research, Saitama
351-0198, Japan  Department of Mechanical Engineering, The University of
Tokyo, Tokyo 113-8656, Japan  K. Nagashio  Department of Materials
Engineering, The University of Tokyo, Tokyo 113-8656, Japan  S. Okada
Department of Physics, University of Tsukuba, Ibaraki 305-8571, Japan  Y. K.
Kato<EMAIL_ADDRESS>Nanoscale Quantum Photonics Laboratory, RIKEN
Cluster for Pioneering Research, Saitama 351-0198, Japan  Quantum
Optoelectronics Research Team, RIKEN Center for Advanced Photonics, Saitama
351-0198, Japan

###### Abstract

The development of van der Waals heterostructures has introduced
unconventional phenomena that emerge at atomically precise interfaces. For
example, interlayer excitons in two-dimensional transition metal
dichalcogenides show intriguing optical properties at low temperatures. Here
we report on room-temperature observation of interface excitons in mixed-
dimensional heterostructures consisting of two-dimensional tungsten diselenide
and one-dimensional carbon nanotubes. Bright emission peaks originating from
the interface are identified, spanning a broad energy range within the
telecommunication wavelengths. The effect of band alignment is investigated by
systematically varying the nanotube bandgap, and we assign the new peaks to
interface excitons as they only appear in type-II heterostructures. Room-
temperature localization of low-energy interface excitons is indicated by
extended lifetimes as well as small excitation saturation powers, and photon
correlation measurements confirm single-photon emission. With mixed-
dimensional van der Waals heterostructures where band alignment can be
engineered, new opportunities for quantum photonics are envisioned.

##  I Introduction

The discovery of van der Waals (vdW) materials, including two-dimensional (2D)
transition metal dichalcogenides (TMDs) and graphene, has brought about a
revolution in the assembly of artificial heterostructures by allowing for the
combination of two different materials without the constraints of lattice
matching. Such an unprecedented level of flexibility in heterostructure design
has led to the emergence of novel properties not seen in individual materials.
A prime example is twisted bilayer graphene at magic angles, which exhibits
exotic phases such as correlated insulating states [1] and superconductivity
[2]. Another notable development is the stacking of two TMDs, resulting in the
observation of unique excitons known as interlayer excitons, characterized by
electrons and holes located in separate layers [3, 4, 5, 6]. The spatially
indirect nature of interlayer excitons imparts them with distinct properties,
including long exciton lifetimes [3], extended diffusion lengths [7], large
valley polarization [8], and significant modulation by moiré potentials [9,
10].

The existing vdW heterostructures comprise of 2D materials with similar
lattice structure, excitonic characteristics, and inherently identical
dimensions. Development of vdW heterostructures that encompass lower
dimensional materials may give rise to unique interface exciton states
resulting from the mixed dimensionality. Carbon nanotubes (CNTs), a typical
one-dimensional (1D) material, are ideal for such heterostructures as they
have all bonds confined to the tube itself [11, 12]. CNTs interact with 2D
materials through weak vdW forces, resulting in well-defined, atomically sharp
interfaces [13, 14]. The chirality-dependent bandgap of CNTs can be utilized
to tune the band alignment [15], allowing for unambiguous identification of
excitonic states at the 1D-2D interface.

Here we report on the observation of multiple new excitonic peaks in the 1D-2D
CNT/tungsten diselenide (WSe2) heterostructures at room temperature. These
peaks appear exclusively at the interface region with a broad energy range
lower than CNT E11 states, and their dependence on the chirality of CNTs and
the layer number of WSe2 is investigated. The emergence of the peaks is found
to be highly correlated with the band alignment, and they are interpreted as
interface excitons. Prominent linear polarization, low excitation saturation
power, and a long lifetime are characteristic of low-energy interface
excitons, suggesting strong confinement. Through photon correlation
measurements, room-temperature single-photon emission (SPE) has been
confirmed. These findings expand the existing concept of spatially indirect
excitons based on 2D heterostructures to 1D systems, demonstrating significant
potential of the novel interface excitons for nanophotonics and quantum
information processing.

##  II Results and discussion

### Emerging peaks in 1D-2D mixed-dimensional heterostructures.

Figure 1:  Emergence of interface excitons in 1D-2D heterostructures. a, A
schematic of a suspended CNT/WSe2 heterostructure. b, An optical microscope
image of the (9,4) CNT/1L WSe2 heterostructure. The substrate is SiO2/Si. The
scale bar represents 5 $\mu$m. c, The band diagram of the (9,4) CNT/1L WSe2
heterostructure, illustrating the excitons in CNT and WSe2 as well as the
interface exciton. d, A PL spectrum of the pristine, suspended (9,4) CNT prior
to WSe2 transfer. e, A PL spectrum immediately following the transfer of 1L
WSe2 to form the heterostructure. Arrows indicate unstable IX peaks. The inset
depicts the time-trace of the integrated PL intensity for the IX peak
indicated by the orange arrow. f, A PL spectrum from the same sample after
approximately eight hours of optical measurements. g,h, PL intensity maps of
E11 (g) and IX1 (h). The edges of the trench are indicated by the green broken
lines. The scale bar represents 1 $\mu$m. i, A PLE map of the (9,4) CNT/1L
WSe2 heterostructure. The excitation power is 4.5 and 5 $\mu$W for (f) and
(i), and 10 $\mu$W otherwise. The excitation energy is 1.730 eV for (d,e),
1.680 eV for (f), and 1.653 eV for (g,h).

The CNT/WSe2 heterostructures under investigation are entirely free-standing
to preclude substrate effects [5, 16], as depicted in Fig. 1a and b. CNTs are
initially grown over trenches, followed by a placement of a WSe2 flake upon
the tubes using the anthracene-assisted transfer technique [17, 18]. A (9,4)
CNT is selected as a representative case, forming type-II band alignment with
WSe2 [15]. Such alignment should establish new excitonic states between the
CNT conduction band minimum and the WSe2 valence band maximum (Fig. 1c).

Room-temperature photoluminescence (PL) spectroscopy is employed to
investigate the excitonic states present within the heterostructure [19, 20,
21]. The PL spectrum of the pristine, suspended (9,4) CNT displays a singular
peak at 1.143 eV, corresponding to the E11 transition (Fig. 1d) [11]. After
the transfer of the monolayer WSe2 flake, the CNT E11 peak is redshifted to
1.102 eV (Fig. 1e) as a result of the dielectric screening effect [14],
indicating intimate contact between the two materials. Notably, two excitonic
peaks arise with energies lower than E11. These peaks cannot be attributed to
the suspended monolayer (1L) WSe2 emission as only the A exciton peak at 1.658
eV is expected (see Supplementary Fig. 1), suggesting the existence of new
excitonic states in the CNT/WSe2 heterostructure. The newly emerged peaks are
initially unstable and exhibit temporal blinking (inset in Fig. 1e). Following
a spectral development involving fluctuations of the peaks, the unstable
excitonic states vanish and stable states remain for which we perform the
subsequent measurements (see Supplementary Fig. 2).

Figure 1f presents the stable PL spectrum, featuring two prominent peaks at
0.924 eV and 0.821 eV, denoted as IX1 and IX2, respectively. With an energy
difference of approximately 0.278 eV from E11, IX2 is at a lower energy than
any reported dark or defect states in (9,4) CNTs [22, 23, 24]. We hypothesize
these states to be interface excitons, which could possess substantially lower
excitonic energies as determined by the heterostructure band alignment.

The spatial and spectral correlation between E11 excitons and IXs is
investigated by conducting PL imaging measurement and photoluminescence
excitation (PLE) spectroscopy. Figure 1g is an integrated PL image from E11,
exhibiting strong signal above the trench due to the quenching from the
silicon dioxide (SiO2)/silicon (Si) substrate. Both IX1 and IX2 peaks are
observed precisely at the position of the E11 peak, as demonstrated by the IX1
and IX2 images displayed in Fig. 1h and Supplementary Fig. 3, respectively. In
the PLE map (Fig. 1i), E11 shows a strong response to excitation energy
corresponding to the CNT E22 transition. Similar response as E11 is observed
for IX1 and IX2 peaks, implying that the carriers forming the IXs are supplied
from the CNT. In comparison, we do not observe a clear signature of the WSe2 A
exciton peak in the PLE map. Considering IXs only emerge upon transfer of the
WSe2 flake, the spatial and spectral overlap with CNT supports the hypothesis
that they originate from the interface.

### Band alignment effect on IX peaks.

Figure 2:  Band alignment effect on IXs. a, Band diagram of heterostructures
constructed from CNTs with various chiralities. b, Chirality-dependent PL
spectra of 1D-2D heterostructures. We define a heterostructure nomenclature
where (9,4) CNT/2L WSe2 is represented by (9,4)/2L, with other samples
following the same convention. Excitation energy is adjusted to E22 for each
heterostructure. Excitation power values are 4, 6, 5, and 5 $\mu$W for
(9,4)/2L, (12,1)/2L, (8,7)/2L, and (14,3)/3L heterostructures, respectively,
and 10 $\mu$W for other samples. Arrows indicate IX peaks. Samples exhibiting
IX peaks may also host other unstable IX peaks on a longer timescale, which is
not indicated here. The peaks from K-momentum states are carefully checked and
excluded in the analysis of IXs. c, Band diagram of heterostructures
constructed from WSe2 with different layer numbers. d, PL spectra from
(10,5)/1L, (10,5)/3L, and (10,5)/4L samples. The excitation energy is adjusted
to 1.653 eV and the excitation power is 5 $\mu$W for the (10,5)/4L sample and
10 $\mu$W for others. Red arrows indicate IX peaks while the gray arrow
denotes trion peaks T. e, The plot of $N$ as a function of E11 energy for all
samples. It should be noted that $N$ encompasses all IX peaks observed during
measurements, and some of them are not illustrated in (b).

Since interface excitons form between the two materials, manipulating the
heterostructure band alignment should affect the IXs [6, 25]. It is possible
to vary the CNT bandgap (Fig. 2a) by studying different CNT chiralities (Fig.
2b), whereas WSe2 bandgap can be altered (Fig. 2c) through the layer number
(Fig. 2d).

We first investigate the chirality dependence, which significantly modulates
the CNT bandgap. The band alignment is systematically tuned by utilizing
CNT/WSe2 heterostructures with different CNT chiralities as illustrated in
Fig. 2b. WSe2 layer numbers $\leq$ 4 are used for heterostructures since IX
peaks can be observed as shown in the case for (9,4) CNTs with bilayer (2L)
and quadlayer (4L) WSe2. Multiple IX peaks appear in (9,4), (12,1), (8,6),
(8,7), and (14,0) CNTs, which have large E11 energies and therefore large
bandgaps. This is consistent with the expectation that a large bandgap is
favorable for type-II band alignment as depicted in Fig. 2a. The observed IX
peaks span a broad energy range within the telecommunication wavelengths, and
the highest energy peak in each chirality is located close to E11, with a
difference of less than 0.08 eV. Remarkably, a further decrease in the bandgap
leads to the disappearance of IXs. The presence of the IX peaks is determined
by chirality, consistent with the transition in band alignment from type-II to
type-I. We therefore identify IXs as interface excitons.

The IX peaks in the PL spectra from various heterostructures are summarized in
Fig. 2e by plotting the number of the peaks ($N$) observed during time-trace
measurements as a function of E11. Two distinct regions can be seen below and
above 0.95 eV, corresponding to type-I and type-II alignment, respectively. IX
peaks are absent for type-I band alignment, whereas numerous peaks appear for
type-II alignment. Such a band alignment transition has also been observed for
exciton transfer process in the same system [15].

The dependence of IXs on the number of WSe2 layers is more subtle, since the
number of layers does not significantly modulate the bandgap in comparison to
CNT chirality (Fig. 2c). For example, (9,4) CNT/WSe2 heterostructures are
consistent with complete type-II band alignment irrespective of the WSe2 layer
number (Fig. 1f and Fig. 2b). We therefore study (10,5) CNT/WSe2
heterostructures located at the band alignment transition [15], as they should
be sensitive to the small changes in WSe2 bandgap. The PL spectrum of the 1L
WSe2 heterostructure is presented in Fig. 2d, which does not show any
observable IX peaks. In contrast, the trilayer (3L) WSe2 heterostructure
reveals a discernible IX peak in between the E11 exciton and the trion (T)
peaks. Two IX peaks appear for the 4L WSe2 heterostructure, with increased PL
intensity for the higher energy IX peak and an additional lower energy IX peak
besides the trion peak. This layer-number dependent behavior of the IX peaks
can be explained by the band alignment transition as shown in Fig. 2c.

### Optical properties of the interface excitons.

Figure 3:  Optical properties of low-energy interface excitons. a, Emission
polarization dependence of PL emission from E11 (blue circles), IX1 (green
dots), and IX2 (red circles). PL emission is plotted as a function of angle
with respect to the trench. The lines are fits to a cosine squared function.
The degree of polarization is estimated by
$(I_{\text{max}}-I_{\text{min}})/(I_{\text{max}}+I_{\text{min}})$, where
$I_{\text{max}}$ ($I_{\text{min}}$) is the maximum (minimum) PL emission
intensity. The excitation energy is adjusted to E22 of 1.653 eV with a power
of 5 $\mu$W. b, PL decay curves taken from the CNT/WSe2 heterostructures for
E11 (blue dots) and IX2 (red dots). Shortpass (1.033 eV) and longpass filters
(0.919 eV) are used for the measurements of E11 and IX2, respectively. The
pulsed laser is used here with the power adjusted to 2 nW, and the excitation
energy is 1.653 eV. The grey curves are the IRF, and the black curves are the
exponential fitting curves convoluted with the IRF. c, PL spectra at powers of
0.04, 0.4, and 4 $\mu$W from bottom to top. The excitation energy is 1.680 eV.
d, Integrated PL intensity as a function of the laser power for E11 (blue),
IX1 (green), and IX2 (red), respectively.

The interface excitons display several distinct features different from the
E11 excitons. In Fig. 3a, we first examine the emission polarization
dependence of IX1 and IX2 for the (9,4) CNT/1L WSe2 sample used in Fig. 1. PL
from IXs exhibits near ideal linear polarization of $>$95%, consistent with
confinement in the 1D channel [11, 26]. We also note that the polarization
angle of IX1 and IX2 deviates from that of E11 polarization by 13.7∘,
potentially suggesting some distortion of the optical dipole moment in the IXs
(see Supplementary Fig. 4).

Time-resolved PL is then performed for E11 and IX2 excitons, as the lifetime
of IXs is expected to be long due to the spatially indirect nature [3, 7]. PL
decay curves corresponding to E11 and IX2 are measured as shown in Fig. 3b,
and the lifetime is extracted by reconvolution fitting using the instrument
response function (IRF). Two decay components are obtained for the E11 PL
decay curve, as is the case for suspended CNTs: A main fast component with a
lifetime of 59 ps associated with the bright states, and a small slow
component with a lifetime of 646 ps associated with the dark states [12]. In
contrast, only one decay component is observed for IX2 with a long lifetime of
673 ps, consistent with the reduced optical dipole moment.

The interface excitons exhibit considerably bright emission at low excitation
powers as shown in Fig. 3c. At a low power of 0.04 $\mu$W, both IX1 and IX2
display bright emission with the IX1 PL even exceeding the E11 PL. Such high
intensity emission from interface excitons is unexpected at room temperature.
Generally, indirect excitons exhibit weak PL emission due to diminished dipole
coupling, which is the case for 2D-2D heterostructures where interlayer
excitons can hardly be observed at room temperature [3, 4, 6]. The evident PL
from interface excitons in our system can be ascribed to two primary factors.
Firstly, we employ a fully suspended structure, which reduces the substrate-
induced screening effect and helps sustain the dipole strength. Secondly, the
wavefunction of $\pi$-orbitals in CNTs, which extend significantly out of the
tube, could reduce the spatially indirect nature of the interface excitons.

Emission intensity of the interface excitons exhibit intriguing power
dependence as shown in Fig. 3d. Both IX1 and IX2 PL nearly saturate with a low
threshold power of approximately 0.6 $\mu$W, while the E11 PL increases
substantially. The saturation observed is much more pronounced compared to
interlayer excitons in 2D-2D heterostructures [3]. It is suspected that the
IXs are further confined to a lower dimension, that is, 0D. Interface excitons
in CNTs may be more readily localized than interlayer excitons in 2D-2D
heterostructures because of the lower dimensionality. In general, localized
states show much stronger saturation behavior than free states because of the
state-filling effects [27, 28]. In other samples where interface excitons also
emerge, we find that the saturation behavior depends on their energies (see
Supplementary Fig. 5). The IX peaks with energies substantially lower than the
E11 energy display stronger saturation behaviors. This could be explained by
the deeper trap potential in the confinement of the interface excitons. The
localization is also supported by the observed blinking noise from the
unstable IXs that host only “on” and “off” states (see Supplementary Fig.
2,6). The pure two-level noise is exclusively observed in quasi-0D systems,
such as quantum dots and single molecules [29, 30], strongly suggesting that
the interface exciton state could function as a single-photon emitter.

### Room-temperature single-photon emission from interface excitons.

Figure 4:  Room-temperature quantum emission from low-energy interface
excitons. a-c, Second-order correlation statistics of the IX peak during (a)
short and (b) long time scales, and of (c) the E11 peak for the (9,4) CNT/2L
WSe2 sample. The excitation energy is adjusted to E22 of 1.653 eV with a power
of 2 $\mu$W. Shortpass (1.033 eV) and longpass filters (0.886 eV) are used for
the measurements of E11 and IX, respectively. The grey lines are experimental
results, and the red lines are the fittings. The second-correlation statistics
data in (b) and (c) are binned to reduce the noise.

To elucidate the quantum nature of interface excitons, we conduct photon
correlation measurements on a (9,4) CNT/2L WSe2 sample under continuous-wave
laser excitation. Two primary peaks are observed in this sample, originating
from the stable low-energy IX and E11 (see Supplementary Fig. 2,7). The
second-order correlation $g^{(2)}(\tau)$ from the IX is shown in Fig. 4a,
after background correction. A distinct antibunching dip is observed at zero
delay with $g^{(2)}(0)=0.467$. Falling below the 0.5 threshold, this
unequivocally demonstrates single-photon emission from the interface excitons
at room temperature.

Over a longer timescale, we also observe a bunching peak (Fig. 4b). The
bunching behavior is often observed in other single-photon sources and is
associated with the dynamics between excited states and other dark, charged,
or meta-stable states [31, 32]. We employ the equation
$g^{(2)}(\tau)=[1-\alpha\exp(-|\tau|/\tau_{A})]*[1+\beta\exp(-|\tau|/\tau_{B})]$
to fit the observed $g^{(2)}(\tau)$ statistics, where factors $\alpha$ and
$\beta$ quantify the degree of antibunching and bunching with values of 0.720
and 0.666, respectively [33]. $\tau_{A}$ and $\tau_{B}$ indicate the
timescales of the antibunching dip and the bunching peak, with values of 0.148
ns and 3.752 ns from the fitting, respectively. The value of $1-\alpha$ is
0.280, revealing high single-photon purity by considering the effect of
bunching behavior. It is noteworthy that most of the low-energy interface
excitons display similar peak linewidths and power saturation behavior (see
Supplementary Fig. 5), implying that each of them acts as a single-photon
emitter. This is further supported by the high reproducibility of the
antibunching behavior in other samples (see Supplementary Fig. 8).

For comparison, $g^{(2)}(\tau)$ from E11 excitons does not exhibit any
antibunching or bunching behavior, as shown in Fig. 4c. Under pulsed laser
excitation, the E11 excitons are known to go through an efficient exciton-
exciton annihilation (EEA) process that could result in SPE [34]. The exciton
density is generally lower with continuous-wave excitation, hindering the SPE
through EEA. The confinement effect is crucial for SPE, and the absence of any
antibunching behavior therefore indicates the 1D free feature of E11 excitons.

We now discuss the possible origins of the localized interface exciton states,
which would require a potential depth exceeding a few multiples of the thermal
energy. The confinement can be provided by defect states, for example in
materials such as diamond [35], silicon carbide [36], hexagonal boron nitride
[32], and CNTs [37]. It is unlikely that defects in CNTs play a role, since we
use pristine suspended CNTs containing negligible exciton quenching sites [11,
34]. We note that they have been characterized under low-power conditions
within dry nitrogen gas environment, precluding the formation of defects in
the CNTs [11]. In comparison, WSe2 flakes inherently encompass a range of
defect states, spanning from single vacancies to complex vacancy clusters [38,
39]. Although most defects in WSe2 yield shallow energy traps and exhibit SPE
only under cryogenic conditions, these defects may be responsible for the
trapping potential for the interface excitons.

As another possible explanation, inhomogeneous strain could also contribute to
the localization of interface excitons as in the case of monolayer WSe2 [40].
Strain might also impact the sample through many aspects in a manner similar
to 2D-2D heterostructures, from van der Waals gap fluctuation [41] to lattice
reconstruction [42]. While the exact impact of strain remains unclear, the
spatially indirect nature renders interface excitons more susceptible to the
aforementioned effects than intralayer excitons.

In conclusion, we observe numerous IX peaks at the CNT/WSe2 interface below
the CNT E11 energy, spanning the telecommunication wavelengths. By
systematically varying the chirality of CNTs and the layer number of WSe2, we
are able to assign the peaks to interface excitons as they only appear for
type-II band alignment. The low saturation power and the long lifetime
indicate that low-energy interface excitons are strongly confined, and single-
photon emission is confirmed with $g^{(2)}(0)<0.5$. The observation of
interface excitons as room-temperature quantum emitters at telecommunication
bands opens up new opportunities for applications in quantum technologies and
optoelectronics, underscoring the emerging potential of mixed-dimensional
heterostructures.

## Methods

### Air-suspended carbon nanotubes.

We prepare air-suspended CNTs using trenched SiO2/Si substrates [11]. First,
we pattern alignment markers and trenches with lengths of 900 $\mu$m and
widths ranging from 0.5 to 3.0 $\mu$m onto the Si substrates using electron-
beam lithography, followed by dry etching. We then thermally oxidize the
substrate to form a SiO2 film, with a thickness ranging from 60 to 70 nm.
Another electron-beam lithography process is used to define catalyst regions
along the edges of the trenches. A 1.5 Å thick iron (Fe) film is deposited as
a catalyst for CNT growth using an electron beam evaporator. CNTs are
synthesized by alcohol chemical vapor deposition at 800∘C for 1 minute. The Fe
film thickness is optimized to control the yield for preparing isolated CNTs.
We select isolated, fully suspended chirality-identified CNTs with lengths
ranging from 0.5 to 2.0 $\mu$m to form the heterostructures with WSe2.

### Anthracene crystal growth.

For transferring WSe2 flakes onto CNTs, we grow anthracene crystals through an
in-air sublimation process [17, 18]. Anthracene powder is heated to 80∘C on a
glass slide, while another glass slide is placed 1 mm above the anthracene
source. Thin and large-area single crystals are then grown on the glass
surface. To promote the growth of large-area single crystals, we pattern the
glass slides using ink from commercial markers. The typical growth time for
anthracene crystals is 10 hours.

### Transfer of WSe2 by anthracene crystals.

First, WSe2 (HQ graphene) flakes are prepared on 90-nm-thick SiO2/Si
substrates using mechanical exfoliation, and the layer number is determined by
optical contrast. An anthracene single crystal is picked up with a glass-
supported PDMS sheet to form an anthracene/PDMS stamp. Next, the WSe2 flakes
are picked up by pressing the anthracene/PDMS stamp against a substrate with
the target WSe2 flakes. The stamp is quickly separated ($>$ 10 mm/s) to ensure
that the anthracene crystal remains attached to the PDMS sheet. The stamp is
then applied to the receiving substrate with the desired chirality-identified
CNT, whose position is determined by a prior measurement. Precise position
alignment is accomplished with the aid of markers prepared on the substrate.
By slowly peeling off the PDMS ($<$ 0.2 $\mu$m/s), the anthracene crystal with
the WSe2 flake is released on the receiving substrate. Sublimation of
anthracene in air at 110∘C for 10 minutes removes the anthracene crystal,
leaving behind a clean suspended CNT/WSe2 heterostructure. This all-dry
process eliminates contamination from solvents, and the solid single-crystal
anthracene protects the 2D flakes and the CNT during the transfer, ensuring
that the CNT/WSe2 heterostructure experiences minimal strain [17, 18].

### PL measurements.

A homebuilt confocal microscopy system is employed to perform PL measurements
for interface excitons and E11 excitons at room temperature in dry nitrogen
gas [11, 14]. We utilize a wavelength-tunable continuous-wave Ti:sapphire
laser for excitation, with its power controlled by neutral density filters.
The excitation polarization angle is adjusted to be parallel to the CNT axis
and the emission polarization angle dependence is measured using a half-wave
plate followed by a polarizer placed in front of a spectrometer. The laser
beam is focused on the samples with an objective lens that has a numerical
aperture of 0.65 and a working distance of 4.5 mm. The $1/e^{2}$ spot diameter
and the collection spot size defined by a confocal pinhole are approximately
1.2 and 5.4 $\mu$m, respectively. PL is collected through the same objective
lens and detected using a liquid-nitrogen-cooled 1024-pixel InGaAs diode array
connected to the spectrometer. A 150-lines/mm grating is used to obtain a
dispersion of 0.52 nm/pixel at a wavelength of 1340 nm. For photoluminescence
measurements of WSe2 A excitons, a 532-nm laser and a charge-coupled device
camera are employed.

### Time-resolved and photon correlation measurements.

Approximately 100 femtosecond pulses at a repetition rate of 76 MHz from a
Ti:sapphire laser is utilized for time-resolved measurements. The excitation
laser beam is focused onto the sample using an objective lens with a numerical
aperture of 0.85 and a working distance of 1.48 mm. The PL from the center of
the nanotube within the heterostructure is coupled to a superconducting
single-photon detector with an optical fiber, and a time-correlated single-
photon counting module is used to collect the data. IRFs dependent on the
detection wavelength are acquired by dispersing supercontinuum white light
pulses with a spectrometer. Photon correlation measurements are carried out
using a Hanbury-Brown-Twiss setup with a 50:50 fiber coupler under excitation
with a continuous-wave laser. The experiments are conducted at room
temperature.

###### Acknowledgements.

Parts of this study are supported by JSPS (KAKENHI JP22K14624, JP22K14625,
JP21K14484, JP22F22350, JP22K14623, JP22H01893, JP21H05233, JP23H00262,
JP20H02558) and MEXT (ARIM JPMXP1222UT1135). Y.R.C. is supported by JSPS
(International Research Fellow). N.F. and C.F.F. are supported by RIKEN
Special Postdoctoral Researcher Program. We thank the Advanced Manufacturing
Support Team at RIKEN for technical assistance.

## Author Contributions

N.F. carried out sample preparation and performed measurements on the samples.
Y.R.C. assisted in sample preparation. Y.R.C., C.F.F., and K.N. assisted in
optical measurements. D.Y., S.F., and D.K. contributed to the time-resolved PL
and photon correlation measurements. M. M., Y.G., and S.O. performed density
functional theory calculations. K.O. aided in the development of the
anthracene-assisted dry transfer method. Y.K.K. supervised the project. N.F.
and Y.K.K. co-wrote the manuscript, with all authors providing input and
comments on the manuscript.

## References

  * Cao  _et al._ [2018a] Y. Cao, V. Fatemi, A. Demir, S. Fang, S. L. Tomarken, J. Y. Luo, J. D. Sanchez-Yamagishi, K. Watanabe, T. Taniguchi, E. Kaxiras, R. C. Ashoori, and P. Jarillo-Herrero, Correlated insulator behaviour at half-filling in magic-angle graphene superlattices, Nature 556, 80 (2018a).
  * Cao  _et al._ [2018b] Y. Cao, V. Fatemi, S. Fang, K. Watanabe, T. Taniguchi, E. Kaxiras, and P. Jarillo-Herrero, Unconventional superconductivity in magic-angle graphene superlattices, Nature 556, 43 (2018b).
  * Rivera  _et al._ [2015] P. Rivera, J. R. Schaibley, A. M. Jones, J. S. Ross, S. Wu, G. Aivazian, P. Klement, K. Seyler, G. Clark, N. J. Ghimire, J. Yan, D. G. Mandrus, W. Yao, and X. Xu, Observation of long-lived interlayer excitons in monolayer MoSe2-WSe2 heterostructures, Nat. Commun. 6, 6242 (2015).
  * Xiong  _et al._ [2023] R. Xiong, J. H. Nie, S. L. Brantly, P. Hays, R. Sailus, K. Watanabe, T. Taniguchi, S. Tongay, and C. Jin, Correlated insulator of excitons in MoSe2-WSe2 moiré superlattices, Science 380, 860 (2023).
  * Sun  _et al._ [2022] X. Sun, Y. Zhu, H. Qin, B. Liu, Y. Tang, T. Lü, S. Rahman, T. Yildirim, and Y. Lu, Enhanced interactions of interlayer excitons in free-standing heterobilayers, Nature 610, 478 (2022).
  * Ubrig  _et al._ [2020] N. Ubrig, E. Ponomarev, J. Zultak, D. Domaretskiy, V. Zólyomi, D. Terry, J. Howarth, I. Gutiérrez-Lezama, A. Zhukov, Z. R. Kudrynskyi, Z. D. Kovalyuk, A. Patané, T. Taniguchi, K. Watanabe, R. V. Gorbachev, V. I. Fal’ko, and A. F. Morpurgo, Design of van der Waals interfaces for broad-spectrum optoelectronics, Nat. Mater. 19, 299 (2020).
  * Unuchek  _et al._ [2018] D. Unuchek, A. Ciarrocchi, A. Avsar, K. Watanabe, T. Taniguchi, and A. Kis, Room-temperature electrical control of exciton flux in a van der Waals heterostructure, Nature 560, 340 (2018).
  * Rivera  _et al._ [2016] P. Rivera, K. L. Seyler, H. Yu, J. R. Schaibley, J. Yan, D. G. Mandrus, W. Yao, and X. Xu, Valley-polarized exciton dynamics in a 2D semiconductor heterostructure, Science 351, 688 (2016).
  * Jin  _et al._ [2019] C. Jin, E. C. Regan, A. Yan, M. I. B. Utama, D. Wang, S. Zhao, Y. Qin, S. Yang, Z. Zheng, S. Shi, K. Watanabe, T. Taniguchi, S. Tongay, A. Zettl, and F. Wang, Observation of moiré excitons in WSe2/WS2 heterostructure superlattices, Nature 567, 76 (2019).
  * Seyler  _et al._ [2019] K. L. Seyler, P. Rivera, H. Yu, N. P. Wilson, E. L. Ray, D. G. Mandrus, J. Yan, W. Yao, and X. Xu, Signatures of moiré-trapped valley excitons in MoSe2/WSe2 heterobilayers, Nature 567, 66 (2019).
  * Ishii  _et al._ [2015] A. Ishii, M. Yoshida, and Y. K. Kato, Exciton diffusion, end quenching, and exciton-exciton annihilation in individual air-suspended carbon nanotubes, Phys. Rev. B 91, 125427 (2015).
  * Ishii  _et al._ [2019] A. Ishii, M. Yoshida, and Y. K. Kato, High efficiency dark-to-bright exciton conversion in carbon nanotubes, Phys. Rev. X 9, 041048 (2019).
  * Jariwala  _et al._ [2016] D. Jariwala, T. J. Marks, and M. C. Hersam, Mixed-dimensional van der Waals heterostructures, Nat. Mater. 16, 170 (2016).
  * Fang  _et al._ [2020] N. Fang, K. Otsuka, A. Ishii, T. Taniguchi, K. Watanabe, K. Nagashio, and Y. K. Kato, Hexagonal boron nitride as an ideal substrate for carbon nanotube photonics, ACS Photonics 7, 1773 (2020).
  * Fang  _et al._ [2023] N. Fang, D. Yamashita, S. Fujii, M. Maruyama, Y. Gao, Y.-R. Chang, C. F. Fong, K. Otsuka, K. Nagashio, S. Okada, and Y. K. Kato, Resonant exciton transfer in mixed-dimensional heterostructures for overcoming dimensional restrictions in optical processes, arXiv:2307.07124 (2023).
  * Lefebvre  _et al._ [2003] J. Lefebvre, Y. Homma, and P. Finnie, Bright band gap photoluminescence from unprocessed single-walled carbon nanotubes, Phys. Rev. Lett. 90, 217401 (2003).
  * Otsuka  _et al._ [2021] K. Otsuka, N. Fang, D. Yamashita, T. Taniguchi, K. Watanabe, and Y. K. Kato, Deterministic transfer of optical-quality carbon nanotubes for atomically defined technology, Nat. Commun. 12, 3138 (2021).
  * Fang  _et al._ [2022] N. Fang, D. Yamashita, S. Fujii, K. Otsuka, T. Taniguchi, K. Watanabe, K. Nagashio, and Y. K. Kato, Quantization of mode shifts in nanocavities integrated with atomically thin sheets, Adv. Opt. Mater. 10, 2200538 (2022).
  * Jiang  _et al._ [2015] M. Jiang, Y. Kumamoto, A. Ishii, M. Yoshida, T. Shimada, and Y. K. Kato, Gate-controlled generation of optical pulse trains using individual carbon nanotubes, Nat. Commun. 6, 6335 (2015).
  * Uda  _et al._ [2016] T. Uda, M. Yoshida, A. Ishii, and Y. K. Kato, Electric-field induced activation of dark excitonic states in carbon nanotubes, Nano Lett. 16, 2278 (2016).
  * Otsuka  _et al._ [2019] K. Otsuka, A. Ishii, and Y. K. Kato, Super-resolution fluorescence imaging of carbon nanotubes using a nonlinear excitonic process, Opt. Express 27, 17463 (2019).
  * Matsunaga  _et al._ [2010] R. Matsunaga, K. Matsuda, and Y. Kanemitsu, Origin of low-energy photoluminescence peaks in single carbon nanotubes: ${K}$-momentum dark excitons and triplet dark excitons, Phys. Rev. B 81, 033401 (2010).
  * Kozawa  _et al._ [2022] D. Kozawa, X. Wu, A. Ishii, J. Fortner, K. Otsuka, R. Xiang, T. Inoue, S. Maruyama, Y. Wang, and Y. K. Kato, Formation of organic color centers in air-suspended carbon nanotubes using vapor-phase reaction, Nat. Commun. 13, 2814 (2022).
  * Yu  _et al._ [2022] B. Yu, S. Naka, H. Aoki, K. Kato, D. Yamashita, S. Fujii, Y. K. Kato, T. Fujigaya, and T. Shiraki, ortho-substituted aryldiazonium design for the defect configuration-controlled photoluminescent functionalization of chiral single-walled carbon nanotubes, ACS Nano 16, 21452 (2022).
  * Tan  _et al._ [2021] Q. Tan, A. Rasmita, S. Li, S. Liu, Z. Huang, Q. Xiong, S. A. Yang, K. Novoselov, and W.-b. Gao, Layer-engineered interlayer excitons, Sci. Adv. 7, 1 (2021).
  * Bai  _et al._ [2020] Y. Bai, L. Zhou, J. Wang, W. Wu, L. J. McGilly, D. Halbertal, C. F. B. Lo, F. Liu, J. Ardelean, P. Rivera, N. R. Finney, X.-C. Yang, D. N. Basov, W. Yao, X. Xu, J. Hone, A. N. Pasupathy, and X. Zhu, Excitons in strain-induced one-dimensional moiré potentials at transition metal dichalcogenide heterojunctions, Nat. Mater. 19, 1068 (2020).
  * Settele  _et al._ [2021] S. Settele, F. J. Berger, S. Lindenthal, S. Zhao, A. A. El Yumin, N. F. Zorn, A. Asyuda, M. Zharnikov, A. Högele, and J. Zaumseil, Synthetic control over the binding configuration of luminescent sp3-defects in single-walled carbon nanotubes, Nat. Commun. 12, 2119 (2021).
  * Shinokita  _et al._ [2021] K. Shinokita, Y. Miyauchi, K. Watanabe, T. Taniguchi, and K. Matsuda, Resonant coupling of a moiré exciton to a phonon in a WSe2/MoSe2 heterobilayer, Nano Lett. 21, 5938 (2021).
  * Nirmal  _et al._ [1996] M. Nirmal, B. O. Dabbousi, M. G. Bawendi, J. Macklin, J. Trautman, T. Harris, and L. E. Brus, Fluorescence intermittency in single cadmium selenide nanocrystals, Nature 383, 802 (1996).
  * Moerner and Orrit [1999] W. E. Moerner and M. Orrit, Illuminating single molecules in condensed matter, Science 283, 1670 (1999).
  * Sallen  _et al._ [2009] G. Sallen, A. Tribu, T. Aichele, R. André, L. Besombes, C. Bougerol, S. Tatarenko, K. Kheng, and J. P. Poizat, Exciton dynamics of a single quantum dot embedded in a nanowire, Phys. Rev. B 80, 085310 (2009).
  * Tran  _et al._ [2016] T. T. Tran, K. Bray, M. J. Ford, M. Toth, and I. Aharonovich, Quantum emission from hexagonal boron nitride monolayers, Nat. Nanotechnol. 11, 37 (2016).
  * Gao  _et al._ [2017] K. Gao, I. Solovev, M. Holmes, M. Arita, and Y. Arakawa, Nanosecond-scale spectral diffusion in the single photon emission of a gan quantum dot, AIP Adv. 7, 125216 (2017).
  * Ishii  _et al._ [2017] A. Ishii, M. Yoshida, and Y. K. Kato, Room-temperature single-photon emission from micrometer-long air-suspended carbon nanotubes, Phys. Rev. Applied 8, 054039 (2017).
  * Babinec  _et al._ [2010] T. M. Babinec, B. J. Hausmann, M. Khan, Y. Zhang, J. R. Maze, P. R. Hemmer, and M. Lončar, A diamond nanowire single-photon source, Nat. Nanotechnol. 5, 195 (2010).
  * Castelletto  _et al._ [2014] S. Castelletto, B. C. Johnson, V. Ivády, N. Stavrias, T. Umeda, A. Gali, and T. Ohshima, A silicon carbide room-temperature single-photon source, Nat. Mater. 13, 151 (2014).
  * Ma  _et al._ [2015] X. Ma, N. F. Hartmann, J. K. S. Baldwin, S. K. Doorn, and H. Htoon, Room-temperature single-photon generation from solitary dopants of carbon nanotubes, Nat. Nanotechnol. 10, 671 (2015).
  * Linhart  _et al._ [2019] L. Linhart, M. Paur, V. Smejkal, J. Burgdörfer, T. Mueller, and F. Libisch, Localized intervalley defect excitons as single-photon emitters in WSe2, Phys. Rev. Lett. 123, 146401 (2019).
  * Zhang  _et al._ [2017] S. Zhang, C.-G. Wang, M.-Y. Li, D. Huang, L.-J. Li, W. Ji, and S. Wu, Defect structure of localized excitons in a WSe2 monolayer, Phys. Rev. Lett. 119, 046101 (2017).
  * Parto  _et al._ [2021] K. Parto, S. I. Azzam, K. Banerjee, and G. Moody, Defect and strain engineering of monolayer WSe2 enables site-controlled single-photon emission up to 150 K, Nat. Commun. 12, 3585 (2021).
  * Shin  _et al._ [2016] B. G. Shin, G. H. Han, S. J. Yun, H. M. Oh, J. J. Bae, Y. J. Song, C.-Y. Park, and Y. H. Lee, Indirect bandgap puddles in monolayer MoS2 by substrate-induced local strain, Adv. Mater. 28, 9378 (2016).
  * Zhao  _et al._ [2023] S. Zhao, Z. Li, X. Huang, A. Rupp, J. Göser, I. A. Vovk, S. Y. Kruchinin, K. Watanabe, T. Taniguchi, I. Bilgin, A. S. Baimuratov, and A. Högele, Excitons in mesoscopically reconstructed moiré heterostructures, Nat. Nanotechnol. 18, 572 (2023).


# Shifted Diffusion for Text-to-image Generation

Yufan Zhou 1, Bingchen Liu 2, Yizhe Zhu 2, Xiao Yang 2, Changyou Chen 1,
Jinhui Xu 1
1 State University of New York at Buffalo 2 ByteDance
{yufanzho, changyou<EMAIL_ADDRESS>{bingchenliu, yizhe.zhu,
<EMAIL_ADDRESS>Performed this work during internship at ByteDance

###### Abstract

We present Corgi, a novel method for text-to-image generation. Corgi is based
on our proposed shifted diffusion model, which achieves better image embedding
generation from input text. Unlike the baseline diffusion model used in DALL-E
2, our method seamlessly encodes prior knowledge of the pre-trained CLIP model
in its diffusion process by designing a new initialization distribution and a
new transition step of the diffusion. Compared to the strong DALL-E 2
baseline, our method performs better in generating image embedding from the
text in terms of both efficiency and effectiveness, resulting in better text-
to-image generation. Extensive large-scale experiments are conducted and
evaluated in terms of both quantitative measures and human evaluation,
indicating a stronger generation ability of our method compared to existing
ones. Furthermore, our model enables semi-supervised and language-free
training for text-to-image generation, where only part or none of the images
in the training dataset have an associated caption. Trained with only $1.7\%$
of the images being captioned, our semi-supervised model obtains FID results
comparable to DALL-E 2 on zero-shot text-to-image generation evaluated on MS-
COCO. Corgi also achieves new state-of-the-art results across different
datasets on downstream language-free text-to-image generation tasks,
outperforming the previous method, Lafite, by a large margin.

##  1 Introduction

“AI-generated content” has attracted increasingly more public awareness thanks
to the significant progress in recent research of high-fidelity text-aligned
image synthetic tasks. [20, 21, 19, 22, 30]. Particularly, models trained on
web-scale datasets have demonstrated their impressive ability to generate out-
of-distribution images from arbitrary text inputs that describe unseen
combinations of visual concepts.

Starting from DALL-E [20], researchers have proposed a variety of approaches
to further advance the state-of-the-art (SOTA) of text-to-image generation in
terms of both generation quality and efficiency. Latent Diffusion Model [21]
trains a diffusion model in the latent space of auto-encoder instead of pixel
space, leading to better generation efficiency. GLIDE [14] adopts a
hierarchical architecture, which consists of diffusion models at different
resolutions. Such a model design strategy has shown to be effective and has
been adopted by many follow-up works. DALL-E 2 [19] further introduces an
extra image embedding input. Such an image embedding not only improves the
model performance in text-to-image generation but also enables applications,
including image-to-image generation and generation under multi-modal
conditions. Imagen [22] makes use of a rich pre-trained text encoder [18],
demonstrating that a frozen text encoder pre-trained on the large-scale text-
only dataset can help text-to-image generation models in understanding the
semantics of text descriptions. Parti [30] shows a further successful scale-up
of the generative model, leading to impressive improvement in text-to-image
consistency with transformer structure.

The aforementioned approaches focus on improving text-to-image generation by
either scaling up trainable/frozen modules or designing better model
architectures. In this work, we explore an orthogonal direction, where we
propose novel techniques to improve the diffusion process itself and make it
more suitable and effective for text-to-image generation.

(a) A corgi in front of a blackboard with “Bridge!” on it.

(b) A corgi with a neon light reads “Bridge!”.

(c) A realistic photo about corgi with word “Bridge!”.

(d) A magazine about Corgi, which has “Bridge” on it.

(e) A corgi is smiling, in front of a whiteboard with “Bridge the Gap” on it.

(f) A corgi is giving a speech about “Bridge the Gap” in cartoon style.

Figure 1: We propose Corgi, a novel diffusion model designed for flexible
text-to-image generation which can “bridge the gap”.

Specifically, we propose Corgi (Clip-based shifted diffusiOn model bRidGIng
the gap), a novel diffusion model designed for a flexible text-to-image
generation. Our model can perform text-to-image generation under all the
supervised, semi-supervised, and language-free settings. By “bridging the
gap,” we emphasize two key novelties in our method: (1) our model bridges the
image-text modality gap [11] so as to train a better generative model.
Modality gap is a critical concept discovered in pre-trained models such as
CLIP [16], which captures the phenomenon that multi-modality representations
do not align in the joint embedding space. By bridging the gap with Corgi, we
can better utilize CLIP in text-to-image generation, which is shown in the
experiment to achieve better generation; (2) our model bridges the gap of data
availability for different researchers/communities. In our design, Corgi can
naturally enable semi-supervised and language-free text-to-image generation,
where only a small portion or even none of the images in the training dataset
are captioned. This is important because the cost of constructing a high-
quality large image-text-paired dataset could be prohibitive, especially in
the case where hundreds of millions of images need to be captioned. We show
that by merely using an image-only dataset and the public CC15M dataset [24,
2], Corgi achieves promising results that are comparable to the SOTA models on
open-domain text-to-image generation tasks. To summarize, our contributions
are:

  * •

We propose Corgi, a novel diffusion model that seamlessly incorporates prior
knowledge from the pre-trained model (e.g.,CLIP) into its diffusion process;

  * •

Our method is general and can be applied in different settings of text-to-
image generation, e.g., it naturally enables semi-supervised and language-free
text-to-image generation;

  * •

Extensive and large-scale experiments are conducted. Both quantitative and
qualitative results illustrate the effectiveness of the proposed method. We
especially demonstrate that, with only $1.7\%$ captioned images in the
training dataset, it is possible to achieve results comparable to the SOTA
method. In addition, our model achieves SOTA results under language-free
setting across different downstream datasets.

(a) Train a decoder which generates images from image embeddings.

(b) Train a shifted diffusion model which generates embeddings from text.

(c) Generate image with text input at inference.

Figure 2: Illustration of our framework. The trainable modules are colored in
green, and the frozen modules are colored in blue.

##  2 Methodology

We start by illustrating the proposed general framework for text-to-image
generation. Our framework is shown in Figure 2, which consists of three key
components: (1) a pre-trained image encoder that maps images to their
embeddings; (2) a decoder that generates images from the corresponding
embeddings; and (3) a prior model that generates image embeddings from the
corresponding text captions. In our implementation, we use the pre-trained
CLIP image encoder because its output space is a multi-modal embedding space
that has been demonstrated to benefit the text-to-image generation task [33].
The decoder can be either a diffusion model or a generative adversarial
network (GAN). Note that if one chooses the decoder as a hierarchical
diffusion model and makes it conditioned on both image embedding and text, our
final structure will be similar to DALL-E 2 [19].

We adopt this framework because of its flexibility: it can perform different
types of generation tasks such as text-to-image generation, image-to-image
generation, and generation conditioned on both image and text. In addition,
our framework naturally enables semi-supervised training, i.e., the training
dataset can be a mixture of image-text pairs and pure images that are not
captioned. In this setting, the image-text pairs will be used to train the
prior model, and all the pure images will be used to train the decoder. Such a
semi-supervised training setting is important in practice, especially when
training a text-to-image generation model on new domains with a limited
budget. As discussed in [33], constructing high-quality image-text pairs could
be very expensive and requires a heavy human workload. Our framework provides
the community with the flexibility of choosing the number of images to be
captioned based on their budget. As it will be shown in the experiments, semi-
supervised training can obtain impressive results, which are even comparable
to those of supervised training.

Figure 3: Illustration of diffusions inside the multi-modal joint space of
CLIP. We use the cone in red dotted lines to represent the effective output
space of CLIP image encoder. The target embedding is represented by the black
star. The green dashed arrow represents the sampling process of the baseline
diffusion model, whose starting point is random noise. The blue solid arrow
represents the sampling process of our shifted diffusion, whose starting point
is an image embedding inside the red cone.

In this paper, we focus on improving the prior model, which is less exploited
in previous works. As shown in DALL-E 2 [19], a diffusion-based prior model is
introduced to generate the target CLIP image embedding
$\operatorname{\mathbf{z}}_{0}$ via the following sequential sampling process:
$\operatorname{\mathbf{z}}_{t-1}\sim
p_{{\bm{\theta}}}(\operatorname{\mathbf{z}}_{t-1}|\operatorname{\mathbf{z}}_{t},\operatorname{\mathbf{y}})$
for $t=T,...,1$, where
$\operatorname{\mathbf{z}}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, $t$
denotes timestep, $\operatorname{\mathbf{y}}$ denotes text caption, and
$p_{{\bm{\theta}}}(\cdot|\cdot)$ is the inverse transition distribution
induced from the diffusion model. Although it is shown in [19] that this prior
model can benefit generation in terms of both image-text alignment and image
fidelity, we suspect that this vanilla sampling process may not be appropriate
for generating high-quality CLIP image embeddings (which will be the inputs of
decoder thus greatly influence the generation quality). The reason is that, as
revealed in [11], the effective output space of the CLIP image encoder is
actually a very small region of the whole embedding space, as shown in Figure
3. Consequently,
$\operatorname{\mathbf{z}}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, which
is the starting point of sampling, might be far away from the target
embedding. Intuitively, if $\operatorname{\mathbf{z}}_{T}$ is closer to the
target $\operatorname{\mathbf{z}}_{0}$, we may be able to well approximate
$\operatorname{\mathbf{z}}_{0}$ within fewer sampling steps. Likewise, we may
better approximate the target within the same number of steps, if the
initialization is closer to the target.

Based on this motivation, we propose shifted diffusion, a novel diffusion
model that considers prior knowledge contained in the pre-trained CLIP image
encoder. Specifically, the noise distribution
$p(\operatorname{\mathbf{z}}_{T})$ of shifted diffusion is a parametric
distribution rather than standard Gaussian
$\mathcal{N}(\mathbf{0},\mathbf{I})$. We present the details of our method in
the following.

###  2.1 Shifted Diffusion

Let $q(\operatorname{\mathbf{z}}_{0})$ be the distribution of ground-truth
image embeddings in the joint latent space of the CLIP model. Because of the
fact that valid image embeddings only occupy a small region of the whole
embedding space [11], in order to achieve better generation in a diffusion
process, $\operatorname{\mathbf{z}}_{T}\sim q(\operatorname{\mathbf{z}}_{0})$
is expected to be closer to the target $\operatorname{\mathbf{z}}_{0}$,
compared to the standard sampling of
$\operatorname{\mathbf{z}}_{T}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$.
However, $q(\operatorname{\mathbf{z}}_{0})$ is an intractable distribution,
making it challenging to train a diffusion model based on it.

To tackle the aforementioned problem, we consider the initial distribution
$p(\operatorname{\mathbf{z}}_{T})$ to be a parametric Gaussian distribution as
$\mathcal{N}(\operatorname{\mathbf{z}}_{T};{\bm{\mu}},\mathbf{\Sigma})$, which
approximates $q(\operatorname{\mathbf{z}}_{0})$, and can be obtained by simply
analyzing the training dataset***In practice, we set $\mathbf{\Sigma}$ to be a
diagonal matrix with its element $\mathbf{\Sigma}_{i,i}=\kappa\sigma_{i}$.
$\sigma_{i}$ is the standard deviation of $i^{th}$ element of all image
embeddings from the training dataset, $\kappa>0$ is a constant for scaling..
We design the transition
$q(\operatorname{\mathbf{z}}_{t}|\operatorname{\mathbf{z}}_{t-1})$ to be a
Gaussian as

$\displaystyle
q(\operatorname{\mathbf{z}}_{t}|\operatorname{\mathbf{z}}_{t-1})\coloneqq\mathcal{N}(\operatorname{\mathbf{z}}_{t};\sqrt{1-\beta_{t}}\operatorname{\mathbf{z}}_{t-1}+\operatorname{\mathbf{s}}_{t},\beta_{t}\mathbf{\Sigma}),$
(1)

where $\beta_{t}$ is a constant following [7]. Compared to the vanilla
diffusion with

$q(\operatorname{\mathbf{z}}_{t}|\operatorname{\mathbf{z}}_{t-1})\coloneqq\mathcal{N}(\operatorname{\mathbf{z}}_{t};\sqrt{1-\beta_{t}}\operatorname{\mathbf{z}}_{t-1},\beta_{t}\mathbf{I}),$

our diffusion process introduces a shift term $\operatorname{\mathbf{s}}_{t}$
at every timestep $t$, thus termed shifted diffusion.

One can show that
$q(\operatorname{\mathbf{z}}_{t}|\operatorname{\mathbf{z}}_{0})$ has a closed-
form expression (all proofs are provided in the Appendix):

$\displaystyle
q(\operatorname{\mathbf{z}}_{t}|\operatorname{\mathbf{z}}_{0})=\mathcal{N}(\operatorname{\mathbf{z}}_{t};\sqrt{\bar{\alpha}_{t}}\operatorname{\mathbf{z}}_{0}+\sum_{i=1}^{t}\operatorname{\mathbf{s}}_{i}\sqrt{\bar{\alpha}_{t}/\bar{\alpha}_{i}},(1-\bar{\alpha}_{t})\mathbf{\Sigma}),$

where $\bar{\alpha}_{t}=\prod_{i=1}^{t}(1-\beta_{i})$. Specifically, we choose
$\operatorname{\mathbf{s}}_{t}=(1-\sqrt{1-\beta_{t}}){\bm{\mu}}$, leading to

$\displaystyle
q(\operatorname{\mathbf{z}}_{t}|\operatorname{\mathbf{z}}_{0})=\mathcal{N}(\operatorname{\mathbf{z}}_{t};\sqrt{\bar{\alpha}_{t}}\operatorname{\mathbf{z}}_{0}+(1-\sqrt{\bar{\alpha}_{t}}){\bm{\mu}},(1-\bar{\alpha}_{t})\mathbf{\Sigma}).$
(2)

###### Remark 1

As we can see, by selecting $\\{\beta_{t}\\}_{t=1}^{T}$ such that
$\alpha_{T}\approx 0$,
$q(\operatorname{\mathbf{z}}_{T}|\operatorname{\mathbf{z}}_{0})$ can
approximate
$\mathcal{N}(\operatorname{\mathbf{z}}_{T};{\bm{\mu}},\mathbf{\Sigma})$, the
distribution of the image embeddings. In other words, shifted diffusion
process is a process that transfers a ground-truth image embedding into a
random image embedding; whereas the vanilla diffusion process is a process
that transfers an image embedding into a random Gaussian noise, which is
certainly not what we expect.

From Equation (1) and (2), we can get the closed-form expression of
$q(\operatorname{\mathbf{z}}_{t}|\operatorname{\mathbf{z}}_{t-1}),q(\operatorname{\mathbf{z}}_{t}|\operatorname{\mathbf{z}}_{0}),q(\operatorname{\mathbf{z}}_{t-1}|\operatorname{\mathbf{z}}_{0})$.
By the property of Gaussian distribution [1] and some simple derivations, we
can get the posterior distribution

$\displaystyle
q(\operatorname{\mathbf{z}}_{t-1}|\operatorname{\mathbf{z}}_{t},\operatorname{\mathbf{z}}_{0})=\mathcal{N}(\operatorname{\mathbf{z}}_{t-1};{\bm{\nu}},\mathbf{\Lambda}),$
(3)
$\displaystyle{\bm{\nu}}=\gamma(\operatorname{\mathbf{z}}_{t}-\operatorname{\mathbf{s}}_{t})+\eta\operatorname{\mathbf{z}}_{0}+\tau(1-\sqrt{\bar{\alpha}_{t-1}}){\bm{\mu}},$
$\displaystyle\mathbf{\Lambda}=(1-\bar{\alpha}_{t-1})\beta_{t}\mathbf{\Sigma}/(1-\bar{\alpha}_{t}),$

where

$\displaystyle\gamma=(1-\bar{\alpha}_{t-1})\sqrt{\alpha_{t}}/(1-\bar{\alpha}_{t}),$
$\displaystyle\eta=\beta_{t}\sqrt{\bar{\alpha}_{t-1}}/(1-\bar{\alpha}_{t}),$
$\displaystyle\tau=\beta_{t}/(1-\bar{\alpha}_{t}).$

With Equations (2) and (3), we can easily optimize the loss function of the
proposed shifted diffusion

$\displaystyle\mathbf{L}_{{\bm{\theta}}}=\mathbb{E}_{q}\\{$
$\displaystyle\text{D}_{\text{KL}}(q(\operatorname{\mathbf{z}}_{T}|\operatorname{\mathbf{z}}_{0})\|p(\operatorname{\mathbf{z}}_{T}))-\log
p_{{\bm{\theta}}}(\operatorname{\mathbf{z}}_{0}|\operatorname{\mathbf{z}}_{1})$
(4)
$\displaystyle+\sum_{t>1}\text{D}_{\text{KL}}(q(\operatorname{\mathbf{z}}_{t-1}|\operatorname{\mathbf{z}}_{t},\operatorname{\mathbf{z}}_{0})\|p_{{\bm{\theta}}}(\operatorname{\mathbf{z}}_{t-1}|\operatorname{\mathbf{z}}_{t}))\\}~{},$

where $\text{D}_{\text{KL}}$ denotes KL-divergence. Since both
$q(\operatorname{\mathbf{z}}_{t}|\operatorname{\mathbf{z}}_{0})$ and
$q(\operatorname{\mathbf{z}}_{t-1}|\operatorname{\mathbf{z}}_{t},\operatorname{\mathbf{z}}_{0})$
are Gaussian distributions by our design, the KL-divergence terms have closed-
form solutions, enabling easy stochastic optimization.

Algorithm 1 Shifted Diffusion

1: // Training

2: Require:
$\\{p_{i}(\operatorname{\mathbf{z}}_{T})=\mathcal{N}(\operatorname{\mathbf{z}}_{T};{\bm{\mu}}_{i},\mathbf{\Sigma}_{i})\\}_{i=1}^{k}$,
a diffusion model parameterized by ${\bm{\theta}}$, CLIP image encoder
$f_{\text{img}}$ and text encoder $f_{\text{txt}}$.

3: while not converge do

4: Sample image-text data pair
$(\operatorname{\mathbf{x}}_{0},\operatorname{\mathbf{y}})$,
$\operatorname{\mathbf{z}}_{0}=f_{\text{img}}(\operatorname{\mathbf{x}}_{0})$

5: Select corresponding $p_{c_{\operatorname{\mathbf{y}}}}$ by (5)

6: Update ${\bm{\theta}}$ by gradient descent w.r.t $\text{L}_{{\bm{\theta}}}$

7: if ${\bm{\mu}}_{i},\mathbf{\Sigma}_{i}$ are learn-able then

8: Update ${\bm{\mu}}_{i},\mathbf{\Sigma}_{i}$ by gradient descent w.r.t
$\text{L}_{p}$

9: end if

10: end while

11: // Inference

12: Given a text caption $\operatorname{\mathbf{y}}$, select its corresponding
distribution $p_{c_{\operatorname{\mathbf{y}}}}$, sample corresponding
$\operatorname{\mathbf{z}}_{T}$

13: for t=T,…,1 do

14: Sample $\operatorname{\mathbf{z}}_{t-1}\sim
p_{{\bm{\theta}}}(\operatorname{\mathbf{z}}_{t-1}|\operatorname{\mathbf{z}}_{t})$

15: end for

16: Return $\operatorname{\mathbf{z}}_{0}$

Although setting $p(\operatorname{\mathbf{z}}_{T})$ to be a Gaussian
distribution leads to a simplified loss that can be conveniently optimized, it
also introduces some drawbacks as the ground-truth distribution of image
embedding is not a single-mode Gaussian. To tackle this problem, we propose to
use a collection of Gaussian distributions, denoted as
$\\{p_{i}(\operatorname{\mathbf{z}}_{T})\\}_{i=1}^{k}$ with
$p_{i}(\operatorname{\mathbf{z}}_{T})\coloneqq\mathcal{N}(\operatorname{\mathbf{z}}_{T};{\bm{\mu}}_{i},\mathbf{\Sigma}_{i})$.
Let $\operatorname{\mathbf{z}}_{0}$ and $\operatorname{\mathbf{y}}$ be the
ground-truth image embedding and its associated text caption, respectively.
For each pair of $(\operatorname{\mathbf{z}}_{0},\operatorname{\mathbf{y}})$,
we select its corresponding Gaussian $p_{c_{\operatorname{\mathbf{y}}}}$ by
the top-1 cosine similarity as

$\displaystyle
c_{\operatorname{\mathbf{y}}}=\operatorname*{\mathop{\mathrm{argmax}}}_{1\leq
i\leq k}\text{Sim}({\bm{\mu}}_{i},f_{\text{txt}}(\operatorname{\mathbf{y}})),$
(5)

where ${\mbox{\boldmath$f$}}_{\text{txt}}$ is the pre-trained CLIP text
encoder, and $c_{\operatorname{\mathbf{y}}}$ denotes the index of the selected
Gaussian. One may also use

$\displaystyle
c_{\operatorname{\mathbf{y}}}=\operatorname*{\mathop{\mathrm{argmax}}}_{1\leq
i\leq k}\mathbb{E}_{{\bm{\epsilon}}_{i}\sim
p_{i}}\left[\text{Sim}({\bm{\epsilon}}_{i},f_{\text{txt}}(\operatorname{\mathbf{y}}))\right]~{},$
(6)

which requires more computation because of the expectation. After selecting
$p_{c_{\operatorname{\mathbf{y}}}}(\operatorname{\mathbf{z}}_{T})$, its
parameters ${\bm{\mu}}_{c_{\operatorname{\mathbf{y}}}}$ and
$\mathbf{\Sigma}_{c_{\operatorname{\mathbf{y}}}}$ will be used in Equation (2)
and (3) for optimization. An extra positional embedding representing
$c_{\operatorname{\mathbf{y}}}$ is also injected into the diffusion model,
with a similar implementation as the time embedding for $t$. Compared to a
single-mode Gaussian distributions,
$\\{p_{i}(\operatorname{\mathbf{z}}_{T})\\}_{i=1}^{k}$ is supposed to have
better expressive ability, and $p_{c_{\operatorname{\mathbf{y}}}}$ is expected
to better initialize $\operatorname{\mathbf{z}}_{T}$ to make it closer to the
target $\operatorname{\mathbf{z}}_{0}$.

In our implementation, $\\{p_{i}(\operatorname{\mathbf{z}}_{T})\\}_{i=1}^{k}$
is estimated by performing clustering on the training dataset. Similar to
existing quantization methods [26], we can also learn
$({\bm{\mu}}_{i},\mathbf{\Sigma}_{i})$ by optimization. Specifically, we
propose to update ${\bm{\mu}}_{i},\mathbf{\Sigma}_{i}$ during training with
the following loss function

$\displaystyle\mathbf{L}_{p}=$
$\displaystyle-\mathbb{E}_{\operatorname{\mathbf{z}}}\\{\mathbb{E}_{\operatorname{\mathbf{z}}_{T}\sim
p_{c_{\operatorname{\mathbf{y}}}}(\operatorname{\mathbf{z}}_{T})}\\{\text{Sim}(\operatorname{\mathbf{z}}_{T},\operatorname{\mathbf{z}}_{0})\\}\\}$
(7) $\displaystyle+\xi\dfrac{\sum_{i\neq
j}}{k(k-1)}\mathbb{E}_{\operatorname{\mathbf{z}}_{T}\sim
p_{i}(\operatorname{\mathbf{z}}_{T})}\\{\mathbb{E}_{\operatorname{\mathbf{z}}^{\prime}_{T}\sim
p_{j}(\operatorname{\mathbf{z}}_{T})}\\{\text{Sim}(\operatorname{\mathbf{z}}_{T},\operatorname{\mathbf{z}}_{T}^{\prime})\\}\\},$

where $\xi$ is a hyper-parameter. The first term of $\mathbf{L}_{p}$ forces
the sample $\operatorname{\mathbf{z}}_{T}$ from Gaussian
$p_{c_{\operatorname{\mathbf{y}}}}$ to be close to the corresponding ground-
truth $\operatorname{\mathbf{z}}_{0}$, and the second term ensures that
$p_{i}(\operatorname{\mathbf{z}}_{T})$ does not overlap each other too much.
Note that $\\{p_{i}(\operatorname{\mathbf{z}}_{T})\\}_{i=1}^{k}$ is only
optimized with $\mathbf{L}_{p}$, i.e., we manually stop the gradient back-
propagation from $\text{L}_{{\bm{\theta}}}$ to
$\\{{\bm{\mu}}_{i},\mathbf{\Sigma}_{i}\\}_{i=1}^{k}$. Our algorithm is
summarized in Algorithm 1, with more implementation details provided in the
experiments section.

(a) A photo of a light bulb in outer space traveling the galaxy with a sailing
boat inside the light bulb.

(b) A high-resolution photo of a chicken working out in a gym.

(c) New York Skyline with ’Hello World’ written with fireworks on the sky.

(d) A television made of water that displays an image of a cityscape at night.

(e) A photo of llama wearing sunglasses standing on the deck of a spaceship
with the Earth in the background.

(f) A high contrast portrait photo of a fluffy hamster wearing an orange
beanie and sunglasses holding a sign that says “Let’s PAINT!”.

(g) A photo of an astronaut riding a horse in the forest. There is a river in
front of them with water lilies.

(h) A portrait photo of a kangaroo wearing an orange hoodie and blue
sunglasses standing on the grass in front of the Sydney Opera House holding a
sign on the chest that says Welcome Friends!

Figure 4: Some generated examples on DrawBench and PartiPrompts.

##  3 Experiments

###  3.1 Zero-shot Text-to-image Generation

We first test the zero-shot text-to-image generation ability of Corgi. We
prepare a dataset containing 900 million image-text pairs, which is composed
of some commonly used datasets such as Conceptual Captions (CC3M) [24],
Conceptual Captions 12M (CC12M) [2], filtered LAION-5B [23] and some image-
text pairs collected by ourselves. Note that we make sure our dataset does not
overlap with MS-COCO [12], CUB [27], Localized Narratives [15] and Multi-modal
CelebA-HQ (MM-CelebA-HQ) [28], because we will test zero-shot or language-free
performance on these downstream datasets.

#### Decoder

Following [14, 19, 22], our decoder adopts a model architecture consisting of
a hierarchy of diffusion models, which has shown impressive ability in text-
to-image generation. Specifically, three diffusion models are trained on 64,
256, and 1024 resolutions, respectively. All the 900M images are used to train
the decoder. During training, each image is processed by three different pre-
trained CLIP models: ViT-B/16, ViT-B/32, and RN-101. The outputs from these
three models will be concatenated into a single 1536-dimensional embedding,
which is then projected into eight vectors and fed into the decoder (in the
same manner as DALL-E 2).

#### Prior model

To handle the image-text alignment, we train a shifted diffusion model to
generate image embeddings from captions. Our shifted diffusion model is a
decoder-only transformer whose input is a sequence consisting of encoded text
from T5 [17], CLIP text embedding, an embedding representing diffusion
timestep, an embedding representing the index of corresponding Gaussian, a
noised CLIP image embedding and a final embedding which will be used to
predict the target CLIP image embedding. We train two variants on datasets
with different scales: one prior is trained on the full 900M image-text pairs;
the other one is trained on CC15M†††collection of CC3M and CC12M, which is a
subset of our full dataset.

#### Final model

Our two different prior models correspond to two final models. The decoders
for the prior models (which are respectively trained on 900M and 15M image-
text pairs) are trained on the 900M images. In other words, the two final
models can be regarded as being trained in supervised and semi-supervised
manners, where the training dataset of the first one consists of 900M image-
text pairs, while the other one consists of 900M images but with only
$1.7\%=15/900$ of them associating with captions. We denote these two variants
as Corgi and Corgi-Semi, respectively. More implementation details are
provided in the Appendix.

(a) A panda making latte art.

(b) Illustration of a mouse using a mushroom as an umbrella.

Figure 5: Comparison of our final models trained in supervised (left) and
semi-supervised manner (right).

Methods | Supervised FID $\downarrow$ | Zero-shot FID $\downarrow$
---|---|---
AttnGAN [29] | $33.10$ |
DF-GAN [25] | $21.42$ |
XMC-GAN [31] | $9.33$ |
Lafite [33] | $8.12$ | $26.94$
DALL-E [20] |  | $27.50$
CogView [4] |  | $27.10$
GLIDE [14] |  | $12.24$
LDM [21] |  | $12.63$
Make-A-Scene [6] |  | $11.84$
DALL-E 2 [19] |  | $10.39$
CogView 2 [5] |  | $24.00$
Imagen [22] |  | $7.27$
Parti [30] |  | $7.23$
Re-Imagen [3] |  | $6.88$
Corgi (Ours) |  | $10.88$
Corgi-Semi (Ours) |  | $10.60$

Table 1: Text-to-image generation results on MS-COCO.

In Table 1, we report zero-shot Fréchet Inception Distance (FID) evaluated on
MS-COCO. Following previous works, the FID is calculated using 30,000
generated images, which corresponds to 30,000 randomly sampled captions from
the validation set of MS-COCO. The results illustrate that our models obtain
strong results, even when it is trained in a semi-supervised manner.
Interestingly, our semi-supervised model obtains better FID than the
supervised one. However, as we can see from Figure 5, the model trained with
more image-text pairs leads to better image-text alignment and better image
quality, as expected. One reason that it obtains relatively worse FID could be
due to the dataset bias: CC15M might contain many samples that are similar to
samples in MS-COCO, while this bias faded when they were merged to construct
our full 900M dataset.

We note that most large-scale text-to-image generation models are trained on
different datasets, but all tested on MS-COCO, ignoring potential dataset
bias. Comparing these models simply by FID might not be appropriate. Thus,
comparing models by actual generation quality is necessary. We follow previous
works and choose to evaluate our model on DrawBench [22] and PartiPrompts
[30]. Some generated examples with complicated scenes are shown in Figure 4.
We compare our model with DALL-E 2 and Stable Diffusion, as these are the only
models allowing public access. For a fair comparison, we fine-tune our model
for an extra 300,000 iterations so that the decoder can take both image
embeddings and text as inputs, which is similar to DALL-E 2. Some visual
comparisons are provided in Figure 7. More results are provided in the
Appendix. As can be seen, our model leads to better generation in most cases
in terms of image-text alignment and image fidelity. Furthermore, we conduct a
human evaluation on DrawBench and PartiPrompts. Specifically, we first
generate four images for each prompt, then ten random human laborers are asked
to judge which model is better (or comparable) in terms of image-text
alignment and fidelity. The results are shown in Figure 6, where our model is
shown to perform better in the two evaluation metrics consistently.

Figure 6: Human evaluation on DrawBench and PartiPrompts.

| MS-COCO | CUB | LN-COCO
---|---|---|---
Methods | IS $\uparrow$ | FID $\downarrow$ | IS $\uparrow$ | FID $\downarrow$ | IS $\uparrow$ | FID $\downarrow$
Lafite [33] | $27.20$ | $18.04$ | $4.32$ | $27.53$ | $18.49$ | $39.85$
Lafite-2[32] | $31.16$ | $10.26$ | $4.93$ | $16.87$ | $23.18$ | $25.51$
Corgi (Ours) | $\mathbf{34.14}$ | $10.33$ | $\mathbf{5.08}$ | $\mathbf{15.80}$ | $\mathbf{28.71}$ | $\mathbf{16.16}$

Table 2: Language-free results on different datasets.

(a) Infinity.

(b) A hamster dragon.

(c) A map of the United States made out sushi. It is on a table next to a
glass of red wine.

(d) A portrait of a statue of the Egyptian god Anubis wearing aviator goggles,
white t-shirt and leather jacket. A full moon over the city of Los Angeles is
in the background at night.

(e) A statue of Abraham Lincoln wearing an opaque and shiny astronaut’s
helmet. The statue sits on the moon, with the planet Earth in the sky.

Figure 7: Comparison with DALL-E 2 and Stable Diffusion. More results are
provided in the Appendix.

###  3.2 Language-free Text-to-image Generation

[33] is the pioneering work to train text-to-image generation models on image-
only datasets, termed language-free training as no associated captions are
provided. With a pre-trained shifted diffusion model, we can also perform
language-free training on any downstream dataset. Given an image-only dataset,
we can train or fine-tune a generative model that generates images from image
embeddings. Consequently, we can directly use the pre-trained shifted
diffusion model at inference time to perform text-to-image generation. Because
our shifted diffusion model is pre-trained on large-scale image-text pairs, it
is expected to generalize well on any downstream domain.

We compare our method with [33, 32]. For a fair comparison, our decoder uses
the same network architecture as [33], which is a StyleGAN2-based model
instead of a diffusion model. Some quantitative results are provided in Table
2, from which we can see that our method leads to better results in general.
More results are provided in the Appendix. Although Lafite-2 [32] achieves a
competitive FID on the MS-COCO dataset, it was trained on pseudo captions that
require extra human workload. Specifically, to train Lafite-2, domain-specific
vocabularies and prompts are needed, which requires human prior knowledge for
each downstream domain. On the other hand, our shifted diffusion model is pre-
trained and can be directly plugged into any domain without further training
or fine-tuning.

###  3.3 Ablation Study

#### Baseline vs. shifted diffusion

Although we have shown that our Corgi model obtains better generation quality
than DALL-E 2 in the previous experiments, there is no direct clue suggesting
that the improvement is due to the advantages of shifted diffusion over the
baseline diffusion. This is because, as we pointed out, other important
factors, such as the potential dataset bias and different implementation
details, may also affect model performance. Therefore, to better compare our
shifted diffusion model with the baseline diffusion model, we conduct an
ablation study where we train these two diffusion models with the same
implementation and training dataset. Specifically, CC15M dataset is used for
both models. We choose $k=1$ for shifted diffusion, i.e., only one Gaussian
distribution is used in the shifted diffusion model for a fair comparison (a
larger $k$ leads to better improvements). More implementation details are
provided in the Appendix.

Firstly, we compare the cosine similarity of the generated image embeddings to
the ground-truth image embeddings. We randomly sample 10,000 image-text pairs
from the validation set of MS-COCO to prevent overlapping between training and
testing datasets. The sampled captions are fed into different diffusion models
to generate the corresponding image embeddings. Intuitively, we expect higher
cosine similarity, as this implies that the model can better approximate the
ground truth. The results are shown in Figure 11. We can see that our shifted
diffusion model leads to higher similarity scores than the baseline, implying
a better image embedding generation.

Secondly, we compare the similarity scores of the generated embeddings to
ground truth at different timesteps during the sampling process. We use 64
strided steps for both the baseline and the shifted diffusion model. The
results are shown in Figure 8, from which we can see that our method leads to
higher similarity scores, especially at initialization. This indicates that
our starting point is much closer to the target, consistent with our design
intention.

Figure 8: The evolution of similarity of generated embedding to ground-truth
during sampling.

(a) A close-up photo of a person holding a cup and a doughnut

(b) A lady sitting on a park bench reading a book.

Figure 9: Examples generated with baseline diffusion (left) and shifted
diffusion (right).

Thirdly, we compare the two models by their text-to-image generation results
evaluated on the MS-COCO dataset. Similar to previous experiments, 30,000
captions are randomly sampled from the validation set of MS-COCO. We then use
the baseline and the shifted diffusion models to generate 30,000 embeddings,
respectively. The embeddings will be fed into our pre-trained diffusion-based
decoder to generate 30,000 images for calculating the zero-shot FID score on
MS-COCO. The FID results are shown in Figure 11, from which we can see that
shifted diffusion model indeed leads to better quantitative results. We also
provide some generated examples in Figure 9. From the generated images, we can
see that the baseline model can fail to capture some details described in the
text.

Figure 10: Results of shifted diffusion with different settings.

Figure 11: Comparison between baseline and shifted diffusion. Shifted
diffusion leads to image embedding more similar to ground truth, which results
in a better downstream generation.

#### Different settings for shifted diffusion

Recall that our shifted diffusion model adopts one or more Gaussian
distributions as its initialization for the sampling process. We investigate
the influence of the number of distributions in this experiment. We train our
shifted diffusion model with 1, 16, 128, and 1024 Gaussians, respectively.
Furthermore, we train a model with 1024 learnable mean vectors and covariance
matrices. All the models are trained on CC15M. We calculate the similarity of
generated image embeddings to the ground-truth embeddings on the validation
set of MS-COCO. The results are shown in Figure 10, from which we can see that
using more Gaussian distributions leads to better results; furthermore, making
the parameters learnable gains further performance improvement. Some more
discussions are provided in the Appendix.

##  4 Conclusion

We propose Corgi, a novel and general diffusion model that benefits text-to-
image generation under different settings. Extensive large-scale experiments
are conducted. Strong quantitative and qualitative results are obtained,
illustrating the effectiveness of the proposed method.

## References

  * [1] Christopher M Bishop and Nasser M Nasrabadi.  Pattern recognition and machine learning, volume 4.  Springer, 2006.
  * [2] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3558–3568, 2021.
  * [3] Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen.  Re-imagen: Retrieval-augmented text-to-image generator.  arXiv preprint arXiv:2209.14491, 2022.
  * [4] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang.  Cogview: Mastering text-to-image generation via transformers, 2021.
  * [5] Ming Ding, Wendi Zheng, Wenyi Hong, and Jie Tang.  Cogview2: Faster and better text-to-image generation via hierarchical transformers.  arXiv preprint arXiv:2204.14217, 2022.
  * [6] Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman.  Make-a-scene: Scene-based text-to-image generation with human priors.  arXiv preprint arXiv:2203.13131, 2022.
  * [7] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [8] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
  * [9] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4401–4410, 2019.
  * [10] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [11] Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou.  Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning.  arXiv preprint arXiv:2203.02053, 2022.
  * [12] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [13] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In International Conference on Learning Representations, 2018.
  * [14] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  arXiv preprint arXiv:2112.10741, 2021.
  * [15] Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari.  Connecting vision and language with localized narratives.  In European Conference on Computer Vision, pages 647–664. Springer, 2020.
  * [16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  arXiv preprint arXiv:2103.00020, 2021.
  * [17] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of Machine Learning Research, 21(140):1–67, 2020.
  * [18] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al.  Exploring the limits of transfer learning with a unified text-to-text transformer.  J. Mach. Learn. Res., 21(140):1–67, 2020.
  * [19] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  arXiv preprint arXiv:2204.06125, 2022.
  * [20] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  arXiv preprint arXiv:2102.12092, 2021.
  * [21] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  arXiv preprint arXiv:2112.10752, 2021.
  * [22] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  arXiv preprint arXiv:2205.11487, 2022.
  * [23] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  arXiv preprint arXiv:2111.02114, 2021.
  * [24] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In ACL, 2018.
  * [25] Ming Tao, Hao Tang, Songsong Wu, Nicu Sebe, Xiao-Yuan Jing, Fei Wu, and Bingkun Bao.  Df-gan: Deep fusion generative adversarial networks for text-to-image synthesis, 2021.
  * [26] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 6309–6318, 2017.
  * [27] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * [28] Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu.  Tedigan: Text-guided diverse face image generation and manipulation.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [29] Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He.  Attngan: Fine-grained text to image generation with attentional generative adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1316–1324, 2018.
  * [30] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al.  Scaling autoregressive models for content-rich text-to-image generation.  arXiv preprint arXiv:2206.10789, 2022.
  * [31] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang.  Cross-modal contrastive learning for text-to-image generation, 2021.
  * [32] Yufan Zhou, Chunyuan Li, Changyou Chen, Jianfeng Gao, and Jinhui Xu.  Lafite2: Few-shot text-to-image generation.  ArXiv, abs/2210.14124, 2022.
  * [33] Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun.  Lafite: Towards language-free training for text-to-image generation.  arXiv preprint arXiv:2111.13792, 2021.

##  Appendix A Derivation of Shifted Diffusion

Recall that we define

$q(\operatorname{\mathbf{z}}_{t}|\operatorname{\mathbf{z}}_{t-1})\coloneqq\mathcal{N}(\operatorname{\mathbf{z}}_{t};\sqrt{1-\beta_{t}}\operatorname{\mathbf{z}}_{t-1}+\operatorname{\mathbf{s}}_{t},\beta_{t}\mathbf{\Sigma}),$

we use deduction to prove that

$\displaystyle
q(\operatorname{\mathbf{z}}_{t}|\operatorname{\mathbf{z}}_{0})=\mathcal{N}(\operatorname{\mathbf{z}}_{t};\sqrt{\bar{\alpha}_{t}}\operatorname{\mathbf{z}}_{0}+\sum_{i=1}^{t}\operatorname{\mathbf{s}}_{i}\sqrt{\bar{\alpha}_{t}/\bar{\alpha}_{i}},(1-\bar{\alpha}_{t})\mathbf{\Sigma}).$
(8)

When $t=1$, we know that

$q(\operatorname{\mathbf{z}}_{1}|\operatorname{\mathbf{z}}_{0})=\mathcal{N}(\operatorname{\mathbf{z}}_{1};\sqrt{1-\beta_{1}}\operatorname{\mathbf{z}}_{0}+\operatorname{\mathbf{s}}_{1},\beta_{1}\mathbf{\Sigma}).$

We can re-write

$\displaystyle\sqrt{1-\beta_{1}}\operatorname{\mathbf{z}}_{0}+\operatorname{\mathbf{s}}_{1}$
$\displaystyle=\sqrt{1-\beta_{1}}\operatorname{\mathbf{z}}_{0}+\sum_{i=1}^{1}\operatorname{\mathbf{s}}_{i}\sqrt{\bar{\alpha}_{1}/\bar{\alpha}_{i}}$
$\displaystyle=\sqrt{\bar{\alpha}_{1}}\operatorname{\mathbf{z}}_{0}+\sum_{i=1}^{1}\operatorname{\mathbf{s}}_{i}\sqrt{\bar{\alpha}_{1}/\bar{\alpha}_{i}}$

and

$\beta_{1}\mathbf{\Sigma}=\\{1-(1-\beta_{1})\\}\mathbf{\Sigma}=(1-\bar{\alpha}_{1})\mathbf{\Sigma},$

which means (8) holds when $t=1$.

Now we prove the case when $t>1$. Assume Equation 8 holds for time $t=s$,
which means

$q(\operatorname{\mathbf{z}}_{s}|\operatorname{\mathbf{z}}_{0})=\mathcal{N}(\operatorname{\mathbf{z}}_{s};\sqrt{\bar{\alpha}_{s}}\operatorname{\mathbf{z}}_{0}+\sum_{i=1}^{s}\operatorname{\mathbf{s}}_{i}\sqrt{\bar{\alpha}_{s}/\bar{\alpha}_{i}},(1-\bar{\alpha}_{s})\mathbf{\Sigma}).$

We also know that

$q(\operatorname{\mathbf{z}}_{s+1}|\operatorname{\mathbf{z}}_{s})\coloneqq\mathcal{N}(\operatorname{\mathbf{z}}_{s+1};\sqrt{1-\beta_{s}}\operatorname{\mathbf{z}}_{s}+\operatorname{\mathbf{s}}_{s+1},\beta_{s}\mathbf{\Sigma}),$

by the property of Gaussian distribution [1], we know that
$q(\operatorname{\mathbf{z}}_{s+1}|\operatorname{\mathbf{z}}_{0})$ is also a
Gaussian distribution with mean

$\displaystyle(\sqrt{\bar{\alpha}_{s}}\operatorname{\mathbf{z}}_{0}+\sum_{i=1}^{s}\operatorname{\mathbf{s}}_{i}\sqrt{\bar{\alpha}_{s}/\bar{\alpha}_{i}})\sqrt{1-\beta_{s+1}}+\operatorname{\mathbf{s}}_{s+1}$
$\displaystyle=$
$\displaystyle(\sqrt{\bar{\alpha}_{s}}\operatorname{\mathbf{z}}_{0}+\sum_{i=1}^{s}\operatorname{\mathbf{s}}_{i}\sqrt{\bar{\alpha}_{s}/\bar{\alpha}_{i}})\sqrt{1-\beta_{s+1}}+\operatorname{\mathbf{s}}_{s+1}\sqrt{\bar{\alpha}_{t+1}/\bar{\alpha}_{t+1}}$
$\displaystyle=$
$\displaystyle\sqrt{\bar{\alpha}_{s+1}}\operatorname{\mathbf{z}}_{0}+\sum_{i=1}^{s}\operatorname{\mathbf{s}}_{i}\sqrt{\bar{\alpha}_{s+1}/\bar{\alpha}_{i}}+\operatorname{\mathbf{s}}_{s+1}\sqrt{\bar{\alpha}_{t+1}/\bar{\alpha}_{t+1}}$
$\displaystyle=$
$\displaystyle\sqrt{\bar{\alpha}_{s+1}}\operatorname{\mathbf{z}}_{0}+\sum_{i=1}^{s+1}\operatorname{\mathbf{s}}_{i}\sqrt{\bar{\alpha}_{s+1}/\bar{\alpha}_{i}},$

and covariance matrix (a diagonal matrix)

$\displaystyle\beta_{s+1}\mathbf{\Sigma}+\sqrt{1-\beta_{s+1}}(1-\bar{\alpha}_{s})\mathbf{\Sigma}\sqrt{1-\beta_{s+1}}$
$\displaystyle=$
$\displaystyle\beta_{s+1}\mathbf{\Sigma}+(1-\beta_{s+1})(1-\bar{\alpha}_{s})\mathbf{\Sigma}$
$\displaystyle=$
$\displaystyle\mathbf{\Sigma}\\{\beta_{s+1}+(1-\beta_{s+1})(1-\bar{\alpha}_{s})\\}$
$\displaystyle=$
$\displaystyle\mathbf{\Sigma}(\beta_{s+1}+1-\beta_{s+1}-\bar{\alpha}_{s}+\beta_{s+1}\bar{\alpha}_{s})$
$\displaystyle=$
$\displaystyle\mathbf{\Sigma}(1-\bar{\alpha}_{s}+\beta_{s+1}\bar{\alpha}_{s})$
$\displaystyle=$
$\displaystyle\mathbf{\Sigma}\\{1-(1-\beta_{s+1})\bar{\alpha}_{s}\\}$
$\displaystyle=$ $\displaystyle\mathbf{\Sigma}(1-\bar{\alpha}_{s+1}),$

which means Equation 8 holds for $t=s+1$ when it holds for $t=s$. By
deduction, we know that since it holds for $t=1$, it holds for any integer.

Specifically, if we choose
$\operatorname{\mathbf{s}}_{t}=(1-\sqrt{1-\beta_{t}}){\bm{\mu}}$, we get

$q(\operatorname{\mathbf{z}}_{t}|\operatorname{\mathbf{z}}_{0})=\mathcal{N}(\operatorname{\mathbf{z}}_{t};\sqrt{\bar{\alpha}_{t}}\operatorname{\mathbf{z}}_{0}+(1-\sqrt{\bar{\alpha}_{t}}){\bm{\mu}},(1-\bar{\alpha}_{t})\mathbf{\Sigma}),$

because

$\displaystyle\sqrt{\bar{\alpha}_{t}}\operatorname{\mathbf{z}}_{0}+\sum_{i=1}^{t}\operatorname{\mathbf{s}}_{i}\sqrt{\dfrac{\bar{\alpha}_{t}}{\bar{\alpha}_{i}}}$
$\displaystyle=$
$\displaystyle\sqrt{\bar{\alpha}_{t}}\operatorname{\mathbf{z}}_{0}+\sum_{i=1}^{t}(1-\sqrt{1-\beta_{i}}){\bm{\mu}}\sqrt{\dfrac{\bar{\alpha}_{t}}{\bar{\alpha}_{i}}}$
$\displaystyle=$
$\displaystyle\sqrt{\bar{\alpha}_{t}}\operatorname{\mathbf{z}}_{0}+{\bm{\mu}}\sum_{i=1}^{t}(1-\sqrt{1-\beta_{i}})\sqrt{\dfrac{\bar{\alpha}_{t}}{\bar{\alpha}_{i}}}$
$\displaystyle=$
$\displaystyle\sqrt{\bar{\alpha}_{t}}\operatorname{\mathbf{z}}_{0}+{\bm{\mu}}(\sqrt{\dfrac{\bar{\alpha}_{t}}{\bar{\alpha}_{t}}}-\sqrt{\dfrac{\bar{\alpha}_{t}}{\bar{\alpha}_{t-1}}}+\sqrt{\dfrac{\bar{\alpha}_{t}}{\bar{\alpha}_{t-1}}}-\sqrt{\dfrac{\bar{\alpha}_{t}}{\bar{\alpha}_{t-2}}}+...+\sqrt{\dfrac{\bar{\alpha}_{t}}{\bar{\alpha}_{2}}}-\sqrt{\dfrac{\bar{\alpha}_{t}}{\bar{\alpha}_{1}}}+\sqrt{\dfrac{\bar{\alpha}_{t}}{\bar{\alpha}_{1}}}-\sqrt{\bar{\alpha}_{t}})$
$\displaystyle=$
$\displaystyle\sqrt{\bar{\alpha}_{t}}\operatorname{\mathbf{z}}_{0}+(1-\sqrt{\bar{\alpha}_{t}}){\bm{\mu}}.$

We are now able to get the closed-form expression of posterior
$q(\operatorname{\mathbf{z}}_{t-1}|\operatorname{\mathbf{z}}_{t},\operatorname{\mathbf{z}}_{0})$.
We know that

$\displaystyle q(\operatorname{\mathbf{z}}_{t}|\operatorname{\mathbf{z}}_{0})$
$\displaystyle=\mathcal{N}(\operatorname{\mathbf{z}}_{t};\sqrt{\bar{\alpha}_{t}}\operatorname{\mathbf{z}}_{0}+(1-\sqrt{\bar{\alpha}_{t}}){\bm{\mu}},(1-\bar{\alpha}_{t})\mathbf{\Sigma})$
$\displaystyle
q(\operatorname{\mathbf{z}}_{t-1}|\operatorname{\mathbf{z}}_{0})$
$\displaystyle=\mathcal{N}(\operatorname{\mathbf{z}}_{t-1};\sqrt{\bar{\alpha}_{t-1}}\operatorname{\mathbf{z}}_{0}+(1-\sqrt{\bar{\alpha}_{t-1}}){\bm{\mu}},(1-\bar{\alpha}_{t-1})\mathbf{\Sigma})$
$\displaystyle
q(\operatorname{\mathbf{z}}_{t}|\operatorname{\mathbf{z}}_{t-1})$
$\displaystyle=\mathcal{N}(\operatorname{\mathbf{z}}_{t};\sqrt{1-\beta_{t}}\operatorname{\mathbf{z}}_{t-1}+(1-\sqrt{1-\beta_{t}}){\bm{\mu}},\beta_{t}\mathbf{\Sigma}).$

From [1], we know that

$\displaystyle
q(\operatorname{\mathbf{z}}_{t-1}|\operatorname{\mathbf{z}}_{t},\operatorname{\mathbf{z}}_{0})=\mathcal{N}(\operatorname{\mathbf{z}}_{t-1};{\bm{\nu}},\mathbf{\Lambda})$

where

$\displaystyle{\bm{\nu}}$
$\displaystyle=\mathbf{\Lambda}\\{\sqrt{1-\beta_{t}}\\{\operatorname{\mathbf{z}}_{t}-(1-\sqrt{1-\beta_{t}}){\bm{\mu}}\\}\mathbf{\Sigma}^{-1}/\beta_{t}+\\{\sqrt{\bar{\alpha}_{t-1}}\operatorname{\mathbf{z}}_{0}+(1-\sqrt{\bar{\alpha}_{t-1}}){\bm{\mu}}\\}\mathbf{\Sigma}^{-1}/(1-\bar{\alpha}_{t-1})\\},$
$\displaystyle\mathbf{\Lambda}$
$\displaystyle=\\{\mathbf{\Sigma}^{-1}/(1-\bar{\alpha}_{t-1})+(1-\beta_{t})\mathbf{\Sigma}^{-1}/\beta_{t}\\}^{-1}.$

Equation (3) can be obtained by simple derivation.

##  Appendix B More Experimental Results

#### Language-free text-to-image generation

We provide more results on language-free text-to-image generation in Table 3.
Results in Table 3 are models trained from scratch on corresponding datasets.
Results of MM-CelebA-HQ in [32] is based on a model which is first pre-trained
on FFHQ dataset [9]. For fair comparison, we use the code provided by the
authors and train Lafite-2 from scratch on MM-CelebA-HQ, without pre-training
on FFHQ.

| MS-COCO | CUB | LN-COCO | MM-CelebA-HQ
---|---|---|---|---
Methods | IS $\uparrow$ | FID $\downarrow$ | IS $\uparrow$ | FID $\downarrow$ | IS $\uparrow$ | FID $\downarrow$ | IS $\uparrow$ | FID $\downarrow$
Lafite [33] | $27.20$ | $18.04$ | $4.32$ | $27.53$ | $18.49$ | $39.85$ | $2.89$ | $32.75$
Lafite-2[32] | $31.16$ | $10.26$ | $4.93$ | $16.87$ | $23.18$ | $25.51$ | $2.91$ | $21.89$
Corgi (Ours) | $\mathbf{34.14}$ | $10.33$ | $\mathbf{5.08}$ | $\mathbf{15.80}$ | $\mathbf{28.71}$ | $\mathbf{16.16}$ | $\mathbf{3.06}$ | $\mathbf{19.74}$

Table 3: Language-free results on MS-COCO, CUB, LN-COCO and MM-CelebA-HQ
datasets.

#### Ablation study on shifted diffusion

(a) 1024 fixed Gaussian

(b) 1024 learn-able Gaussian

Figure 12: Frequency of clusters being selected.

We already know that learn-able Gaussians may lead to higher similarity of
generated image embedding to ground-truth embedding, we would like to further
compare them. We train 2 shifted diffusion models, with 1024 fixed/learn-able
Gaussian distributions. After training, we feed random text captions from the
validation set of MS-COCO to the model, and calculate the frequency of each
Gaussian being selected by (5). We rank and plot the frequency in Figure 12.
Intuitively, when we set $k=1024$, we are expecting 1024 representative
clusters with different semantics, thus we hope every Gaussian has a chance of
being selected. However, as we can see in the figure, in the case where 1024
fixed Gaussians are introduced, some Gaussians are never selected which means
they contribute nothing to the model. On the contrary, with learned mean and
covariance matrix, every Gaussian has a chance of being selected.

##  Appendix C Implementation Details

Our prior model trained on 900M dataset is a decoder-only transformer. We set
the width, depth, number of attention heads to be 2048, 20, 32 respectively.
The model is trained for 500,000 iterations with a batch size of 4096. For the
prior model trained on CC15M and baseline prior model in ablation study, we
reduce the transformer depth from 20 to 16, while keeping the width and number
of attention heads unchanged. The smaller prior models are trained for 40,000
iterations with batch size of 4096 on CC15M dataset. AdamW [13] optimizer with
learning rate of $1.2\times 10^{-4}$, $\beta=(0.9,0.96)$, $\epsilon=10^{-6}$.
We drop the encoded text with probability of 0.1 to enable classifier-free
guidance sampling. [8]

Our diffusion-based decoder follows DALL-E 2 [19]. Batch sizes are set to be
2048, 1024, 512 for diffusion models at 64, 256, 1024 resolutions
respectively. The models are trained for 1,000,000 iterations. AdamW optimizer
with $\beta=(0.9,0.999),\epsilon=10^{-8}$ is used for all three models.
Learning rate is set to be $10^{-4}$ for the model at 1024 resolution, while
it is set to be $1.2\times 10^{-4}$ for the other two models. We drop the
encoded text with probability of 0.1 to enable classifier-free guidance
sampling.

Our GAN-based decoder follows the design in [33]. The batch size is set to be
64. Adam optimizer [10] with learning rate 0.0025,
$\beta=(0,0.99),\epsilon=10^{-8}$ is used for both generator and
discriminator.

##  Appendix D More Generated Examples

We provide more generated examples and comparisons with DALL-E 2 and Stable
Diffusion here.

(a) Infinity.

(b) The Starry Night.

(c) A photo of a teddy bear made of water.

(d) A heart made of water.

(e) A television made of water that displays an image of a cityscape at night.

(f) The city of London.

(g) New York Skyline with ’Hello World’ written with fireworks on the sky.

Figure 13: Comparison with DALL-E 2 and Stable Diffusion.

(a) A portrait of a statue of the Egyptian god Anubis wearing aviator goggles,
white t-shirt and leather jacket. A full moon over the city of Los Angeles is
in the background at night.

(b) A photo of an astronaut riding a horse in the forest. There is a river in
front of them with water lilies.

(c) A photo of a Ming Dynasty vase on a leather topped table.

(d) A photo of a light bulb in outer space traveling the galaxy with a sailing
boat inside the light bulb.

(e) A statue of Abraham Lincoln wearing an opaque and shiny astronaut’s
helmet. The statue sits on the moon, with the planet Earth in the sky.

(f) Darth Vader playing with raccoon in Mars during sunset.

(g) Ground view of the Great Pyramids and Sphinx on the moon’s surface. The
back of an astronaut is in the foreground. The planet Earth looms in the sky.

Figure 14: Comparison with DALL-E 2 and Stable Diffusion.

(a) A tiger wearing a tuxedo.

(b) A photograph of an ostrich wearing a fedora and singing soulfully into a
microphone.

(c) A photo of llama wearing sunglasses standing on the deck of a spaceship
with the Earth in the background.

(d) A teddy bear wearing a motorcycle helmet and cape is riding a motorcycle
in Rio de Janeiro with Dois Irmãos in the background.

(e) A squirrel driving a toy car.

(f) A smiling sloth wearing a leather jacket, a cowboy hat and a kilt.

(g) A punk rock squirrel in a studded leather jacket shouting into a
microphone while standing on a lily pad.

Figure 15: Comparison with DALL-E 2 and Stable Diffusion.

(a) An iPhone case.

(b) An appliance or compartment which is artificially kept cool and used to
store food and drink.

(c) A cute wooden owl statue holding a large globe of the Earth above its
head.

(d) A laptop screen showing a bunch of photographs.

(e) A chopper decorated with the Stars and Stripes..

(f) A paranoid android freaking out and jumping into the air because it is
surrounded by colorful Easter eggs.

(g) A type of digital currency in which a record of transactions is maintained
and new units of currency are generated by the computational solution of
mathematical problems, and which operates independently of a central bank.

Figure 16: Comparison with DALL-E 2 and Stable Diffusion.

(a) A map of the United States made out sushi. It is on a table next to a
glass of red wine.

(b) A high resolution photo of a large bowl of ramen. There are several
origami boats in the ramen of different colors.

(c) A giant cobra snake made from sushi.

(d) A high resolution photo of a chicken working out in a gym.

(e) Close-up portrait of a smiling businesswoman holding a cell phone, oil
painting in the style of Rembrandt.

(f) A young badger delicately sniffing a yellow rose, richly textured oil
painting.

(g) A gundam stands tall with its sword raised. A city with tall skyscrapers
is in the distance, with a mountain and ocean in the background. A dark moon
is in the sky. realistic high-contrast anime illustration.

Figure 17: Comparison with DALL-E 2 and Stable Diffusion.

(a) A high contrast portrait photo of a fluffy hamster wearing an orange
beanie and sunglasses holding a sign that says Let’s PAINT!

(b) A flower with a cat’s face in the middle.

(c) A zebra with blue stripes.

(d) A wine glass on top of a dog.

(e) A blue colored dog.

(f) A chimpanzee wearing a bowtie and playing a piano.

(g) A cat standing on a horse.

Figure 18: Comparison with DALL-E 2 and Stable Diffusion.