# Pictures Of MIDI: Controlled Music Generation via Graphical Prompts for
Image-Based Diffusion Inpainting

###### Abstract

Recent years have witnessed significant progress in generative models for
music, featuring diverse architectures that balance output quality, diversity,
speed, and user control. This study explores a user-friendly graphical
interface enabling the drawing of masked regions for inpainting by an
Hourglass Diffusion Transformer (HDiT) model trained on MIDI piano roll
images. To enhance note generation in specified areas, masked regions can be
“repainted” with extra noise. The non-latent HDiT’s linear scaling with pixel
count allows efficient generation in pixel space, providing intuitive and
interpretable controls such as masking throughout the network and removing the
need to operate in compressed latent spaces such as those provided by
pretrained autoencoders. We demonstrate that, in addition to inpainting of
melodies, accompaniment, and continuations, the use of repainting can help
increase note density yielding musical structures closely matching user
specifications such as rising, falling, or diverging melody and/or
accompaniment, even when these lie outside the typical training data
distribution. We achieve performance on par with prior results while operating
at longer context windows, with no autoencoder, and can enable complex
geometries for inpainting masks, increasing the options for machine-assisted
composers to control the generated music.

##  0 Introduction

Music generation with deep learning models has seen tremendous progress in
recent years, enabling algorithmic composition of creative and expressive
musical pieces. However, a key challenge that remains is providing users with
intuitive control over the generative process. While text prompts and
conditioning on attributes like melodies, chords, or timbres have shown
promising results [1, 2, 3], there is a need to explore richer interactive
interfaces that extend user control of constraints on the generative process
beyond mere text inputs.

The investigation in this paper is inspired by a typical songwriting
composition task in which we seek to create a melody given some accompaniment
and a rough idea of when or where we would like the melody to rise and fall
[4]. We imagine an application in which the user could draw on the piano roll
a rough profile for the “shapes” of the regions in which to melody is to be
added, and have the generative model fill in the notes in way that sounds
appropriate, given the accompaniment. This process is somewhat akin to the
“graphic notation” movement in 20th-century music composition championed by
composers such as John Cage [5], Cornelius Cardew [6], and Karlheinz
Stockhausen [7], by which performers were encouraged to improvise within
certain confines indicated by the shapes drawn by the composer. In this study
we imagine not so much the performers improvising, but rather the model
generating a composition. Such visual interfaces for generative models have
been present in the image-generation domain for quite some time, such as with
NVIDIA’s Gau-GAN [8], and continue today as with the recent “Semantic Image
Synthesis” work of Park et al[9].

An example of such a “prompt” is shown graphically in Figure 1. One may view
the prompt as a constraint on the space of generative possibilities. Such
constraints have long been linked with creativity and composition [10], as
they both limit the field and provide structure around which to generate. For
a task such as this, one may then ask the question which generative model
architectures are best suited for this kind of control. Transformer-based
models for musical audio generation such as MusicGen [11] so far have
incorporated text prompts and conditioning on melody, however the introduction
of arbitrary inpainting masks poses significant challenges for such an
architecture.

Figure 1: The Motivating Idea. Top: MIDI piano roll image of a sample
“graphical prompt” of rough shapes (in blue) of pitches for melody generation
given accompaniment (green lines). Bottom: Sample generated output.

Diffusion models have emerged as a powerful framework for generative modeling,
achieving state-of-the-art results in domains like image synthesis [12, 13]
and audio generation [14, 15, 3], and even symbolic music generation [16]. A
particularly exciting capability is their support for inpainting or infilling,
where partial data is provided as a conditioning context, and the model fills
in the missing regions [15, 17, 18, 19]. This opens up opportunities for user
control via sketching or sculpting of the desired output of the kind shown in
Figure 1.

Image-based control schemes exemplified by ControlNet [20] have inspired
musical audio counterparts [21, 22]. The “direct” adaption of image-based
music generation was demonstrated notably by Riffusion [23] for audio and
Polyffusion [24] for MIDI. The latter set a new standard for quality,
diversity, and controlability of outputs. MIDI data offers a convenient set of
human-readable priors and very compact data representation, making it an
excellent testbed for further research in contollable music generation.
Polyffusion and others [25] explored diffusion models operating on piano roll
representations, treating musical scores as image-like data. While highly
effective, these models are typically limited to short sequences due to memory
constraints, and may involve the use of compressed latent representations
which typically involve basis functions that fail to correspond with humans’
intuitive representations. Our work employs recent advances in high-definition
image synthesis with hierarchical diffusion models [26, 27] to overcome this
limitation and enable longer musical sequences.

Our key interest lies in exploring intuitive interfaces for constrained music
infilling, beyond just text prompts. For example, users could sketch a rough
target melody profile which the model then turns into a fully-realized
composition [28]. Inpainting over the compact and human-interpretable piano
roll domain allows natural specifications of structural constraints.

The key prior works similar to this study are the aforementioned Polyffusion
[24] for the image-based generative diffusion of piano roll images, and
Benetatos et al [29] offering a drawing-based interface for controllable
melody generation. Features which this paper lacks are conditioning on chords,
rhythm, and/or texture, which have been explored to great effect the
aforementioned prior works. Our current codebase does not yet have these
features fully implemented. We also focus only on single-instrument generation
rather than multi-instrument compositions.

We believe the following are unique aspects of this paper that contribute to
and advance the dialogue on controllable music generation:

  1. 1.

We replicate the results of Polyffusion [24], achieving comparable outputs in
terms of objective and subjective metrics, yet extending beyond earlier work
to include

    1. (a)

(4x) larger images and thus (4x) longer sequences

    2. (b)

complex shapes for inpainting masks so users can specify regions and
directions of melodic and harmonic development.

    3. (c)

explicit modeling of note velocity

  2. 2.

We apply the recent Hourglass Diffusion Transformer (HDiT) [26] that
efficiently operates in pixel space on large images, thus

    1. (a)

removing the need for a separate autoencoder as in [24]’s use of latent
diffusion.

    2. (b)

allowing for the straightforward application of complex inpainting mask shapes
(that might otherwise need to be remapped into the autoencoder’s latent
space).

  3. 3.

We explore and develop techniques (RePaint [17], nucleation) to encourage note
inpainting when outside of training data distribution.

  4. 4.

We investigate the limitations of inpainting-only approaches. For example, we
find that while inpainting is useful for a variety of tasks, we do not find it
to be an effective substitute for chord (progression) conditioning.

  5. 5.

We wish to emphasize that these high quality results have been obtained using
the unified, simple, even “naive” approach of simply re-appropriating an
existing image diffusion codebase, without introducing specialized
architectures or mathematical formalism (as in [29]), and applying inpainting
methods commonly seen in image spaces.

In summary, this paper presents a simple, unified, powerful and flexible
framework for controllable and interactive music generation by combining the
capabilities of diffusion models, hierarchical representations, and intuitive
inpainting constraints specified through a visual interface.

We provide an demo website with listening examples111Demo website:
https://picturesofmidi.github.io/PicturesOfMIDI

##  1 Methods

###  1.1 Data Preparation

Following Polyffusion [24], we use the POP909 MIDI dataset [30] of piano
arrangements of Chinese pop songs. We normalize all tempos to 120 beats per
minute to standardize the piano roll representation, and perform data
augmentation by transposing up to +/- 12 semitones, expanding the dataset from
the initial 909 MIDI files to 22725 files. Piano roll images are rendered such
that each pixel corresponds to a 16th note, and as in [24] we color note
durations in green with brightness corresponding to note velocity, and add red
onset markers to add an additional layer of verification to rule out spurious
diffusion artifacts.

We apply the same chord detector from the Polyffusion code to all songs,
through which is its found that 529 different chords exist in the augmented
(i.e. variably transposed) dataset. We encode the chord information as borders
along the top and bottom 8 pixels of the images, using a base-9 arithmetic
scheme to separate each color by 30 color-index-values (ranging from 0 to 255)
in each R,G,B channel of the image. The border size of 8 was chosen so as not
to overwrite any notes, which given transpositions occur at MIDI pitches as
low as 9 and as high as 115. This leaves the bottom 8 pixels and top 12 pixels
available to use for other notations such as embedding chord information.

Each song in each key is rendered as one long $N_{t}\times 128$ pixel image
where $N_{t}$ is the total number of 16th notes in the song. From these full
song images, will train a model by grabbing random 512-pixel windows (padding
the ending with zeros as needed) for each batch of data. Since the HDiT
expects square images, we convert each 512x128 image to 256x256 by putting the
second half of the original image below the first. To help the network
communicate effectively across the cut in the middle, we reverse the lower
half of the image horizontally. Figure 2 shows this process. A window width of
512 pixels at 1 pixel per 16th note results in 32 measures, providing
approximately 1 minute per image at the standardized tempo of 120 BPM.

Figure 2: Sample 512x128 MIDI piano roll image. Following Polyffusion [24], we
denote notes in green with onsets in red. We also add color-coded chord
embeddings as borders along the top and bottom of the image. The right half of
the image (after the dashed line) is “folded” underneath the left half to
produce a square image suitable for example Hourglass Diffusion Transformer
(HDiT) [26] pipelines. After generation, the images are restored to their
rectangular format. (Although this “folding” causes a reversal of direction
vs. of a simplw copy-paste, we do this so that information need not propagate
all the way across the image to maintain musical continuity. In practice we
observe no issues with continuity at fold boundary – the model quickly learns
to adapt.

###  1.2 Neural Network Architecture

We use an image-based deep learning system to learn and generate piano rolls.
Employing image modalities for music representations has a long history in
deep learning research, such as the use of spectrogram images for
classification [31]. The closest musical audio analog of this present study
might be Riffusion [23], which used Stable Diffusion [12] essentially
unaltered to produce images of spectrograms that were then converted to audio
via other means.

Instead of Stable Diffusion which operates in a latent space defined by a
(VQ-)VAE, we use the recent Hourglass Diffusion Transformer (HDiT) of Crowson
et al [26] and the code in the official  k-diffusion
repository222https://github.com/crowsonkb/k-diffusion with minimal
modifications that allow inpainting support. HDiT operates in pixel space and
requires no VAE. From Crowson’s code repository, we minimally modify the
example configuration file for the 256x256 pixel Oxford 102 flowers dataset
[32] to our current needs. During training, we dynamically grab batches of
random 512-pixel-wide windows from each $N_{t}\times 128$ MIDI piano roll
image. Then, as shown in Figure 2, we “fold” the right half of each windowed
image under the left half to supply a 256x256 image to supply for the HDiT
code.

###  1.3 Training

Training was performed for 48 hours on a pair of NVIDIA A6000 GPUs with a
batch size of 192. Even in as little as the first 1000 training steps, the
model is already putting out discernible notes in green with matching onsets
in red, and corresponding chord annotation borders along the top and bottom
which mirror each other. The music is sheer cacophony at this point, and the
remaining two days improve the musicality. We typically see polynomial
convergence of the loss function (relative to the number of steps), or
slightly worse. However, eventually the model starts memorizing the data, at
which point the convergence becomes exponential. We halt the training shortly
after exponential convergence is first detected.

###  1.4 Sampling and Inpainting

Inference for diffusion models is typically referred to as sampling. It is
during sampling that we do the inpainting described below. The  k-diffusion
code supports a variety of higher-order sampling algorithms. We found the
Linear Multistep (LMS) sampler to yield poor results when inpainting, and
settled on the stochastic variant of the DPM-Solver++(2M) solver [33] instead.
(The 3M SDE version of the DPM-Solver++ also produced comparable results.)

Important musical inpainting work includes that of CRASH [15], the question of
achieving usable long-term inpainted music was has received special attention
in the literature [34, 35]. Our inpainting will rely on a basic binary mask as
was outlined in the Polyffusion paper [24], although we must adapt the RePaint
algorithm [17] slightly for the  k-diffusion package’s implementation of the
mathematical formalism by Karras et al[27]. To wit: we compute the $\beta_{t}$
parameters for Repaint using the $\sigma_{t}$ variables of Karras via

$\beta_{t}=(\sigma_{t}/\sigma_{\rm max})^{2},$

where as with comparable HDiT configurations, we use $\sigma_{\rm max}=160$.

RePaint involves repeatedly stepping backwards once re-introducing noise at
each stage of the reverse diffusion process, with the number of repaint loops
(or “repaints“) given by some parameter $U$. For generations in Section 2.1 we
use $U=1$ (i.e., regular diffusion sampling with no re-paints), but in Section
2.2 we increase $U$ as a way to generate higher note densities in drawn
inpainting regions.

Somewhat inspired by the recent work of Lin et al [36] who “algorithmically
render all symbolic controls into the audio format before inputting them to
MusicGen,” we pass chord information into the model as part of each image. Our
hope was to achieve results comparable to those obtained from chord-based
conditioning but using inpainting alone instead. Rather than supply chords as
a conditioning signal, we render “color embeddings” of the chords and affix
them to “borders” of unused very high and low pitches, which we observe from
our (transposed) dataset to comprise the top and bottom 8 pixels of our
128-pixel-tall MIDI piano roll images.

##  2 Results

###  2.1 “Standard” Inpainting Tasks

Figure 3 shows sample outputs from letting the model generate in an
undirectected fashion, based on the learned distribution of training data,
with no inpainting. We see differences in melody, accompaniment, and chords.
Regarding the latter: although the model is able to reliably produce colored
chord-marker borders that match on the top and bottom of each image, we find
that these colors, when decoded, do not correspond to the chords as detected
independently using a chord-extraction code (e.g. from [24]. This demonstrates
a limitation of the “all-inpainting” approach in this paper: any chord borders
shown in this paper are thus “decorative” and may be cropped out.

Figure 3: Undirected generation. Here we see model generates various
assortments of melody, accompaniment, and “chord borders”. Refer to the demo
website for listening examples.  Figure 4: Melody inpainting. The top portion
of the piano roll is masked out in blue (top image), and the model generates
melodies (shown in the bottom 3 images) that fit the accompaniment notes and
the chords (denoted by colored bars along the top and bottom).

Figures 4 through 6 show examples of common musical inpainting tasks. In
Figure 4, we mask out the notes in the upper half of the piano roll. The model
fills in melody lines consistent with the underlying chord structure and
accompaniment. Figure 5 shows the reverse, generating accompaniment given a
melody.

Figure 5: Accompaniment inpainting, given a melody. It is noteworthy that the
chords shown in the top and bottom borders are the same for each output even
though the accompaniment notes differ. Figure 6: Continuation / "outpainting,"
offering novel continuations from the same starting music.

Not shown are the negative results from our inpainting studies: anything
involving the chord markers – inpainting notes given chords and vice versa –
does not work, in the sense that the generated chord markers appear to be
arbitrary and do not match the notes at in any significant sense, and the
notes generated do not follow the chords requested. It is possible that a
revised inpainting scheme such as RePaint [17] would be sufficient to allow
these tasks to be completed via inpainting alone, although finding the proper
implementation of RePaint for multi-step k-diffusion [27] integration is thus
far an open question. Otherwise, inpainting alone will need to be abandoned in
favor of chord based conditioning, the effectiveness of which was demonstrated
by Min et al. [24].

a)
b)

Figure 7: Generation constrained by “creative drawing” of inpainting mask,
Case 1.

###  2.2 “Creative” Inpainting Tasks

We now consider cases where a user may have an idea of the overall shape that
they would like the music to follow, expressed via drawing the mask via some
graphical interface. (For this paper, these were drawn via GIMP and/or the
Mac’s Preview app.333We also have a Gradio GUI demo available on the Demo
website. The model is tasked with adding notes that “make sense” and meet the
user’s criteria.

The examples shown in Figures 7 and 8 are deliberately “extreme” in the sense
of featuring pitch ranges that fluctuate over larger intervals and shorter
timescales than is found in the training data distribution. As a result of
this domain mismatch, the model typically produces relatively few notes
meeting the user’s criteria, instead outputting mostly silence.

It is possible to “brute force” the creation of satisfactory(-looking) outputs
by generating a large number of example outputs and ranking them according to
the area of the mask filled in by notes (i.e., by taking the dot-product of
the mask and the generated image). These satisfactory outputs are therefore
low-probability events. The examples shown in Figures 7 and 8 are thus
automatically cherry-picked via this ranking method, obtained by generating
100 outputs at a time and taking only the top 2 outputs in the ranking. An
alternative to this brute force method, namely “seeding nucleation,” is
explored in the next subsection.

Given that these examples differ significantly from the training data
distribution, it is perhaps unsurprising that the generated music is not
especially “musical” or aesthetically pleasing in the sense of pop tunes. Thus
the utility of such extreme masking cases may yet have little practical
utility without additional and more varied training data, conditioning
signals, and/or diffusion guidance. This remains an area for further study.

a)
b)

Figure 8: Generation constrained by “creative drawing” of inpainting mask,
Case 3. Here the user requests that the bass line and melody diverge leaving a
sizable gap in between, and then converge.

###  2.3 Increasing Inpainted Note Density

Depending on how closely or poorly the user’s visual prompt conforms to
typical data in the training data distribution, the model may have a
respectively higher or lower probability of providing matching outputs. In
Figure 9a), we see an attempt to generate notes in a mask shape that extends
higher than typical melodic ranges for POP909. The results of these
generations can be “hit or miss” in that there can be a low probability of
notes appearing at these high pitch values as demonstrated in Figure 9b).
Applying the automated brute-forced cherry-picking described in Section 2.2,
we can get more notes as shown in Figure 9c), however this is computationally
needlessly expensive. More effective is to combine the ranked cherry-picking
method for a smaller number of generated outputs in which we increase the
RePaint parameter $U$ from 1 to 2. These results are shown in 9d).

a)
b)
c)
d)

Figure 9: Driving Inpainting. a) The user’s promp veers a bit high compared to
the training data. b) Typical output shows few notes generated. c) By brute
force (cf. Section 2.2) we can get more notes. d) More efficiently, we can get
more notes by increasing the RePaint parameter $U$ from 1 to 2. As we increase
$U$ to 3, 4, or more, we get more notes but they increasingly show little
musical coherence and tend toward apparent randomness. Our attempts to spell
out words in the MIDI in the style of Jacob Collier [37] have so far produced
nothing close to his level of musicality.

###  2.4 Evaluation

Name of Dataset / Model | $\sigma_{P}\uparrow$ | IQR${}_{D}\uparrow$ | $\mathcal{D}_{P}\uparrow$ | $\mathcal{D}_{D}\uparrow$ | $\mathcal{D}_{\mathrm{KL},P}\downarrow$ | $\mathcal{D}_{\mathrm{KL},D}\downarrow$
---|---|---|---|---|---|---
Original MIDI | 0.46 | 10.6 | n/a | n/a | n/a | n/a
Piano Roll Images $(\pm 12$ semitones) | 0.88 | 13.6 | n/a | n/a | n/a | n/a
PoM (ours) | 1.0 | 11.7 | 0.87 | 0.95 | 0.036 | 0.14
Polyffusion (our execution) | 0.38 | 11.0 | 0.88 | 0.95 | 0.0050 | 0.13
Table 1: Objective Metrics. For MIDI pitches (P) and note durations (D), we
measure the diversity of values via the standard deviation of pitches
$\sigma_{\mathrm{P}}$ and the inter-quartile range of durations (IQRD).
Similarity to the training data is computed via the average overlapped area of
pitch distribution ( $\mathcal{D}_{P}$) and duration distribution (
$\mathcal{D}_{D}$), as well as Kullback–Leibler divergences
$\mathcal{D}_{\mathrm{KL}}$. All values should be taken as +/- 1 on the last
digit. Figure 10:  Subjective evaluation for unconditional generation. The
error bars may suggest that, for the examples provided, listeners found all
examples to be comparable. “GT” denotes ground truth music from the POP909
MIDI Dataset. The “PoM-Drawn” results are for melodies generated to follow
pre-made user-drawn inpainting masks as in Figure 9, whereas “PoM” alone
denotes “typical” unconditioned melody generation as in Figure 4.

In theory, a desirable model would generate music that is both of high quality
and high diversity. While the diversity of outputs can be measured
statistically, it is difficult to objectively define “quality.” Thus as a
proxy for quality, we tend to objectively measure the statistical similarity
of the generated data to the training data and supplement this with subjective
evaluations using human preferences measured via listening tests.

For objective diversity measurements, it’s appropriate to use the standard
deviation of MIDI pitches $\sigma_{\mathrm{P}}$ since these are approximately
normally distributed. The note durations, however, are heavily skewed toward
small values, and thus a metric such as the inter-quartile range IQRD444This
IQR covers 25% to 75% of the sorted data points. is better suited to convey
the statistical spread of data rather than the standard deviation (which can
be heavily influenced by the the long tail). Table 1 summarizes these
measurements for various generative outputs. For our objective similarity
measurements, we compare the training data with generated data by computing
the average overlapped area of pitch distribution ($\mathcal{D}_{P}$ ) and
duration distribution ( $\mathcal{D}_{D}$) as in [24]. In addition to these we
compute Kullback–Leibler divergences $\mathcal{D}_{\mathrm{KL}}$ of
probability density functions (PDF) for each data distribution, obtained via
kernel density estimation on histograms of the MIDI pitches
($\mathcal{D}_{\mathrm{KL,P}}$) and note durations
($\mathcal{D}_{\mathrm{KL,D}}$).

Subjective listening evaluations were performed by 54 volunteers, half of whom
were senior Audio Engineering undergraduate majors, and the remaining half
were drawn from the general population. We repeated the Polyffusion paper’s
criteria of “Creativity,” “Naturalness” (i.e., “how likely a human musician
composed the music”), overall “Musicality,” and “Fitness” of a generated
melody over a fixed accompaniment. Evaluators listened to two 16-second long
examples drawn from the “ground truth” POP909 MIDI dataset, generations from
our execution of the Polyffusion code (using their pretrained checkpoints),
and our model. Results are shown in Figure 10, which indicate that listeners
found all examples to be comparable in quality. Notably, the “Fitness”
evaluation included examples of the “creative inpainting” variety with a
Repaint value of 2 and with drawn melody shapes that, unlike those shown
earlier, were of a modest range of notes, thus staying within the typical
range of the training data distribution. One further finding is that requiring
red onset markers for our model makes no difference to our metrics, which
agrees with the Polyffusion paper [24]: “In practice, the generation process
of 160 8-bar samples report zero invalid notes.” Taken together these
evaluations suggest that the PoM model generates output comparable to
Polyffusion [24] while supporting longer contexts and arbitrary inpainting
mask shapes.

##  3 Conclusions

We have presented a unified method for image-based diffusion inpainting that
performs comparably to prior inpainting work, yet requires no additional
autoencoder due to its use of a Hourglass Diffusion Transformer (HDiT). The
HDiT allows for efficient processing of large images allowing longer sequence
contexts than previous approaches, and also allows for the straightforward and
interpretable use of complex mask geometries for inpainting. These
simplifications and extensions, along with the relative ease of appropriating
the  k-diffusion codebase, suggest that image-based piano roll diffusion shows
increasing promise for enabling composers to exert greater control during the
generative music creation process. Future studies may include scaling images
up, semi-transparent masks, and multi-instrument piano rolls.

## References

  * [1] A. Agostinelli, T. I. Denk, Z. Borsos, J. Engel, M. Verzetti, A. Caillon, Q. Huang, A. Jansen, A. Roberts, M. Tagliasacchi, M. Sharifi, N. Zeghidour, and C. Frank, “Musiclm: Generating music from text,” 2023.
  * [2] Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour, “Audiolm: a language modeling approach to audio generation,” _arXiv preprint arXiv:2209.03143_ , 2022.
  * [3] Z. Evans, C. Carr, J. Taylor, S. H. Hawley, and J. Pons, “Fast timing-conditioned latent audio diffusion,” 2024.
  * [4] L. Bell, “Hitmaking For Producers: The Professional Method For Producing Radio-Ready Songs. 2. Writing the Melody,” May 2021. [Online]. Available: https://studio.com/louis-bell/hitmaking-for-producers
  * [5] J. Cage, Ed., _Notations_ , first edition ed. Something Else Press, Jan. 1969.
  * [6] B. Dennis, “Cardew’s ‘treatise’ (mainly the visual aspects),” _Tempo_ , no. 177, p. 10–16, 1991.
  * [7] T. Sauer and M. Perry, _Notations 21_. New York, NY: Mark Batty Publisher, Apr. 2009.
  * [8] T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu, “Semantic image synthesis with spatially-adaptive normalization,” 2019.
  * [9] J. Lee, D. S. Jung, K. Lee, and K. M. Lee, “Streammultidiffusion: Real-time interactive generation with region-based semantic control,” 2024.
  * [10] I. Stravinsky, _Poetics of music: in the form of six lessons_ , 16th ed. Cambridge, Mass: Harvard University Press, 1970.
  * [11] J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y. Adi, and A. Défossez, “Simple and controllable music generation,” 2024.
  * [12] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” _CoRR_ , vol. abs/2112.10752, 2021.
  * [13] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, T. Salimans, J. Ho, D. J. Fleet, and M. Norouzi, “Photorealistic text-to-image diffusion models with deep language understanding,” 2022.
  * [14] Z. Kong, W. Ping, J. Huang, K. Zhao, and B. Catanzaro, “Diffwave: A versatile diffusion model for audio synthesis,” in _International Conference on Learning Representations_ , 2021. [Online]. Available: https://openreview.net/forum?id=a-xFK8Ymz5J
  * [15] S. Rouard and G. Hadjeres, “CRASH: Raw audio score-based generative modeling for controllable high-resolution drum sound synthesis,” in _Proc. ISMIR_ , Nov. 2021.
  * [16] G. Mittal, J. Engel, C. Hawthorne, and I. Simon, “Symbolic music generation with diffusion models,” in _Proc. ISMIR_ , 2021.
  * [17] A. Lugmayr, M. Danelljan, A. Romero _et al._ , “RePaint: Inpainting using denoising diffusion probabilistic models,” in _Proc. IEEE/CVF Conf. CVPR_ , Jun. 2022, pp. 11 461–11 471.
  * [18] E. Moliner and V. Välimäki, “Diffusion-based audio inpainting,” 2023.
  * [19] C.-J. Chang, C.-Y. Lee, and Y.-H. Yang, “Variable-length music score infilling via xlnet and musically specialized positional encoding,” 2021.
  * [20] L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in _IEEE International Conference on Computer Vision (ICCV)_ , 2023.
  * [21] S.-L. Wu, C. Donahue, S. Watanabe, and N. J. Bryan, “Music controlnet: Multiple time-varying controls for music generation,” 2023.
  * [22] Z. Novack, J. McAuley, T. Berg-Kirkpatrick, and N. J. Bryan, “DITTO: Diffusion inference-time t-optimization for music generation,” 2024.
  * [23] S. Forsgren and H. Martiros, “Riffusion - Stable diffusion for real-time music generation,” 2022. [Online]. Available: https://riffusion.com/about
  * [24] L. Min, J. Jiang, G. Xia, and J. Zhao, “Polyffusion: A Diffusion Model for Polyphonic Score Generation with Internal and External Controls,” Jul. 2023, arXiv:2307.10304 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2307.10304
  * [25] H. Wang, “DiffuseRoll: Multi-track multi-category music generation based on diffusion model,” Mar. 2023, arXiv:2303.07794 [cs, eess]. [Online]. Available: http://arxiv.org/abs/2303.07794
  * [26] K. Crowson, S. A. Baumann, A. Birch, T. M. Abraham, D. Z. Kaplan, and E. Shippole, “Scalable high-resolution pixel-space image synthesis with hourglass diffusion transformers,” 2024. [Online]. Available: https://arxiv.org/abs/2401.11605
  * [27] T. Karras, M. Aittala, T. Aila, and S. Laine, “Elucidating the design space of diffusion-based generative models,” in _Proc. NeurIPS_ , Dec. 2022.
  * [28] C.-P. Tan, A. W. Y. Su, and Y.-H. Yang, “Melody infilling with user-provided structural context,” in _Proc. ISMIR_ , 2022.
  * [29] C. Benetatos and Z. Duan, “Draw and listen! A sketch-based system for music inpainting,” in _Proc. ISMIR_ , 2022.
  * [30] Z. Wang, K. Chen, J. Jiang, Y. Zhang, M. Xu, S. Dai, G. Bin, and G. Xia, “Pop909: A pop-song dataset for music arrangement generation,” in _Proc. ISMIR_ , 2020.
  * [31] K. Choi, G. Fazekas, and M. B. Sandler, “Automatic tagging using deep convolutional neural networks,” in _Proc. ISMIR_ , vol. abs/1606.00298, 2016. [Online]. Available: http://arxiv.org/abs/1606.00298
  * [32] M.-E. Nilsback and A. Zisserman, “Automated flower classification over a large number of classes,” in _Indian Conference on Computer Vision, Graphics and Image Processing_ , Dec 2008.
  * [33] C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu, “Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models,” 2023.
  * [34] E. Moliner, J. Lehtinen, and V. Välimäki, “Solving audio inverse problems with a diffusion model,” in _Proc. IEEE ICASSP_ , 2023, pp. 1–5.
  * [35] K. Liu, W. Gan, and C. Yuan, “Maid: A conditional diffusion model for long music audio inpainting,” in _Proc. IEEE ICASSP_ , 2023, pp. 1–5.
  * [36] L. Lin, G. Xia, Y. Zhang, and J. Jiang, “Arrange, inpaint, and refine: Steerable long-term music audio generation and editing via content-based controls,” 2024.
  * [37] “Jacob collier can spell out words with a piano,” Jan 2021. [Online]. Available: https://www.youtube.com/watch?v=LszYGO22azA


# Debias the Training of Diffusion Models

Hu Yu1 , Li Shen2, Jie Huang1, Man Zhou1, Hongsheng Li3, Feng Zhao 1
1University of Science and Technology of China 2Alibaba Group
3The Chinese University of Hong Kong
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Work performed during internship at Alibaba DAMO Academy.Corresponding author.

###### Abstract

Diffusion models have demonstrated compelling generation quality by optimizing
the variational lower bound through a simple denoising score matching loss. In
this paper, we provide theoretical evidence that the prevailing practice of
using a constant loss weight strategy in diffusion models leads to biased
estimation during the training phase. Simply optimizing the denoising network
to predict Gaussian noise with constant weighting may hinder precise
estimations of original images. To address the issue, we propose an elegant
and effective weighting strategy grounded in the theoretically unbiased
principle. Moreover, we conduct a comprehensive and systematic exploration to
dissect the inherent bias problem deriving from constant weighting loss from
the perspectives of its existence, impact and reasons. These analyses are
expected to advance our understanding and demystify the inner workings of
diffusion models. Through empirical evaluation, we demonstrate that our
proposed debiased estimation method significantly enhances sample quality
without the reliance on complex techniques, and exhibits improved efficiency
compared to the baseline method both in training and sampling processes.

##  1 Introduction

Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) have emerged
as powerful generative models that garner significant attention recently.
Their popularity stems from the remarkable ability to generate diverse and
high-quality samples (Dhariwal & Nichol, 2021; Rombach et al., 2022; Ramesh et
al., 2022; Nichol & Dhariwal, 2021) as well as the training-stable loss form,
compared to the adversarial training paradigms used in Generative Adversarial
Networks (GANs) (Goodfellow et al., 2014). Diffusion models often serve as a
fundamental building block and have exhibited impressive success on a wide
rang of tasks (Ruiz et al., 2023; Saharia et al., 2022). While, it is usually
employed as a black-box component in these works. Some methods delve into the
methodology of diffusion models, particularly focusing on the reverse sampling
process. They typically aim to minimize the number of steps in the sampling
process to accelerate generation, achieved through designing more efficient
noise schedule (Song et al., 2020; Liu et al., 2021; Lu et al., 2022) or
progressive distilling (Salimans & Ho, 2022; Meng et al., 2023). These works
make commendable attempts to enhance fundamental diffusion models and provide
insights into the mechanisms concealed within the black box.

Here we focus on the perspective of training diffusion models, which
traditionally retain an elegantly simple loss function, i.e., the L2 loss with
constant weight between the Gaussian noise and the predicted one as follows:

$L=\sum_{t}\mathbb{E}_{x_{0},\epsilon}\left[||\epsilon-\epsilon_{\theta}(x_{t},t)||^{2}\right].$
(1)

Several works have recognized the issue of using this constant weighting and
proposed alternative weights and objectives (Choi et al., 2022; Salimans & Ho,
2022; Hang et al., 2023). Nonetheless, it is still an open question: whether
such a constant weighting form is not optimal and, if so, why and how it
affects the model’s performance.

In this paper, we theoretically demonstrate the suboptimality of the constant-
weighting loss formulation, revealing its potential to introduce biased
estimations during training and diminish the model’s performance.
Consequently, we propose an effective and elegant debiased loss weight to
address the issue, adhering to the debiased principle. Apart from the
theoretical proof and the debiased solution, more importantly, we figure out
several key questions crucial for achieving a systematical understanding of
the bias problem in conventional diffusion models, from its existence, impact,
and reasons. First, we show the existence of the biased estimation problem in
the training process. Concretely, the output of denoising network can be very
close to the target Gaussian noise at every step $t$, while, the corresponding
estimated $\hat{x}_{0}$ may severely deviates from $x_{0}$, which grows as $t$
gets larger. Then, we analyse the impact of the biased estimation problem on
the sampling process, which we call biased generation. Biased generation
mainly attributes to the chaos and inconsistency in the early few sampling
steps, which affects the final generation via error propagation effect. And
the final results of biased generation usually come with poor details, color
shift and global inconsistency. Additionally, we unravel the underlying causes
of biased estimation. The importance and optimization difficulty of the
denoising network is vastly different at different step $t$.

We empirically show that the proposed debiased estimation method is capable of
addressing the above problems and substantially elevates the sample quality.
Our method shows enhanced efficiency in both the training and sampling,
achieving superior performance to previous constant weighting strategy with
much less training iterations and sampling steps. All these are achieved by
slightly revising the loss weight strategy with only one additional line of
code, which is orthogonal to existing sampling accelerating methods. We expect
the in-depth analysis of dissecting the bias issue can provide a further
understanding for the research of diffusion models.

##  2 Background

Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) transform
complex data distribution $p_{data}(x)$ into simple noise distribution
$\mathcal{N}(0,\mathbf{I})$ and learn to recover data from noise. It contains
two processes: the forward diffusion process and the reverse denoise process.

The forward diffusion process starts from a clean data sample $x_{0}$ and
repeatedly injects Gaussian noise according to the transition kernel
$q(x_{t}|x_{t-1})$ as follows:

$q(x_{t}|x_{t-1})=N(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}I),\\\ $ (2)

where $\beta_{t}$ can be learned or held constant as hyper-parameters,
controlling the variance of noise added at each step. For example, Ho et al.
(Ho et al., 2020) employed a linear noise schedule, and Nichol et al. (Nichol
& Dhariwal, 2021) applied a cosine schedule. From the Gaussian diffusion
process, we can derive closed-form expressions for the marginal1 distribution
$q(x_{t}|x_{0})$ as follows:

$x_{t}=\sqrt{\alpha_{t}}x_{0}+\sqrt{1-\alpha_{t}}\epsilon,$ (3)

where $\epsilon\sim\mathcal{N}(0,\mathbf{I})$ and
$\alpha_{t}:=\prod_{s=1}^{t}(1-\beta_{s})$. Note that the above-defined
forward diffusion formulation has no learnable parameters, and the reverse
diffusion step cannot be applied due to having no access to $x_{0}$ in the
inference stage. Therefore, we further introduce the learnable reverse denoise
process for estimating $x_{0}$ from $x_{T}$.

The reverse denoise process is trained to reverse the forward diffusion
process in Eq. 2 by learning the denoise network, with the current de facto
training objective being Eq. 1. Given a randomly sampled Gaussian noise
$x_{T}\sim\mathcal{N}(0,\mathbf{I})$, the sample iteratively gets less noisy
as follows:

$x_{t-1}=\frac{1}{\sqrt{1-\beta_{t}}}(x_{t}-\frac{\beta_{t}}{\sqrt{1-\alpha_{t}}}\epsilon_{\theta}(x_{t},t))+\sigma_{t}z,$
(4)

where $\sigma_{t}^{2}$ is a variance and $z\sim\mathcal{N}(0,\mathbf{I})$. Ho
et al. (Ho et al., 2020) used $\beta_{t}$ as $\sigma_{t}^{2}$.

Kingma et al. (Kingma et al., 2021) proposed the use of signal-to-noise ratio
(SNR) to represent the noise schedules in diffusion models. The SNR of the
intermediate noisy sample $x_{t}$ is calculated as the ratio of the squared
mean and variance, which is expressed as:

$\text{SNR}(t)=\alpha_{t}/(1-\alpha_{t}).$ (5)

##  3 Theoretical Exploration of the Inherent Bias

###  3.1 Training Objectives of Conventional Diffusion Models

Diffusion models are trained by optimizing a variational lower bound (VLB).
For each step $t$, the denoising score matching loss $L_{t}$ is the distance
between two Gaussian distributions, which can be rewritten as:

$\displaystyle
L_{t}=D_{KL}(q(x_{t-1}|x_{t},x_{0})~{}||~{}p_{\theta}(x_{t-1}|x_{t})),$ (6)

where the reverse diffusion step $q(x_{t-1}|x_{t},x_{0})$ and
$p_{\theta}(x_{t-1}|x_{t})$ can be expressed as follows:

$\displaystyle
q(x_{t-1}|x_{t},x_{0})=N(x_{t-1};\tilde{\mu}_{t}(x_{t},x_{0}),\tilde{\beta}_{t}\mathbf{I}),$
(7) $\displaystyle
p_{\theta}(x_{t-1}|x_{t})=N(x_{t-1};\mu_{\theta}(x_{t},t),{\textstyle\sum_{\theta}}(x_{t},t)),$

where
$\tilde{\mu}_{t}(x_{t},x_{0}):=\frac{\sqrt{\alpha_{t-1}}\beta_{t}}{1-\alpha_{t}}x_{0}+\frac{\sqrt{1-\beta_{t}}(1-\alpha_{t-1})}{1-\alpha_{t}}x_{t}$,
$\tilde{\beta}_{t}:=\frac{1-\alpha_{t-1}}{1-\alpha_{t}}\beta_{t}$, and the
variance ${\textstyle\sum_{\theta}}(x_{t},t)=\sigma_{t}^{2}\mathbf{I}$. Ho et
al. (Ho et al., 2020) set $\sigma_{t}^{2}=\beta_{t}$. Thus, we can rewrite
$L_{t}$ as follows:

$\displaystyle L_{t}$
$\displaystyle=\mathbb{E}_{x_{0},\epsilon}\left[\frac{1}{2\sigma_{t}^{2}}\left\|\frac{\sqrt{\alpha_{t-1}}\beta_{t}}{1-\alpha_{t}}x_{0}+\frac{\sqrt{1-\beta_{t}}(1-\alpha_{t-1})}{1-\alpha_{t}}x_{t}-\bm{\mu}_{\theta}\left(\mathbf{x}_{t},t\right)\right\|^{2}\right]+C$
(8)
$\displaystyle=\mathbb{E}_{x_{0},\epsilon}\left[\frac{\beta_{t}^{2}}{(1-\beta_{t})(1-\alpha_{t})}||\epsilon-\epsilon_{\theta}(x_{t},t)||^{2}\right]+C.$

The denoising network is indeed optimized to approach $x_{0}$, and $\epsilon$
can also be employed as training target with a deterministic relationship to
$x_{0}$. Ho et al. (Ho et al., 2020) demonstrated that using $\epsilon$ as the
training target empirically outperforms training directly to predict $x_{0}$.
Additionally, they empirically observed that the simplified objective (Eq. 1)
with constant-weighting form yields better sample quality, which subsequently
becomes the default training objective of diffusion models.

###  3.2 Constant Weighting Introduces Bias in Training

We treat $\epsilon$ as the explicit and direct target, and $x_{0}$ as the
implicit but intrinsic target. Given the predicted noise
$\epsilon_{\theta}(x_{t},t)$ of the denoising network, we can derive
$\hat{x}_{0}$ from Eq. 3 as follows:

$\displaystyle\hat{x}_{0}$
$\displaystyle=\frac{1}{\sqrt{\alpha_{t}}}x_{t}-\frac{\sqrt{1-\alpha_{t}}}{\sqrt{\alpha_{t}}}\epsilon_{\theta}(x_{t},t)$
(9)
$\displaystyle=\frac{1}{\sqrt{\alpha_{t}}}(\sqrt{\alpha_{t}}x_{0}+\sqrt{1-\alpha_{t}}\epsilon)-\frac{\sqrt{1-\alpha_{t}}}{\sqrt{\alpha_{t}}}\epsilon_{\theta}(x_{t},t)$
$\displaystyle=x_{0}+\frac{\sqrt{1-\alpha_{t}}}{\sqrt{\alpha_{t}}}(\epsilon-\epsilon_{\theta}(x_{t},t))$
$\displaystyle=x_{0}+\frac{1}{\sqrt{SNR(t)}}(\epsilon-\epsilon_{\theta}(x_{t},t)).$

Further, we can rewrite Eq. 9 to express $x_{0}$ in terms of two components:
the $\mathit{estimated\ \hat{x}_{0}}$ part and the $\mathit{amplified\ error}$
part.

$x_{0}=\underbrace{\hat{x}_{0}}_{estimated\
\hat{x}_{0}}+\underbrace{\frac{1}{\sqrt{SNR(t)}}(\epsilon_{\theta}(x_{t},t)-\epsilon)}_{amplified\
error}.$ (10)

Although the difference between the predicted $\epsilon_{\theta}(x_{t},t)$ and
the target Gaussian noise $\epsilon$ may be very small at every step, the
amplification coefficient $\frac{1}{\sqrt{\text{SNR($t$)}}}$ is expected to be
significantly larger as the step $t$ increases (as shown in Fig. 1), which
would result in a substantial deviation of the estimated $\hat{x}_{0}$ from
the target $x_{0}$. We also visualize the estimated $\hat{x}_{0}$ and the
amplified error at different timesteps via feeding
$x_{t}=\sqrt{\alpha_{t}}x_{0}+\sqrt{1-\alpha_{t}}\epsilon$ into the denoising
network once. The $\mathit{estimated\ \hat{x}_{0}}$ increasingly deviates from
the ground-truth $x_{0}$ when $t$ grows larger, meanwhile the
$\mathit{amplified\ error}$ becomes larger and even gradually approaches
$x_{0}$. In this regard, we can find that the constant training weight
strategy is indeed biased, and optimizing the explicit target $\epsilon$
uniformly across different timesteps cannot guarantee approaching the implicit
target $x_{0}$ in an optimal manner.

Figure 1: Left: The visualization of SNR($t$) and amplification coefficient
$\frac{1}{\sqrt{\text{SNR($t$)}}}$ at different timesteps. Right: The upper
row is the input $x_{t}$ at different timesteps. We employ the diffusion model
(Dhariwal & Nichol, 2021) pretrained on ImageNet dataset to obtain the
$\mathit{estimated\ \hat{x}_{0}}$ part and $\mathit{amplified\ error}$ part of
each input $x_{t}$. The second row is the $\mathit{estimated\ \hat{x}_{0}}$.
The bottom row is the corresponding $\mathit{amplified\ error}$ part.
Apparently, as step $t$ gets larger, the $\mathit{estimated\ \hat{x}_{0}}$
severely deviates from $x_{0}$ and the $\mathit{amplified\ error}$ part
gradually approaches $x_{0}$.

###  3.3 Debiased Training Strategy

The above theoretical analysis provides a principled guidance to eliminate the
biased estimation problem. Concretely, we not only expect the loss function to
reach the explicit target $\epsilon$ but also encourage the estimated
$\hat{x}_{0}$ to approach the implicit target $x_{0}$. Therefore, it is
essential to take into account the varying impact of noise prediction at
different steps $t$ when designing the loss function. In this regard, we
propose a debiased loss formulation which is simple but effective by taking
the amplified coefficient into account:

$L=\sum_{t}\mathbb{E}_{x_{0},\epsilon}\left[\frac{1}{\sqrt{SNR(t)}}||\epsilon-\epsilon_{\theta}(x_{t},t)||^{2}\right].$
(11)

In other word, we assign higher weight as the step $t$ increases (i.e., when
adding more noise to $x_{0}$), thereby compelling the noise error
$(\epsilon_{\theta}(x_{t},t)-\epsilon)$ to decrease more significantly at
larger step $t$. Note that we use the above form instead of the seemingly more
reasonable weighting $\frac{1}{SNR}$ as it would hinder the optimization of
the explicit target $\epsilon$. We explain it in detail in Appendix A.

##  4 Comprehensive understanding the bias problem

In this section, We aim to address several key questions crucial for achieving
a systematical understanding of the bias problem in conventional diffusion
models: Why is the bias problem important? What are its effects? And what is
the underlying cause ? We believe answering these questions is essential for
unraveling and dissecting the black box of diffusion models.

###  4.1 Biased estimation in the training process

First, we illustrate the one-step estimation $\hat{x}_{0}$ in Fig. 2 to
compare the results obtained using the original constant weighting and our
variant. There is a general tendency for the estimated $\hat{x}_{0}$ of both
weighting strategies to gradually deviate from the original $x_{0}$ as the
step $t$ increases, which is inevitable due to the increasing noise in the
input $x_{t}$. However, when utilizing the constant weighting loss for
training, noticeable color shifts and inferior arrangement of human faces can
be observed in the early steps ($t=999$ and $t=950$), severely deviating from
the target $x_{0}$. In contrast, our strategy effectively reduces the bias,
achieving greater consistency with the targets across various timesteps, even
under relatively high noise levels (e.g., at $t=999$ and $t=900$). These
findings indicate that the proposed debiased formulation facilitates training
in a more appropriate direction. More analyses are available in Appendix B.

Figure 2: We present the one-step estimation results of $\hat{x}_{0}$ using
different input samples $x_{t}$, where the diffusion models are pretrained on
the FFHQ dataset (Karras et al., 2019) with different loss weighting
strategies. One-step estimation: start from a clean image and add noise to get
$x_{t}$ according to Eq. 3. Then put $x_{t}$ into the denoising network once
to get the estimated noise, and finally get the estimated $\hat{x}_{0}$ by
reversing Eq. 3. The top row displays the results obtained using a well-
trained constant weighting model, while the bottom row showcases the results
achieved with our well-trained debiased weighting model.

###  4.2 Biased generation on the sampling process

We further analyse the detrimental effects of the biased estimation problem
introduced by the constant weighting loss for model inference, i.e., biased
generation on the sampling process. As seen in the first two rows in Fig. 3,
biased generation primarily attributes to the chaos and inconsistency in the
early few sampling steps, which substantially affects the final generation
through error propagation. We particularly observe pronounced color shifting
in biased generation when employing a small number of sampling steps (e.g.,
$T=2$), which remains challenging to correct even with an extended sampling
process (e.g., $T=1000$). In contrast, training with our strategy can
essentially prevent the issue (e.g., the shown images with $T=2$), eliminating
the need for a lengthy correction process. Moreover, generated images using
our strategy show enhanced details and global consistency compared to the
baseline method. More analyses are available in Appendix C.

Figure 3: Sampling results with different total steps $T$. The upper two rows
are the generated samples of constant weight. The bottom two rows are the
generated samples of our improved version.

###  4.3 Unraveling the Underlying Causes of Biased Estimation

Finally, we take one step further to unravel the underlying causes of biased
estimation. Specifically, the optimization difficulty and importance of the
denoising network is vastly different across step $t$.

#### Different optimization difficulty

Intuitively, the input $x_{t}$ is closer to the target as step $t$ becomes
larger. Consequently, the network encounters varying levels of fitting
difficulty across different values of $t$, with larger values of $t$ being
relatively easier. To verify this, we plot the Mean squared error (MSE)-step
curve under several settings in Fig. 4. In the “Initial” setting, the MSE
value is directly computed between the network input $x_{t}$ and the target
Gaussian noise. The remaining two settings compute the MSE value between the
network output and the target Gaussian noise, with “Constant” representing the
constant weighting method and “Ours” representing the debiased weighting
strategy. The distribution of MSE value under “Initial” mode is extremely
unbalanced, in which the MSE value is negligible when $t>600$. Consequently,
this imbalance endows different optimization difficulty across step $t$ and
renders the constant weighting strategy suboptimal. Specifically, when $t$ is
sufficiently large, the MSE value between the network input and the target
becomes extremely small, allowing the network to “do nothing” while still
maintaining a low MSE loss.

The above analysis is verified in Fig. 4. For $t>950$, the MSE value in
constant weight mode surpasses that of the “Initial” mode, indicating the
output deviates even further from the target than the input. This observation
illustrates that the denoising network in constant weight setting fails to
identify the noise pattern in the input and, therefore, cannot effectively
handle the denoising task. In contrast, our weight strategy consistently
yields MSE values lower than those of the “Initial” mode, demonstrating its
exceptional denoising capability, particularly for highly noisy inputs.

Figure 4: MSE-step curve under several settings. “Initial” mode is calculated
between input and target. Obviously, the optimization difficulty is vastly
different across step $t$. “Constant” and “Ours” modes are calculated between
network output and target. “Constant” denotes constant weight strategy and
“Ours” stands for our debiased weight strategy. Note that “Constant” and
“Ours” visually overlap in the left figure due to large scale .

#### Different importance

The importance of the denoising network varies across step $t$. Intuitively,
initial steps are important for both training and sampling process. For
training, the initial steps pose greater difficulty due to the presence of
high noise levels in the input. For sampling, the initial steps serve as the
foundation for subsequent steps, contributing to error propagation.
Theoretically, we have verified that initial steps should be emphasized to
reach the implicit target $x_{0}$ in section 3. Additionally, we also find
evidence supporting the crucial role of initial steps in diffusion models
(Nichol & Dhariwal, 2021; Wang et al., 2023). For example, Nichol et al.
(Nichol & Dhariwal, 2021) demonstrated that the first few steps of the
diffusion process contribute the most to the variational lower bound. Wang et
al. (Wang et al., 2023) found that reusing update directions from initial
steps with adaptive momentum sampler can generate images with enhanced low-
level details. The constant weighting strategy assumes equal importance across
all steps. While, our method assigns higher weights to the initial steps,
which is consistent with both intuition and theory.

##  5 Experiments

###  5.1 Setup

#### Datasets.

We perform experiments on unconditional image generation using the FFHQ
(Karras et al., 2019), AFHQ-dog (Choi et al., 2020), and MetFaces (Karras et
al., 2020a) datasets. These datasets contain approximately 70k, 50k, and 1k
images respectively. We resize and center-crop data to 256×256, following the
pre-processing performed by ADM (Dhariwal & Nichol, 2021).

#### Training Details.

We set T = 1000 for all experiments. We implement the proposed approach on top
of ADM (Dhariwal & Nichol, 2021), which offers well-designed architecture and
efficient sampling. We train our model for 500K iterations with a batch size
of 8.

#### Evaluation Settings.

Following the common practice (Song & Ermon, 2020), we utilize an Exponential
Moving Average (EMA) model with a rate of 0.9999 for all experiments. Besides,
we generate 50K samples for each trained model and use the full training set
to compute the reference distribution statistics, following (Ho et al., 2020;
Choi et al., 2022). During inference, we obtain results with fewer sampling
steps than T by employing the respacing technique. For quantitative
evaluations, we employ the Fréchet Inception Distance (FID) (Heusel et al.,
2017).

###  5.2 Comparison to existing weighting strategies

#### Unified perspective on existing weighting strategies

Figure 5: Left: Visualization of various weighting strategies. P2 and Min-SNR
starts from the basis of constant weight and lower the weight down for small
$t$. Right: Sampling results with different total sampling steps $T$. From top
to bottom, they are constant, P2, Min-SNR, and our method. Evidently, P2 and
Min-SNR still suffer from bias and artifacts during the initial generation
stage.

The majority of existing methods (Ho et al., 2020; Dhariwal & Nichol, 2021;
Rombach et al., 2022) adhere to the prevailing training objective of
predicting Gaussian noise using a constant weight. Several prior methods (Choi
et al., 2022; Kingma et al., 2021; Salimans & Ho, 2022; Hang et al., 2023)
investigate alternative training targets and weighting strategies. For
example, P2 (Choi et al., 2022) proposes a weighting scheme to prioritize
higher noise levels that recover content information. Min-SNR (Hang et al.,
2023) treats the diffusion training as a multi-task learning problem and
designs a weighting strategy to avoid the model focusing too much on small
noise levels. Salimans et al. (Salimans & Ho, 2022) proposed some more
weighting strategies. We present a visualization of these distinct weighting
strategies in Fig. 5 and refer to them as SNR-aware weighting strategies.

Our method differs from these existing methods in several key aspects. (1)
Most importantly, they struggle to uncover the core: existing constant
weighting strategy is biased and sub-optimal. Thus, they lack the ability to
provide insights and future directions to the followers.(2) They intuitively
modified the weight on the basis of the sub-optimal constant weighting. For
instance, compared with constant weight, they only lower the weight for small
$t$, keeping the weight unchanged for the remaining substantial portion of the
steps. Consequently, they encounter difficulties in establishing general
principles for guiding the design of the weighting strategy.

A unified perspective can be adopted on these existing SNR-aware weighting
strategies within the framework of the debiased principle. Our theoretically
unbiased principle elucidates that the weight should monotonically increase
and assign higher weights to large $t$, similar to the red curve depicted in
Figure 5. And prior methods tend to assign lower weights to small $t$ values,
adhering to the principle overall. This also elucidates the underlying nature
and rationale for their superior performance compared to constant weight.
However, without identifying the inherent biased problems, these intuitively
designed weighting methods still cannot achieve optimal performance. We also
compare different training targets in Appendix D.

#### Quantitative comparison.

Tab. 1 presents a quantitative performance comparison of various weighting
strategies across different sampling steps $T$. Our method achieves the
highest performance across multiple datasets and sampling steps. It is worth
noting that the performance gain is particularly pronounced with smaller
datasets and shorter sampling steps. This observation indicates the
generalization and robustness of our method.

Table 1: Quantitative comparison. The experimental results are reported in terms of FID under a fair setting, with the only distinction being the loss weighting strategy. * denotes the results reported in the original paper. However, as certain essential training details of P2* (e.g., training iterations) are unknown, its reported values are used for reference only. Dataset | Step $T$ | Constant | P2 | P2* | Min-SNR | Ours
---|---|---|---|---|---|---
FFHQ | 1000 | 10.8636 | 6.5173 | 6.92 | 6.5012 | 6.3537
500 | 11.0266 | 6.7919 | 6.97 | 6.8733 | 6.7055
250 | 11.7802 | 7.4777 | - | 7.7219 | 7.3854
100 | 15.6705 | 10.8546 | - | 11.3910 | 10.8154
50 | 22.3752 | 16.5376 | - | 17.3284 | 15.3447
20 | 41.2703 | 34.3992 | - | 34.6515 | 29.3796
AFHQ-dog | 1000 | 18.2999 | 17.0680 | 11.55 | 17.3418 | 14.9284
500 | 18.6062 | 17.4743 | - | 17.6393 | 14.9461
250 | 19.1036 | 17.7591 | 11.66 | 17.9216 | 15.0327
| 100 | 20.4464 | 18.3439 | - | 18.4205 | 15.8210
MetFaces | 1000 | 41.4184 | 14.2044 | - | 30.8761 | 9.1683
500 | 42.1150 | 14.4476 | - | 31.1675 | 9.4289
250 | 42.3241 | 14.7384 | 36.80 | 31.3396 | 9.8486
| 100 | 42.6236 | 14.9937 | - | 31.6259 | 10.3884

#### Qualitative comparison.

Fig. 6 presents the qualitative results. As anticipated, the biased constant
weighting strategy produces images with inferior global structure and color
alignment. P2 and Min-SNR enhance the sample quality by building upon the
constant weight foundation. However, they still produce images with inferior
global structure. This is due to their significant bias and chaotic behavior
during the initial sampling steps, as depicted in Fig. 5.

Figure 6: Visual results of different weighting strategies on different
datasets. We randomly choose the first nine generated images without cherry-
pick. The first row is trained on FFHQ dataset and the second row is on AFHQ-
dog dataset.

#### High efficiency.

Fig. 7 illustrates the FID-training iterations curve and the FID-sampling
steps curve for the FFHQ dataset. The training curve clearly demonstrates the
superior efficiency and potential of our method. For instance, our weighting
strategy matches the performance of 1000k iterations of constant weight
training with only 400k iterations. In terms of sampling, our method surpasses
all existing weight strategies across all sampling steps. Moreover, consistent
with the analysis in section 4.2, the performance gains are more pronounced
with fewer sampling steps.

Figure 7: Left: FID-training iterations curve. Right: FID-sampling steps
curve. These two curves are obtained on FFHQ dataset. Our method is more
efficient and high-performing. Note that, we use DDPM to denote the constant
weighting strategy.

###  5.3 Comparison to the Prior Literature

Generative models have enabled unprecedented and photorealistic synthesis
(Brock et al., 2018; Karras et al., 2019; Sauer et al., 2021; Esser et al.,
2021; Rombach et al., 2022; Ho et al., 2022; Bao et al., 2023). These methods
achieve high performance through meticulously crafted architectures and
designs. Our method offers a general strategy for diffusion models, achieving
competitive performance without relying on intricate techniques. Additionally,
our contribution complements these methods and can further extend their
performance potential. For example, we achieve substantial improvements by
only changing the loss weight on top of ADM. We provide a quantitative
comparison and discussion in Appendix E.

###  5.4 Broader impact and future work

Given that diffusion models usually serve as fundamental building blocks for
various application-oriented works, our method provides valuable inspiration
and insights for these endeavors. Additionally, we identify several potential
avenues for future research within the community. (1) The elucidated mechanism
behind the biased problem offers valuable insights for downstream tasks, such
as editing and restoration, facilitating the integration of the bias issue
into specific tasks. (2) The biased problem can be investigated from other
perspectives, such as noise schedule (Ning et al., 2023; Chen, 2023). It is
encouraging to discuss the defects of diffusion models from a unified
perspective. For example, one can explore the characteristics and shortcomings
from the design space of diffusion models (Karras et al., 2022), and elucidate
their relationships and correlations.

##  6 Conclusion

This paper provides theoretical analyses and comprehensive studies to
demonstrate that the traditional uniform weighting loss function is prone to
causing biased estimations during the training of diffusion models, by
examining the existence, impact, and underlying reasons behind this issue. To
mitigate this problem, we employ a simple yet highly effective weighting
strategy that adheres to the theoretically unbiased principle. Empirical
studies conducted on multiple datasets, along with comparisons with existing
weight methods, further validate the effectiveness of our proposed approach.
We also believe these analyses contribute to a deeper understanding of the
underlying mechanism of diffusion models.

#### Reproducibility Statement

We are highly confident of the reproducibility of this work. We strictly
follow the baseline setting P2 (Choi et al., 2022), with detailed setting
description in section 5.1. We employ the official code of P2 and only modify
the loss weight with one additional line of code as follows:

# original constant weight mse_loss = (target - model_output) ** 2

# ours weight = 1/torch.sqrt(SNR) mse_loss = weight * (target - model_output)
** 2

## References

  * Bao et al. (2023) Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu.  All are worth words: A vit backbone for diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 22669–22679, 2023.
  * Brock et al. (2018) Andrew Brock, Jeff Donahue, and Karen Simonyan.  Large scale gan training for high fidelity natural image synthesis.  _arXiv preprint arXiv:1809.11096_ , 2018.
  * Chen (2023) Ting Chen.  On the importance of noise scheduling for diffusion models.  _arXiv preprint arXiv:2301.10972_ , 2023.
  * Choi et al. (2022) Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon.  Perception prioritized training of diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 11472–11481, 2022.
  * Choi et al. (2020) Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha.  Stargan v2: Diverse image synthesis for multiple domains.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 8188–8197, 2020.
  * Dhariwal & Nichol (2021) Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  _Advances in Neural Information Processing Systems_ , 34:8780–8794, 2021.
  * Esser et al. (2021) Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 12873–12883, 2021.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  _Advances in neural information processing systems_ , 27, 2014.
  * Gu et al. (2022) Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo.  Vector quantized diffusion model for text-to-image synthesis.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 10696–10706, 2022.
  * Hang et al. (2023) Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo.  Efficient diffusion training via min-snr weighting strategy.  _arXiv preprint arXiv:2303.09556_ , 2023.
  * Hertz et al. (2022) Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  _arXiv preprint arXiv:2208.01626_ , 2022.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  _Advances in neural information processing systems_ , 33:6840–6851, 2020.
  * Ho et al. (2022) Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans.  Cascaded diffusion models for high fidelity image generation.  _The Journal of Machine Learning Research_ , 23(1):2249–2281, 2022.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 4401–4410, 2019.
  * Karras et al. (2020a) Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila.  Training generative adversarial networks with limited data.  _Advances in neural information processing systems_ , 33:12104–12114, 2020a.
  * Karras et al. (2020b) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 8110–8119, 2020b.
  * Karras et al. (2022) Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine.  Elucidating the design space of diffusion-based generative models.  _Advances in Neural Information Processing Systems_ , 35:26565–26577, 2022.
  * Kingma et al. (2021) Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models.  _Advances in Neural Information Processing Systems_ , 34:21696–21707, 2021.
  * Liu et al. (2021) Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao.  Pseudo numerical methods for diffusion models on manifolds.  In _International Conference on Learning Representations_ , 2021.
  * Lu et al. (2022) Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.  _Advances in Neural Information Processing Systems_ , 35:5775–5787, 2022.
  * Meng et al. (2023) Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans.  On distillation of guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 14297–14306, 2023.
  * Nichol et al. (2021) Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen.  Glide: Towards photorealistic image generation and editing with text-guided diffusion models.  _arXiv preprint arXiv:2112.10741_ , 2021.
  * Nichol & Dhariwal (2021) Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pp. 8162–8171. PMLR, 2021.
  * Ning et al. (2023) Mang Ning, Enver Sangineto, Angelo Porrello, Simone Calderara, and Rita Cucchiara.  Input perturbation reduces exposure bias in diffusion models.  _International Conference on Machine Learning_ , 2023.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10684–10695, 2022.
  * Ruiz et al. (2023) Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 22500–22510, 2023.
  * Saharia et al. (2022) Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(4):4713–4726, 2022.
  * Salimans & Ho (2022) Tim Salimans and Jonathan Ho.  Progressive distillation for fast sampling of diffusion models.  In _International Conference on Learning Representations_ , 2022.
  * Sauer et al. (2021) Axel Sauer, Kashyap Chitta, Jens Müller, and Andreas Geiger.  Projected gans converge faster.  _Advances in Neural Information Processing Systems_ , 34:17480–17492, 2021.
  * Schonfeld et al. (2020) Edgar Schonfeld, Bernt Schiele, and Anna Khoreva.  A u-net based discriminator for generative adversarial networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 8207–8216, 2020.
  * Sohl-Dickstein et al. (2015) Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli.  Deep unsupervised learning using nonequilibrium thermodynamics.  In _International Conference on Machine Learning_ , pp. 2256–2265. PMLR, 2015.
  * Song et al. (2020) Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  In _International Conference on Learning Representations_ , 2020.
  * Song & Ermon (2020) Yang Song and Stefano Ermon.  Improved techniques for training score-based generative models.  _Advances in neural information processing systems_ , 33:12438–12448, 2020.
  * Tumanyan et al. (2023) Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel.  Plug-and-play diffusion features for text-driven image-to-image translation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 1921–1930, 2023.
  * Wang et al. (2023) Xiyu Wang, Anh-Dung Dinh, Daochang Liu, and Chang Xu.  Boosting diffusion models with an adaptive momentum sampler.  _arXiv preprint arXiv:2308.11941_ , 2023.

Appendix

##  Appendix A Discussion on the theoretically optimal weight: $\frac{1}{SNR}$

As we have mentioned, we didn’t employ the seemingly more reasonable loss
weight $\frac{1}{SNR}$. We use “SNR” to denote the weight of $\frac{1}{SNR}$.
We demonstrate the reason from two aspects. (1) SNR weight damages the
explicit target without further boosting the implicit target as shown in Fig.
8. Concretely, the MSE of the SNR mode completely overlaps with our
$\frac{1}{\sqrt{SNR}}$ weigh strategy for large $t$, indicating that SNR
weight can’t further boost the implicit target. However, the MSE of the SNR
mode is substantially larger than all other weighting strategies for small
$t$, indicating that the explicit target is seriously violated. The reason
behind this is the excessive range field of SNR weight ranging from $10^{-4}$
to $10^{4}$, which causes the denoising network excessively focusing on few
early steps. (2) Empirically, the SNR mode performs terribly, as shown in Fig.
9. Min-SNR (Hang et al., 2023) also explores predicting Gaussian noise with
the weight of $\frac{1}{SNR}$, and they find that this setting leads to
divergence. Thus, our experimental result is also consistent with the
conclusion of Min-SNR.

Figure 8: MSE analysis of different weighting strategies. We use “SNR” to
denote the weight of $\frac{1}{SNR}$. SNR weight can’t further lower the MSE
for large $t$. On the contrary, it’s denoising ability is worsen for small $t$
with large MSE. The reason behind this is the excessive range field of SNR
weight ranging from $10^{-4}$ to $10^{4}$, which causes the denoising network
excessively focusing on few early steps. For example, for batchsize=8, if one
$t$ is large and the remaining seven $t$ are small, the network will pay
excessive attention to the large $t$ with high weight, while at the cost of
sacrificing the seven small $t$s. In contrast, our method achieves the lowest
MSE across these weighting strategies at different step $t$, only slightly
larger than the constant weight for $t<50$. Figure 9: First ten generated
sample of ”SNR” weight. This weight strategy leads to divergence and poor
sample quality.

##  Appendix B More analyses of the biased estimation in the training process

In the main manuscript, we show the existence of biased estimation in the
training process with the one-step estimated $\hat{x}_{0}$. In this section,
we further illustrate the bias problem exploiting the intermediate feature
maps at different step $t$ in Fig. 10. Fig. 10 is achieved in the same way as
Fig. 2 via visualizing the feature maps of the one-step estimation process at
various step $t$.

The intermediate feature maps can reflect the structures underlying the noisy
samples (Hertz et al., 2022; Tumanyan et al., 2023). Consistent with the
conclusion in Fig. 2, the constant weight mode struggles to generate facial
structures for large step $t$ ($t>$900). In contrast, our method can generate
clear facial layout even with the most noisy $x_{999}$ as input.

Figure 10: The intermediate feature maps at different steps $t$. Constant
strategy struggles to generate clear facial architecture with noisy $x_{t}$ as
input ($t>$900). In contrast, our method can generate clear facial layout even
with the most noisy $x_{999}$ as input.

##  Appendix C More analyses of the biased generation

In the main manuscript, we indicate the biased generation with generated
samples of different total sampling steps $T$. In this section, we show more
analyses and visualization of the biased generation. As shown in Fig. 11 and
Fig. 12, we respectively show the estimated $\hat{x}_{0}$ and intermediate
feature maps of different weighting strategies at different step $t$. The
feature visualization method is similar to prompt-to-prompt (Hertz et al.,
2022).

From Fig. 11, we observe that constant weight exhibits artifacts and color
shift in the first generation step, resulting in the final generated images
with color and structure distortion. P2 and Min-SNR also show global artifacts
and inconsistency in the first generation step. Thus, their generated images
suffer from poor structures. On the contrary, our method is free of artifacts
and color distortion in the whole generation process.

From Fig. 12, we observe that constant, P2, and Min-SNR weight strategies
struggle to generate clear facial architecture in the early generation steps.
Besides, their intermediate feature maps of the final step also demonstrate
poor global consistency. In contrast, our method demonstrates clear facial
architecture even at very early steps, and the final feature maps are also
more visually pleasing.

Figure 11: Biased generation: the estimated $\hat{x}_{0}$ of different
weighting strategies at different step $t$. Constant weight exhibits artifacts
and color shift in the first generation step, resulting in the final generated
images with color and structure distortion. P2 and Min-SNR also show global
artifact and inconsistency in the first generation step. Thus, their generated
images suffer from poor structures. Figure 12: Biased generation: the
intermediate feature maps of different weighting strategies at different steps
$t$. Constant, P2, and Min-SNR weight strategies struggle to generate clear
facial architecture in the early generation steps. Our method shows clear
facial architecture even at very early steps.

##  Appendix D Different training targets

In this section, we delve into the difference between $x_{0}$ prediction,
$\epsilon$ prediction and $v$ prediction. Most previous works (Nichol &
Dhariwal, 2021; Dhariwal & Nichol, 2021; Nichol et al., 2021; Rombach et al.,
2022) follow DDPM (Ho et al., 2020) to predict the noise $\epsilon$. Some
works (Salimans & Ho, 2022; Gu et al., 2022) use reparameterization to predict
$x_{0}$. And some other works (Salimans & Ho, 2022) employ the network to
predict $v\equiv\alpha_{t}\epsilon-\sigma_{t}x_{0}$.

Predicting different targets is mathematically equivalent. However, different
prediction targets inherently correspond to different optimizing difficulty.
$\epsilon$ prediction is theoretically easiest as the distribution of the
optimizing target is simple and fixed. This also explains why predicting
Gaussian noise $\epsilon$ with constant weight is most widely employed and
becomes the de facto component of diffusion models. Significantly, this
further validates the importance and meaning of our work unlocking the biased
problem in $\epsilon$ prediction mode.

We also show the performance comparison of different training targets in Tab.
2. Obviously, $\epsilon$ prediction is superior to $x_{0}$ prediction and $v$
prediction. This is also consistent with the conclusion of (Hang et al.,
2023), which finds that predicting $\epsilon$ yields higher quality.

Table 2: Quantitative comparison of different training targets.  |  $x_{0}$ prediction |  $v$ prediction |  $\epsilon$ prediction | Ours ($\epsilon$ prediction with debiased weight)
---|---|---|---|---
FID | 17.8148 | 19.1426 | 10.8636 | 6.3537

##  Appendix E Comparison to the Prior Literature

We conducted a comparative analysis between diffusion models trained using our
approach and existing models on the FFHQ dataset (Karras et al., 2019), as
presented in Tab. 3. Previous generative models(Brock et al., 2018; Karras et
al., 2019; Sauer et al., 2021; Esser et al., 2021; Rombach et al., 2022; Ho et
al., 2022; Bao et al., 2023) have achieved exceptional results in
photorealistic synthesis by employing meticulously designed architectures and
methodologies.

In contrast, our method achieves competitive performance with a simple loss
weight strategyinstead of relying on intricate techniques. Besides, our method
is a general strategy of diffusion models and has the potential to enhance
their performance limits. For instance, we achieved substantial improvements
by solely adjusting the loss weight on top of ADM (Dhariwal & Nichol, 2021),
reducing the FID score from 10.86 to 6.35. Moreover, our method offers the
capability to achieve even higher performance. Firstly, we can extend the
training duration. For instance, with 500k iterations, our method achieves a
FID of 6.35, while with 1000k iterations, it achieves a FID of 4.97.
Additionally, we have the flexibility to replace the codebase ADM with a
stronger model, such as stable diffusion.

Table 3: Quantitative comparison to prior generative models on FFHQ dataset. Our method is on top of ADM with only one additional line of code, yet achieving substantial performance lift. Dataset | Method | Type | FID
---|---|---|---
FFHQ | BigGAN (Brock et al., 2018) | GAN | 12.4
UNet GAN (Schonfeld et al., 2020) | GAN | 10.9
StyleGAN (Karras et al., 2019) | GAN | 4.16
StyleGAN2 (Karras et al., 2020b) | GAN | 3.73
VQGAN (Esser et al., 2021) | GAN+AR | 9.6
LDM (Rombach et al., 2022) | Diffusion model | 4.98
ADM (Baseline) (Dhariwal & Nichol, 2021) | Diffusion model | 10.86
Ours (500k iterations) | Diffusion model | 6.35
| Ours (1000k iterations) | Diffusion model | 4.97

##  Appendix F Different samplers

Our weighting strategy is orthogonal to samplers. In this part, we analyse the
effect of DDIM sampler (Song et al., 2020) on the sample quality. As shown in
Fig. 13, we visualize the generated samples of DDIM sampler under four
different weighting strategies. Similar to the conclusion in Fig. 6 of the
main manuscript, our method achieves the highest performance with DDIM sampler
among these four weighting strategies.

Figure 13: The generated samples of DDIM sampler under four different
weighting strategies.

##  Appendix G More visual results

In this part, we show more visual results of different weighting strategies on
various datasets to further validate the effectiveness and robustness of our
method. Fig. 14, 15 , and 16 show the visual results on FFHQ (Karras et al.,
2019), AFHQ-dog (Choi et al., 2020), and MetFaces (Karras et al., 2020a)
datasets, respectively.

Figure 14: More visual results on FFHQ dataset. Figure 15: More visual results
on AFHQ-dog dataset. Figure 16: More visual results on MetFaces dataset.