# Balanced SNR-Aware Distillation for Guided Text-to-Audio Generation

###### Abstract

Diffusion models have demonstrated promising results in text-to-audio
generation tasks. However, their practical usability is hindered by slow
sampling speeds, limiting their applicability in high-throughput scenarios. To
address this challenge, progressive distillation methods have been effective
in producing more compact and efficient models. Nevertheless, these methods
encounter issues with unbalanced weights at both high and low noise levels,
potentially impacting the quality of generated samples. In this paper, we
propose the adaptation of the progressive distillation method to text-to-audio
generation tasks and introduce the Balanced SNR-Aware (BSA) method, an
enhanced loss-weighting mechanism for diffusion distillation. The BSA method
employs a balanced approach to weight the loss for both high and low noise
levels. We evaluate our proposed method on the AudioCaps dataset and report
experimental results showing superior performance during the reverse diffusion
process compared to previous distillation methods with the same number of
sampling steps. Furthermore, the BSA method allows for a significant reduction
in sampling steps from 200 to 25, with minimal performance degradation when
compared to the original teacher models.

Index Terms—  text-to-audio, progressive distillation, latent diffusion

##  1 Introduction

Diffusion-based text-to-audio (TTA) generation techniques have emerged as a
promising approach to AI content creation[1, 2, 3, 4, 5] that utilizes a
diffusion model to transform text into high-quality audio[3, 6, 1, 7, 8] that
matches the content being created, which is valuable in domains such as movie
production and gaming, where immersive audio experiences are critical. However
its inference requires many sampling steps, hence is slow, which hinders its
potential utilization. To address this challenge, Salimans et al.[9]
introduced the concept of progressive distillation, aimed to increase the
sampling rate. However, their method tended to exhibit a bias towards an
excessive focus on low noise levels during distillation, resulting in models
that underperformed when subjected to high noise levels. Subsequent attempts,
such as Hang et al.’s[10] adjustment to the training loss weight function,
alleviated the overemphasis on low noise levels during training. However, this
method overlooked the long-term exposure of models to high noise levels in
later stages of progressive distillation, thereby leading to a bias in the
model’s predictive capabilities.

In response to these challenges, we introduce a Balanced SNR-Aware (BSA)
method to weights loss functions in a SNR balanced manner. BSA method
mitigates the overemphasis on small noise levels while preserving the model’s
effectiveness in predicting audio samples in high noise levels. We incorporate
this loss-weighting strategy in the framework of progressive distillation. On
this framework we redefine the progressive distillation algorithm to make it
suitable for text-to-audio production tasks. Without loss of generality, we
perform progressive distillation on Tango [8], which is a language-guided
diffusion-based TTA model. Experimental results show that our proposed method
outperformes previous methods with same sampling steps ranging from 100 to 25.
In addition, we can reduce the sampling steps from 200 to 25 with little
performance degradation compared with original teacher models.

In summary, the contributions of this paper are as follows:

  * •

We provide a novel representation of how the progressive distillation method
can be adapted to Text-to-Audio (TTA) generation tasks. This application
significantly reduces sampling steps of latent diffusion models, making them
more practical for high-throughput scenarios.

  * •

We propose a Balanced Signal-to-Noise-Ratio-Aware (BSA) method, which is a
loss-weighting strategy that allows the model to balance the signal-to-noise
ratio during distillation, which can effectively improve performance of
distilled models.

We organize this paper as follows. Section 2 presents related works on
diffusion models and knowledge distillation. Section 3.1 describes the
proposed Balanced Signal-to-Noise Ratio Perceived Loss. Section 4 presents our
experimental settings and results. Section 5 summarizes our work.

Fig. 1: In the process of distillation, we start with audio-text pairs from
the dataset. After encoding the audio data and introducing noise in the
forward process, a teacher model uses two steps of DDIM guided by the text
data to predict $\tilde{z}_{0}$ from the noisy data $z_{t}$. This prediction
$\tilde{z}_{0}$ becomes the target for the student model. The student model
then generates $\hat{z}_{0}$ in a single step using text guidance, and this
result is used to calculate the loss in conjunction with the teacher model’s
output.

##  2 Related Works

###  2.1 Diffusion

For TTA generation, Diffsound[6] employed a discrete representation of the
compressed Mel-spectrum to efficiently model the data, resulting in a
generation speed five times faster than an autoregressive (AR) token decoder.
However, the TTA generation process in Diffsound still involves hundreds of
sampling steps. In contrast, AudioLDM[7] took a different approach, utilizing
a latent-representation-based method. Their results showed that utilizing
contrastive language audio pretraining (CLAP) as an audio encoder to allow
audio-only data to be used in the training process can improve the
computational efficiency of the TTA system. However, it is worth noting that
the generation process of AudioLDM is still not improved. Tango[8] utilized
FLAN-T5[11] to replace the previous CLAP. Although the quality of the
generated samples has improved, there is still no improvement in generation
speed.

###  2.2 Knowledge Distillation

Hinton et al.[12] and Oord et al.[13] introduced knowledge distillation,
allowing for smaller and more efficient models while maintaining performance.
Salimans et al.[9] proposed progressive distillation, reconstructing the loss
weight function to train student models with one-step predictions matching the
teacher model’s two-step predictions. However, this approach lacks an upper
limit on the loss weight function, leading to high sensitivity to minor
changes in predictions, especially during initial training stages with high
SNR. Hang et al.[10] improved training by introducing an upper limit for the
loss weight function, reducing the model’s focus on low noise levels. However,
they didn’t establish a lower limit, causing challenges during later training
iterations with high noise levels and low SNR. This increased uncertainty
makes training less effective when faced with high noise levels in later
stages.

In addition, in the field of text-to-audio generation, a distinctive
characteristic sets it apart from image generation[14, 15]: the utilization of
an intermediary representation, where the choice of this representation, such
as the mel-spectrogra. It is important to underscore the absence of
established progressive distillation methodologies tailored explicitly for
text-to-audio generation.

##  3 Method

###  3.1 Latent diffusion model

We employ diffusion models to approximate the genuine conditional data
distribution $q(z_{0}\mid\tau)$, using a model distribution
$p_{\theta}(z_{0}\mid\tau)$. Here, $z_{0}$ exists in a latent space, which is
a compression of the mel-spectrogram $X\in\mathbb{R}^{T\times F}$ and is
associated with an audio sample $x$. The dimensionality of $z_{0}$ is given by
$C\times\frac{T}{r}\times\frac{F}{r}$, where $r$ signifies the compression
level. Additionally, $C$ represents the number of channels in the latent
representation, $X$ corresponds to the mel-spectrogram, and $T$ and $F$ denote
the temporal and frequency dimensions, respectively. The text embedding $\tau$
is acquired through a pre-trained text encoder in FLAN-T5[11].

The diffusion model has two processes, a forward and a reverse processes. In
the forward process[16], the data distribution is systematically converted to
a standard Gaussian distribution according to a predefined noise schedule[7].
In the reverse process, the audio $priori\ z_{0}$ is created through a
stepwise denoising procedure with the guide text $\tau$. This denoising
process is controlled by the following loss function[8]:

$\displaystyle
L_{DM}=\sum_{n=1}^{N}\gamma_{n}\mathbb{E}_{\epsilon_{n}\sim\mathcal{N}(0,\boldsymbol{I}),z_{0}}\|\epsilon_{n}-\hat{\epsilon}_{\theta}^{(n)}(z_{n},\tau)\|^{2}_{2}$
(1)

where $N$ is the number of sampling steps. $z_{n}$ is obtained by adding noise
$\epsilon_{n}\sim\mathcal{N}(0,\boldsymbol{I})$ to $z_{0}$. $\gamma_{n}$ is
the weight of the loss function $n$[10] and a measure of the SNR. This
denoising process begins with an initial Gaussian distribution denoted as
$p(z_{N})$, generating the audio $priori\ z_{0}$ step by step as follows[8]:

$\displaystyle
p_{\theta}(z_{0:N}\mid\tau)=p(z_{N})\prod_{n=1}^{N}p_{\theta}(z_{n-1}\mid
z_{n},\tau),$ (2) $\displaystyle p_{\theta}(z_{n-1}\mid
z_{n},\tau)=\mathcal{N}(\mu_{\theta}^{(n)}(z_{n},\tau),\tilde{\beta}^{(n)}).$
(3)

The mean function and variance are parameterized as[8]:

$\displaystyle\mu_{\theta}^{(n)}(z_{n},\tau)=\frac{1}{\sqrt{\alpha_{n}}}\left[z_{n}-\frac{1-\alpha_{n}}{\sqrt{1-\bar{\alpha}_{n}}}\hat{\epsilon}_{\theta}^{(n)}(z_{n},\tau)\right],$
(4)
$\displaystyle\tilde{\beta^{(n)}}=\frac{1-\bar{\alpha}_{n-1}}{1-\bar{\alpha}_{n}}\beta_{n}.$
(5)

The structure of the diffusion model is adopted from U-Net[17], which contains
a cross-attention[18] module going to the input text guidance $\tau$.

###  3.2 Progressive Distiallation for TTA

Figure 1 illustrates the asymptotic refinement process, where the only thing
to be trained is the student model. The whole process differs from traditional
training methods because the audio embedding $\tilde{z}_{0}$ used in the loss
function during training is not the pure audio data $x$ from the dataset.
Throughout the distillation process, the BSA method calculates the loss by
comparing the results of the two sampling steps of the teacher model with the
single sampling step of the student model, during which equilibrium in the
model training is achieved by a loss weighting function, and then updating the
student model. Iterating this process results in a model that produces similar
output but with half the sampling steps of the teacher model.

Algorithm 1 illustrates the asymptotical refining process of progressive
distillation for TTA in detail.A student model is initialized using the
teacher model $\hat{x}_{\eta}(z_{t},\tau)$. let we start with $z_{t}$ till we
reach $z_{t-1/N}$, where $N$ denotes the number of steps sampled by the
student. We then perform the denoising procedure from $z_{t}$ to $z_{t-1/N}$,
which allows the student model to learn a single step from teacher’s two
DDIM[10] steps each time accompanying with a textual embedding $\tau$ input.
It is worth noting that given the teacher model and the starting point $z_{t}$
along with the textual embedding $\tau$, the prediction
$\tilde{x}(z_{t},\tau)$ from the teacher model is fully deterministic.
However, with raw data $z_{0}$ and the textual embedding $\tau$ as inputs, the
output of the forward process, $z_{t}$ is not fully deterministic due to
multiple possible raw data inputs leading to the same noisy data $z_{t}$. This
implies that the predictions of the original denoising model are a weighted
average of $z_{0}$, leading to ambiguous predictions. Unlike the conventional
way of training diffusion models, our approach provides both the teacher model
and the starting point $z_{t}$. Consequently, the student model trained using
our method can directly predict $z_{0}$, thereby avoiding intermediate steps,
resulting in faster model sampling. As demonstrated in Section 4.4, our
student model maintains the teacher model’s performance while halving the
number of steps. The student model then becomes the teacher model in the next
distillation cycle.

Algorithm 1 Progressive distillation for TTA

Trained teacher model $\hat{x}_{\eta}(z_{t},\tau)$

The matched text dataset $\mathcal{D}_{t}$ and audio dataset $\mathcal{D}_{a}$

Loss weight function $w()$

Student sampling steps $N$

for $K$ iterations do

$\theta\leftarrow\eta$ $\triangleright$ Init student from teacher

while not converged do

$y\sim\mathcal{D}_{t},x\sim\mathcal{D}_{a}$ $\triangleright$ Textual
description $y$, audio sample $x$

$\tau=f_{FLAN-T5}(y)$ $\triangleright$ $\tau$ is the output of FLAN-T5

$z_{0}=f_{AudioEncoder}(x)$ $\triangleright$ output of AudioEncoder

$t=i/N,i\sim Cat\left[1,2,\ldots,N\right]$

$\epsilon\sim\mathcal{N}(0,\boldsymbol{I})$

$z_{t}=\alpha_{t}z_{0}+\sigma_{t}\epsilon$ $\triangleright$ Signal level
$\alpha_{t}$, noise level $\sigma_{t}$ at time $t$

# 2 steps of DDIM with teacher model

$t^{{}^{\prime}}=t-0.5/N,t^{{}^{\prime\prime}}=t-1/N$

$z_{t^{{}^{\prime}}}=\alpha_{t^{{}^{\prime}}}\hat{x}_{\eta}(z_{t},\tau)+\frac{\sigma_{{t}^{{}^{\prime}}}}{\sigma_{t}}(z_{t}-\alpha_{t}\hat{x}_{\eta}(z_{t},\tau))$

$z_{t^{{}^{\prime\prime}}}=\alpha_{{t}^{{}^{\prime\prime}}}\hat{x}_{\eta}(z_{t^{{}^{\prime}}},\tau)+\frac{\sigma_{{t}^{{}^{\prime\prime}}}}{\sigma_{t^{{}^{\prime}}}}(z_{t^{{}^{\prime}}}-\alpha_{t^{{}^{\prime}}}\hat{x}_{\eta}(z_{t^{{}^{\prime}}},\tau))$

$\tilde{z}_{0}=\frac{z_{t^{{}^{\prime\prime}}}-(\sigma_{t^{{}^{\prime\prime}}}/\sigma_{t})z_{t}}{\alpha_{t^{{}^{\prime\prime}}}-(\sigma_{t^{{}^{\prime\prime}}}/\sigma_{t})\alpha_{t}}$
$\triangleright$ Teacher $\tilde{z}_{0}$ target

$L_{\theta}=w(\gamma_{t})\mathbb{E}_{\epsilon,t,\tilde{z}_{0}}\|\tilde{z}_{0}-\hat{x}_{\theta}(z_{t},\tau)\|^{2}_{2}$

$\triangleright$ student prediction $\hat{x}_{\theta}(z_{t},\tau)$ is
equivalent to $\hat{z}_{0}$

$\theta\leftarrow\theta-\gamma\bigtriangledown_{\theta}L_{\theta}$

end while

$\eta\leftarrow\theta$ $\triangleright$ Student becomes next teacher

$N\leftarrow N/2$ $\triangleright$ Halve number of sampling steps

end for

###  3.3 Advantages of Balanced SNR-Aware Method

Ho et al.[19] used Equation (6) as the loss function for the diffusion model:

$\displaystyle
L_{\theta}=\|\epsilon-\hat{\epsilon}_{\theta}(z_{t})\|^{2}_{2}=\frac{\alpha_{t}^{2}}{\sigma_{t}^{2}}\|x-\hat{x}_{\theta}(z_{t})\|^{2}_{2}.$
(6)

[9] showed that Equation (6) is unsuitable in distillation. During the
distillation process, the signal-to-noise ratio steadily approaches zero. In
the extreme scenario where the signal level $\alpha_{t}$ approaches zero, the
noise data $z_{t}$ ceases to convey any information about $z_{0}$, and the
predicted value $\hat{\epsilon}_{\theta}(z_{t})$ no longer contains meaningful
data-related information.

As the number of progressive distillation iterations increases, the span of
each step widens, more noise is eliminated, and the information implicitly
encoded in $z_{t}$ decreases. This information decrease adversely impacts the
model’s performance, necessitating the introduction of a reconstructive loss
function. To address this challenge, [9] proposed two solutions:

$\displaystyle
L_{\theta}=max(\frac{\alpha_{t}^{2}}{\sigma_{t}^{2}},1)\|x-\hat{x}_{t}\|^{2}_{2},$
(7) $\displaystyle
L_{\theta}=(1+\frac{\alpha_{t}^{2}}{\sigma_{t}^{2}})\|x-\hat{x}_{t}\|^{2}_{2}.$
(8)

Both methods effectively handle the problem of loss function weights reaching
zero. However, they exhibit a limitation: they result in large weights for the
progressive distillation loss function during the early stages of high SNR
levels. Consequently, the entire weight adjustment process becomes centered
around the early training phases, leading to degraded model performance.

In response to this challenge, [10] introduced a solution in Equation 9, which
assigns maximum weights to the loss function to prevent the concentration of
model training weights at high SNR. However, this approach does not address
the problem of the loss function weights dropping to zero at low SNR. The
importance of late-stage training is that the low information content in this
phase leads to a denominator in the model prediction
$\hat{x}_{\theta}(z_{t})=\frac{1}{\alpha_{t}}(z_{t}-\sigma_{t}\hat{\epsilon}_{\theta}(z_{t}))$
approaching zero, making the model highly sensitive to small perturbations.

$\displaystyle
L_{\theta}=min(\frac{\alpha_{t}^{2}}{\sigma_{t}^{2}},\gamma)\|x-\hat{x}_{t}\|^{2}_{2}.$
(9)

To address these limitations, we propose our Balanced SNR-Aware(BSA) method as
follows:

$\displaystyle
L_{\theta}=min(\frac{\alpha_{t}^{2}}{\sigma_{t}^{2}}+1,\gamma)\|x-\hat{x}_{t}\|^{2}_{2}.$
(10)

In BSA method, a hyperparameter $\gamma$ is introduced, which is set to $5$
empirically. The proposed method weights the loss function in a balanced SNR
manner. During training, when SNR is high, the weight locks to $\gamma$, which
prevents excessive focus on low noise information. When SNR is low, the weight
approaches 1, which prevents ignoring the high noise information.

##  4 Experiments

###  4.1 Datasets

In this paper, we use the AudioCaps dataset[20] for evaluating BSA. The
dataset contains a training set with 49,286 audio-text pairs and a test set
with 975 audio clips, each accompanied by five manually written subtitles,
totaling 4,785 audio-text pairs. All the audio clips in the dataset are 10
seconds long and were collected from YouTube. These clips were edited to
ensure a consistent 10-second duration and were paired with text by
annotators.

###  4.2 Experimental process

We assess the effectiveness of three different loss weight strategies: the
conventional Loss Weight Strategy, the Min-SNR-$\gamma$ Loss Weight Strategy,
and our proposed BSA, using the AudioCaps dataset. While we employ Tango[8]
during the experiments, it’s important to note that our BSA is versatile and
can be applied universally to latent diffusion models. We perform three
iterations of the potential diffusion model, ensuring convergence at each
iteration. Subsequently, we test the models generated by each strategy on the
test set five times, with the aim of reducing experimental variability.
Finally, to mitigate experimental errors, we construct a 95% confidence
interval around the test results. Detailed experimental results are presented
in Section 4.4.

###  4.3 Evaluation Metrics

In this work, we use Frechet Audio Distance (FAD)[21] and Frechet Distance
(FD). FAD is the use of the Frechet Initial Distance (FID) metric for the
audio domain, which is a perceptual metric capable of measuring the perceptual
effects of a variety of distortions.FD denotes the similarity between the
production sample and the similarity between the production and target
samples.

###  4.4 Results

In Table 1, we conduct varying numbers of inference steps (200, 100, 50, and
25) using Tango’s best model for generating data samples, as evaluated with
objective metrics from the AudioCaps test set. We then use the BSA method to
progressively distillate this model into student models with 100, 50, and 25
inference steps, respectively. For comparison, we use methods[9, 10] to do the
same work. Our student model consistently outperformed others on the AudioCaps
dataset in terms of the objective metric FAD and FD for the same inference
steps. Specifically, our 25-step student model generated samples with a FAD of
1.672, which is lower than the samples generated by models trained by other
methods. This shows that the BSA method generates higher quality samples than
the existing methods, and at the same time, the BSA method can achieve eight
times faster sampling speed with a model before distillation.

Figure 2 illustrates the performance of various methods across distillation
iterations. Notably, our approach consistently surpasses both the traditional
DDIM sampler and the method introduced by Salimans et al. Following three
distillation iterations, our model attains a FAD score of 1.672 for the
25-step model. In comparison, the DDIM sampler yields a FAD of 3.455,
Salimans’ method produces a FAD of 2.274 for the 25-step model, and Hang’s
method results in a FAD of 1.866. Our method consistently maintains robust and
dependable performance during the evaluation.

Table 1: Performance of the teacher model and the student model with different number of step, including the data from the 200-step teacher Our reported metrics encompass both FAD and FD, along with their respective 95% confidence intervals. The efficacy of the BAS method stands out at 25 steps. After just three iterations, it enables us to produce higher-quality samples than would have been possible otherwise, achieving FAD = 1.672 and FD = 21.847. Model | Sampling steps | FAD$\downarrow$ | FD$\downarrow$
---|---|---|---
| 200 | 1.5706 | 21.3416
Teacher[8] | 100 |  1.694$\pm 0.015$ |  21.643$\pm 0.140$
| 50 |  2.049$\pm 0.031$ |  22.523$\pm 0.268$
| 25 |  3.455$\pm 0.041$ |  25.722$\pm 0.106$
| 100 |  1.661$\pm 0.028$ |  21.627$\pm 0.188$
Student[9] | 50 |  1.751$\pm 0.007$ |  21.981$\pm 0.324$
| 25 |  2.274$\pm 0.174$ |  23.133$\pm 0.492$
| 100 |  1.631$\pm 0.023$ |  21.794$\pm 0.303$
Student[10] | 50 |  1.711$\pm 0.014$ |  21.819$\pm 0.056$
| 25 |  1.866$\pm 0.096$ |  22.147$\pm 0.492$
| 100 |  1.593$\pm 0.026$ |  21.444$\pm 0.090$
Student | 50 |  1.607$\pm 0.017$ |  21.706$\pm 0.156$
(ours) | 25 |  1.672$\pm 0.052$ |  21.847$\pm 0.038$
Fig. 2: Comparing the FAD of the various methods for training the model to
convergence at the same number of sampling steps, the BSA method gives the
least performance in exchange for the corresponding sampling rate at the
25-step result.

##  5 Conclusions

In this paper, we adapt progressive distillation to latent-diffusion-based
Text-to-Audio (TTA) generation, marking it a practical implementation to
distill knowledge from a TTA teacher model for a student model that can
expedite sampling speed. We further propose a Balanced SNR-Aware (BSA) method
that can balance the weights of a loss function so that the student model
prevents excessive focus on low-noise information and ignorance of high-noise
information. Through three iterations of distillation, we obtain a student
model with only 25 sampling steps, which is eight times faster than the
original teacher model. Experimental results show that the student model has
little performance degradation compared to the teacher model and outperforms
the previous best method by approximately 11.6% in FAD and 1.3% in FD.

##  6 Acknowledgement

This work was supported in part by Grant “XJTLU RDF-22-01-084”. For the
purpose of open access, the authors have applied a Creative Commons
Attribution (CC BY) licence to any Author Accepted Manuscript version arising.

## References

  * [1] Rongjie Huang, Jiawei Huang, Dongchao Yang, Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao,  “Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models,”  arXiv preprint arXiv:2301.12661, 2023.
  * [2] Flavio Schneider, Zhijing Jin, and Bernhard Schölkopf,  “Mo$\rm\hat{u}$sai: Text-to-music generation with long-context latent diffusion,”  arXiv preprint arXiv:2301.11757, 2023.
  * [3] Yuancheng Wang, Zeqian Ju, Xu Tan, Lei He, Zhizheng Wu, Jiang Bian, and Sheng Zhao,  “Audit: Audio editing by following instructions with latent diffusion models,”  arXiv preprint arXiv:2304.00830, 2023.
  * [4] Yi Yuan, Haohe Liu, Xubo Liu, Xiyuan Kang, Peipei Wu, Mark D Plumbley, and Wenwu Wang,  “Text-driven foley sound generation with latent diffusion model,”  arXiv preprint arXiv:2306.10359, 2023.
  * [5] Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo,  “Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 10219–10228.
  * [6] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu,  “Diffsound: Discrete diffusion model for text-to-sound generation,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023.
  * [7] Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley,  “Audioldm: Text-to-audio generation with latent diffusion models,”  arXiv preprint arXiv:2301.12503, 2023.
  * [8] Deepanway Ghosal, Navonil Majumder, Ambuj Mehrish, and Soujanya Poria,  “Text-to-audio generation using instruction-tuned llm and latent diffusion model,”  arXiv preprint arXiv:2304.13731, 2023.
  * [9] Tim Salimans and Jonathan Ho,  “Progressive distillation for fast sampling of diffusion models,”  arXiv preprint arXiv:2202.00512, 2022.
  * [10] Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo,  “Efficient diffusion training via min-snr weighting strategy,”  arXiv preprint arXiv:2303.09556, 2023.
  * [11] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al.,  “Scaling instruction-finetuned language models,”  arXiv preprint arXiv:2210.11416, 2022.
  * [12] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean,  “Distilling the knowledge in a neural network,”  arXiv preprint arXiv:1503.02531, 2015.
  * [13] Aaron Oord, Yazhe Li, Igor Babuschkin, Karen Simonyan, Oriol Vinyals, Koray Kavukcuoglu, George Driessche, Edward Lockhart, Luis Cobo, Florian Stimberg, et al.,  “Parallel wavenet: Fast high-fidelity speech synthesis,”  in International conference on machine learning. PMLR, 2018, pp. 3918–3926.
  * [14] Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans,  “On distillation of guided diffusion models,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 14297–14306.
  * [15] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al.,  “Muse: Text-to-image generation via masked generative transformers,”  arXiv preprint arXiv:2301.00704, 2023.
  * [16] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole,  “Score-based generative modeling through stochastic differential equations,”  arXiv preprint arXiv:2011.13456, 2020.
  * [17] Olaf Ronneberger, Philipp Fischer, and Thomas Brox,  “U-net: Convolutional networks for biomedical image segmentation,”  in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18. Springer, 2015, pp. 234–241.
  * [18] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Advances in neural information processing systems, vol. 30, 2017.
  * [19] Jonathan Ho, Ajay Jain, and Pieter Abbeel,  “Denoising diffusion probabilistic models,”  Advances in neural information processing systems, vol. 33, pp. 6840–6851, 2020.
  * [20] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim,  “Audiocaps: Generating captions for audios in the wild,”  in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 119–132.
  * [21] Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi,  “Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms.,”  in INTERSPEECH, 2019, pp. 2350–2354.