# Text2Data: Low-Resource Data Generation with Textual Control

Shiyu Wang  Yihao Feng  Tian Lan  Ning Yu  Yu Bai  Ran Xu  Huan Wang  Caiming
Xiong  Silvio Savarese

###### Abstract

Natural language serves as a common and straightforward control signal for
humans to interact seamlessly with machines. Recognizing the importance of
this interface, the machine learning community is investing considerable
effort in generating data that is semantically coherent with textual
instructions. While strides have been made in text-to-data generation spanning
image editing, audio synthesis, video creation, and beyond, low-resource areas
characterized by expensive annotations or complex data structures, such as
molecules, motion dynamics, and time series, often lack textual labels. This
deficiency impedes supervised learning, thereby constraining the application
of advanced generative models for text-to-data tasks. In response to these
challenges in the low-resource scenario, we propose Text2Data, a novel
approach that utilizes unlabeled data to understand the underlying data
distribution through an unsupervised diffusion model. Subsequently, it
undergoes controllable finetuning via a novel constraint optimization-based
learning objective that ensures controllability and effectively counteracts
catastrophic forgetting. Comprehensive experiments demonstrate that Text2Data
is able to achieve enhanced performance regarding controllability across
various modalities, including molecules, motions and time series, when
compared to existing baselines. The code can be found at
https://github.com/salesforce/text2data.

Machine Learning, ICML

Salesforce Research, 181 Lytton Ave, Palo Alto, CA 94301, USA

##  1 Introduction

Autonomy and controllability stand as twin primary pillars of generative AI
(Gozalo-Brizuela & Garrido-Merchan, 2023; Wang et al., 2022). While the
challenge of autonomy has been substantially addressed through the rapid
advancements of generative models, controllability is now ascending as a
fervently explored arena within the machine learning community. As natural
languages are one of the most common and simplest control signal for human
beings to interact with machines, the machine learning community has
increasingly focused on generating data that aligns semantically with textual
descriptions, given its wide-ranging applications such as image editing (Zhang
et al., 2023; Kawar et al., 2023), audio synthesis (Liu et al., 2023; Huang et
al., 2023), video generation (Li et al., 2018; Hu et al., 2022), and many more
(Tevet et al., 2023; Sanghi et al., 2022).

Recent breakthroughs in text-to-data generative models, particularly those
using diffusion techniques (Li et al., 2023; Yang et al., 2023; Kumari et al.,
2023), have demonstrated remarkable proficiency by harnessing the rich
semantic insights from vast datasets of data-text pairs. Despite the broad
applicability of text-to-data generative models, not all modalities can meet
the substantial data-text pair requirements for achieving optimal
controllability during model training. This is often due to costly annotations
or intricate data structures, a scenario we refer to as the low-resource
situation. The lack of text labels in certain areas, such as molecules
(Ramakrishnan et al., 2014; Irwin et al., 2012), motions (Guo et al., 2020;
Mahmood et al., 2019), and time series (Du et al., 2020), primarily restricts
supervised learning and hinders the use of advanced generative models for
text-to-data generation tasks. The low-resource situation when training
generative models unsurprisingly results in issues like undesirable generation
quality, model overfitting, bias, and lack of diversity. As a result, the
optimization for scarce text representations to improve the alignment between
generated data and input texts in generative models is still under-explored.

To mitigate the issues in the low-resource scenario, the strategies such as
data augmentation (Hedderich et al., 2020; Meng et al., 2021), semi-supervised
learning (Thomas et al., 2013; Cheuk et al., 2021), and transfer learning
(Tits et al., 2020; Yi et al., 2018) are utilized. Yet, each comes across
challenges. Data augmentation, for example, cannot always replicate genuine
data fidelity to align accurately with initial text descriptions, and
potentially leads to overfitting due to over-reliance on augmented samples.
This method also exacerbates the training complexity, intensifying the already
high computational demands of diffusion models. For semi-supervised learning,
text inherently carries nuances, ambiguities, and multiple meanings. Ensuring
that the model infers the correct interpretation when leveraging unlabeled
data is not straightforward. Lastly, while transfer learning offers a solution
for limited datasets, it is prone to catastrophic forgetting (Iman et al.,
2023), where previous knowledge diminishes as new information (i.e., text
descriptions) is introduced.

Alternative to existing solutions, we propose Text2Data, a diffusion-based
framework achieving enhanced text-to-data controllability even under low-
resource situation. Specially, Text2Data operates in two pivotal stages: (1)
Distribution mastery by leveraging unlabeled data. Distinct from conventional
semi-supervised learning methods, Text2Data does not aim to deduce labels for
unlabeled data. Instead, this step uses unlabeled data to discern the
overarching data distribution via an unsupervised diffusion model, eliminating
the semantic ambiguity often associated with semi-supervised approaches. (2)
Controllable finetuning on text-labeled data. The learned diffusion model is
then finetuned by text-labeled data. Distinct from methods reliant on data
augmentation, Text2Data abstains from inflating the training dataset. Instead,
we introduce a novel constraint optimization-based learning objective, aiming
to mitigate catastrophic forgetting by regularizing the model parameter space
closely to its preliminary space before finetuning. Our contributions are
summarized as follows:

  * •

We introduce Text2Data, a novel framework designed for text-to-data generation
in the low-resource scenario. This approach maintains the fine-grained data
distribution by fully harnessing both labeled and unlabeled data.

  * •

We design a novel learning objective based on constraint optimization to
achieve controllability and overcome catastrophic forgetting during
finetuning.

  * •

We theoretically validate our optimization constraint selection and
generalization bounds for our learning objective.

  * •

We compile real-world datasets across three modalities and conduct
comprehensive experiments to show the effectiveness of Text2Data. The results
demonstrate that Text2Data achieves superior performance baselines regarding
both generation quality and controllability.

##  2 Related works

###  2.1 Text-to-data diffusion-based generation

Diffusion models, notably divided into classifier-guided (Dhariwal & Nichol,
2021) and classifier-free (Ho & Salimans, 2022) categories, have significantly
impacted data generation across various domains. (Hoogeboom et al., 2022; Yang
et al., 2023; Ho et al., 2022; Voleti et al., 2022). The classifier-guided
diffusion guides the model during inference phase by independently training a
classifier and supervising the model with its gradient, which is inefficient
when computing gradient at each time step and sometimes the generation quality
is deficient as the guidance is not involved in the training. By contrast,
classifier-free diffusion guidance blends score estimates from both a
conditional diffusion model and an unconditional one with time step as a
parameter, exemplified by E(3) Equivariant Diffusion Model (EDM) (Hoogeboom et
al., 2022) and Motion Diffusion Model (MDM) (Tevet et al., 2023) for
controllable molecule and motion generation, respectively. Furthermore, since
natural languages are a prevalent medium for human to communicate with the
world, the text-to-data generation paradigm has gained traction, with
diffusion models being instrumental in generating high-quality data aligned
with textual inputs. The extensive applications encompass text-to-image
generation (Ruiz et al., 2023; Zhang & Agrawala, 2023), text-to-speech
generation (Huang et al., 2022; Kim et al., 2022), text-to-shape generation
(Li et al., 2023; Lin et al., 2023), and more, leveraging the abundant text
descriptions for training potent generative models. Despite advancements in
generating data from text across various modalities, many other modalities may
not satisfy the stringent requirements for sufficient data-text pairs
essential for attaining optimal controllability during the training of models.

###  2.2 Low-resource learning

In response to the challenges of low-resource training for controllable
generative models, several strategies have been formulated. For instance, Yin
et al. (2023) proposes Text-to-Text-to-Image Data Augmentation that employed
both large-scale pretrained Text-to-Text and Text-to-Image generative models
for data augmentation to generate photo-realistic labeled images in a
controllable manner. Zang & Wan (2019) utilizes a semi-supervised approach to
augment the training of both the encoder and decoder with both labeled and
unlabeled data. Tu et al. (2019) proposes to learn a mapping between source
and target linguistic symbols and employs transfer learning to transfer
knowledge from a high-resource language to low-resource language.
Nevertheless, all those strategies have their own limitations, such as high
computational complexity for data augmentation, difficulty in maintaining the
correct interpretation of text when leveraging unlabeled data during semi-
supervised learning, and the potential catastrophic forgetting issues in
transfer learning. Additionally, most of works on text-to-data generation lie
in the modality of image, speeches, and texts, which have plenty of labeled
datasets (Ito & Johnson, 2017; Pratap et al., 2020; Lin et al., 2014; Jiang et
al., 2021; Wang et al., 2020) to train the models nowadays. Yet it is far
under-explored for the low-source modalities such as molecules, motions and
time series. Therefore, we propose Text2Data, a diffusion-based framework
adept at harnessing limited text-labeled data to enhance the controllability
of the model in text-to-data generation.

##  3 Problem formulation

Suppose the dataset ${\mathcal{D}}=\\{{\mathbf{x}},{\mathbf{c}}\\}$ contains
$N$ independent samples in total, where
${\mathbf{x}}=\\{{\mathbf{x}}_{i}\\}_{i=1}^{N}$ is the data samples such as
molecules, motions, time series, etc. We assume that there is only a
proportion of data in ${\mathbf{x}}$ that has corresponding text description
${\mathbf{c}}=\\{{\mathbf{c}}_{i}\\}_{i=1}^{N_{p}}$ where $N_{p}\leq N$. We
denote that data with text description is contained in ${\mathcal{D}}_{p}$ and
${\mathcal{D}}_{p}\subset{\mathcal{D}}$. Using both text-labeled and unlabeled
data in ${\mathcal{D}}$, we aim to learn a generative model,
$p_{\theta}({\mathbf{x}}|{\mathbf{c}})$ parameterized by $\theta$ that is able
to generate data ${\mathbf{x}}\sim p_{\theta}({\mathbf{x}}|{\mathbf{c}})$
corresponding to specific text description ${\mathbf{c}}={\mathbf{c}}^{*}$.

Figure 1: Overview of Text2Data. The model leverages unlabeled data (i.e.,
blue module) to discern the overall data distribution while the optimal set of
model parameters $\Theta$ is obtained. Then the model is finetuned on labeled
data (i.e., red module) by constraint optimization that gives the optimal set
of parameters as $\Theta\cap\Theta^{\prime}$, where $\Theta^{\prime}$ is the
optimal set of parameters if finetune the model without constraint.

##  4 Methods

Controllable data generation seeks to learn the conditional data distribution
$p_{\theta}({\mathbf{x}}|{\mathbf{c}})$ during training and subsequently draw
samples from this assimilated distribution during the inference stage.
Consequently, our primary objective during the training phase is to optimize
the following:

$\displaystyle\min_{\theta}~{}\mathbb{E}_{{\mathbf{x}},{\mathbf{c}}\sim
p_{{\mathcal{D}}_{p}}({\mathbf{x}},{\mathbf{c}})}[-\log
p_{\theta}({\mathbf{x}}|{\mathbf{c}})]\,.$ (1)

While the training of such generative models is contingent upon the
supervision of text descriptions present in the dataset (i.e.,
${\mathcal{D}}_{p}$), it is not always feasible to obtain an adequate number
of data-text pairs to ensure optimal controllability (i.e.,
$|{\mathcal{D}}_{p}|<|{\mathcal{D}}|$), especially in modalities like
molecular structures, motion patterns and time series. Such constraints can
precipitate complications, including model overfitting when optimizing
according to Eq. 1. Given the challenges, devising strategies to capitalize on
the unlabeled data within ${\mathcal{D}}$—which is often more accessible and
cost-effective—is pivotal to effectively learn
$p_{\theta}({\mathbf{x}}|{\mathbf{c}})$.

Notably, the marginal distributions learned from unlabeled data closely
resemble those obtained from labeled data:

$\displaystyle p_{\theta}({\mathbf{x}})\approx\int
p_{\theta}({\mathbf{x}}|{\mathbf{c}})p_{{\mathcal{D}}_{p}}({\mathbf{c}})d{\mathbf{c}}=\mathbb{E}_{{\mathbf{c}}\sim
p_{{\mathcal{D}}_{p}}({\mathbf{c}})}[p_{\theta}({\mathbf{x}}|{\mathbf{c}})],$
(2)

where $p_{{\mathcal{D}}_{p}}({\mathbf{c}})$ is the true underlying text
generating distribution corresponding to ${\mathbf{x}}\in{\mathcal{D}}_{p}$.
Hence, as per Figure 1, we are inspired to initially utilize the unlabeled
data in ${\mathcal{D}}$ to learn $p_{\theta}({\mathbf{x}})$ and obtain the
optimal set of model parameters $\hat{\theta}\in\Theta$, which serves as a
robust approximation of $p_{\theta}({\mathbf{x}}|{\mathbf{c}})$. Then, we
finetune it using the data-text pairs in ${\mathcal{D}}_{p}$ to achieve
desired model controllability (Section 4.1). Crucially, we ensure that the
parameters remain in close proximity to those established when learning
$p_{\theta}({\mathbf{x}})$ via constraint optimization to make Eq. 2 hold
while mitigating catastrophic forgetting. Figure 1 shows our constraint to
keep the finetuned parameters $\hat{\theta}^{\prime}$ within
$\Theta\cap\Theta^{\prime}$, where $\Theta^{\prime}$ represents the optimal
parameters from unconstrained finetuning. Then we plug the generative
diffusion implementation to our framework in Section 4.2. Finally, we use
Section 4.3 to offer theoretical insights into the selection of the
optimization constraint, underpinned by generalization bounds for our learning
objective.

###  4.1 Learning text-to-data generation under low-resource situation

To capture a general distribution of data, we leverage all data in
${\mathcal{D}}$ incorporating NULL tokens as conditions (i.e., ${\mathbf{c}}=$
NULL) to facilitate subsequent finetuning. Specifically, we train the
generative model $p_{\theta}({\mathbf{x}}|\emptyset)$, where $\theta$
parameterizes the model and $\emptyset$ represents the NULL token in practice.
As the NULL token is independent to ${\mathbf{x}}$, we also have
$p_{\theta}({\mathbf{x}}|\emptyset)=p_{\theta}({\mathbf{x}})$. Hence, we are
equivalently optimizing the following:

$\displaystyle\min_{\theta}~{}\mathbb{E}_{{\mathbf{x}}\sim
p_{\mathcal{D}}({\mathbf{x}})}[-\log p_{\theta}({\mathbf{x}})],$ (3)

where $p_{\mathcal{D}}({\mathbf{x}})$ is the true underlying data generating
distribution. Having discerned the general data distribution through Eq. 3, we
proceed to finetune $p_{\theta}({\mathbf{x}}|{\mathbf{c}})$ using text labels
${\mathbf{c}}$ in ${\mathcal{D}}_{p}$ to achieve the controllability of model.
In alignment with Eq. 2, the anticipated finetuned parameter should
approximate the parameter optimized in Eq. 3. We achieve this by finetuning
$p_{\theta}({\mathbf{x}}|{\mathbf{c}})$ using the labeled data in
${\mathcal{D}}_{p}$ within the optimal set obtained in Eq. 3, leading to the
subsequent learning objective for the finetuning phase:

$\displaystyle\min_{\theta}~{}$
$\displaystyle\mathbb{E}_{{\mathbf{x}},{\mathbf{c}}\sim
p_{{\mathcal{D}}_{p}}({\mathbf{x}},{\mathbf{c}})}[-\log
p_{\theta}({\mathbf{x}}|{\mathbf{c}})]$ s.t.
$\displaystyle\mathbb{E}_{{\mathbf{x}}\sim
p_{{\mathcal{D}}_{p}}({\mathbf{x}})}[-\log p_{\theta}({\mathbf{x}})]\leq\xi,$
$\displaystyle\xi=\inf_{\theta\in\Theta}\mathbb{E}_{{\mathbf{x}}\sim
p_{{\mathcal{D}}}({\mathbf{x}})}[-\log p_{\theta}({\mathbf{x}})]$ (4)

where $p_{{\mathcal{D}}_{p}}({\mathbf{x}},{\mathbf{c}})$ is the true
underlying data-text joint distribution. $\Theta$ denotes a localized
parameter space where a minimum can be located. Specifically, we minimize
$\mathbb{E}_{{\mathbf{x}},{\mathbf{c}}\sim
p_{{\mathcal{D}}_{p}}({\mathbf{x}},{\mathbf{c}})}[-\log
p_{\theta}({\mathbf{x}}|{\mathbf{c}})]$ using the labeled data in
${\mathcal{D}}_{p}$ within the optimal set
$\\{\theta:\mathbb{E}_{{\mathbf{x}}\sim
p_{{\mathcal{D}}_{p}}({\mathbf{x}})}[-\log
p_{\theta}({\mathbf{x}})]\leq\xi\\}$ to make the parameter not far from
learned via Eq. 3, so that catastrophic forgetting is mitigated. The trade-off
between the learning objective and its constraint aligns with the nature of a
lexicographic optimization problem (Gong & Liu, 2021) and therefore can be
solved in that context.

###  4.2 Generative objective on empirical samples

Eq. 4 is the population-level learning objective while empirically we follow
the standard loss function (i.e., transformed evidence lower bound of Eq. 4)
of the classifier-free diffusion guidance (Ho & Salimans, 2022) by optimizing:

$\displaystyle\min_{\theta}~{}$
$\displaystyle{\mathcal{L}}_{2}(\theta)~{}~{}~{}~{}\text{s.t.}~{}{\mathcal{L}}_{1}^{\prime}(\theta)\leq\xi,~{}\xi=\inf_{\theta\in\Theta}{\mathcal{L}}_{1}(\theta),$
(5)

where we have: ${\mathcal{L}}_{1}(\theta)=\mathbb{E}_{{\mathbf{x}}\sim
p_{{\mathcal{D}}}({\mathbf{x}}),t}[||{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},t)-{\bm{\epsilon}}||^{2}]$,
${\mathcal{L}}_{1}^{\prime}(\theta)=\mathbb{E}_{{\mathbf{x}}\sim
p_{{\mathcal{D}}_{p}}({\mathbf{x}}),t}[||{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},t)-{\bm{\epsilon}}||^{2}]$
and ${\mathcal{L}}_{2}(\theta)=\mathbb{E}_{{\mathbf{x}},{\mathbf{c}}\sim
p_{{\mathcal{D}}_{p}}({\mathbf{x}},{\mathbf{c}}),t}[||{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t)-{\bm{\epsilon}}||^{2}]$.
Also, $t$ is sampled from uniform between 1 and $T$, $T$ is the total
diffusion steps, ${\bm{\epsilon}}$ is the standard Gaussian random variable,
and ${\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},t)$ and
${\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t)$ are functions we
aim to fit at the $t$-th diffusion step. Note that
${\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},t)$ and
${\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t)$ share the same
parameters but are just trained at different stages: distribution mastery on
unlabeled data and controllable finetuning on labeled data, respectively. The
framework of classifier-free diffusion model and the derivation of
${\mathcal{L}}_{1}(\theta)$, ${\mathcal{L}}_{1}^{\prime}(\theta)$ and
${\mathcal{L}}_{2}(\theta)$ are introduced in Appendix A.

As the true data generating distributions $p_{{\mathcal{D}}}({\mathbf{x}})$,
$p_{{\mathcal{D}}_{p}}({\mathbf{x}})$ and
$p_{{\mathcal{D}}_{p}}({\mathbf{x}},{\mathbf{c}})$ are unknown, we instead
optimize the following empirical loss:

$\displaystyle\min_{\theta}~{}\hat{{\mathcal{L}}}_{2}(\theta)~{}~{}~{}~{}\text{s.t.}~{}\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)\leq\hat{\xi},~{}\hat{\xi}=\inf_{\theta\in\hat{\Theta}}\hat{{\mathcal{L}}}_{1}(\theta),$
(6)

where we have
$\hat{{\mathcal{L}}}_{1}(\theta)=\mathbb{E}_{{\mathbf{x}}\sim\hat{p}_{{\mathcal{D}}}({\mathbf{x}}),t}[||{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},t)-{\bm{\epsilon}}||^{2}]$,
$\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)=\mathbb{E}_{{\mathbf{x}}\sim\hat{p}_{{\mathcal{D}}_{p}}({\mathbf{x}}),t}[||{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},t)-{\bm{\epsilon}}||^{2}]$
and
$\hat{{\mathcal{L}}}_{2}(\theta)=\mathbb{E}_{{\mathbf{x}},{\mathbf{c}}\sim\hat{p}_{{\mathcal{D}}_{p}}({\mathbf{x}},{\mathbf{c}}),t}[||{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t)-{\bm{\epsilon}}||^{2}]$.
$\hat{p}_{{\mathcal{D}}}({\mathbf{x}})$,
$\hat{p}_{{\mathcal{D}}_{p}}({\mathbf{x}})$ and
$\hat{p}_{{\mathcal{D}}_{p}}({\mathbf{x}},{\mathbf{c}})$ are corresponding
empirical data generating distributions. $\hat{\Theta}$ is the localized
parameter space where a minimum can be located for
$\hat{{\mathcal{L}}}_{1}(\theta)$. The lexicographic optimization of Eq. 6 is
presented in Algorithm 1, which relies on learning the update direction on
model parameter (i.e.,
$\nabla\hat{\mathcal{L}}_{2}(\theta)+\lambda\nabla\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)$
in Algorithm 1) to balance the trade-off between the minimization of
$\hat{{\mathcal{L}}}_{2}(\theta)$ and
$\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)$ using a dynamic gradient descent
(Gong & Liu, 2021):

$\displaystyle\theta\leftarrow\theta-\omega\cdot(\nabla\hat{\mathcal{L}}_{2}(\theta)+\lambda\nabla\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)),$
(7)

where $\omega$ is predefined positive step size, and $\lambda$ is calculated
based on whether the constraint is satisfied at the current gradient step:

$\displaystyle\lambda=$
$\displaystyle\max(\frac{\phi(\theta)-\nabla\hat{\mathcal{L}}_{2}(\theta)^{T}\nabla\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)}{\|\nabla\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)\|^{2}},0)$
$\displaystyle\phi(\theta)=$
$\displaystyle\min(\alpha(\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)-\gamma\cdot\hat{\xi}),\beta\|\nabla\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)\|^{2}),$
(8)

where $\alpha$, $\beta$ and $\gamma$ are predefined positive hyperparameters.
More details and intuition of lexicographic optimization are explained in
Appendix B.

Input: $\hat{\xi}=\inf_{\theta\in\hat{\Theta}}\hat{{\mathcal{L}}}_{1}(\theta)$
by pretraining on ${\mathcal{D}}$; Total diffusion steps $T$; Scheduled
forward variance $\\{\beta_{t}\\}_{t=1}^{T}$;
$\\{\alpha_{t}=1-\beta_{t}\\}_{t=1}^{T}$; Predefined positive hyperparameters
$\alpha$, $\beta$, $\gamma$ and $\omega$; probability of unconditional
training $p_{uncond}$

1 while  _Not converged_ do

2 Sample
${\mathbf{x}},{\mathbf{c}}\sim\hat{p}_{{\mathcal{D}}_{p}}({\mathbf{x}},{\mathbf{c}})$,
${\bm{\epsilon}}\sim{\mathcal{N}}(\bm{0},\bm{I})$, $t\sim U(1,T)$

3 Compute $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{t}$

4 Diffuse
${\mathbf{x}}^{(t)}=\sqrt{\bar{\alpha}_{t}}{\mathbf{x}}+\sqrt{1-\bar{\alpha}_{t}}{\bm{\epsilon}}$

5 Replace ${\mathbf{c}}$ with $\emptyset$ with probability $p_{uncond}$

6 Compute
$\hat{{\mathcal{L}}_{2}}(\theta)=\|{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},t)-{\bm{\epsilon}}\|^{2}$,
$\hat{\mathcal{L}}_{1}^{\prime}(\theta)=\|{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t)-{\bm{\epsilon}}\|^{2}$

7 Compute
$\phi(\theta)=\min(\alpha(\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)-\gamma\cdot\hat{\xi}),\beta\|\nabla\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)\|^{2})$

8 Compute
$\lambda=\max(\frac{\phi(\theta)-\nabla\hat{\mathcal{L}}_{2}(\theta)^{T}\nabla\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)}{\|\nabla\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)\|^{2}},0)$

9 Update $\theta$ by
$\theta-\omega\cdot(\nabla\hat{\mathcal{L}}_{2}(\theta)+\lambda\nabla\hat{{\mathcal{L}}}_{1}^{\prime}(\theta))$

Algorithm 1 Lexicographic optimization on Eq. 4

Furthermore, the lexicographic optimization-based constraint
$\hat{\xi}=\inf_{\theta\in\hat{\Theta}}\hat{{\mathcal{L}}}_{1}(\theta)$ in Eq.
6 may be overly strict and could require relaxation to ease the training
process. While we anticipate that the parameters derived from Eq. 6 should be
close to those from Eq. 5, they do not necessarily have to be an exact subset
of the parameters from Eq. 5.

###  4.3 Generalization bound of learning constraint

In this section, we deduce a confidence bound for the constraint to
demonstrate that the optimal set of minimizing the empirical loss within the
derived confidence bound encapsulates the true one, and guide the further
relaxation on the constraint if needed. First, we define sub-Gaussian random
variable in Definition 4.1 followed by the deducted confidence bound in
Theorem 3.

######  Definition 4.1 (Sub-Gaussian random variable).

The random variable $X$ with mean 0 is sub-Gaussian with variance $\sigma^{2}$
if $\forall s\in\mathbb{R}$,
$\mathbb{E}_{X}[\exp(sX)]\leq\exp(\frac{\sigma^{2}s^{2}}{2})$.

Based on the fact that ${\mathbf{x}}^{(t)}$ is diffused from the random
variable ${\mathbf{x}}$ and the standard Gaussian noise ${\bm{\epsilon}}$
(i.e., Algorithm 1), therefore,
${\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},t)$ and
${\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}}^{(t)},t)$ are also
random variables. Meanwhile, minimizing the loss in Eq. 5 pushes
${\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},t)$ and
${\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}}^{(t)},t)$ towards
${\bm{\epsilon}}$, which has the mean of 0. Then we introduce the following
theorem 1 while assuming ${\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},t)$ and
${\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}}^{(t)},t)$ are sub-
Gaussian random variables.

######  Theorem 4.2.

For every $\theta$ and $t$, assume
${\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},t)$ and
${\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}}_{i},t)$ are sub-
Gaussian random variables with mean 0 and variance $\sigma^{2}$, and $\Theta$
is finite. Let
$\Theta^{*}=\\{\theta:{\mathcal{L}}_{1}^{\prime}(\theta)\leq\xi\\}$,
$\hat{\Theta}^{*}=\\{\theta:\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)\leq\hat{\xi}+\epsilon\\}$
where $\epsilon$ is the confidence bound with the probability of $1-\delta$.
Let $\theta^{*}$ be the solution to Eq. 5 and $\hat{\theta}^{*}$ be the
solution to empirical Eq. 6, then we have the following:

  1. 1.

$\Theta^{*}\subseteq\hat{\Theta}^{*}$: the set of $\theta$ by optimizing Eq. 6
within the confidence bound covers the true one.

  2. 2.

${\mathcal{L}}_{2}(\hat{\theta}^{*})\leq{\mathcal{L}}_{2}(\theta^{*})+2\epsilon_{N_{p}}$:
$\theta^{*}$ and $\hat{\theta}^{*}$ compete well on
${\mathcal{L}}_{2}(\theta)$.

  3. 3.

${\mathcal{L}}_{1}^{\prime}(\hat{\theta}^{*})\leq\xi+2\epsilon_{N_{p}}+2\epsilon_{N}$:
$\hat{\theta}^{*}$ does not violate constraint of Eq. 5 too much on
${\mathcal{L}}_{1}^{\prime}(\theta)$.

where $\epsilon=\epsilon_{N}+\epsilon_{N_{p}}$,
$\epsilon_{N}=\sqrt{C\tilde{\sigma}^{2}}\cdot\sqrt{\frac{\log|\Theta|+\log\frac{2}{\delta}}{N}}\vee
C\tilde{\sigma}^{2}\cdot\frac{\log|\Theta|+\log\frac{2}{\delta}}{N}$,
$\epsilon_{N_{p}}=\sqrt{C\tilde{\sigma}^{2}}\cdot\sqrt{\frac{\log|\Theta|+\log\frac{2}{\delta}}{N_{p}}}\vee
C\tilde{\sigma}^{2}\cdot\frac{\log|\Theta|+\log\frac{2}{\delta}}{N_{p}}$,
$\tilde{\sigma}^{2}=\sigma^{2}+1$ and $C=8\sqrt{2}$.

Theorem 3 can be proved starting from the Bernstein’s inequality and the union
bound inequality on squared zero-mean sub-Gaussian random variable. More
detailed proof is in Appendix C. As a large amount of unlabeled data (i.e.,
$N$) is usually easy to obtain by either manual collection or simulation,
$\epsilon_{N}$ is not large after taking logarithm on the number of model
parameters (i.e., $|\Theta|$), even though it is usually much larger than $N$.
Additionally, $\log|\Theta|$ is not significantly larger than $N_{p}$ so that
$\epsilon_{N_{p}}$ should not be large as well. For instance, in our
experiments, around 45,000 samples and 14 million model parameters for motion
generation result in rather small $\epsilon_{N}$ and $\epsilon_{N_{p}}$. In
practice, we use
$\xi=\rho\cdot\inf_{\theta\in\Theta}{\mathcal{L}}_{1}(\theta)$ in Eq. 5 (i.e.,
$\hat{\xi}=\rho\cdot\inf_{\theta\in\hat{\Theta}}\hat{{\mathcal{L}}}_{1}(\theta)$
in Eq. 6) to relax the constraint, where $\rho$ is an hyperparameter to keep
the constraint within the confidence interval.

##  5 Experiments

###  5.1 Datasets

We employ datasets from three modalities that may suffer from low-resource
scenario.

Molecules. We extract 130,831 molecules from QM9 dataset (Ramakrishnan et al.,
2014) with six molecular properties: polarizability ($\alpha$), highest
occupied molecular orbital energy ($\epsilon_{\text{HOMO}}$), lowest
unoccupied molecular orbital energy ($\epsilon_{\text{LUMO}}$), the energy
difference between HOMO and LUMO ($\Delta_{\epsilon}$), dipole moment ($\mu$)
and heat capacity at 298.15K ($C_{v}$).

Motions. We employ HumanML3D that contains textually re-annotating motions
captured from AMASS (Mahmood et al., 2019) and HumanAct12 (Guo et al., 2020).
It contains 14,616 motions annotated by 44,970 textual descriptions.

Time Series. We assemble 24 stocks from Yahoo
Finance111https://finance.yahoo.com/ (Appendix D) from their IPO date to July
8, 2023. We further tailor data to the length of 120 by slicing on the opening
price for every 120 days, and scale by min-max normalization following Yoon et
al. (2019). Totally, 210,964 time series are produced. We then extract their
features including frequency, skewness, mean, variance, linearity (i.e.,
$R^{2}$), and the number of peaks via “tsfresh” in Python.

Each dataset is divided into training and testing sets at the ratio of $80\%$
and $20\%$, respectively. We curate each dataset to have varying proportions
(i.e., $2\%,4\%,6\%,8\%,10\%,20\%,30\%,40\%$) of text labels. Details
regarding constructing text descriptions for Molecules and Time Series are
introduced in Appendix D.

###  5.2 Baseline models

We compare Text2Data with a representative classifier-free diffusion model in
each modality as the baselines.

E(3) Equivariant Diffusion Model (EDM) (Hoogeboom et al., 2022).To handle
Molecule dataset, we employ EDM as the baseline. EDM utilizes an equivariant
network to denoise diffusion processes by concurrently processing both
continuous (i.e., atom coordinates) and categorical data (i.e., atom types).
The controllability on molecular properties is realized by the classifier-free
diffusion guidance conditioned on the embedding of text descriptions, which is
encoded by a pretrained T5 (Raffel et al., 2020) encoder (i.e., t5-large).

Motion Diffusion Model (MDM) (Tevet et al., 2023). We use MDM, a classifier-
free diffusion model, for text-to-human motion generation. The text
descriptions are embedded by a pretrained T5 encoder to guide the motion
generation, providing the mechanism for controllability.

Generation diffusion for time series (DiffTS). To generate time series, we
design the classifier-free diffusion model conditioned on text embeddings
encoded from pretrained T5 encoder. We employ the backbone of (Ho & Salimans,
2022) by substituting image data to one-dimensional time series, and replacing
the U-Net with one-dimensional convolutional neural network.

During implementation, we train the baselines on a specific proportion of
labeled data. Text2Data is modified from the baseline model with a
pretraining+finetuning strategy following Eq. 6. Additionally, we conduct
ablation study where each baseline is still finetuned but without the
constraint in Eq. 6 as a transfer learning-based baseline, or directly applied
to augmented text-data pairs. More details are in Appendix E.

(a) $\alpha$ of generated molecules

(b) $\epsilon_{HOMO}$ of generated molecules

(c) $\epsilon_{LUMO}$ of generated molecules

(d) $\Delta_{\epsilon}$ of generated molecules

(e) $\mu$ of generated molecules

(f) $C_{v}$ of generated molecules

Figure 2: Evaluate controllability on Molecule dataset according to different
proportions of paired data. Green solid line corresponds to Text2Data and two
dashed lines are baseline comparisons, in which blue line is EDM and orange
line is EDM-finetune. Properties of generated molecules are predicted by
classifier $\phi_{c}$. MAE is computed between properties of generated
molecules and intended properties. Lower MAE indicates better performance.
Figure 3: Visualization of generated molecules when the polarizability
increases from “very low” to “very high”.

###  5.3 Evaluation metrics

We follow different strategies to evaluate both the controllability and the
generation quality of the propose approach compared to other baselines.

Controllability. We compare the controllability between Text2Data and the
baselines by the similarity between generated data and the ground truth. To
assess the generated molecules, we follow (Hoogeboom et al., 2022) and train a
classifier for each property to extract specific properties from generated
data. Then, we calculate the Mean Absolute Error (MAE) between the extracted
property and the ground truth. To assess the controllability of motion
generation, we compute R precision and Multimodal distance that measure the
relevancy of the generated motions to the input prompts (Guo et al., 2022b).
To evaluate the controllability of time series generation, we extract
properties via ”tsfresh” and compute the MAE between properties of generated
data and that of ground truth. Additionally, we also visualize generated data
according to the specified properties.

Generation quality. The evaluation of generation quality varies based on the
modality. For molecular generation, we compute $-\log$ likelihood ($-\log p$)
(Hoogeboom et al., 2022) and validity of generated molecules (Jin et al.,
2018), molecular stability and atom stability (Garcia Satorras et al., 2021)
to evaluate their overall generation quality. For motion generation, we use
FID score and Diversity following (Tevet et al., 2023). For time series
generation, we follow (Yoon et al., 2019) and (Lim et al., 2023) by drawing
t-SNE plots to visualize the overlap between generated data and the ground-
truth. Better model tends to have a larger overlap, indicating more similar
distribution.

Table 1: Evaluate controllability on HumanML3D dataset by R Precision and
Multimodal Distance according to different proportions of paired data.

Proportion (%) R Precision $\uparrow$ Multimodal Dist. $\downarrow$ Text2Data
MDM-finetune MDM Text2Data MDM-finetune MDM 2 0.34$\pm$0.01 0.37$\pm$0.01
0.31$\pm$0.01 6.48$\pm$0.06 6.19$\pm$0.05 6.67$\pm$0.02 4 0.39$\pm$0.01
0.42$\pm$0.01 0.38$\pm$0.01 5.99$\pm$0.05 5.83$\pm$0.04 6.01$\pm$0.04 6
0.43$\pm$0.01 0.43$\pm$0.01 0.40$\pm$0.02 5.85$\pm$0.06 5.78$\pm$0.05
6.01$\pm$0.06 8 0.44$\pm$0.01 0.43$\pm$0.01 0.42$\pm$0.01 5.65$\pm$0.04
5.75$\pm$0.05 5.90$\pm$0.04 10 0.45$\pm$0.01 0.45$\pm$0.01 0.44$\pm$0.01
5.74$\pm$0.07 5.76$\pm$0.07 5.84$\pm$0.06 20 0.47$\pm$0.01 0.47$\pm$0.01
0.45$\pm$0.01 5.61$\pm$0.04 5.68$\pm$0.11 5.73$\pm$0.14 30 0.48$\pm$0.01
0.47$\pm$0.01 0.45$\pm$0.11 5.61$\pm$0.05 5.66$\pm$0.06 5.80$\pm$0.09 40
0.49$\pm$0.01 0.46$\pm$0.01 0.45$\pm$0.01 5.61$\pm$0.05 5.63$\pm$0.09
5.90$\pm$0.04

Table 2: Evaluate controllability on time series by MAE on testing set,
according to different proportions of paired data. Lower MAE indicates better
performance.

Proportion (%) Frequency ($\times 10^{-1}$) Skewness Mean ($\times 10^{-2}$)
Text2Data DiffTS-finetune DiffTS Text2Data DiffTS-finetune DiffTS Text2Data
DiffTS-finetune DiffTS 2 2.59$\pm$0.20 2.62$\pm$0.20 2.60$\pm$0.17
1.68$\pm$0.20 2.34$\pm$0.30 1.84$\pm$0.22 0.63$\pm$0.39 0.63$\pm$0.39
0.79$\pm$0.41 4 2.55$\pm$0.18 2.59$\pm$0.19 2.59$\pm$0.19 1.63$\pm$0.14
1.77$\pm$0.29 2.80$\pm$0.28 0.60$\pm$0.40 0.61$\pm$0.39 0.73$\pm$0.40 6
2.52$\pm$0.18 2.57$\pm$0.19 2.57$\pm$0.19 1.00$\pm$0.16 1.85$\pm$0.22
1.81$\pm$0.21 0.56$\pm$0.38 0.58$\pm$0.38 0.71$\pm$0.36 8 2.54$\pm$0.18
2.56$\pm$0.19 2.57$\pm$0.18 1.10$\pm$0.18 1.56$\pm$0.18 1.78$\pm$0.09
0.57$\pm$0.38 0.63$\pm$0.40 0.62$\pm$0.36 10 2.54$\pm$0.19 2.57$\pm$0.18
2.55$\pm$0.20 0.87$\pm$0.13 1.20$\pm$0.17 1.12$\pm$0.11 0.55$\pm$0.37
0.57$\pm$0.36 0.62$\pm$0.39 20 2.54$\pm$0.18 2.55$\pm$0.18 2.58$\pm$0.21
1.05$\pm$0.15 1.06$\pm$0.12 1.26$\pm$0.14 0.55$\pm$0.40 0.57$\pm$0.37
0.65$\pm$0.38 30 2.53$\pm$0.18 2.56$\pm$0.17 2.56$\pm$0.18 1.03$\pm$0.12
1.16$\pm$0.25 1.71$\pm$0.23 0.51$\pm$0.33 0.53$\pm$0.33 0.59$\pm$0.34 40
2.53$\pm$0.18 2.55$\pm$0.18 2.55$\pm$0.19 1.03$\pm$0.11 1.15$\pm$0.24
1.19$\pm$0.18 0.51$\pm$0.33 0.57$\pm$0.32 0.57$\pm$0.36

###  5.4 Comparisons on controllability

Figure 2 illustrates the MAE trend between the specific property of generated
molecules and the intended one as the proportion of labeled training data
rises. Text2Data achieves superior performance than EDM-finetune and EDM on
all six properties by a remarkable margin. The results also indicate that
certain properties, such as $\epsilon_{LUMO}$ and $Cv$, are more readily
controllable. For these properties, the performance of the three models
converges as the amount of labeled training data becomes sufficiently large
(Figure 2). We further increase the proportion of available labels in the
dataset up to $100\%$ and, as indicated in Appendix Table 3, the MAE keeps
increasing when more labels are involved during training, and it gradually
converges in the end. Appendix Table 4 also suggests that Text2Data surpasses
the data augmentation-based method, which may suffer from the potentially poor
alignment between text and data and the potential overfitting.

We depict the molecules generated as the text descriptor for polarizability
shifts from “very low” to “very high” in Figure 3. Polarizability indicates
the inclination of the molecule to form an electric dipole moment under an
external electric field. As $\alpha$ values rise, we expect to see molecules
with less symmetrical forms, as evidenced in Figure 3. This trend suggests the
validity of generated molecules by Text2Data and its fine-grained
controllability.

As suggested in Table 1, Text2Data also outperforms MDM-finetune and MDM in
the controllable generation of motions from text descriptions. While MDM-
finetune is slightly better than Text2Data when the proportion of labeled
training data is small-owing to milder catastrophic forgetting during
finetuning with a smaller sample size-Text2Data consistently surpasses both
MDM-finetune and MDM as the volume of labeled training data increases.
Specifically, in this situation, Text2Data surpasses MDM-finetune and MDM in R
Precision with average margins of 2.31$\%$ and 5.57$\%$, respectively, and in
Multimodal Distance with average margins of 0.93$\%$ and 3.30$\%$,
respectively. The results also indicate that an increase in labeled training
data enhances the performance of controllability, which is expected as more
supervision is involved.

Additionally, we evaluate the controllability of Text2Data, along with its
baseline comparisons, by utilizing MAE to measure the congruence between the
property of generated data and the intended one within the Time Series
dataset. As indicated in Table 2, Text2Data consistently excels over two
baselines, DiffTS-finetune and DiffTS, across all three properties assessed
during the experiments. Results of another three properties are presented in
Appendix Table 5, which suggests the similar conclusion. Specifically,
Text2Data and DiffTS-finetune both show a marked improvement over DiffTS in
controlling frequency, variance, and skewness. They also exhibit a slight edge
in controlling mean, number of peaks, and linearity. The enhanced performance
of Text2Data correlates with its proficiency in alleviating the issue of
catastrophic forgetting while maintaining a pursuit of controllability.

(a) Text2Data

(b) DiffTS-finetune

(c) DiffTS

Figure 4: t-SNE visualization on time series data generated by Text2Data,
DiffTS-finetune model and DiffTS. Red denotes ground truth, and blue denotes
generated data.

###  5.5 Comparisons on generation quality

Text2Data not only demonstrates superior performance in controllable text-to-
data generation but also sustains competitive generation quality relative to
baseline models.

When generating molecules from Text2Data and its baseline comparisons, as
shown in Appendix Table 6, we compute $-\log p$ and validity to evaluate
generation quality. The performance of Text2Data is consistently better. It
surpasses EDM-finetune and EDM by average margins of 19.07$\%$ and 58.03$\%$,
respectively. It is 1.98$\%$ and 10.59$\%$ better than EDM-finetune and EDM on
average, respectively, regarding validity of molecules on average. Besides, we
evaluate Text2Data compared with EDM-finetune and EDM on molecular stability
and atom stability (Appendix Table 7). Text2Data exceeds EDM-finetune and EDM
by average margins of 2.34$\%$ and 17.31$\%$, respectively, in terms of
molecular stability. It is also 0.29$\%$ and 1.21$\%$ better than EDM-finetune
and EDM on average, respectively, regarding atom stability in molecules. The
consistent improvements on all the three models result from our superior
performance of Text2Data on properties (e.g., molecular stability) that are
hard to control.

As indicated in Appendix Table 8, quantitative assessment of motion generation
from text shows that Text2Data surpasses the baseline methods in both quality
and diversity. Particularly, Text2Data outperforms MDM-finetune and MDM by
2.73$\%$ and 36.05$\%$ on average, respectively, regarding FID score.
Regarding diversity, Text2Data surpasses MDM-finetune and MDM by average
margins of 0.81$\%$ and 3.71$\%$, respectively. Enhanced performance is
derived from the ability of Text2Data to fully leverage all samples in the
dataset, while effectively mitigating catastrophic forgetting during
finetuning.

We evaluate the overall quality of generating time series by making t-SNE
plots of generated time series against ground truth. Substantial overlap
between the generated time series and the ground truth suggests a closer
distribution alignment, implying a better performance of Text2Data. As
demonstrated in Figure 4, specifically, the red pattern represents the t-SNE
of the ground-truth time series, whereas the blue pattern represents the t-SNE
of the generated time series according to the same text description. Compared
with DiffTS-finetune and DiffTS, Text2Data corresponds to the largest overlap
between distributions of the generated and the ground-truth time series,
suggesting its superior ability to precisely generate data according to the
text description. The non-overlapping part may result from the diversity of
generated time series or some other properties not labeled in the dataset so
that not controlled by text description. The inferior performance of DiffTS
stems from its training solely on labeled data, potentially leading to an
incomplete understanding of the overall data distribution and a risk of
overfitting due to limited size of labeled data. DiffTS may only partially
capture the data distribution based on textual descriptions due to its
susceptibility to catastrophic forgetting, which also heightens the risk of
overfitting.

##  6 Conclusion

In this paper, we propose a novel constraint optimization-based framework,
Text2Data, to improve the quality and controllability of text-to-data
generation for various modalities in low-resource scenarios using diffusion
models. Text2Data employs unlabeled data to capture the prevailing data
distribution using an unsupervised diffusion model. It is then finetuned on
text-labeled data, employing a novel constraint optimization-based learning
objective to ensure controllability while reducing catastrophic forgetting.
Comprehensive experiments are conducted on real-world datasets and our model
shows consistently superior performance to recent baselines. While Text2Data
is presented as a diffusion-based framework in this article, it can be
seamlessly adapted to other generative models such as generative adversarial
networks.

##  7 Ethical statement

We develop our method from publicly available QM9 (Ramakrishnan et al., 2014)
and HumanML3D (Guo et al., 2022a) datasets, and stock data from public Yahoo
Finance222https://finance.yahoo.com/. It is important to note that, like other
text-to-data models, our implementation will likely reflect the socioeconomic
and entity biases inherent in datasets that we use. Additionally, although our
method is designed for controllable data generation from text, we are not able
to control the prompt that user inputs, which may contain improper contents.

## References

  * Cheuk et al. (2021) Cheuk, K. W., Herremans, D., and Su, L.  Reconvat: A semi-supervised automatic music transcription framework for low-resource real-world data.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pp. 3918–3926, 2021.
  * Dhariwal & Nichol (2021) Dhariwal, P. and Nichol, A.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ , 34:8780–8794, 2021.
  * Du et al. (2020) Du, S., Li, T., Yang, Y., and Horng, S.-J.  Multivariate time series forecasting via attention-based encoder–decoder framework.  _Neurocomputing_ , 388:269–279, 2020.
  * Garcia Satorras et al. (2021) Garcia Satorras, V., Hoogeboom, E., Fuchs, F., Posner, I., and Welling, M.  E (n) equivariant normalizing flows.  _Advances in Neural Information Processing Systems_ , 34:4181–4192, 2021.
  * Gong & Liu (2021) Gong, C. and Liu, X.  Bi-objective trade-off with dynamic barrier gradient descent.  _Advances in Neural Information Processing Systems_ , 2021.
  * Gozalo-Brizuela & Garrido-Merchan (2023) Gozalo-Brizuela, R. and Garrido-Merchan, E. C.  Chatgpt is not all you need. a state of the art review of large generative ai models.  _arXiv preprint arXiv:2301.04655_ , 2023.
  * Guo et al. (2020) Guo, C., Zuo, X., Wang, S., Zou, S., Sun, Q., Deng, A., Gong, M., and Cheng, L.  Action2motion: Conditioned generation of 3d human motions.  In _Proceedings of the 28th ACM International Conference on Multimedia_ , pp. 2021–2029, 2020.
  * Guo et al. (2022a) Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., and Cheng, L.  Generating diverse and natural 3d human motions from text.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 5152–5161, June 2022a.
  * Guo et al. (2022b) Guo, C., Zou, S., Zuo, X., Wang, S., Ji, W., Li, X., and Cheng, L.  Generating diverse and natural 3d human motions from text.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5152–5161, 2022b.
  * Hedderich et al. (2020) Hedderich, M. A., Lange, L., Adel, H., Strötgen, J., and Klakow, D.  A survey on recent approaches for natural language processing in low-resource scenarios.  _arXiv preprint arXiv:2010.12309_ , 2020.
  * Ho & Salimans (2022) Ho, J. and Salimans, T.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. (2022) Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al.  Imagen video: High definition video generation with diffusion models.  _arXiv preprint arXiv:2210.02303_ , 2022.
  * Honorio & Jaakkola (2014) Honorio, J. and Jaakkola, T.  Tight bounds for the expected risk of linear classifiers and pac-bayes finite-sample guarantees.  In _Artificial Intelligence and Statistics_ , pp. 384–392. PMLR, 2014.
  * Hoogeboom et al. (2022) Hoogeboom, E., Satorras, V. G., Vignac, C., and Welling, M.  Equivariant diffusion for molecule generation in 3d.  In _International conference on machine learning_ , pp. 8867–8887. PMLR, 2022.
  * Hu et al. (2022) Hu, Y., Luo, C., and Chen, Z.  Make it move: controllable image-to-video generation with text descriptions.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 18219–18228, 2022.
  * Huang et al. (2022) Huang, R., Zhao, Z., Liu, H., Liu, J., Cui, C., and Ren, Y.  Prodiff: Progressive fast diffusion model for high-quality text-to-speech.  In _Proceedings of the 30th ACM International Conference on Multimedia_ , pp. 2595–2605, 2022.
  * Huang et al. (2023) Huang, R., Huang, J., Yang, D., Ren, Y., Liu, L., Li, M., Ye, Z., Liu, J., Yin, X., and Zhao, Z.  Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models.  _arXiv preprint arXiv:2301.12661_ , 2023.
  * Iman et al. (2023) Iman, M., Arabnia, H. R., and Rasheed, K.  A review of deep transfer learning and recent advancements.  _Technologies_ , 11(2):40, 2023.
  * Irwin et al. (2012) Irwin, J. J., Sterling, T., Mysinger, M. M., Bolstad, E. S., and Coleman, R. G.  Zinc: a free tool to discover chemistry for biology.  _Journal of chemical information and modeling_ , 52(7):1757–1768, 2012.
  * Ito & Johnson (2017) Ito, K. and Johnson, L.  The lj speech dataset.  https://keithito.com/LJ-Speech-Dataset/, 2017.
  * Jiang et al. (2021) Jiang, Y., Huang, Z., Pan, X., Loy, C. C., and Liu, Z.  Talk-to-edit: Fine-grained facial editing via dialog.  In _Proceedings of International Conference on Computer Vision (ICCV)_ , 2021.
  * Jin et al. (2018) Jin, W., Barzilay, R., and Jaakkola, T.  Junction tree variational autoencoder for molecular graph generation.  In _International conference on machine learning_ , pp. 2323–2332. PMLR, 2018.
  * Kawar et al. (2023) Kawar, B., Zada, S., Lang, O., Tov, O., Chang, H., Dekel, T., Mosseri, I., and Irani, M.  Imagic: Text-based real image editing with diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6007–6017, 2023.
  * Kim et al. (2022) Kim, H., Kim, S., and Yoon, S.  Guided-tts: A diffusion model for text-to-speech via classifier guidance.  In _International Conference on Machine Learning_ , pp. 11119–11133. PMLR, 2022.
  * Kong et al. (2020) Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B.  Diffwave: A versatile diffusion model for audio synthesis.  _arXiv preprint arXiv:2009.09761_ , 2020.
  * Kumari et al. (2023) Kumari, N., Zhang, B., Zhang, R., Shechtman, E., and Zhu, J.-Y.  Multi-concept customization of text-to-image diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 1931–1941, 2023.
  * Li et al. (2023) Li, M., Duan, Y., Zhou, J., and Lu, J.  Diffusion-sdf: Text-to-shape via voxelized diffusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 12642–12651, 2023.
  * Li et al. (2018) Li, Y., Min, M., Shen, D., Carlson, D., and Carin, L.  Video generation from text.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32, 2018.
  * Lim et al. (2023) Lim, H., Kim, M., Park, S., and Park, N.  Regular time-series generation using sgm.  _arXiv preprint arXiv:2301.08518_ , 2023.
  * Lin et al. (2023) Lin, C.-H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.-Y., and Lin, T.-Y.  Magic3d: High-resolution text-to-3d content creation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 300–309, 2023.
  * Lin et al. (2014) Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pp. 740–755. Springer, 2014.
  * Liu et al. (2023) Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D.  AudioLDM: Text-to-audio generation with latent diffusion models.  In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pp. 21450–21474. PMLR, 23–29 Jul 2023.
  * Ma et al. (2023) Ma, J., Zhao, M., Chen, C., Wang, R., Niu, D., Lu, H., and Lin, X.  Glyphdraw: Learning to draw chinese characters in image synthesis models coherently.  _arXiv preprint arXiv:2303.17870_ , 2023.
  * Mahmood et al. (2019) Mahmood, N., Ghorbani, N., Troje, N. F., Pons-Moll, G., and Black, M. J.  Amass: Archive of motion capture as surface shapes.  In _Proceedings of the International Conference on Computer vision (ICCV)_ , pp. 5442–5451, 2019.
  * Meng et al. (2021) Meng, L., Xu, J., Tan, X., Wang, J., Qin, T., and Xu, B.  Mixspeech: Data augmentation for low-resource automatic speech recognition.  In _ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 7008–7012. IEEE, 2021.
  * Oord et al. (2016) Oord, A. v. d., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K.  Wavenet: A generative model for raw audio.  _arXiv preprint arXiv:1609.03499_ , 2016.
  * Pratap et al. (2020) Pratap, V., Xu, Q., Sriram, A., Synnaeve, G., and Collobert, R.  Mls: A large-scale multilingual dataset for speech research.  _ArXiv_ , abs/2012.03411, 2020.
  * Raffel et al. (2020) Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67, 2020.  URL http://jmlr.org/papers/v21/20-074.html.
  * Ramakrishnan et al. (2014) Ramakrishnan, R., Dral, P. O., Rupp, M., and Von Lilienfeld, O. A.  Quantum chemistry structures and properties of 134 kilo molecules.  _Scientific data_ , 1(1):1–7, 2014.
  * Rasul et al. (2021) Rasul, K., Seward, C., Schuster, I., and Vollgraf, R.  Autoregressive denoising diffusion models for multivariate probabilistic time series forecasting.  In _International Conference on Machine Learning_ , pp. 8857–8868. PMLR, 2021.
  * Ruiz et al. (2023) Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 22500–22510, 2023.
  * Sanghi et al. (2022) Sanghi, A., Chu, H., Lambourne, J. G., Wang, Y., Cheng, C.-Y., Fumero, M., and Malekshan, K. R.  Clip-forge: Towards zero-shot text-to-shape generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 18603–18613, 2022.
  * Tevet et al. (2023) Tevet, G., Raab, S., Gordon, B., Shafir, Y., Cohen-or, D., and Bermano, A. H.  Human motion diffusion model.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=SJ1kSyO2jwu.
  * Thomas et al. (2013) Thomas, S., Seltzer, M. L., Church, K., and Hermansky, H.  Deep neural network features and semi-supervised training for low resource speech recognition.  In _2013 IEEE international conference on acoustics, speech and signal processing_ , pp. 6704–6708. IEEE, 2013.
  * Tits et al. (2020) Tits, N., El Haddad, K., and Dutoit, T.  Exploring transfer learning for low resource emotional tts.  In _Intelligent Systems and Applications: Proceedings of the 2019 Intelligent Systems Conference (IntelliSys) Volume 1_ , pp. 52–60. Springer, 2020.
  * Tu et al. (2019) Tu, T., Chen, Y.-J., Yeh, C.-c., and Lee, H.-Y.  End-to-end text-to-speech for low-resource languages by cross-lingual transfer learning.  _arXiv preprint arXiv:1904.06508_ , 2019.
  * Voleti et al. (2022) Voleti, V., Jolicoeur-Martineau, A., and Pal, C.  Mcvd-masked conditional video diffusion for prediction, generation, and interpolation.  _Advances in Neural Information Processing Systems_ , 35:23371–23385, 2022.
  * Wang et al. (2020) Wang, C., Wu, A., and Pino, J.  Covost 2: A massively multilingual speech-to-text translation corpus, 2020.
  * Wang et al. (2022) Wang, S., Du, Y., Guo, X., Pan, B., Qin, Z., and Zhao, L.  Controllable data generation by deep learning: A review.  _arXiv preprint arXiv:2207.09542_ , 2022.
  * Yang et al. (2023) Yang, D., Yu, J., Wang, H., Wang, W., Weng, C., Zou, Y., and Yu, D.  Diffsound: Discrete diffusion model for text-to-sound generation.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2023.
  * Yi et al. (2018) Yi, J., Tao, J., Wen, Z., and Bai, Y.  Language-adversarial transfer learning for low-resource speech recognition.  _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 27(3):621–630, 2018.
  * Yin et al. (2023) Yin, Y., Kaddour, J., Zhang, X., Nie, Y., Liu, Z., Kong, L., and Liu, Q.  Ttida: Controllable generative data augmentation via text-to-text and text-to-image models.  _arXiv preprint arXiv:2304.08821_ , 2023.
  * Yoon et al. (2019) Yoon, J., Jarrett, D., and Van der Schaar, M.  Time-series generative adversarial networks.  _Advances in neural information processing systems_ , 32, 2019.
  * Zang & Wan (2019) Zang, H. and Wan, X.  A semi-supervised approach for low-resourced text generation.  _arXiv preprint arXiv:1906.00584_ , 2019.
  * Zhang & Agrawala (2023) Zhang, L. and Agrawala, M.  Adding conditional control to text-to-image diffusion models.  _Proceedings of the International Conference on Computer Vision (ICCV)_ , 2023.
  * Zhang et al. (2023) Zhang, Z., Han, L., Ghosh, A., Metaxas, D. N., and Ren, J.  Sine: Single image editing with text-to-image diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 6027–6037, 2023.

##  Appendix A Derivation of diffusion model

We introduce the framework of the classifier-free diffusion model (Ho &
Salimans, 2022) that we use, including the forward process, reverse process
and its learning objective. Let ${\mathbf{x}}^{0:T}$ be
$\\{{\mathbf{x}},{\mathbf{x}}^{(1)},...,{\mathbf{x}}^{(T)}\\}$. Diffusion
models have the form $p_{\theta}({\mathbf{x}})=\int
p_{\theta}({\mathbf{x}},{\mathbf{x}}^{1:T})d{\mathbf{x}}^{0:T}$, where
${\mathbf{x}}^{(1)},{\mathbf{x}}^{(2)},...,{\mathbf{x}}^{(T)}$ are latents of
the same dimensionality as the data ${\mathbf{x}}$.

###  A.1 Forward process

First, the data ${\mathbf{x}}$ is diffused up to time $T$ by adding noise. The
posterior $q({\mathbf{x}}^{1:T}|{\mathbf{x}})$ is fixed to a Markov Chain that
gradually adds noise to the data according to a variance schedule
$\beta_{1},\beta_{2},...,\beta_{T}$:

$\displaystyle
q({\mathbf{x}}^{(1)},{\mathbf{x}}^{(2)},...,{\mathbf{x}}^{(T)}|{\mathbf{x}})$
$\displaystyle=\prod_{t=1}^{T}q({\mathbf{x}}^{(t)}|{\mathbf{x}}^{(t-1)})$ (9)
$\displaystyle q({\mathbf{x}}^{(t)}|{\mathbf{x}}^{(t-1)})$
$\displaystyle=\mathcal{N}({\mathbf{x}}^{(t)};\sqrt{1-\beta_{t}}{\mathbf{x}}_{t-1},\beta_{t}\bm{I})$
(10)

Let $\alpha_{t}=1-\beta_{t}$ and $\bar{\alpha}_{t}=\prod_{s=1}^{t}\alpha_{s}$,
we can derive:

$\displaystyle
q({\mathbf{x}}^{(t)}|{\mathbf{x}})=\mathcal{N}({\mathbf{x}}^{(t)};\sqrt{\bar{\alpha}_{t}}{\mathbf{x}},(1-\bar{\alpha}_{t})\bm{I})$
(11)

###  A.2 Reverse process

The diffusion model aims to learn
$p_{\theta}({\mathbf{x}}^{0:T},{\mathbf{c}})$ for the paired data s.t.
${\mathbf{x}},{\mathbf{c}}\in{\mathcal{D}}_{p}$, which is defined as a Markov
Chain with learned Gaussian transitions starting from
$p({\mathbf{x}}^{(T)})=\mathcal{N}({\mathbf{x}}^{(T)};\bm{0},\bm{I})$.

$\displaystyle p_{\theta}({\mathbf{x}}^{0:T},{\mathbf{c}})$
$\displaystyle=p({\mathbf{x}}^{(T)},{\mathbf{c}})\prod_{t=1}^{T}p_{\theta}({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{c}})=p({\mathbf{x}}^{(T)},{\mathbf{c}})\prod_{t=1}^{T}p_{\theta}({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{c}})$
(12) $\displaystyle
p_{\theta}({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{c}})$
$\displaystyle={\mathcal{N}}({\mathbf{x}}^{(t-1)};{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t),\Sigma_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t))$
(13)

###  A.3 Derivation of learning objective

To maximize the joint likelihood $p_{\theta}({\mathbf{x}},{\mathbf{c}})$, we
derive evidence variational lower bound (ELBO) as the learning objective.
Specifically, we have:

$\displaystyle\log p_{\theta}({\mathbf{x}},{\mathbf{c}})$
$\displaystyle=\log\int
p_{\theta}({\mathbf{x}},{\mathbf{x}}^{(1)},...,{\mathbf{x}}^{(T)},{\mathbf{c}})d{\mathbf{x}}^{1:T}$
(14) $\displaystyle=\log\int
p_{\theta}({\mathbf{x}}^{0:T},{\mathbf{c}})d{\mathbf{x}}^{1:T}$ (15)
$\displaystyle=\log\int\frac{p_{\theta}({\mathbf{x}}^{0:T},{\mathbf{c}})q({\mathbf{x}}^{1:T}|{\mathbf{x}})}{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}d{\mathbf{x}}^{1:T}$
(16)
$\displaystyle=\log\mathbb{E}_{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}[\frac{p_{\theta}({\mathbf{x}}^{0:T},{\mathbf{c}})}{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}]$
(17)

Based on Jensen’s inequality, the ELBO of Eq. LABEL:eq:logp is as below:

$\displaystyle\log p_{\theta}({\mathbf{x}},{\mathbf{c}})$
$\displaystyle=\log\mathbb{E}_{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}[\frac{p_{\theta}({\mathbf{x}}^{0:T},{\mathbf{c}})}{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}]$
(19)
$\displaystyle\geq\mathbb{E}_{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}[\log\frac{p_{\theta}({\mathbf{x}}^{0:T},{\mathbf{c}})}{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}]$
(20) $\displaystyle=\mathbb{E}_{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}[\log
p_{\theta}({\mathbf{x}}^{(T)},{\mathbf{c}})+\sum_{t\geq
1}\log\frac{p_{\theta}({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{c}})}{q({\mathbf{x}}^{(t)}|{\mathbf{x}}^{(t-1)})}]$
(21)

Since $p({\mathbf{x}}^{(T)})=\mathcal{N}({\mathbf{x}}^{(T)};\bm{0},\bm{I})$
and not dependent on ${\mathbf{c}}$, as a result, $\log
p({\mathbf{x}}^{(T)},{\mathbf{c}})=\log
p({\mathbf{x}}^{(T)}|{\mathbf{c}})p({\mathbf{c}})=\log
p({\mathbf{x}}^{(T)})p({\mathbf{c}})=\log p({\mathbf{x}}^{(T)})+\log
p({\mathbf{c}})$. Then Eq. 21 above becomes:

$\displaystyle\log p_{\theta}({\mathbf{x}},{\mathbf{c}})$
$\displaystyle\geq\mathbb{E}_{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}[\log
p_{\theta}({\mathbf{x}}^{(T)},{\mathbf{c}})+\sum_{t\geq
1}\log\frac{p_{\theta}({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{c}})}{q({\mathbf{x}}^{(t)}|{\mathbf{x}}^{(t-1)})}]$
(22) $\displaystyle=\mathbb{E}_{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}[\log
p({\mathbf{x}}^{(T)})+\log p({\mathbf{c}})+\sum_{t\geq
1}\log\frac{p_{\theta}({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{c}})}{q({\mathbf{x}}^{(t)}|{\mathbf{x}}^{(t-1)})}]$
(23)

Given the posterior formed as the Markov Chain (Eq. 10), we have
$q({\mathbf{x}}_{t}|{\mathbf{x}}_{t-1})=q({\mathbf{x}}^{(t)}|{\mathbf{x}}^{(t-1)},{\mathbf{x}})$.
Therefore, we minimize the loss for the paired data of
$\\{{\mathbf{x}},{\mathbf{c}}\\}\in{\mathcal{D}}_{p}$:

$\displaystyle{\mathcal{L}}_{ELBO}$
$\displaystyle=-\mathbb{E}_{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}[\log
p({\mathbf{x}}^{(T)})+\log p({\mathbf{c}})+\sum_{t\geq
1}\log\frac{p_{\theta}({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{c}})}{q({\mathbf{x}}^{(t)}|{\mathbf{x}}^{(t-1)})}]$
(24) $\displaystyle=\mathbb{E}_{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}[-\log
p({\mathbf{x}}^{(T)})-\log
p({\mathbf{c}})-\sum_{t>1}\log\frac{p_{\theta}({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{c}})}{q({\mathbf{x}}^{(t)}|{\mathbf{x}}^{(t-1)})}-\log\frac{p_{\theta}({\mathbf{x}}|{\mathbf{x}}^{(1)},{\mathbf{c}})}{q({\mathbf{x}}^{(1)}|{\mathbf{x}})}]$
(25) $\displaystyle=\mathbb{E}_{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}[-\log
p({\mathbf{x}}^{(T)})-\log
p({\mathbf{c}})-\sum_{t>1}\log\frac{p_{\theta}({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{c}})}{q({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{x}})}\cdot\frac{q({\mathbf{x}}^{(t-1)}|{\mathbf{x}})}{q({\mathbf{x}}^{(t)}|{\mathbf{x}})}-\log\frac{p_{\theta}({\mathbf{x}}|{\mathbf{x}}^{(1)},{\mathbf{c}})}{q({\mathbf{x}}^{(1)}|{\mathbf{x}})}]$
(26)
$\displaystyle=\mathbb{E}_{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}[-\log\frac{p({\mathbf{x}}^{(T)})}{q({\mathbf{x}}^{(T)}|{\mathbf{x}})}-\log
p({\mathbf{c}})-\sum_{t>1}\log\frac{p_{\theta}({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{c}})}{q({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{x}})}-\log
p_{\theta}({\mathbf{x}}|{\mathbf{x}}^{(1)},{\mathbf{c}})]$ (27)
$\displaystyle=D_{KL}(q({\mathbf{x}}^{(T)}|{\mathbf{x}})||p({\mathbf{x}}^{(T)}))+\sum_{t>1}D_{KL}(q({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{x}})||p_{\theta}({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{c}}))-\mathbb{E}_{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}[\log
p_{\theta}({\mathbf{x}}|{\mathbf{x}}^{(1)},{\mathbf{c}})]-\log
p({\mathbf{c}})$ (28) $\displaystyle=L_{T}+\sum_{t>1}L_{t-1}+L_{0}-\log
p({\mathbf{c}})$ (29) $\displaystyle\propto L_{T}+\sum_{t>1}L_{t-1}+L_{0},$
(30)

where we have:

$\displaystyle L_{T}$
$\displaystyle=D_{KL}(q({\mathbf{x}}^{(T)}|{\mathbf{x}})||p({\mathbf{x}}^{(T)}))$
(31) $\displaystyle L_{t-1}$
$\displaystyle=D_{KL}(q({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{x}})||p_{\theta}({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{c}}))$
(32) $\displaystyle L_{0}$
$\displaystyle=-\mathbb{E}_{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}[\log
p_{\theta}({\mathbf{x}}|{\mathbf{x}}^{(1)},{\mathbf{c}})]$ (33)

We can directly derive
$q({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{x}})$ as:

$\displaystyle q({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{x}})$
$\displaystyle=\frac{q({\mathbf{x}}^{(t-1)},{\mathbf{x}}^{(t)},{\mathbf{x}})}{q({\mathbf{x}}^{(t)},{\mathbf{x}})}$
(34)
$\displaystyle=\frac{q({\mathbf{x}}^{(t)}|{\mathbf{x}}^{(t-1)},{\mathbf{x}})q({\mathbf{x}}^{(t-1)},{\mathbf{x}})}{q({\mathbf{x}}^{(t)},{\mathbf{x}})}$
(35)
$\displaystyle=\frac{q({\mathbf{x}}^{(t)}|{\mathbf{x}}^{(t-1)})q({\mathbf{x}}^{(t-1)}|{\mathbf{x}})}{q({\mathbf{x}}^{(t)}|{\mathbf{x}})}$
(36)
$\displaystyle\propto\exp[-\frac{1}{2}(\frac{({\mathbf{x}}^{(t)}-\sqrt{\alpha_{t}}{\mathbf{x}}^{(t-1)2})}{\beta_{t}}+\frac{({\mathbf{x}}^{(t-1)}-\sqrt{\bar{\alpha}_{t-1}}{\mathbf{x}})^{2}}{1-\bar{\alpha}_{t-1}}-\frac{({\mathbf{x}}^{(t)}-\sqrt{\bar{\alpha}_{t}}{\mathbf{x}})^{2}}{1-\bar{\alpha}_{t}})]$
(37)
$\displaystyle=\exp[-\frac{1}{2}((\frac{\alpha_{t}}{\beta_{t}}+\frac{1}{1-\bar{\alpha}_{t-1}}){\mathbf{x}}^{(t-1)2}-2(\frac{\sqrt{\alpha_{t}}}{\beta_{t}}{\mathbf{x}}^{(t)}+\frac{\sqrt{\bar{\alpha}_{t-1}}}{1-\bar{\alpha}_{t-1}}{\mathbf{x}}){\mathbf{x}}^{(t-1)}+C({\mathbf{x}}^{(t)},{\mathbf{x}}))$
(38)
$\displaystyle=\exp[-\frac{1}{2}(A{\mathbf{x}}^{(t-1)2}-2B{\mathbf{x}}^{(t-1)}+C({\mathbf{x}}^{(t)},{\mathbf{x}}))],$
(39)

where $A=\frac{\alpha_{t}}{\beta_{t}}+\frac{1}{1-\bar{\alpha}_{t-1}}$ and
$B=\frac{\sqrt{\alpha_{t}}}{\beta_{t}}{\mathbf{x}}^{(t)}+\frac{\sqrt{\bar{\alpha}_{t-1}}}{1-\bar{\alpha}_{t-1}}{\mathbf{x}}$.
Then we have:

$\displaystyle\Sigma=\frac{1}{A}=\frac{1}{\frac{\alpha_{t}}{\beta_{t}}+\frac{1}{1-\bar{\alpha}_{t-1}}}=\frac{(1-\bar{\alpha}_{t-1})\beta_{t}}{(1-\bar{\alpha}_{t-1})\alpha_{t}+\beta_{t}}=\frac{(1-\bar{\alpha}_{t-1})\beta_{t}}{\alpha_{t}+\beta_{t}-\bar{\alpha}_{t-1}\alpha_{t}}=\frac{(1-\bar{\alpha}_{t-1})\beta_{t}}{1-\bar{\alpha}_{t}}$
(40)

Also we have:

$\displaystyle\mu({\mathbf{x}}^{(t)},{\mathbf{x}})$
$\displaystyle=\frac{B}{A}=B\Sigma$ (41)
$\displaystyle=(\frac{\sqrt{\alpha_{t}}}{\beta_{t}}{\mathbf{x}}^{(t)}+\frac{\sqrt{\bar{\alpha}_{t-1}}}{1-\bar{\alpha}_{t-1}}{\mathbf{x}})\Sigma$
(42)
$\displaystyle=(\frac{\sqrt{\alpha_{t}}}{\beta_{t}}{\mathbf{x}}^{(t)}+\frac{\sqrt{\bar{\alpha}_{t-1}}}{1-\bar{\alpha}_{t-1}}{\mathbf{x}})\frac{(1-\bar{\alpha}_{t-1})\beta_{t}}{1-\bar{\alpha}_{t}}$
(43)
$\displaystyle=\frac{(1-\bar{\alpha}_{t-1})\sqrt{\alpha_{t}}}{1-\bar{\alpha}_{t}}{\mathbf{x}}^{(t)}+\frac{\sqrt{\bar{\alpha}_{t-1}}\beta_{t}}{1-\bar{\alpha}_{t}}{\mathbf{x}}$
(44)

Based on Eq. 11, we can reparameterize ${\mathbf{x}}^{(t)}$ as:
${\mathbf{x}}^{(t)}=\sqrt{\bar{\alpha}_{t}}{\mathbf{x}}-\sqrt{1-\bar{\alpha}_{t}}{\bm{\epsilon}}$
where ${\bm{\epsilon}}\sim\mathcal{N}(\bm{0},\bm{I})$. Then, plug
${\mathbf{x}}=\frac{1}{\sqrt{\bar{\alpha}_{t}}}({\mathbf{x}}^{(t)}-\sqrt{1-\bar{\alpha}_{t}}{\bm{\epsilon}})$
in to Eq. 44, we have:

$\displaystyle\mu({\mathbf{x}}^{(t)},{\mathbf{x}})$
$\displaystyle=\frac{(1-\bar{\alpha}_{t-1})\sqrt{\alpha_{t}}}{1-\bar{\alpha}_{t}}{\mathbf{x}}^{(t)}+\frac{\sqrt{\bar{\alpha}_{t-1}}\beta_{t}}{1-\bar{\alpha}_{t}}\frac{1}{\sqrt{\bar{\alpha}_{t}}}({\mathbf{x}}^{(t)}-\sqrt{1-\bar{\alpha}_{t}}{\bm{\epsilon}})$
(45)
$\displaystyle=\frac{(1-\bar{\alpha}_{t-1})\sqrt{\alpha_{t}}}{1-\bar{\alpha}_{t}}{\mathbf{x}}^{(t)}+\frac{\beta_{t}}{(1-\bar{\alpha}_{t})\sqrt{\alpha_{t}}}({\mathbf{x}}^{(t)}-\sqrt{1-\bar{\alpha}_{t}}{\bm{\epsilon}})$
(46)
$\displaystyle=\frac{(1-\bar{\alpha}_{t-1})\alpha_{t}+\beta_{t}}{(1-\bar{\alpha}_{t})\sqrt{\alpha_{t}}}{\mathbf{x}}^{(t)}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}\sqrt{\alpha_{t}}}{\bm{\epsilon}}$
(47)
$\displaystyle=\frac{(\alpha_{t}+\beta_{t})-\bar{\alpha}_{t-1}\alpha_{t}}{(1-\bar{\alpha}_{t})\sqrt{\alpha_{t}}}{\mathbf{x}}^{(t)}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}\sqrt{\alpha_{t}}}{\bm{\epsilon}}$
(48)
$\displaystyle=\frac{1-\bar{\alpha}_{t}}{(1-\bar{\alpha}_{t})\sqrt{\alpha_{t}}}{\mathbf{x}}^{(t)}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}\sqrt{\alpha_{t}}}{\bm{\epsilon}}$
(49)
$\displaystyle=\frac{1}{\sqrt{\alpha_{t}}}({\mathbf{x}}^{(t)}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}{\bm{\epsilon}})$
(50)

Therefore,
$q({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{x}})\sim\mathcal{N}({\mathbf{x}}^{(t-1)};\frac{1}{\sqrt{\alpha_{t}}}({\mathbf{x}}^{(t)}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}{\bm{\epsilon}}),\frac{(1-\bar{\alpha}_{t-1})\beta_{t}}{1-\bar{\alpha}_{t}}\bm{I})$,
where ${\bm{\epsilon}}\sim{\mathcal{N}}(\bm{0},\bm{I})$. We fix $\beta_{t}$ to
constants. As a result, the posterior $q(\cdot)$ has no learnable parameters,
and $L_{T}$ does not need to be trained and can be disregarded during
training.

For
$p_{\theta}({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{c}})=\mathcal{N}({\mathbf{x}}^{(t-1)};{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t),\Sigma_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t))$
where $1<t<T$, let
$\Sigma_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t)=\sigma_{t}^{2}\bm{I}$ be
time-dependent constants. With
$p_{\theta}({\mathbf{x}}^{(t-1)}|{\mathbf{x}}^{(t)},{\mathbf{c}})=\mathcal{N}({\mathbf{x}}^{(t-1)};{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t),\sigma_{t}^{2}\bm{I})$,
we have:

$\displaystyle
L_{t-1}=\mathbb{E}_{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}[\frac{1}{2\sigma_{t}^{2}}||\tilde{{\bm{\epsilon}}}_{t}({\mathbf{x}}^{(t)},{\mathbf{x}})-{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t)||^{2}]+C,\
\ \text{s.t.}\ \ \\{{\mathbf{x}},{\mathbf{c}}\\}\in\mathcal{D}_{p}$ (51)

where $C$ is a constant not related to ${\mathbf{c}}$,
$\tilde{{\bm{\epsilon}}}_{t}({\mathbf{x}}^{(t)},{\mathbf{x}})=\frac{1}{\sqrt{\alpha_{t}}}({\mathbf{x}}^{(t)}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}}_{t}}{\bm{\epsilon}})$
based on Eq. 44, ${\bm{\epsilon}}\sim\mathcal{N}(\bm{0},\bm{I})$. Plug
$\tilde{{\bm{\epsilon}}}_{t}({\mathbf{x}}^{(t)},{\mathbf{x}})=\frac{1}{\sqrt{\alpha_{t}}}({\mathbf{x}}^{(t)}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}{\bm{\epsilon}})$
and
$\tilde{{\bm{\epsilon}}}_{t}({\mathbf{x}}^{(t)},{\mathbf{x}})=\frac{1}{\sqrt{\alpha_{t}}}({\mathbf{x}}^{(t)}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t))$,
we have:

$\displaystyle
L_{t-1}=\mathbb{E}_{q({\mathbf{x}}^{1:T}|{\mathbf{x}})}[\frac{\beta_{t}}{2\sigma_{t}^{2}\sqrt{\alpha_{t}(1-\bar{\alpha}_{t})}}||{\bm{\epsilon}}-{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t)||^{2}]+C,\
\ \text{s.t.}\ \ \\{{\mathbf{x}},{\mathbf{c}}\\}\in{\mathcal{D}}_{p}$ (52)

A simpler version of empirical loss in our implementation:

$\displaystyle\mathcal{L}_{1}(\theta,{\mathbf{x}},{\mathbf{c}})=\mathbb{E}_{{\mathbf{x}},{\mathbf{c}}\sim
p_{{\mathcal{D}}_{p}}({\mathbf{x}},{\mathbf{c}}),t}||{\bm{\epsilon}}-{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t)||^{2},$
(53)

which is the expression of ${\mathcal{L}}_{2}(\theta)$ in Eq. 5. When compute
${\mathcal{L}}_{1}(\theta)$ in Eq. 5, replace ${\mathbf{c}}$ with $\emptyset$.
When it comes to classifier-free diffusion model, we can train unconditional
denoising diffusion model via ${\mathcal{L}}_{1}(\theta)$ together with the
conditional model via ${\mathcal{L}}_{2}(\theta)$. The occurrence of
$\emptyset$ is sampled with the probability $p_{uncond}$.

##  Appendix B Lexicographic optimization

The model is firstly trained on unlabeled data, and then finetuned on labeled
data under the framework of lexicographic optimization (Gong & Liu, 2021).
Lexicographic optimization is used to balance the optimization of the learning
objective (i.e., $\hat{{\mathcal{L}}}_{2}(\theta)$) and its constraint (i.e.,
$\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)$). Specifically, lexicographic
optimization leverage a dynamic gradient descent (Gong & Liu, 2021):

$\displaystyle\theta\leftarrow\theta-\omega\cdot(\nabla\hat{\mathcal{L}}_{2}(\theta)+\lambda\nabla\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)),$
(54)

where $\omega$ is predefined positive step size, and $\lambda$ is designed to
satisfy the following desiderata:

(1) When the constraint is not satisfied (i.e.,
$\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)>\hat{\xi}$), we should let the
optimization focus on decreasing $\hat{\mathcal{L}}_{1}^{\prime}(\theta)$ to
meet the constraint as fast as possible. At the same time, the learning
objective $\\\ hatgL_{2}(\theta)$ should serve as the secondary objective and
minimized to the degree that it does not hurt the descent of
$\hat{\mathcal{L}}_{1}^{\prime}(\theta)$.

(2) When the constraint is satisfied (i.e.,
$\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)\leq\hat{\xi}$), we should focus on
optimizing $\hat{\mathcal{L}}_{2}(\theta)$. However, the increasing rate of
$\hat{\mathcal{L}}_{1}^{\prime}(\theta)$ should in properly controlled so that
$\theta$ stays inside or nearby the feasible set while we minimize
$\hat{\mathcal{L}}_{2}(\theta)$.

Both properties can be satisfied if $\lambda$ is selected by the following
optimization:

$\displaystyle\lambda\leftarrow\operatorname*{arg\,min}_{\lambda}\\{\|\nabla\hat{\mathcal{L}}_{2}(\theta)-\lambda\|~{}~{}~{}s.t.~{}~{}~{}\nabla\hat{\mathcal{L}}_{1}^{\prime}(\theta)^{T}\lambda\geq\phi(\theta)\\},$
(55)

where we want $\lambda$ to be as close to
$\nabla\hat{\mathcal{L}}_{2}(\theta)$ as much as possible, but subject to a
lower bound on the inner product of
$\nabla\hat{\mathcal{L}}_{1}^{\prime}(\theta)$ and $\lambda$ to make sure that
the change of $\hat{\mathcal{L}}_{1}^{\prime}(\theta)$ is controlled by the
location of $\theta$. The $\phi(\theta)$ is the dynamic barrier function which
balance the loss minimization with constraint satisfaction by controlling the
inner product between $\hat{\mathcal{L}}_{1}^{\prime}(\theta)$ and $\lambda$.
To achieve the desiderata on $\lambda$, we should keep $\phi(\theta)$ to have
the same sign as $\hat{\mathcal{L}}_{1}^{\prime}(\theta)-\hat{\xi}$ so that
the constraint $\hat{\mathcal{L}}_{1}^{\prime}(\theta)\leq\hat{\xi}$ is
equivalent to $\\{\theta:\phi(\theta)\leq 0\\}$, which is:

$\displaystyle
sign(\phi(\theta))=sign(\hat{\mathcal{L}}_{1}^{\prime}(\theta)-\hat{\xi})$
(56)

It is straight to see that the following choice of $\lambda$ satisfies the
dual problem of Eq. 55:

$\displaystyle\lambda=\operatorname*{arg\,min}_{\lambda\geq
0}\\{\|\nabla{\mathcal{L}}_{2}(\theta)+\lambda{\mathcal{L}}_{1}^{\prime}(\theta)\|^{2}-\lambda\phi(\theta)\\}=\max(\frac{\phi(\theta)-\nabla\hat{\mathcal{L}}_{2}(\theta)^{T}\nabla\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)}{\|\nabla\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)\|^{2}},0)$
(57)

with the choice of $\phi(\theta)$:

$\displaystyle\phi(\theta)=$
$\displaystyle\min(\alpha(\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)-\gamma\cdot\hat{\xi}),\beta\|\nabla\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)\|^{2}),$
(58)

where $\alpha$, $\beta$ and $\gamma$ are predefined positive hyperparameters.

##  Appendix C Proof of Theorem 1

First, we define sub-exponential random variable and then and elucidate its
relationship with sub-Gaussian random variable.

######  Definition C.1 (Sub-exponential random variable).

The random variable $X$ with mean $\mu$ is sub-exponential with parameters
$(v,b)$ if for $\forall\lambda<\frac{1}{b}$,
$\mathbb{E}_{X}[\exp\\{\lambda(X-\mu)\\}]\leq\exp(\frac{v^{2}\lambda^{2}}{2})$.

######  Lemma C.2.

The square of a zero-mean sub-Gaussian random variable with parameter
$\sigma^{2}$ is a sub-exponential random variable with parameter
$(4\sqrt{2}\sigma^{2},4\sigma^{2})$ (Honorio & Jaakkola, 2014).

###### Proof.

Given that ${\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},t)$ and
${\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t)$ are sub-Gaussian
random variables parameterized by $\sigma^{2}$, and ${\bm{\epsilon}}$ is the
multivariate standard Gaussian random variable, then their subtraction,
${\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},t)-{\bm{\epsilon}}$ and
${\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t)-{\bm{\epsilon}}$,
are still sub-Gaussian random variables parameterized by $\sigma^{2}+1$. Then
$\hat{{\mathcal{L}}}_{1}(\theta)=\mathbb{E}_{{\mathbf{x}}\sim\hat{p}_{{\mathcal{D}}}({\mathbf{x}}),t}[||{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},t)-{\bm{\epsilon}}||^{2}]$
and
$\hat{{\mathcal{L}}}_{2}(\theta)=\mathbb{E}_{{\mathbf{x}},{\mathbf{c}}\sim\hat{p}_{{\mathcal{D}}_{p}}({\mathbf{x}},{\mathbf{c}}),t}||{\bm{\epsilon}}_{\theta}({\mathbf{x}}^{(t)},{\mathbf{c}},t)-{\bm{\epsilon}}||^{2}$,
the square of sub-Gaussian random variables, are sub-exponential random
variables parameterized by
$(4\sqrt{2}\tilde{\sigma}^{2},4\tilde{\sigma}^{2})$, based on the Lemma above.

Then, based on Bernstein’s inequality, we have:

$\displaystyle
p(|{\mathcal{L}}_{2}(\theta)-\hat{{\mathcal{L}}}_{2}(\theta)|>\epsilon)\leq
2\exp(-\frac{N_{p}\epsilon^{2}}{8\sqrt{2}\tilde{\sigma}^{2}}\wedge\frac{N_{p}\epsilon}{8\tilde{\sigma}^{2}})$
(59)

Based on the union bound inequility, we further have:

$\displaystyle
p(\sup_{\theta\in\Theta}|{\mathcal{L}}_{2}(\theta)-\hat{{\mathcal{L}}}_{2}(\theta)|>\epsilon)\leq$
$\displaystyle\sum_{\theta\in\Theta}p(|{\mathcal{L}}_{2}(\theta)-\hat{{\mathcal{L}}}_{2}(\theta)|>\epsilon)$
$\displaystyle\leq$ $\displaystyle
2|\Theta|\exp(-\frac{N_{p}\epsilon^{2}}{8\sqrt{2}\tilde{\sigma}^{2}}\wedge\frac{N_{p}\epsilon}{8\tilde{\sigma}^{2}})$
(60)

Following the same way, we have:

$\displaystyle
p(\sup_{\theta\in\Theta}|{\mathcal{L}}_{1}^{\prime}(\theta)-\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)|>\epsilon)\leq$
$\displaystyle
2|\Theta|\exp(-\frac{N_{p}\epsilon^{2}}{8\sqrt{2}\tilde{\sigma}^{2}}\wedge\frac{N_{p}\epsilon}{8\tilde{\sigma}^{2}})$
(61) $\displaystyle
p(\sup_{\theta\in\Theta}|{\mathcal{L}}_{1}(\theta)-\hat{{\mathcal{L}}}_{1}(\theta)|>\epsilon)\leq$
$\displaystyle
2|\Theta|\exp(-\frac{N\epsilon^{2}}{8\sqrt{2}\tilde{\sigma}^{2}}\wedge\frac{N\epsilon}{8\tilde{\sigma}^{2}})$
(62)

Let the probability on the RHS of Eq. 60, Eq. 61 and Eq. 62 be $\delta$, then
we compute $\epsilon$ and plug in into LHS, then with the probability of
$1-\delta$ we have:

$\displaystyle\sup_{\theta\in\Theta}|{\mathcal{L}}_{2}(\theta)-\hat{{\mathcal{L}}}_{2}(\theta)|\leq\epsilon_{N_{p}}$
(63)

$\displaystyle\sup_{\theta\in\Theta}|{\mathcal{L}}_{1}^{\prime}(\theta)-\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)|\leq\epsilon_{N_{p}}$
(64)

$\displaystyle\sup_{\theta\in\Theta}|{\mathcal{L}}_{1}(\theta)-\hat{{\mathcal{L}}}_{1}(\theta)|\leq\epsilon_{N}$
(65)

where
$\epsilon_{N}=\sqrt{8\sqrt{2}\tilde{\sigma}^{2}}\cdot\sqrt{\frac{\log|\Theta|+\log\frac{2}{\delta}}{N}}\vee
8\tilde{\sigma}^{2}\cdot\frac{\log|\Theta|+\log\frac{2}{\delta}}{N}$ and
$\epsilon_{N_{p}}=\sqrt{8\sqrt{2}\tilde{\sigma}^{2}}\cdot\sqrt{\frac{\log|\Theta|+\log\frac{2}{\delta}}{N_{p}}}\vee
8\tilde{\sigma}^{2}\cdot\frac{\log|\Theta|+\log\frac{2}{\delta}}{N_{p}}$ are
used to simplify the notation.

Let $\epsilon=\epsilon_{N}+\epsilon_{N_{p}}$. From Eq. 65, we have:

$\displaystyle|\hat{\xi}-\xi|\leq\epsilon_{N}$ (66)
$\displaystyle\Longrightarrow$ $\displaystyle\xi\leq\hat{\xi}+\epsilon_{N}$
(67) $\displaystyle\Longrightarrow$
$\displaystyle\xi+\epsilon_{N_{p}}\leq\hat{\xi}+\epsilon_{N_{p}}+\epsilon_{N}$
(68)

Based on Eq. 64, similarly, we have
$\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)\leq{\mathcal{L}}_{1}^{\prime}(\theta)+\epsilon_{N_{p}}$.
Since $\xi=\inf_{\theta\in\Theta}{\mathcal{L}}_{1}(\theta)$, then for
$\forall\theta^{*}$ s.t. ${\mathcal{L}}_{1}^{\prime}(\theta^{*})=\xi$,
according to Eq. 68, we can obtain:

$\displaystyle\hat{{\mathcal{L}}}_{1}^{\prime}(\theta)\leq{\mathcal{L}}_{1}^{\prime}(\theta)+\epsilon_{N_{p}}\leq\xi+\epsilon_{N_{p}}\leq\hat{\xi}+\epsilon_{N_{p}}+\epsilon_{N}.$
(69)

Let $\hat{\theta}^{*}$ be the solution of Eq. 6 and $\theta^{*}$ be the
solution of Eq. 5. Additionally, let
$\Theta^{*}=\\{\theta:{\mathcal{L}}_{1}^{\prime}(\theta)\leq\xi\\}$, and
$\hat{\Theta}^{*}=\\{\theta:\hat{{\mathcal{L}}}_{1}^{\prime}(\theta;{\mathbf{x}})\leq\hat{\xi}+\epsilon\\}$.
We state that $\Theta^{*}\subseteq\hat{\Theta}^{*}$ based on Eq. 69.

Next, we prove that $\hat{\theta}^{*}$ competes well with $\theta^{*}$ on
$\hat{{\mathcal{L}}}_{2}(\theta;{\mathbf{x}},{\mathbf{c}})$ and
${\mathcal{L}}_{2}(\theta)$:

$\displaystyle{\mathcal{L}}_{2}(\hat{\theta}^{*})\leq$
$\displaystyle\hat{{\mathcal{L}}}_{2}(\hat{\theta}^{*};{\mathbf{x}},{\mathbf{c}})+\epsilon_{N_{p}}$
based on Eq. 63 $\displaystyle=$
$\displaystyle\min_{\theta\in\hat{\Theta}^{*}}\hat{{\mathcal{L}}}_{2}(\theta;{\mathbf{x}},{\mathbf{c}})+\epsilon_{N_{p}}$
$\hat{\theta}$ is the solution of Eq. 6 $\displaystyle\leq$
$\displaystyle\min_{\theta\in\Theta^{*}}\hat{{\mathcal{L}}}_{2}(\theta;{\mathbf{x}},{\mathbf{c}})+\epsilon_{N_{p}}$
$\Theta^{*}\subseteq\hat{\Theta}^{*}$ $\displaystyle\leq$
$\displaystyle\min_{\theta\in\Theta^{*}}{\mathcal{L}}_{2}(\theta)+2\epsilon_{N_{p}}$
based on Eq. 63 $\displaystyle=$
$\displaystyle{\mathcal{L}}_{2}(\theta^{*})+2\epsilon_{N_{p}}$ (70)

Next, we prove that $\hat{\theta}^{*}$ does not violate constraint too much:

$\displaystyle{\mathcal{L}}_{1}^{\prime}(\hat{\theta}^{*})\leq$
$\displaystyle\hat{{\mathcal{L}}}_{1}^{\prime}(\hat{\theta}^{*};{\mathbf{x}})$
based on Eq. 64 $\displaystyle\leq$
$\displaystyle\hat{\xi}+\epsilon+\epsilon_{N_{p}}$ based on definition of
$\hat{\Theta}^{*}$ $\displaystyle=$
$\displaystyle\hat{\xi}+2\epsilon_{N_{p}}+\epsilon_{N}$
$\epsilon=\epsilon_{N_{P}}+\epsilon_{N}$ $\displaystyle\leq$
$\displaystyle\xi+2\epsilon_{N_{p}}+2\epsilon_{N}$ based on Eq. 68 (71)

∎

##  Appendix D More details regarding dataset

###  D.1 Stocks employed to form time series data

We construct the dataset by assembling 24 stocks from Yahoo Finance during
their IPO date to July 8, 2023, including Ethereum USD, NVIDIA Corporation,
AT&T, Accenture plc, The Boeing Company, The Coca-Cola Company, Simon Property
Group Inc., NIKE Inc., Sony Group Corporation, Chegg Inc., UnitedHealth Group
Incorporated, General Motors Company, Russell 2000, JPMorgan Chase & Co.,
Salesforce Inc., Lockheed Martin Corporation, Walmart Inc., NASDAQ Composite,
Shell plc, Pfizer Inc., Bitcoin USD, Apple Inc., Amazon.com Inc., Alphabet
Inc.

###  D.2 Constructing text descriptions

We construct the text descriptions of molecules by two templates: (1) exact
description, e.g., “A molecule with the heat capacity of -0.11, the lomo of
0.87, the homo of -0.21, the polarizability of 0.95, the dipole moment of
-1.61 and the energy gap between homo and lumo as 0.94.”, and (2) general
description, e.g., “A molecule with a high homo value, a very low heat
capacity, a medium polarizability, a high energy difference between homo and
lomo, a very high lomo value and a high dipole moment.”. These text
descriptions are then refined by GPT 3.5 and the order of different properties
is shuffled. We form the text description for time series data using three
templates: (a) exact description, e.g., “A time series with the frequency of
0.017, the mean of 3.12e-05, 19 peaks, the variance of 1.18e-11, the linear
trend of 0.12 and the skewness of -6.15.”; (b) general description, e.g., “A
time series with large average, medium frequency, nearly equal large and small
values, medium negative linearity, a few peaks and large variance” and (c)
description of trend, e.g., “A time series that first stays stable, then
increases with the slope of 1.25”. Then those descriptions are further refined
by GPT-3.5.

##  Appendix E Implementation details

###  E.1 Details of training process

We leverage a cross-attention layer to enhance the alignment between data and
text embeddings (Ma et al., 2023). Specifically, we maximize the cosine-
similarity between text embeddings obtained from the pretrained LLMs encoder
and data embeddings diffused from original data, where the last layer of LLMs
encoder is finetuned during training. Baselines models are trained for 3,000
epoches. Their finetuned version and Text2Data are both pretrained for 1,000
epoches and finetuned for another 2,000 epoches. All experiments are conduced
on A100 hardware.

For data augmentation as an ablation study (i.e., EDM-DA), we employed GPT-4
to modify the textual descriptions to augment the text-data pairs.

###  E.2 Score functions

For time series generation, we use the same score function as in (Rasul et
al., 2021) that consists of conditional 1-dim dilated ConvNets with residual
connections adapted from the WaveNet (Oord et al., 2016) and Diffwave (Kong et
al., 2020). For molecule generation, we employ the same score function from
EDM (Hoogeboom et al., 2022) which is a Multilayer Perceptron. For motion
generation, we also leverage the same score function in (Tevet et al., 2023)
as a straightforward transformer.

##  Appendix F Additional results

Table 3 shows that the MAE of predicted properties against intended properties
of generated molecules from 2$\%$ to 100$\%$ available labels. Table 4 shows
that two additional ablation studies on Molecules dataset evaluating the
controllability of Text2Data. EDM-finetune-unlabel is pretrained on only
unlabeled data and finetuned on labeled data; EDM-DA is trained on text-data
paired augmented by GPT-4.

Table 3: Evaluate controllability on Molecules dataset by MAE according to
$\alpha$ and $\epsilon_{HOMO}$ of molecules.

Proportion (%) $\alpha$ $\epsilon_{HOMO}$ Text2Data EDM-finetune EDM Text2Data
EDM-finetune EDM 2 0.66 0.84 0.83 0.58 0.92 1.00 4 0.62 0.74 0.69 0.56 0.84
1.07 6 0.44 0.68 0.69 0.44 0.92 1.00 8 0.52 0.67 0.69 0.47 0.90 0.95 10 0.34
0.66 0.75 0.33 0.89 0.92 20 0.41 0.60 0.67 0.42 0.82 0.91 40 0.43 0.55 0.55
0.42 0.78 0.81 60 0.41 0.50 0.52 0.43 0.61 0.59 80 0.40 0.41 0.43 0.42 0.50
0.51 100 0.40 0.40 0.40 0.42 0.43 0.42

Table 4: Evaluate controllability on Molecules dataset by MAE according to
$\alpha$ and $\epsilon_{HOMO}$ of molecules. Two additional ablation studies
are added: EDM-finetune-unlabel is pretrained on only unlabeled data and
finetuned on labeled data; EDM-DA is trained on text-data paired augmented by
GPT-4.

Proportion (%) $\alpha$ $\epsilon_{HOMO}$ Text2Data EDM-finetune EDM-finetune-
unlabel EDM-DA EDM Text2Data EDM-finetune EDM-finetune-unlabel EDM-DA EDM 2
0.66 0.84 0.67 0.80 0.83 0.58 0.92 0.58 0.95 1.00 4 0.62 0.74 0.65 0.69 0.69
0.56 0.84 0.58 0.93 1.07 6 0.44 0.68 0.56 0.67 0.69 0.44 0.92 0.50 0.90 1.00 8
0.52 0.67 0.54 0.66 0.69 0.47 0.90 0.50 0.88 0.95 10 0.34 0.66 0.46 0.61 0.75
0.33 0.89 0.43 0.88 0.92

The results of evaluating the controllability of Text2Data and its baseline
comparisons regarding variance, number of peaks and linearity of generated
time series are in Table 5.

Table 5: Evaluate controllability on time series by MAE on testing set,
according to different proportions of paired data. Lower MAE indicates better
performance.

Proportion (%) Variance ($\times 10^{-5}$) Number of peaks Linearity Text2Data
DiffTS-finetune DiffTS Text2Data DiffTS-finetune DiffTS Text2Data DiffTS-
finetune DiffTS 2 5.28$\pm$11.40 5.35$\pm$11.20 5.70$\pm$11.40 12.94$\pm$0.88
12.95$\pm$0.87 13.01$\pm$0.83 0.61$\pm$0.04 0.61$\pm$0.04 0.62$\pm$0.04 4
5.37$\pm$11.28 5.48$\pm$11.24 6.08$\pm$11.31 12.91$\pm$0.84 12.90$\pm$0.86
12.93$\pm$0.84 0.61$\pm$0.04 0.61$\pm$0.04 0.62$\pm$0.04 6 5.04$\pm$10.90
5.19$\pm$10.42 5.60$\pm$10.20 12.89$\pm$0.90 12.90$\pm$0.83 12.90$\pm$0.83
0.61$\pm$0.04 0.61$\pm$0.04 0.62$\pm$0.04 8 5.30$\pm$10.95 5.59$\pm$11.19
5.60$\pm$10.40 12.88$\pm$0.86 12.89$\pm$0.85 12.95$\pm$0.87 0.61$\pm$0.04
0.61$\pm$0.04 0.61$\pm$0.04 10 5.08$\pm$20.48 5.41$\pm$16.23 5.80$\pm$11.00
12.75$\pm$0.93 12.87$\pm$0.90 12.90$\pm$0.83 0.61$\pm$0.04 0.61$\pm$0.04
0.61$\pm$0.04 20 5.09$\pm$10.98 5.37$\pm$11.09 6.40$\pm$11.20 12.87$\pm$0.85
12.88$\pm$0.85 12.90$\pm$0.84 0.61$\pm$0.04 0.61$\pm$0.04 0.61$\pm$0.04 30
4.85$\pm$10.58 5.19$\pm$9.90 5.70$\pm$10.20 12.84$\pm$0.83 12.87$\pm$0.87
12.91$\pm$0.88 0.61$\pm$0.04 0.61$\pm$0.04 0.61$\pm$0.04 40 4.77$\pm$10.42
5.11$\pm$14.10 5.34$\pm$13.10 12.84$\pm$0.87 12.88$\pm$0.90 12.89$\pm$0.86
0.60$\pm$0.04 0.61$\pm$0.04 0.61$\pm$0.04

Table 6 shows the results of evaluating the generation quality of Text2Data
and baseline comparisons on Molecule dataset based on $-\log p$ and atom
validity. Table 7 shows the results of evaluating the generation quality of
Text2Data and baseline comparisons on Molecule dataset based on molecular and
atom stability.

Table 6: Evaluate generation quality on Molecule dataset by $-\log p$ and
validity according to different proportions of paired data. Lower $-\log p$
and higher validity indicate better performance.

Proportion (%) -log p $\downarrow$ Validity $\uparrow$ Text2Data EDM-finetune
EDM Text2Data EDM-finetune EDM 2 -111.39$\pm$0.92 -74.88$\pm$1.82
-49.15$\pm$0.96 0.97$\pm$0.07 0.93$\pm$0.11 0.86$\pm$0.09 4 -119.09$\pm$0.30
-87.56$\pm$4.18 -78.72$\pm$2.95 0.95$\pm$0.07 0.96$\pm$0.07 0.83$\pm$0.15 6
-119.55$\pm$0.59 -97.32$\pm$2.09 -69.58$\pm$1.97 0.97$\pm$0.07 0.96$\pm$0.05
0.81$\pm$0.14 8 -119.40$\pm$0.68 -101.31$\pm$1.11 -85.19$\pm$1.58
0.97$\pm$0.07 0.93$\pm$0.07 0.90$\pm$0.11 10 -121.37$\pm$1.24 -104.17$\pm$1.94
-85.73$\pm$1.00 0.96$\pm$0.06 0.95$\pm$0.13 0.88$\pm$0.10 20 -119.58$\pm$1.61
-104.08$\pm$2.03 -76.39$\pm$2.11 0.97$\pm$0.07 0.95$\pm$0.07 0.90$\pm$0.09 30
-121.00$\pm$1.07 -115.58$\pm$0.95 -76.22$\pm$1.26 0.97$\pm$0.07 0.95$\pm$0.07
0.91$\pm$0.09 40 -119.90$\pm$0.97 -114.00$\pm$0.59 -80.97$\pm$0.09
0.97$\pm$0.07 0.95$\pm$0.08 0.90$\pm$0.10

Table 7: Evaluate generation quality on Molecule dataset by molecular and atom
stability of generated molecules according to different proportions of paired
data. Higher molecular stability and atom stability indicate better
performance.

Proportion (%) Mol. Stability $\uparrow$ Atom Stability $\uparrow$ Text2Data
EDM-finetune EDM Text2Data EDM-finetune EDM 2 0.86$\pm$0.14 0.85$\pm$0.04
0.65$\pm$0.04 0.99$\pm$0.01 0.99$\pm$0.01 0.96$\pm$0.01 4 0.87$\pm$0.09
0.83$\pm$0.08 0.66$\pm$0.09 0.99$\pm$0.01 0.98$\pm$0.01 0.97$\pm$0.01 6
0.88$\pm$0.16 0.86$\pm$0.06 0.73$\pm$0.04 0.99$\pm$0.02 0.99$\pm$0.01
0.97$\pm$0.01 8 0.88$\pm$0.11 0.87$\pm$0.04 0.79$\pm$0.05 0.99$\pm$0.01
0.99$\pm$0.01 0.98$\pm$0.01 10 0.88$\pm$0.10 0.83$\pm$0.06 0.77$\pm$0.11
0.99$\pm$0.01 0.98$\pm$0.01 0.98$\pm$0.01 20 0.88$\pm$0.10 0.87$\pm$0.08
0.79$\pm$0.09 0.99$\pm$0.01 0.98$\pm$0.01 0.98$\pm$0.01 30 0.89$\pm$0.10
0.87$\pm$0.07 0.79$\pm$0.09 0.99$\pm$0.01 0.98$\pm$0.01 0.98$\pm$0.01 40
0.89$\pm$0.10 0.86$\pm$0.06 0.79$\pm$0.09 0.99$\pm$0.01 0.98$\pm$0.01
0.98$\pm$0.01

Table 8 shows the results of evaluating the generation quality of Text2Data
and baseline comparisons on motion dataset based on FID and diversity.

Table 8: Evaluate generation quality on HumanML3D dataset by FID and Diversity
according to different proportions of paired data. Low FID and higher
diversity indicate better performance.

Proportion (%) FID $\downarrow$ Diversity $\uparrow$ Text2Data MDM-finetune
MDM Text2Data MDM-finetune MDM 2 1.22$\pm$0.12 1.23$\pm$0.03 3.09$\pm$0.12
9.26$\pm$0.07 9.08$\pm$0.17 8.84$\pm$0.14 4 1.12$\pm$0.11 1.16$\pm$0.17
3.06$\pm$0.21 9.31$\pm$0.21 9.13$\pm$0.20 8.80$\pm$0.18 6 1.00$\pm$0.13
0.64$\pm$0.11 1.13$\pm$0.12 9.40$\pm$0.13 9.39$\pm$0.22 8.91$\pm$0.22 8
1.40$\pm$0.14 1.32$\pm$0.15 1.34$\pm$0.18 9.43$\pm$0.20 9.21$\pm$0.18
8.95$\pm$0.19 10 1.49$\pm$0.19 1.52$\pm$0.13 1.50$\pm$0.15 9.59$\pm$0.09
9.74$\pm$0.13 9.37$\pm$0.15 20 0.92$\pm$0.06 1.02$\pm$0.12 1.07$\pm$0.06
9.77$\pm$0.20 9.72$\pm$0.17 9.66$\pm$0.15 30 0.81$\pm$0.10 0.99$\pm$0.13
1.11$\pm$0.10 9.79$\pm$0.11 9.70$\pm$0.12 9.63$\pm$0.14 40 0.63$\pm$0.12
0.95$\pm$0.11 1.13$\pm$0.15 9.74$\pm$0.13 9.70$\pm$0.20 9.38$\pm$0.15