11institutetext: 1Intelligent Computing and Machine Learning Lab
School of ASEE, Beihang University, Beijing, 100191, China
2School of Biological Science and Medical Engineering
Beihang University, Beijing, 100191, China
∗{zcqin<EMAIL_ADDRESS>

# Text Generation Based on Generative Adversarial Nets with Latent Variable

Heng Wang1 Zengchang Qin1∗ Tao Wan2∗

###### Abstract

In this paper, we propose a model using generative adversarial net (GAN) to
generate realistic text. Instead of using standard GAN, we combine variational
autoencoder (VAE) with generative adversarial net. The use of high-level
latent random variables is helpful to learn the data distribution and solve
the problem that generative adversarial net always emits the similar data. We
propose the VGAN model where the generative model is composed of recurrent
neural network and VAE. The discriminative model is a convolutional neural
network. We train the model via policy gradient. We apply the proposed model
to the task of text generation and compare it to other recent neural network
based models, such as recurrent neural network language model and SeqGAN. We
evaluate the performance of the model by calculating negative log-likelihood
and the BLEU score. We conduct experiments on three benchmark datasets, and
results show that our model outperforms other previous models.

###### Keywords:

Generative Adversarial Net; Variational Autoencoder; VGAN; Text Generation

##  1 Introduction

Automatic text generation is important in natural language processing and
artificial intelligence. For example, text generation can help us write
comments, weather reports and even poems. It is also essential to machine
translation, text summarization, question answering and dialogue system [1].
One popular approach for text generation is by modeling sequence via recurrent
neural network (RNN) [1]. However, recurrent neural network language model
(RNNLM) suffers from two major drawbacks when used to generate text. First,
RNN based model is always trained through maximum likelihood approach, which
suffers from exposure bias [12]. Second, the loss function used to train the
model is at word level but the performance is typically evaluated at sentence
level. There are some research on using generative adversarial net (GAN) to
solve the problems. For example, Yu _et al._ [9] applies GAN to discrete
sequence generation by directly optimizing the discrete discriminator’s
rewards. Li _et al._ [22] applies GAN to open-domain dialogue generation and
generates higher-quality responses. Instead of directly optimizing the GAN
objective, Che _et al._ [21] derives a novel and low-variance objective using
the discriminator’s output that follows corresponds to the log-likelihood. In
GAN, a discriminative net $D$ is learned to distinguish whether a given data
instance is real or not, and a generative net $G$ is learned to confuse $D$ by
generating highly realistic data. GAN has achieved a great success in computer
vision tasks [13], such as image style transfer [24], super resolution and
imagine generation [23]. Unlike image data, text generation is inherently
discrete, which makes the gradient from the discriminator difficult to back-
propagate to the generator [19]. Reinforcement learning is always used to
optimize the model when GAN is applied to the task of text generation [9].

Although GAN can generate realistic texts, even poems, there is an obvious
disadvantage that GAN always emits similar data [14]. For text generation, GAN
usually uses recurrent neural network as the generator. Recurrent neural
network mainly contains two parts: the state transition and a mapping from the
state to the output, and two parts are entirely deterministic. This could be
insufficient to learn the distribution of highly-structured data, such as text
[3]. In order to learn generative models of sequences, we propose to use high-
level latent random variables to model the observed variablity. We combine
recurrent neural network with variational autoencoder (VAE) [2] as generator
$G$.

In this paper, we propose a generative model, called VGAN, by combining VAE
and generative adversarial net to better learn the data distribution and
generate various realistic text. The paper is structured as the following: In
Section 2, we give the preliminary of this research. In Section 3, we
introduce the VGAN model and adversarial training of VGAN. Experimental
results are given and analyzed in Section 4. In Section 5, we conclude this
research and discuss the possible future work.

##  2 Preliminary

###  2.1 LSTM Architecture

A recurrent neural network is a class of artificial neural network where
connections between units form a directed cycle [1]. This allows it to exhibit
dynamic temporal behavior. Long short-term memory (LSTM) is an improved
version of recurrent neural network considering long-term dependency in order
to overcome the vanishing gradient problem. It has been successfully applied
in many tasks, including text generation and speech recognition [20]. LSTM has
a architecture consisting of a set of recurrently connected subnets, known as
memory blocks. Each block contains memory cells and three gate units,
including the input, output and forget gates. The gate units allow the network
to learn when to forget previous information and when to update the memory
cells given new information.

Given a vocabulary $V$ and an embedding matrix $W\in R^{{q}\times{|V|}}$ whose
columns correspond to vectors; $|V|$ and $q$ denote the size of vocabulary and
the dimension of the token vector, respectively. The embedding matrix $W$ can
be initialized randomly or pretrained. Let $x$ denote a token with index $k$,
$e(x)\in R^{|V|\times{1}}$ is a vector with zero in all positions except
$R^{{|V|\times{1}}}_{k}=1$. Given an input sequence
$X_{s}=(x_{1},x_{2},\cdots,x_{T})$, we compute the output sequence of LSTM
$Y_{s}=(y_{1},y_{2},\cdots,y_{T})$. When the input sequence passes through the
embedding layer, each token is represented by a vector:
$v_{i}=W\bm{\cdot}e(x_{i})\in R^{q\times 1}$. The relation between inputs,
memory cells and outputs are defined by the following equations:

$i^{(t)}=\sigma(W_{ix}v^{(t)}+W_{ih}h^{(t-1)}+W_{ic}c^{(t-1)})$ (1)
$f^{(t)}=\sigma(W_{fx}v^{(t)}+W_{fh}h^{(t-1)}+W_{fc}c^{(t-1)})$ (2)
$c^{(t)}=f^{(t)}\odot c^{(t-1)}+i^{(t)}\odot
tanh(W_{cx}v^{(t)}+W_{ch}h^{(t-1)})$ (3)
$o^{(t)}=\sigma(W_{ox}v^{(t)}+W_{oh}h^{(t-1)}+W_{oc}c^{(t-1)})$ (4)
$h^{(t)}=o_{(t)}\odot tanh(c^{t})$ (5)

where $i^{(t)}\in R^{l\times 1}$, $f^{(t)}\in R^{l\times 1}$, $o^{(t)}\in
R^{l\times 1}$ and $h^{(t)}\in R^{l\times 1}$ represent the input gate, forget
gate, output gate, memory cell activation vector and the recurrent hidden
state at time step $t$; $l$ is the dimension of LSTM hidden units, $\sigma$
and $tanh$ are the logistic sigmoid function and hyperbolic tangent function,
respectively. $\odot$ represents element-wise multiplication [20].

###  2.2 Variational Autoencoder

An autoencoder (AE) is an unsupervised learning neural network with the target
values to be equal to the inputs. Typically, AE is mainly used to learn a
representation for the input data, and extracts features and reduces
dimensionality [2]. Recently, autoencoder has been widely used to be a
generative model of image and text. The variational autoencoder is an improved
version based on the standard autoencoder. For variational autoencoder, there
is a hypothesis that data is generated by a directed model and some latent
variables are introduced to capture the variations in the observed variables.
The directed model $p(x)=\int p(x|z)p(z)\,dz$ is optimized by using a
variational upper bound:

$\displaystyle-\log p(x)=$ $\displaystyle-\log\int p(x|z)p(z)\,dz$ (6)
$\displaystyle\leq-KL(q(z|x)$ $\displaystyle||p(z))+E_{z\sim q(z|x)}[\log
p(x|z)]$

where $p(z)$ is a prior distribution over the latent random variable $z$; The
prior distribution is unknown, and we generally assume it to be a normal
distribution. $p(x|z)$ denotes a map from the latent variables $z$ to the
observed variables $x$, and given $z$ it produces a distribution over the
possible corresponding values of $x$. $q(z|x)$ is a variational approximation
of the true posterior distribution. let $KL(q(z|x)||p(z))$ denote the
Kullback-Leibler divergence between $q(z|x)$ and $p(z)$; The introduction of
latent variables makes it intractable to optimize the model directly. We
minimize the upper bound of the negative log-likelihood to optimize VAE. The
training algorithm we use is Stochastic Gradient Variational Bayes (SGVB)
proposed in [2].

###  2.3 Generative Adversarial Nets

For generative adversarial nets, there is a two-sided zero-sum game between a
generator and a discriminator. The training objective of the discriminative
model is to determine whether the data is from the fake data generated by the
generative model or the real training data. For the generative model, its
objective is to generate realistic data, which is similar to the true training
data and the discriminative model can’t distinguish. For the standard
generative adversarial networks, we train the discriminative model $D$ to
maximize the probability of giving the correct labels to both the samples from
the generative model and training examples. We simultaneously train the
generative model $G$ to minimize the estimated probability of being true by
the discriminator. The objective function is:

$\min_{G}\max_{D}V(D,G)=E_{x\sim p_{data}(x)}[\log D(x)]+E_{z\sim
p_{z}(z)}[\log(1-D(G(z)))]$ (7)

where $z$ denotes the noises and $G(z)$ denotes the data generated by the
generator. $D(x)$ denotes the probability that $x$ is from the training data
with emperical distribution $p_{data}(x)$.

##  3 Model Description

###  3.1 The Generative Model of VGAN

The proposed generative model contains a VAE at every timestep. For the
standard VAE, its prior distribution is usually a given standard normal
distribution. Unlike the standard VAE, the current prior distribution depends
on the hidden state $h_{t-1}$ at the previous moment, and adding the hidden
state as an input is helpful to alleviate the long term dependency of
sequential data. It also takes consideration of the temporal structure of the
sequential data [3] [4]. The model is described in Fig. 1. The prior
distribution $z_{t}=p_{1}(z_{t}|h_{t-1})$ is:

$z_{t}\sim N(\mu_{0,t},\sigma^{2}_{0,t}),\quad
where\quad[\mu_{0,t},\sigma_{0,t}^{2}]=\varphi^{prior}(h_{t-1})$ (8)

where $\mu_{0,t}$, $\sigma_{0,t}^{2}$ are the mean and variance of the prior
Gaussian distribution, respectively. The posterior distribution depends on the
current hidden state. For the appriximate posterior distribution, it depends
on the current state $h_{t}$ and the current input $x_{t}$:
$z_{t}^{\prime}=q_{1}(z_{t}|x_{t},h_{t})$.

$z_{t}^{\prime}\sim N(\mu_{1,t},\sigma^{2}_{1,t}),\quad
where\quad[\mu_{1,t},\sigma_{1,t}^{2}]=\varphi^{posterior}(x_{t},h_{t})$ (9)

where $\mu_{1,t}$ , $\sigma_{1,t}^{2}$ are the mean and variance of the
approximate posterior Gaussian distribution, respectively. $\varphi^{prior}$
and $\varphi^{posterior}$ can be any highly flexible functions, for example, a
neural network.

The derivation of the training criterion is done via stochastic gradient
variational Bayes. We achieve the goal of minimizing the negative log-
likelihood in the pre-training stage by minimizing $L(x_{1:T})$:

$\displaystyle L(x_{1:T})=$ $\displaystyle-\log
p(x_{1:T})=-\log\int_{z_{1:T}}\frac{q_{1}(z_{1:T}|x_{1:T},h_{1:T})}{q_{1}(z_{1:T}|x_{1:T},h_{1:T})}\prod_{t=0}^{T-1}p(x_{t+1}|x_{1:t},z_{1:t})\,dz_{1:T}$
(10) $\displaystyle\leq$ $\displaystyle-
KL(q_{1}(z_{1:T}|x_{1:T},h_{1:T})||p_{1}(z_{1:T}|x_{1:T-1},h_{1:T-1}))$
$\displaystyle+$ $\displaystyle E_{z_{1:T}\sim
q_{1}(z_{1:T}|x_{1:T},h_{1:T})}\left[\sum_{t=0}^{T-1}\log
p(x_{t+1}|x_{1:t},z_{1:t})\right]$

where $p_{1}$ and $q_{1}$ represent the prior distribution and the approximate
posterior distribution.

If we directly use the stochastic gradient descent algorithm to optimize the
model, there will be a problem that some parameters of the VAE are not
derivable. In order to solve the problem, we introduce the “reparametrization
trick” [2]. For example, if we want to get the samples from the distribution
$N(\mu_{1,t},\sigma^{2}_{1,t})$, we will sample from a standard normal
distribution $\epsilon\sim N(0,I^{2})$ and get the samples $z_{t}^{\prime}$
via $z_{t}^{\prime}=\mu_{1,t}+\sigma_{1,t}\epsilon$.

Figure 1: The structure of the generator $G_{\theta}$. The generator is
composed of LSTM and VAE. $x_{t}$ denotes the input at timestep $t$; $h_{t}$
denotes the LSTM hidden state; $N(\mu_{0,t},\sigma_{0,t}^{2})$ denotes the
prior distribution; $N(\mu_{1,t},\sigma_{1,t}^{2})$ denotes the appriximate
posterior distribution; $y_{t}$ denotes the target output at timestp $t$,
which is the same as $x_{t+1}$. $o_{t}$ denote the estimated result. The
dotted line denotes the optimization process in the pre-training stage.

Before the adversarial training, we need to pre-train the generative model via
SGVB. For example, given the input $X_{s}=(\textbf{S},i,like,it)$, and the
target output is $Y_{s}=(i,like,it,\textbf{E})$, where S and E are the start
token and the end token of a sentence, respectively.

Figure 2: The illustration of the discriminator. The discriminator $D_{\phi}$
is trained by using the true training data and the fake data generated by
generator. The discriminator contains the convolution layer, the max-pooling
layer and the fully connected layer.

###  3.2 Adversarial Training of VGAN

In this paper, we choose convolutional neural network as the discriminative
model, which has shown a great success in the task of text classification
[5][6]. Let $v_{i}\in R^{k}$ denote a $k$-dimesion vector corresponding to the
$i$-th word in the sentence. $v_{1:n}=v_{1}\oplus v_{2}\oplus\cdots\oplus
v_{n}$ denote a sentence of length $n$, where $\oplus$ is the concatenation
operator and $v_{1:n}\in R^{n\times k}$ is a matrix. Then a filter $w_{1}\in
R^{h\times k}$ is applied to a window of $h$ words to produce a new feature.
For example, $c_{i}=f(w_{1}\bm{\cdot}v_{i:i+h-1}+b)$ where $c_{i}$ is a
feature generated by convolution operation. $f$ denotes a nonlinear function
such as the hyperbolic tangent or sigmoid; $b$ is a bias term. When the filter
is applied to a sentence $v_{1:n}$, a feature map
$\mathbf{c}=[c_{1},c_{2},\cdots,c_{n-h+1}]$ is generated. We can use a variety
of convolution kernels to obtain a variety of feature maps. We apply a max-
over-time pooling operation to the features to get the maximum value
$c^{\prime}=\max\\{\mathbf{c}\\}$. Finally, all features are used as input to
a fully connected layer for classification. After the generator is pre-
trained, we use the generator to generate the negative examples. The negative
samples and the true training data are combined as the input of the
discriminator. The training process of discriminator is showed in Fig. 2. In
the adversarial training, the generator $G_{\theta}$ is optimized via policy
gradient, which is a reinforcement learning algorithm [7] [8]. The training
process is showed in Fig. 3. $G_{\theta}$ can be viewed as an $agent$, which
interacts with the environment. The parameters $\theta$ of this $agent$
defines a policy, which determines the process of generating the sequences.

Figure 3: The training process of the generator via policy gradient. The
dotted line denotes sampling a token from the output distribution. The sampled
token is as the input at the next time. START denotes a start token.

Given a start token S as the input, $G_{\theta}$ samples a token from the
generating distribution. And the sampled word is as the input at the next
time. A whole sentence is generated word by word until an end token E has been
generated or the maximum length is reached. For example, given a start token S
as the input, the sequence $Y_{1:T}=(y_{1},y_{2},\cdots,y_{T})$ is generated
by $G_{\theta}$. In timestep $t$, the state $s_{t}$ is the current produced
tokens $(y_{1},y_{2},\cdots,y_{t-1})$ and the action $a_{t}$ is to select the
next token in the vocabulary. After taking an action $a_{t}$, the agent
updates the state $(y_{1},y_{2},\cdots,y_{t})$. If the agent reaches the end,
the whole sequence has been generated and the reward will be assigned. During
the training, we choose the next token according to the current policy and the
current state. But there is a problem that we can observe the reward after a
whole sequence. When $G_{\theta}$ generates the sequence, we actually care
about the expected accumulative reward from start to end and not only the end
reward. At every timestep, we consider not only the reward brought by the
generated sequence, but also the future reward.

In order to evaluate the reward for the intermediate state, Monte Carlo search
has been employed to sample the remaining unknown tokens. In result, for the
finished sequences, we can directly get the rewards by inputting them to the
discriminator $D_{\phi}$. For the unfinished sequences, we first use the Monte
Carlo search to get estimated rewards [25]. To reduce the variance and get
more accurate assessment of the action value, we employ the Monte Carlo search
for many times and get the average rewards. The objective of the generator
model $G_{\theta}$ is to generate a sequence from the start token to maximize
its expected end reward:

$\max_{\theta}J(\theta)=E[R_{T}|s_{0}]=\sum_{y_{1}\in
V}G_{\theta}(y_{1}|s_{0})\bm{\cdot}Q_{G_{\theta}}^{D_{\phi}}(s_{0},y_{1})$
(11)

where $R_{T}$ denotes the end reward after a whole sequence is generated;
$Q_{G_{\theta}}^{D_{\phi}}(s_{i},y_{i})$ is the action-value function of a
sequence, the expected accumulative reward starting from state $s_{i}$, taking
action $a=y_{i}$; $G_{\theta}(y_{i}|s_{i})$ denotes the generator chooses the
action $a=y_{i}$ when the state $s_{i}=(y_{1},y_{2},\cdots,y_{i-1})$ according
to the policy. The gradient of the objective function $J(\theta)$ can be
derived [9] as:

$\nabla_{\theta}J(\theta)=E_{Y_{1:t-1}\sim G_{\theta}}\left[\sum_{y_{t}\in
V}\nabla_{\theta}G_{\theta}(y_{t}|Y_{1:t-1})\bm{\cdot}Q_{G_{\theta}}^{D_{\phi}}(Y_{1:t-1},y_{t})\right]$
(12)

we then update the parameters of $G_{\theta}$ as:

$\theta\leftarrow\theta+\alpha\bm{\cdot}\nabla_{\theta}J(\theta)$ (13)

where $\alpha$ is the learning rate.

After training the generator by using policy gradient, we will use the
negative samples generated by the updated generator to re-train the
discriminator $D_{\phi}$ as follows:

$\min_{\phi}-E_{Y\sim p_{data}}[\log D_{\phi}(Y)]-E_{Y\sim
G_{\theta}}[\log(1-D_{\phi}(Y))]$ (14)

The pseudo-code of the complete training process is shown in Algorithm 1.

Algorithm 1. Generative Adversarial Network with Latent Variable
---
Require: generaror: $G_{\theta}$;
discriminator: $D_{\phi}$;
True training dataset: $X$;
1 : Pre-train $G_{\theta}$ on $X$ by Eq. (10);
2 : Generate negative samples Y by using $G_{\theta}$;
3 : Pre-train $D_{\phi}$ on $X$ and $Y$ by minimizing the cross entropy;
4 : Repeat:
5 : for 1 $\sim$ m do
6 : generate the data $Y_{1}$ by using $G_{\theta}$;
7 : use $D_{\phi}$ to get the reward of $Y_{1}$;
8 : update the parameters $\theta$ of $G_{\theta}$ by Eq. (13);
9 : end for
10: for 1 $\sim$ n do
11: use $G_{\theta}$ to generate negative samples $Y_{2}$;
12: update the parameters $\phi$ of $D_{\phi}$ on $X$ and $Y_{2}$ by Eq. (14);
13: end for
14: Until: VGAN converges.

In this paper, we propose the VGAN model by combining VAE and GAN for
modelling highly-structured sequences. The VAE can model complex multimodal
distributions, which will help GAN to learn structured data distribution.

##  4 Experimental Studies

In our experiments, given a start token as input, we hope to generate many
complete sentences. We train the proposed model on three datasets: Taobao
Reviews, Amazon Food Reviews and PTB dataset [15]. Taobao dataset is crawled
on taobao.com. The sentence numbers of Taobao dataset, Amazon dataset are 400K
and 300K, respectively. We split the datasets into 90/10 for training and
test. For PTB dataset is relatively small, the sentence numbers of training
data and test data are 42,068 and 3,370.

###  4.1 Training Details

In our paper, we compare our model to two other neural models RNNLM [1] and
SeqGANs [9]. For these two models, we use the random initialized word
embeddings, and they are trained at the level of word. We use 300 for the
dimension of LSTM hidden units. The size of latent variables is 60. For Taobao
Reviews and PTB dataset, the sizes of vocabulary are both 5K. For Amazon Food
Reviews, the size is 20K. The maximum lengths of Taobao Reviews, PTB dataset
and Amazon Food Reviews are 20, 30 and 30, respectively. We drop the LSTM
hidden state with the dropout rate 0.5. All models were trained with the Adam
optimization algorithm [10] with the learning rate 0.001. First, we pre-train
the generator and the discriminator. We pre-train the generator by minimizing
the upper bound of the negative log-likelihood. We use the pre-trained
generator to generate the negative data. The negative data and the true
training data are combined to be the input of the discriminator. Then, we
train the generator and discriminator iteratively. Given that the generator
has more parameters and is more difficult to train than the discriminator, we
perform three optimization steps for the discriminator for every five steps
for the generator. The process is repeated until a given number of epochs is
reached.

Table 1: BLEU-2 score on three benchmark datasets. The best results are highlighted. Numbers of generated sentence |  200 |  400 |  600 |  800 |  1000
---|---|---|---|---|---
RNNLM (Taobao) |  0.965 |  0.967 |  0.967 |  0.967 |  0.967
SeqGAN (Taobao) |  0.968 |  0.970 |  0.970 |  0.968 |  0.968
VGAN-pre (Taobao) |  0.968 |  0.968 |  0.967 |  0.968 |  0.968
VGAN (Taobao) |  0.969 |  0.972 |  0.970 |  0.969 |  0.969
RNNLM (Amazon) |  0.831 |  0.842 |  0.845 |  0.846 |  0.848
SeqGAN (Amazon) |  0.846 |  0.851 |  0.852 |  0.853 |  0.856
VGAN-pre (Amazon) |  0.842 |  0.849 |  0.854 |  0.849 |  0.848
VGAN (Amazon) |  0.876 |  0.874 |  0.866 |  0.868 |  0.868
RNNLM (PTB) |  0.658 |  0.650 |  0.654 |  0.655 |  0.662
SeqGAN (PTB) |  0.712 |  0.705 |  0.701 |  0.702 |  0.681
VGAN-pre (PTB) |  0.680 |  0.690 |  0.694 |  0.695 |  0.671
VGAN (PTB) |  0.715 |  0.709 |  0.714 |  0.715 |  0.695

###  4.2 Results and Evaluation

In this paper, we use the BLEU score [17] and negative log-likelihood as the
evaluation metrics. BLEU score is used to measure the similarity degree
between the generated texts and the human-created texts. We use the whole test
data as the references when calulating the BLEU score via nature language
toolkit (NLTK). For negative log-likelihood, we calulate the value by
inputting the test data. Table 2 shows the NLL values of the test data. VGAN-
pre is the pretrained model of VGAN.

$NLL=-E\left[\sum_{t=1}^{T}\log G_{\theta}(y_{t}|Y_{1:t-1})\right]$ (15)

where $NLL$ denotes the negative log-likelihood; Table 1 shows the
experimental results of BLEU-2 score, and numbers in Table 1 denote the
numbers of the sentence generated. We calulate the average BLEU-2 score
between the generated sentences and the test data. The descent processes of
NLL values during the adversarial training are showed in the Fig 4. Here, we
give some examples generated by the proposed model. Due to the page limit,
only some of generated comments from Amazon Food Reviews are shown in Table 3
and more results will be available online in the final version of the paper.

Table 2: The comparison results (NLL) of VGAN to other models. Dataset |  Taobao Dataset |  Amazon Dataset |  PTB Dataset
---|---|---|---
RNNLM [1] |  219 |  483 |  502
SeqGAN [9] |  212 |  467 |  490
VGAN-pre |  205 |  435 |  465
VGAN |  191 |  408 |  423

From Table 1 and Table 2, we can see the significant advantage of VGAN over
RNNLM and SeqGAN in both metrics. The results in the Fig 4 indicate that
applying adversarial training strategies to generator can breakthrough the
limitation of generator and improve the effect.

(a) Taobao Reviews

(b) Amazon Food Reviews

(c) PTB Data

Figure 4: (a) NLL values of Taobao Reviews. (b) NLL values of Amazon Food Reviews. (c) NLL values of PTB Data. Table 3: Generated examples from Amazon Food Reviews using different models. RNNLM |  | 1\. this is a great product , if you liked canned jerky but this is
---
probably okay because your taste is great too .
2\. we ’ll left never eating a bit of more bars . will definitely buy again .
3\. but my friends and i love it mixes and they are hard in colored tap up ;
4\. we love this product , and our purchase was fast too .
5\. the soup is quite very good , its flax flavoring for the taste … it is
pronounced
and yummy .
SeqGAN |  | 1\. how good it is that i ’ll really cost again , this was my favorite .
---
2\. service was super fast and good timely manner on each order .
3\. the risk is very important , but truly the best so , you ’ll
probably love it !
4\. each bag is n’t lower and use .
5\. i found that the seller was good to my own from amazon bags
practically very frozen .
VGAN |  | 1\. you just ate in first , but that is the best thing .
---
2\. but that did give me a much healthier and healthy .
3\. the tea powder in need is based on the label and is very well as well .
4\. chips ahoy cookies are very hard . this is just not very tasty . not what
they say ?
5\. this is very nice and a little sweet . the red ’s very fresh
6\. i found that the coffee was not bold .
7\. you are not sure why loves these cookies . i will be ordering these again

##  5 Conclusions

In this paper, we proposed the VGAN model to generate realistic text based on
classical GAN model. The generative model of VGAN combines generative
adversarial nets with variational autoencoder and can be applied to the
sequences of discrete tokens. In the process of training, we employ policy
gradient to effectively train the generative model. Our results show that VGAN
outperforms two strong baseline models for text generation and behaves well on
three benchmark datasets. In the future, we plan to use deep deterministic
policy gradient [11] to train the generator better. In the addition, we will
choose other models as the discriminator such as recurrent convolutional
neural network [16] and recurrent neural network [18].

## References

  * [1] Mikolov, Tomas and Karafiat, Martin and Burget, Lukas and Cernocký, Jan and Khudanpur, Sanjeev: Recurrent neural network based language model. Interspeech, 1045–1048, 2010.
  * [2] Kingma, Diederik P and Welling, Max: Auto-encoding variational bayes. arXiv: 1312.6144, 2014.
  * [3] Chung, Junyoung and Kastner, Kyle and Dinh, Laurent and Goel, Kratarth and Courville, Aaron C and Bengio, Yoshua: A recurrent latent variable model for sequential data. NIPS, 2980–2988, 2015.
  * [4] Serban, Iulian Vlad and Sordoni, Alessandro and Lowe, Ryan and Charlin, Laurent and Pineau, Joelle and Courville, Aaron C and Bengio, Yoshua: A hierarchical latent variable encoder-decoder model for generating dialogues. arXiv: 1605.06069, 2016.
  * [5] Kim, Yoon: Convolutional neural networks for sentence classification. EMNLP, 1746–1751, 2014.
  * [6] Kalchbrenner, Nal and Grefenstette, Edward and Blunsom, Phil: A convolutional neural network for modelling sentences. ACL, 655–665, 2014.
  * [7] Sutton, Richard S and Mcallester, David A and Singh, Satinder P and Mansour, Yishay: Policy gradient methods for reinforcement learning with function approximation. NIPS, 1057–1063, 2000.
  * [8] Ranzato, Marcaurelio and Chopra, Sumit and Auli, Michael and Zaremba, Wojciech: Sequence level training with recurrent neural networks. arXiv: 1511.06732, 2016.
  * [9] Yu, Lantao and Zhang, Weinan and Wang, Jun and Yu, Yong: SeqGAN: sequence generative adversarial nets with policy gradient. arXiv:1609.05473, 2016.
  * [10] Kingma, Diederik P and Ba, Jimmy Lei: Adam: a method for stochastic optimization. arXiv:1412.6980, 2015.
  * [11] Lillicrap, Timothy P and Hunt, Jonathan J and Pritzel, Alexander and Heess, Nicolas and Erez, Tom and Tassa, Yuval and Silver, David and Wierstra, Daniel Pieter: Continuous control with deep reinforcement learning. arXiv:1509.02971, 2016.
  * [12] Bengio, Samy and Vinyals, Oriol and Jaitly, Navdeep and Shazeer, Noam: Scheduled sampling for sequence prediction with recurrent Neural networks. NIPS, 1171–1179, 2015.
  * [13] Denton, Emily L and Chintala, Soumith and Szlam, Arthur and Fergus, Robert D: Deep generative image models using a Laplacian pyramid of adversarial networks. NIPS, 1486–1494, 2015.
  * [14] Salimans, Tim and Goodfellow, Ian J and Zaremba, Wojciech and Cheung, Vicki and Radford, Alec and Chen, Xi: Improved techniques for training GANs. NIPS, 2234–2242, 2016.
  * [15] Marcus, Mitchell P and Marcinkiewicz, Mary Ann and Santorini, Beatrice: Building a large annotated corpus of english: the penn treebank. Computational Linguistics, 313–330, 1993.
  * [16] Lai, Siwei and Xu, Liheng and Liu, Kang and Zhao, Jun: Recurrent convolutional neural networks for text classification. AAAI, 2267–2273, 2015.
  * [17] Papineni, Kishore and Roukos, Salim and Ward, Todd and Zhu, Weijing: Bleu: a method for automatic evaluation of machine translation. ACL, 311–318, 2002.
  * [18] Liu, Pengfei and Qiu, Xipeng and Huang, Xuanjing: Recurrent neural network for text classification with multi-task learning. IJCAI, 2873–2879, 2016.
  * [19] Huszar, Ferenc: How (not) to train your generative model: scheduled sampling, likelihood, adversary? arXiv:1511.05101, 2015.
  * [20] Hochreiter, S and Schmidhuber, J: Long short-term memory. Neural Computation 9(8), 1735–1780, 1997.
  * [21] Che, Tong and Li, Yanran and Zhang, Ruixiang and Hjelm, R Devon and Li, Wenjie and Song, Yangqiu and Bengio, Yoshua: Maximum-likelihood augmented discrete generative adversarial networks. arXiv:1702.07983, 2017.
  * [22] Li, Jiwei and Monroe, Will and Shi, Tianlin and Jean, Sébastien and Ritter, Alan and Jurafsky, Dan: Adversarial learning for neural dialogue generation. arXiv:1701.06547, 2017.
  * [23] Radford, Alec and Metz, Luke and Chintala, Soumith: Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv: 1511.06434, 2016.
  * [24] Gatys, Leon A and Ecker, Alexander S and Bethge, Matthias: A neural algorithm of artistic style. arXiv: 1508.06576, 2015.
  * [25] Chaslot, Guillaume and Bakkes, Sander and Szita, Istvan and Spronck, Pieter: Monte-carlo tree search: a new framework for game AI. Artificial Intelligence and Interactive Digital Entertainment Conference, 216-217, 2008.


# Synth2: Boosting Visual-Language Models with
Synthetic Captions and Image Embeddings

Sahand Sharifzadeh  Christos Kaplanis  Shreya Pathak  Dharshan Kumaran
Anastasija Ilic  Jovana Mitrovic  Charles Blundell  Andrea Banino

###### Abstract

The creation of high-quality human-labeled image-caption datasets presents a
significant bottleneck in the development of Visual-Language Models (VLMs). We
propose a novel approach that leverages the strengths of Large Language Models
(LLMs) and image generation models to create synthetic image-text pairs for
efficient and effective VLM training. Our method employs pretraining a text-
to-image model to synthesize image embeddings starting from captions generated
by an LLM. These synthetic pairs are then used to train a VLM. Extensive
experiments demonstrate that the VLM trained with synthetic data exhibits
comparable performance on image captioning, while requiring a fraction of the
data used by models trained solely on human-annotated data. In particular, we
outperform the baseline by 17% through augmentation with a synthetic dataset.
Furthermore, we show that synthesizing in the image embedding space is 25%
faster than in the pixel space. This research introduces a promising technique
for generating large-scale, customizable image datasets, leading to enhanced
VLM performance and wider applicability across various domains, all with
improved data efficiency and resource utilization.

Machine Learning, Computer Vision, Synthetic, VLM, Visual-Language
Model,Visual Language Model, Image Generation, Image Captioning, Synthetic
Data Generation

##  1 Introduction

Figure 1: Traditional dataset curation pipelines require a human in the loop
to collect and annotate images (A). We study whether we can reverse this
pipeline with generative models, i.e. by first sampling synthetic captions
from an LLM and then synthetically generating images from those (B). By
operating in the image embedding space, we also bypass computationally
expensive encoder/decoder steps, optimizing and integrating the process within
visual-language model (VLM) training (C).

Visual-language models (VLMs) are quickly emerging as powerful tools for
understanding visual and textual information. Their ability to combine these
two modalities holds immense promise for application ranging from image
captioning to visual question answering. While VLMs hold significant
potential, their performance is often constrained by limited data
availability. Recent breakthroughs demonstrate that pre-training VLMs on
larger image-text pair datasets leads to significant improvements in
downstream tasks (Li et al., 2022b; Hu et al., 2022). However, creating such
datasets poses several challenges such as scarcity of paired data, high
curation costs, low diversity and high imbalance in semantics, and the
potentially noisy nature of data sourced from the internet (e.g., LAION
(Schuhmann et al., 2021)). These factors often lead to laborious filtering and
extended training times due to low signal-to-noise ratio, thus increasing
overall resource consumption.

This work tackles these limitations by introducing a novel approach that
leverages pre-trained generative text and image models to efficiently create
synthetic paired data for VLMs. Our approach uniquely generates both text and
images synthetically, overcoming the reliance on real-world data and
addressing the challenges of scarcity, cost, and noise.

Table 1: A taxonomy of related work on synthetic data for training vision
models.

Generator | Method | Generator Model | Generator Range | Generated Pairs | Caption Class | Caption Type | Evaluation Setting |
---|---|---|---|---|---|---|---|---
|  Mishra et al. (2022), |  |  | Mixed pairs such as |  |  |  |
|  Greff et al. (2022), |  |  | (Segmentation, Images), |  |  |  |
Simulation/Rendering Engines |  Zheng et al. (2020), | Mix | Mix | (Optimal Flow, Videos), | N/A | N/A | Mix |
| de Melo et al. (2022) |  |  | (Depth Maps, Images) |  |  |  |
| Cascante-Bonilla et al. (2023) | Mix | Pixel | (Captions, Images) | Complex | Rule-based | Vision Encoder |
|  Azizi et al. (2023), | SD, |  |  |  |  |  |
Off-the-shelf Image Generator | Fan et al. (2023) | Imagen, | Pixel | (Class, Images) | Simple | ImageNet Classes | Classifier |
|  | MUSE |  |  |  |  |  |
| Li et al. (2023c) | SD | Pixel | (Captions, Images) | Complex | Human Generated | VLM |
|  |  |  |  |  | LLM Generated |  |
Controlled Image Generator | Synth2 | MUSE | Embedding | (Captions, Images) | Complex | Human Generated | VLM |

While synthetic data generation has been explored for various computer vision
tasks such as image segmentation, optical flow prediction, or image
classification (Mishra et al., 2022; Greff et al., 2022; Azizi et al., 2023;
Fan et al., 2023; Li et al., 2023c), its application to both visual and
textual modalities within VLM training is a significant advancement.

Moreover, instead of using off-the-shelf image generation models, we employ a
controlled image generation approach. That is, we pre-trained our image
generator on a dedicated dataset (also used for VLM training) to prevent
unintended knowledge transfer from the image generator into the VLM. This
strategy ensures a fair evaluation of the impact of synthetic data.

Finally, our framework can operate also at the embedding level; our text-to-
image model generates image embeddings which feed directly into the VLM,
bypassing pixel-space rendering. This paradigm shift dramatically reduces
memory overhead and resource consumption while preserving the quality of
synthetic training data.

To sum up, our research explores the synergy between VLMs and text-to-image
generation models, demonstrating a powerful framework for overcoming data
limitations. This paper presents the following key contributions:

  * •

Fully Synthetic Data Creation: We introduce the first VLM training process
that utilizes a fully synthetic dataset of high-quality text-image embedding
pairs. These pairs are generated by pre-trained generative models,
circumventing the need for massive amounts of real-world data for either
modality.

  * •

Efficient Embedding Space Generation: Our method works with images in
embedding space, avoiding costly image rendering and dramatically improving
efficiency, without comprising performance.

  * •

Fair Evaluation Through Control: By pre-training a text-to-image model on the
same dataset used for VLM training, instead of using a large, off-the-shelf
model, we prevent knowledge transfer from a model trained on vast, external
image-text pairs. This fosters a fair and focused assessment of the benefits
of synthetic data in isolation.

  * •

Demonstrated Effectiveness: Experiments will showcase significant performance
gains in image captioning when using our synthetic data, assessed against real
and synthetic-data baselines.

In conclusion, we offer insights into the future of VLM training, highlighting
the potential for creating customized and potentially infinitely expandable
datasets using text-driven approaches.

##  2 Related Works

##### VLMs

Visual-language models (VLMs) pretrained on large-scale image-text data,
primarily sourced from the web, have shown remarkable capabilities across
numerous vision-language tasks. These tasks include image captioning, visual
question answering, and few-shot learning (Alayrac et al., 2022; Chen et al.,
2022; Li et al., 2022b). Architecturally, VLMs commonly employ an image
encoder coupled with a large language model (LLM). Researchers explore diverse
image encoder designs, ranging from convolutional neural networks (CNNs)
(Alayrac et al., 2022) to transformer-based architectures (Chen et al., 2022).
Additionally, the choice between pretraining the image encoder on separate
image datasets (Alayrac et al., 2022) and training it from scratch alongside
the VLM (Tsimpoukelli et al., 2021) remains an active area of investigation.
Pretraining VLMs involves various objectives, including supervised learning
approaches, as well as contrastive methods that focus either on aligning image
and text representations (Radford et al., 2021) or on aligning different image
representations (Chen et al., 2020b, a; Grill et al., 2020). Similar diversity
exists in LLM choices, with both encoder-decoder (Chen et al., 2022) and
decoder-only (Alayrac et al., 2022) architectures being utilized. The
strategies for combining information extracted by the image encoder and the
language model also exhibit substantial variation. Despite the success
achieved with web-scraped datasets, there’s a growing emphasis on designing
VLM architectures that facilitate training using synthetically generated data.
This capability holds particular significance for applications where data
scarcity or resources availability pose significant challenges.

##### Training From Synthetic Data

The generation of synthetic data for training machine learning models remains
a highly active area of research. While numerous studies (Mishra et al., 2022;
Cascante-Bonilla et al., 2023; Greff et al., 2022; Zheng et al., 2020; de Melo
et al., 2022) explore the use of model-based rendering engines or simulators,
the remarkable advancements in high-quality image generators have ignited a
surge of interest in leveraging generative models for creating training data.
This trend has significantly impacted a wide range of computer vision tasks,
including semantic segmentation (Li et al., 2022a; Ros et al., 2016; Baranchuk
et al., 2021; Tritrong et al., 2021; Li et al., 2021; Chen et al., 2019),
human motion understanding (Varol et al., 2017; Ma et al., 2022; Guo et al.,
2022), and more recently image classification (Azizi et al., 2023; Fan et al.,
2023). Our work investigates the potential of data-driven generative models
within the domain of visual-language models (VLMs). We focus on their
application in downstream tasks such as image captioning, where the ability to
process complex scene descriptions is important. From this aspect, our
approach is closest to a concurrent work by Li et al. (2023c) where the goal
is to replace faulty images in a captioning pipeline with their synthetic
versions. However, our work distinguishes itself from Li et al. (2023c) by
achieving strong performance while utilizing 40 times less paired data and
only a quarter of the parameters. This demonstrates the potential for
achieving both performance and efficiency through our proposed approach.
Furthermore, unlike the mentioned works (Azizi et al. (2023); Fan et al.
(2023); Li et al. (2023c), we avoid using an off-the-self image generator
trained on larger datasets. Our work can also be likened to the concept of
cycle consistency (Zhu et al., 2017) in visual-language models. This
principle, where image-to-text and text-to-image conversion can be employed as
a form of synthetic data training, albeit with extra gradients throughout the
network, exploring image-text cycle consistency during sampling Li et al.
(2023a) or training Li et al. (2023b) has been explored in recent works,
demonstrating promising results.

Finally, we emphasize the efficiency of pipelines that seamlessly connect data
generation and model training. While most prior work has concentrated on
generating images in pixel space, we investigate the generation of image
embeddings that can be directly integrated into the VLM. Our approach aligns
with recent work in the scene graph classification domain (Sharifzadeh et al.,
2021, 2022), which has explored synthetic scene graph embedding generation
using pretrained object representations.

Table 1 provides a taxonomy of some of the mentioned related work to clarify
the differences better. As shown, our study is the first of its kind to
scientifically explore the application of synthetic data generation within a
VLM training pipeline and in particular using image embeddings and by
generating synthetic captions.

Figure 2: Example of synthetic captions and synthetic images generated by LLM
and text-to-image generator.

##  3 Synth2

Given the synthetic generation of text and then images, we refer to our method
as Synth2. Synth2 is a pipeline for training VLMs using generative models in
addition to collected human data. In this section we introduce different
components of this pipeline, namely Caption Generation, Image Generation and
the full Synth2 model.

Figure 3: Qualitatives showing images selected from the validation set of MS-
COCO (Chen et al., 2015) (top row) and their synthetic versions generated by
our text-to-image generator given the ground truth captions (bottom). The
training of the image generator has been done on Conceptual Caption v2
(Changpinyo et al., 2021).

###  3.1 Caption Generation

We leverage the generative capabilities of LLMs to create synthetic captions.
To ensure a wide variety of captions, we adopt a class-based prompting
approach. First, we randomly select a class from the ImageNet21k dataset
(Ridnik et al., 2021). The LLM (Gemini Pro (Team et al., 2023)) is then
prompted with the following text:

Make up a human-annotated description of an image that contains the following
object: $[object]$. The caption should be around 30-40 words long. Describing
the different components of the scene in an objective and unbiased way. Do not
add subjective judgments about the image, it should be as factual as possible.
Do not use fluffy, poetic language. Respond only with the caption itself,
beginning with “This is an image of”.

where we replace “[object]” with the randomly selected class. This class-based
prompting encourages the LLM to generate captions that cover a broad range of
visual concepts. Figure 2 shows some samples from the generated captions.

###  3.2 Image Generation

Off-the-shelf image generator models, trained on vast image-text datasets,
possess an unfair advantage for studying the effect of synthetic images in VLM
training. Their exposure to extensive human annotated data might also enable
them to produce images highly reminiscent of the original training set. To
ensure fair evaluation, we propose pre-training a text-to-image generator in a
controlled environment. This generator is trained on a human annotated image-
caption dataset that will also be utilized for VLM training, alongside the
synthetic data. By doing so, we effectively eliminate the effects of external
data on the VLM training.

Our architecture is similar to Chang et al. (2023) and was chosen for its
superior inference efficiency due to its use of discrete image tokens and its
parallel decoding capability, which dramatically reduces the number of
sampling iterations compared to auto-regressive models. Additionally, MUSE
leverages a transformer architecture, further enhancing efficiency (Chang et
al., 2023). Figure 3 shows some examples of generated images compared with
original ones.

####  3.2.1 Training

To pretrain our image generator given paired image and texts, we embed the
texts with a pre-trained language model and the images with a pre-trained VQ-
GAN (Esser et al., 2021) (refer to A.1.1 in the appendix for the details). The
pre-trained language model is the same used in our VLM and it is a reduced
version of Chinchilla (Hoffmann et al., 2022) that is using 400 million
parameters. Following the approach in Chang et al. (2023), we apply a masking
procedure with a cosine scheduling function. This function gradually increases
the masking ratio during training, biasing the model towards learning from
images with larger masked regions. On average, around 64% of the VQ tokens are
replaced with a special “dropped token”. The noisy VQ tokens and the embedded
text are then fed into a transformer model. The model’s objective is to
predict the original, unmasked VQ tokens. The loss function is the cross-
entropy between the predicted VQ tokens and the masked ones:

$L_{t2i}=\prod_{t\in M}p(z(x)_{t}|y,\\{z(x)_{u}|u\notin M\\}),$

where $z(x)$ denotes the VQ tokens computed from the ground-truth image $x$,
$y$ denotes the ground-truth caption, and $M$ denotes the indices of the
masked tokens.

Our text-to-image generator was trained only on 10.1 millions text-image pairs
from Conceptual Captions V2 (Changpinyo et al., 2021) (see Section 4 for
details).

####  3.2.2 Inference

During inference, we use an iterative procedure to generate images. At first
iteration, the predicted VQ tokens are all initialized with the “dropped
token” representing the masked entries. At each upcoming iteration, the text
embedding and the predicted VQ tokens until that stage are fed into the
transformer and the model predicts the masked tokens. A cosine schedule
determines which masked tokens have the highest prediction confidence during
decoding. These tokens are unmasked, progressively reducing the masked token
set. This process repeats for 24 iterations, gradually refining the generated
VQ-tokens.

Additional details for our text-to-image generator model are reported in A.1.3
in the appendix.

###  3.3 Synth2 VLM architecture

Figure 4: We introduce a VLM framework that leverages LLMs and image
generation models to create synthetic image-text pairs for efficient training.
We can train a VLM from both non-synthetic (A) and synthetic (B) data as shown
in this figure. Our model trained with the added synthetic pairs demonstrates
impressive image captioning performance, significantly reducing the need for
human annotated images.

As described in Section 3.2, our text-to-image model generates image
embeddings (VQ tokens) through an iterative caption-conditioned denoising
process. A VQ-GAN decoder could be used to convert these tokens into human-
viewable images. However, we emphasize that the image embeddings themselves
provide a compressed representation of an image, making the decoder step
unnecessary for certain applications.

Therefore, to enhance efficiency for training the VLM from synthetic data, we
design the VLM to enable the bypass of pixel-space processing. This is
achieved by setting its vision encoder to be identical to the VQ-GAN backbone
used for our image generator. This enables seamless interaction between the
VLM and synthetically generated image embeddings and eliminates the
computationally expensive decoding stage of VQ-GAN for the image generator and
the encoding stage for the VLM when training on synthetic data. At the same
time, it allows us to train from images in pixel space when using human
annotated data. Overall, our VQ-based design removes the need for costly
conversions to and from pixel space, resulting in significant time and disk
space savings as will be shown in Section 4. To harness the efficiency of
discrete tokens during training and sampling, while maintaining the rich image
understanding provided by soft embeddings, we convert each discrete token to
its soft embedding using the codebook before feeding them to the VLM.

On top of the VQ backbone in our VLM, we incorporate a Perceiver Resampler
component (Jaegle et al., 2021) to cross-attend to the VQ tokens extracted
from our backbone. Similar to Flamingo (Alayrac et al., 2022), we employ a
frozen language model to cross-attend to the image representation and
autoregressively generate text. While the self-attention layers in the
language model remain frozen, the cross-attention layers between the language
and vision components are trainable, allowing the model to learn effective
multimodal representations.

The VLM’s objective is to predict the next token in a sequence, conditioned on
the input image. This is formulated as a cross-entropy loss:

$L_{VLM}=\prod_{l=1}^{L}p(y_{l}|y_{<l},x),$

where $x$ denotes the image, $y_{l}$ represents the l-th language token, and
$y_{<l}$ represents all tokens preceding $y_{l}$.

As shown in Figure 4 by combining the components introduced in the previous
parts, we can train VLMs either from human annotated data or synthetic image-
captions pairs such that the VLM is conditioned on the synthetic image
embedding with the cross-entropy loss being between the synthetic captions and
the VLM’s predictions.

##  4 Experiments

###  4.1 Experimental setup

####  4.1.1 Datasets

For training we used the following datasets in our experiments:

Table 2: Zero shot image captioning results when training with ground truth
captions paired with either the original or synthetically generated images.

Method | Real | Synth | #Real Data | #Synth Data | CIDEr-COCO ($\uparrow$) | CIDEr-Flickr-30 ($\uparrow$)
---|---|---|---|---|---|---
Baseline | CCv2 | - | 10.1M | - | 22.1 | 11.3
Synth2 | CCv2 | LTIP | 10.1M | 330M | 28.7 | 12.9
| CCv2 | WebLI | 10.1M | 350M | 27.6 | 12.3
Gold | CCv2+LTIP | - | 340.1M | - | 27.6 | 13.2
| CCv2+WebLI | - | 360.1M | - | 30.7 | 15.1

  * •

Conceptual Captions v2 (Changpinyo et al., 2021) is a dataset with 12 million
image-text pairs that are collected by automatically extracting images and
their corresponding “alt-text” descriptions from various websites. In our
experiments we use Conceptual Captions v2 as the main source of human-
annotated data whereas for the other datasets we might use synthetically
generated images/texts depending on the experiment.

  * •

WebLI (Chen et al., 2022) has around 350 million image and alt text pairs with
high text quality in English.

  * •

LTIP (Alayrac et al., 2022) has 312 million web collected image and text
pairs. For more details refer to Alayrac et al. (2022).

  * •

GenPair refers to 1 million fully synthetic captions that are created as
described in Section 3.1, paired with synthetic images generated on-the-fly
during the VLM training using our text-to-image generator.

For evaluation we used:

  * •

MS-COCO (Chen et al., 2015): we evaluate the performance of our models on the
commonly used test set of MS-COCO under zero-shot and fine-tuning settings.
For the finetuning settings we use the training set MS-COCO. We use the
Karpathy split (Karpathy & Fei-Fei, 2015) for evaluation. We use CIDEr score
(Vedantam et al., 2015) as our metric.

  * •

Flickr-30k (Plummer et al., 2015): again, we evaluate performances on the
Karpathy split.

####  4.1.2 Training details

For both the image generator and the VLM we use a pre-trained and frozen VQ-
GAN (see A.1.1 for details on the network). The images are input at a
resolution of 256x256, with a patch size of 16, which results in a sequence
length of 256 tokens (16x16). Both models use a pre-trained and frozen
Chinchilla 400m (Hoffmann et al., 2022) as LLM. The VLM has a perceiver
resampler (Alayrac et al., 2022; Jaegle et al., 2021) with 6 layers and 64
latent input queries that cross-attend to the image embeddings. There are
cross attention layers between each layer in the LLM and the output of
perceiver resampler similar to the original Flamingo architecture (please
refer to A.1.4 for extra details).

We use ViT-B (Dosovitskiy et al., 2020) for the main text-to-image model in
the image generator (a transformer with 24 layers, embedding dimension of 769,
12 heads and an intermediate latent size of 3072, dropout rate of 0.1). We use
a maximum text length of 64 tokens with guidance scale of 4.0 with 24 refining
steps and sample choice temperature of 32.5. All the models are trained with
AdamW (Loshchilov & Hutter, 2017) with a learning rate of 0.0001 with 5000
warmup steps, a batch size of 512. For finetuning settings (on COCO), we use a
learning rate of 0.00001. Section A.1.3 reports extra details. We pre-train
the image generator for 500k steps at which point it has a Fréchet inception
distance (FID) (Heusel et al., 2017) of 17.1 on MS-COCO test pairs. Our VLMs
training experiments all run for 500k steps. Our models are all implemented in
Jax and trained on 256 TPUs.

###  4.2 Results

####  4.2.1 Synthetic Images

To assess the effectiveness of synthetic images in VLM training, we conduct a
study where human-written captions from existing datasets are paired with
synthetic images generated by a text-to-image model. We train a VLM on these
synthetic image-caption pairs and evaluate its performance against a model
trained on the original real image-caption pairs. This comparison allows us to
investigate whether synthetic images can effectively substitute for real
images in the context of VLM training.

Specifically, we generate synthetic images for the LTIP and WebLI datasets
using their annotations. This provides us with (Real Text, Synthetic Image)
pairs to train a VLM with (Synth2 in Table 2). We compare this model to the
models trained on the original (Real Text, Real Image) pairs (Gold in Table 2)
and one trained from human-annotated data of Conceptual Caption v2 (CCv2)
only, using no data from LTIP and WebLI (Baseline in Table 2). For
consistency, all Synth2 and Gold models are co-trained with CCv2, the same
dataset used by our text-to-image generator and the baseline.

Table 3: Zero shot image captioning results when using synthetically generated
caption and image embedding pairs. Concentration is calculated as the
cumulative distribution on the top-5 clusters, a lower value represent higher
diversity (see Appendix A.3 for more details).

Real | Synth | #Real Data | #Synth Data | Concentration ($\downarrow$) | Entropy ($\uparrow$) | CIDEr-COCO ($\uparrow$)
---|---|---|---|---|---|---
CCv2 | - | 10.1M | - | - | - | 22.1
CCv2 | GenPair | 10.1M | 1M | 57.7% | 3.81 | 25.9
CCv2+WebLI | - | 10.1M+1M | - | 69.8% | 3.43 | 24.4
CCv2+LTIP | - | 10.1M+1M | - | 83.0% | 2.92 | 23.4

As shown in Table 2, synthetic images significantly improve baseline model
performance by over 25%. Importantly, they are effective for VLM training
despite the smaller volume of human-annotated images used. Note that “Gold”
represents the upper performance bound when using the full original dataset.
For LTIP, synthetic images slightly outperform original images, likely due to
increased data diversity. Because original pairs remain static during
training, synthetic images introduce beneficial image augmentation (Azizi et
al., 2023).

These results suggest the potential of generative models to enhance VLM
training, even with limited real image data. While previous work (Azizi et
al., 2023; Fan et al., 2023) demonstrated the use of synthetic images for
single-object image classification (i.e., ImageNet), our findings show that
VLMs can be successfully trained with synthetic datasets containing complex,
multi-object scenes and detailed captions. This highlights the potential of
synthetic images to move beyond simple classification and significantly
improve VLM performance in diverse, challenging scenarios.

####  4.2.2 Synthetic text and image pairs

To further demonstrate the efficacy of using synthetic data for VLM training,
we conducted additional experiments where the entire dataset including
captions and their corresponding images are generated synthetically by an LLM
and the image generator (see Section 3.3 for the explanation of how these
caption where generate and Figure 2 for some examples).

Comparing the first two rows in Table 3 (baseline vs. GenPair) shows that
adding even a small fraction (1M) of purely synthetic image-caption data
(GenPair) significantly improves performance (17.2% increase in CIDEr score on
MS-COCO compared to the baseline model trained only on real data).
Interestingly, the same improvement is not achieved by sampling an additional
1M data points from real datasets like WebLI or LTIP (rows 3 and 4). To
investigate this, we assessed the semantic diversity and balance of each
caption set. First, we used our language model to generate embeddings for the
captions. Then, we employed k-means clustering to analyze these embeddings
(see Appendix A.3 for details). Figure 5 visualizes the cluster distribution,
with the x-axis representing the cluster index and the y-axis representing
data volume from each dataset within a cluster. GenPair shows a more even
distribution across clusters, indicating greater conceptual diversity. It also
exhibits less variance in cluster size, suggesting better balance.

Finally, to quantify the semantic concentration within each dataset, Table 3
presents the ”Concentration” metric: the percentage of data belonging to the
top-5 most populated clusters. GenPair has the lowest concentration, with only
57.5% of its captions in the top-5 clusters. This indicates lower semantic
imbalance in the synthetically generated GenPair and contrasts with the higher
imbalance found in the other datasets (69.9% and 83%). Also, the entropy over
the full cluster distribution, confirms that GenPair has a more uniform
distribution of data across clusters. We postulate that the inherent diversity
within GenPair likely contributes to its robust performance on MS-COCO, as
models trained on diverse data tend to generalize better. (See Appendix A.3
for extra details on how the cluster analysis was performed and for extra
results.)

Figure 5: Semantic diversity. Histogram represent the distribution of cluster
sizes, with GenPair showing a more uniform coverage of semantic concepts. See
A.3 for details on how the histogram was derived.

This evaluation showcases Synth2’s ability to utilize diverse text-based data,
including purely synthetic ones. Overall, our experiments highlight the
significant role that synthetic data can play in enhancing VLM performance by
leveraging synthetic image generation and LLM-generated captions. This
approach not only overcomes data limitations but also demonstrates the
effectiveness of text-only captions in boosting model capabilities, further
emphasizing the value of our technique.

####  4.2.3 Comparison with related work

Furthermore, we investigated the potential of our approach compared to the
state-of-the-art model while taking into account the amount of synthetic or
real training data and the number of model parameters. We combined the two
datasets with the highest diversity, namely, GenPair and WebLI. Table 4
compares our model’s MS-COCO benchmark performance against related methods:
ITIT (Li et al., 2023b) and DC (Li et al., 2023c). Note that, for consistency
with these methods, we used Synth2 finetuned on the MS-COCO training set (see
A.2). ITIT employs cycle consistency for VLM improvement, while DC uses Stable
Diffusion to generate images based on human-defined heuristics.

Synth2 and ITIT share similar parameter counts and paired data usage, making
them directly comparable. As shown in Table 4, Synth2 drastically outperforms
ITIT on the CIDEr score. While DC achieves the highest raw performance, it
requires significantly more parameters, relies on vast amount of real data,
and the filtering procedure is heavily based on human heuristics.

Table 4: Comparison with related work on COCO. The image-captioning
performance is evaluated on the COCO Karpathy split. Bold indicates the best
and underline the second best values.

Model |  $\\#$All Params$\downarrow$ |  $\\#$Real Data$\downarrow$ |  $\\#$Synth Data$\downarrow$ | CIDEr$\uparrow$ (FT)
---|---|---|---|---
ITIT | 11.2B | 3M | 110M | 103.5
DC | 1.7B | 5.5B | 400M | 133.1
Synth2 | 632M | 10.1M | 360M | 126.1

Synth2 strikes a balance between performance and efficiency. By leveraging
text-only data, it achieves competitive results using 40 times less paired
data than DC. This comparison highlights the trade-offs between performance,
efficiency, and data requirements. Synth2 emerges as a compelling solution
when resources are limited, achieving impressive results with reduced data and
computational overhead.

####  4.2.4 Data efficiency and augmentation with fully synthetic data

To further analyze the boost in performance afforded by augmentation with
fully synthetic data, we characterized the performance of the baseline model
(i.e. trained solely on CCv2), and Synth2 which was additionally trained
together with GenPair. Figure 6 visualizes the performance trends for both
models. Notably, the light blue curve representing the baseline exhibits a
gradual increase in performance with increasing training steps, eventually
plateauing at 22.1 CIDER score. In contrast, the purple curve reveals Synth2’s
steeper performance improvement, achieving comparable performance to the
paired only training regime, with roughly 1/3 of the training steps. This
highlights the significant data efficiency gain achieved through fully
synthetic data augmentation in Synth2. Shaded regions surrounding each curve
indicate standard deviation across three random seeds, demonstrating the
robustness of the observed performance trends. These findings demonstrate
Synth2’s ability to effectively leverage fully synthetic data, resulting in
remarkable performance gains.

Our study also investigated the computational efficiency of Synth2. We
compared the case where captions are used to generate image embedding versus
when captions are rendered into actual images. As shown in Table 5, Synth2
trained using image embedding consistently demonstrates faster performance,
running at 2.08 training steps per second, whereas Synth2 trained from pixel
runs at a slower pace of 1.66 steps/second. This performance advantage stems
from Synth2’s efficient utilization of text-only data and embedding-based
image generation using parallel decoding, which reduces the computational
overhead associated with processing pixel-based images as done in the Synth2
model trained from pixels without affecting the performance.

Figure 6:  Performance as a function of training steps. The blue curve shows
the baseline trained solely on paired data (CCv2). The purple curve
demonstrates Synth2’s performance trained additionally on augmentation with
fully synthetic data (GenPair). Synth2 achieves parity with the baseline using
roughly 1/3 of the training steps, showcasing its superior efficiency. Shaded
regions represent standard deviation across 3 random seeds.

##  5 Limitations

The three main limitations of the current work stem from the restricted
quantity of fully synthetic data used, potential biases in the generative
model and the limited exploration of text data sources. Firstly, whilst we
were able to show a substantial gain when training was augmented with fully
synthetic data, our experiments were limited to a relatively low quantity of
data (i.e. 1M image-caption pairs). In the future, it will be important to
examine whether using orders of magnitude more fully synthetic data (e.g.
$\sim$700M) will result in performance gains that surpass that of using solely
paired data. Secondly, the generative model responsible for text-to-image
generation can introduce biases into the synthetic data, reflecting
limitations inherent in its training data, architecture, or procedure. A
possible future step would involve investigating this issue using various
text-to-image models to assess and mitigate potential biases. Thirdly, the
current work primarily explores a restricted set of text-only data sources.
Further research is necessary to investigate the potential of diverse text
sources, including investigating context-specific generation (e.g., medical
data), which could be an exciting avenue for future exploration.

By acknowledging and addressing these limitations, future research can
leverage the strengths of the Synth2 approach and enhance its
generalizability, robustness, and efficiency for a broader range of visual-
language modeling tasks.

Table 5: Model efficiency measured by computing the steps per second of training on the same hardware. Synth2 in embedding space consistently outperforms the Synth2 using pixels, demonstrating superior training efficiency. Model | Step/Sec$\uparrow$ | CIDEr(zs)$\uparrow$
---|---|---
Synth2\- Image Embedding | 2.08 | 25.9
Synth2\- Pixels | 1.66 | 26.0

##  6 Conclusions

Our study presents a novel approach for generating synthetic image-text pairs
to enhance the training of visual language models. By harnessing the
capabilities of large language models and text-to-image generation, we
effectively address the limitations of manual image labeling. Our results
conclusively demonstrate several significant findings:

  * •

Improved VLM Performance: The visual language model trained on our synthetic
and human annotated datasets exhibits a marked improvement in image captioning
tasks compared to a baseline trained exclusively on human-annotated data. This
underscores the potential of our method to augment VLM capabilities
efficiently. This is highly advantageous in scenarios where data acquisition
and annotation are resource-intensive.

  * •

Data Efficiency: Our approach yields comparable performance while utilizing
only a fraction of human-labeled data.

  * •

Customization and Scalability: Our method provides flexibility in generating
customized image datasets tailored to specific domains. Additionally, the
synthetic data generation process is scalable, making it suitable for
supporting large-scale VLM development.

This research introduces a novel technique that has the potential to transform
visual language model training. Our findings highlight the value of synthetic
data generation, paving the way for advancements in numerous fields where
visual language understanding is crucial. Further explorations could
investigate refining the synthetic data generation process with different
objectives in mind.

##  7 Broader Impact

This paper presents work whose goal is to advance the field of Machine
Learning. There are many potential societal consequences of our work, none
which we feel must be specifically highlighted here. In particular biases in
generative models can have a drastic effect on the synthetic data. Similarly
it is important to consider the challenges of privacy when using generative
models.

## Acknowledgements

We would like to thank Sander Dieleman, Ali Razavi, and Benigno Uria for their
insights on VQGAN. We would also like to thank Aida Nematzadeh, Pinelopi
Papalampidi, Han Zhang, Mateusz Malinowski, Valentin De Bortoli, Sahra
Ghalebikesabi, Emanuele Bugliarello, Chris Knutsen, and Murray Shanahan for
their in depth comments throughout the project.

## References

  * Alayrac et al. (2022) Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736, 2022.
  * Azizi et al. (2023) Azizi, S., Kornblith, S., Saharia, C., Norouzi, M., and Fleet, D. J.  Synthetic data from diffusion models improves imagenet classification.  _arXiv preprint arXiv:2304.08466_ , 2023.
  * Baranchuk et al. (2021) Baranchuk, D., Rubachev, I., Voynov, A., Khrulkov, V., and Babenko, A.  Label-efficient semantic segmentation with diffusion models.  _arXiv preprint arXiv:2112.03126_ , 2021.
  * Cascante-Bonilla et al. (2023) Cascante-Bonilla, P., Shehada, K., Smith, J. S., Doveh, S., Kim, D., Panda, R., Varol, G., Oliva, A., Ordonez, V., Feris, R., et al.  Going beyond nouns with vision & language models using synthetic data.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 20155–20165, 2023.
  * Chang et al. (2023) Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T., Rubinstein, M., et al.  Muse: Text-to-image generation via masked generative transformers.  _arXiv preprint arXiv:2301.00704_ , 2023.
  * Changpinyo et al. (2021) Changpinyo, S., Sharma, P., Ding, N., and Soricut, R.  Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 3558–3568, 2021.
  * Chen et al. (2020a) Chen, T., Kornblith, S., Norouzi, M., and Hinton, G.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pp. 1597–1607. PMLR, 2020a.
  * Chen et al. (2020b) Chen, T., Kornblith, S., Swersky, K., Norouzi, M., and Hinton, G. E.  Big self-supervised models are strong semi-supervised learners.  _Advances in neural information processing systems_ , 33:22243–22255, 2020b.
  * Chen et al. (2015) Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollár, P., and Zitnick, C. L.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_ , 2015.
  * Chen et al. (2022) Chen, X., Wang, X., Changpinyo, S., Piergiovanni, A., Padlewski, P., Salz, D., Goodman, S., Grycner, A., Mustafa, B., Beyer, L., et al.  Pali: A jointly-scaled multilingual language-image model.  _arXiv preprint arXiv:2209.06794_ , 2022.
  * Chen et al. (2019) Chen, Y., Li, W., Chen, X., and Gool, L. V.  Learning semantic segmentation from synthetic data: A geometrically guided input-output adaptation approach.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 1841–1850, 2019.
  * de Melo et al. (2022) de Melo, C. M., Torralba, A., Guibas, L., DiCarlo, J., Chellappa, R., and Hodgins, J.  Next-generation deep learning based on simulators and synthetic data.  _Trends in cognitive sciences_ , 2022.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Esser et al. (2021) Esser, P., Rombach, R., and Ommer, B.  Taming transformers for high-resolution image synthesis.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 12873–12883, 2021.
  * Fan et al. (2023) Fan, L., Chen, K., Krishnan, D., Katabi, D., Isola, P., and Tian, Y.  Scaling laws of synthetic images for model training… for now.  _arXiv preprint arXiv:2312.04567_ , 2023.
  * Greff et al. (2022) Greff, K., Belletti, F., Beyer, L., Doersch, C., Du, Y., Duckworth, D., Fleet, D. J., Gnanapragasam, D., Golemo, F., Herrmann, C., et al.  Kubric: A scalable dataset generator.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 3749–3761, 2022.
  * Grill et al. (2020) Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ , 33:21271–21284, 2020.
  * Guo et al. (2022) Guo, X., Wu, W., Wang, D., Su, J., Su, H., Gan, W., Huang, J., and Yang, Q.  Learning video representations of human motion from synthetic data.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 20197–20207, 2022.
  * Heusel et al. (2017) Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Hoffmann et al. (2022) Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al.  Training compute-optimal large language models.  _arXiv preprint arXiv:2203.15556_ , 2022.
  * Hu et al. (2022) Hu, X., Gan, Z., Wang, J., Yang, Z., Liu, Z., Lu, Y., and Wang, L.  Scaling up vision-language pre-training for image captioning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 17980–17989, 2022.
  * Jaegle et al. (2021) Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., and Carreira, J.  Perceiver: General perception with iterative attention.  In _International conference on machine learning_ , pp. 4651–4664. PMLR, 2021.
  * Karpathy & Fei-Fei (2015) Karpathy, A. and Fei-Fei, L.  Deep visual-semantic alignments for generating image descriptions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3128–3137, 2015.
  * Kingma & Ba (2014) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Li et al. (2021) Li, D., Yang, J., Kreis, K., Torralba, A., and Fidler, S.  Semantic segmentation with generative models: Semi-supervised learning and strong out-of-domain generalization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8300–8311, 2021.
  * Li et al. (2022a) Li, D., Ling, H., Kim, S. W., Kreis, K., Fidler, S., and Torralba, A.  Bigdatasetgan: Synthesizing imagenet with pixel-wise annotations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 21330–21340, 2022a.
  * Li et al. (2023a) Li, H., Gu, J., Koner, R., Sharifzadeh, S., and Tresp, V.  Do dall-e and flamingo understand each other?  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 1999–2010, 2023a.
  * Li et al. (2022b) Li, J., Li, D., Xiong, C., and Hoi, S.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pp. 12888–12900. PMLR, 2022b.
  * Li et al. (2023b) Li, T., Bhardwaj, S., Tian, Y., Zhang, H., Barber, J., Katabi, D., Lajoie, G., Chang, H., and Krishnan, D.  Leveraging unpaired data for vision-language generative models via cycle consistency.  _arXiv preprint arXiv:2310.03734_ , 2023b.
  * Li et al. (2023c) Li, W., Lotz, J. F., Qiu, C., and Elliott, D.  Data curation for image captioning with text-to-image generative models.  _arXiv preprint arXiv:2305.03610_ , 2023c.
  * Loshchilov & Hutter (2017) Loshchilov, I. and Hutter, F.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Ma et al. (2022) Ma, J., Bai, S., and Zhou, C.  Pretrained diffusion models for unified human motion synthesis.  _arXiv preprint arXiv:2212.02837_ , 2022.
  * Mishra et al. (2022) Mishra, S., Panda, R., Phoo, C. P., Chen, C.-F. R., Karlinsky, L., Saenko, K., Saligrama, V., and Feris, R. S.  Task2sim: Towards effective pre-training and transfer from synthetic data.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9194–9204, 2022.
  * Plummer et al. (2015) Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2641–2649, 2015.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Ridnik et al. (2021) Ridnik, T., Ben-Baruch, E., Noy, A., and Zelnik-Manor, L.  Imagenet-21k pretraining for the masses.  _arXiv preprint arXiv:2104.10972_ , 2021.
  * Ros et al. (2016) Ros, G., Sellart, L., Materzynska, J., Vazquez, D., and Lopez, A. M.  The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 3234–3243, 2016.
  * Schuhmann et al. (2021) Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * Sharifzadeh et al. (2021) Sharifzadeh, S., Baharlou, S. M., and Tresp, V.  Classification by attention: Scene graph classification with prior knowledge.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pp. 5025–5033, 2021.
  * Sharifzadeh et al. (2022) Sharifzadeh, S., Baharlou, S. M., Schmitt, M., Schütze, H., and Tresp, V.  Improving scene graph classification by exploiting knowledge from texts.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pp. 2189–2197, 2022.
  * Team et al. (2023) Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Tritrong et al. (2021) Tritrong, N., Rewatbowornwong, P., and Suwajanakorn, S.  Repurposing gans for one-shot semantic part segmentation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 4475–4485, 2021.
  * Tsimpoukelli et al. (2021) Tsimpoukelli, M., Menick, J. L., Cabi, S., Eslami, S., Vinyals, O., and Hill, F.  Multimodal few-shot learning with frozen language models.  _Advances in Neural Information Processing Systems_ , 34:200–212, 2021.
  * Varol et al. (2017) Varol, G., Romero, J., Martin, X., Mahmood, N., Black, M. J., Laptev, I., and Schmid, C.  Learning from synthetic humans.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 109–117, 2017.
  * Vedantam et al. (2015) Vedantam, R., Lawrence Zitnick, C., and Parikh, D.  Cider: Consensus-based image description evaluation.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 4566–4575, 2015.
  * Zheng et al. (2020) Zheng, J., Zhang, J., Li, J., Tang, R., Gao, S., and Zhou, Z.  Structured3d: A large photo-realistic dataset for structured 3d modeling.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX 16_ , pp. 519–535. Springer, 2020.
  * Zhu et al. (2017) Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2223–2232, 2017.

##  Appendix A Appendix.

###  A.1 Extra Implementation details

####  A.1.1 VQ-GAN details

Table 6: Configuration and training hyperparameters for VQGAN. Config param | Value
---|---
Perceptual loss weight | 0.05
Adversarial loss weight | 0.01
Codebook size | 8192
Optimizer | Adam (Kingma & Ba, 2014)
Discriminator learning rate | 1e-4
Generator learning rate | 1e-4
Optimizer momentum |  $\beta_{1}$=0.9 $\beta_{2}$=0.99
Batch Size | 256
Learning rate schedule | Cosine
Decay (Loshchilov & Hutter, 2017) Warmup steps | 10000
Training steps | 500000

VQGAN Architecture: Our VQGAN architecture is similar to the previous work
(Esser et al., 2021). It consists of several residual blocks, downsample
(encoder) and upsample (decoder) blocks. The main difference is that we remove
the non-local block to make the encoder and decoder fully convolutional to
support different image sizes. In the base VQGAN model, we apply 2 residual
blocks in each resolution and the base channel dimension is 128. For the
finetuned decoder, we apply 4 residual blocks in each resolution and we also
make the base channel dimension to be 256.

####  A.1.2 Optimization details for Synth2

Config param | Value
---|---
Optimizer | AdamW (Loshchilov & Hutter, 2017)
Learning rate | 1e-4
Warmup steps | 5e3
Weight Decay | 1e-4
Gradient Clip | 1.0

####  A.1.3 Text to image generator details

Config param | Value
---|---
Num Layers | 12
Num Heads | 12
Embedding hidden dim | 768
MLP hidden dim | 3072
Dropout | 0.1
Codebook size | 8192
Max Sample Length | 64
Guidance Scale | 4
Sample Temperature | 32.5
Activation | GeLU

####  A.1.4 Perceiver details

Table 7: Perceiver details Config param | Value
---|---
Num Layers | 6
Num Heads | 16
Embedding hidden dim | 1024
Activation | GeLU

###  A.2 Finetuning details

Following the pre-training stage, our model undergoes fine-tuning for various
downstream tasks. The fine-tuning process employs the AdamW optimizer with
similar Beta values as in pre-training. The learning rate was 1e-5.

To enhance generalization capabilities during fine-tuning, we utilize
regularization methods such as Dropout (set to 0.1).

In accordance with established practices, we use the development split to
determine optimal fine-tuning settings. The performance results are then
reported based on the test split.

###  A.3 Clustering details

To determine the optimal number of clusters for our analysis, we employed the
Elbow Method in conjunction with the Within-Cluster Sum of Squares (WCSS)
metric. We iteratively applied the K-means clustering algorithm while varying
the number of clusters, calculating the WCSS for each iteration. A clear
‘elbow’ point was observed in the WCSS plot, indicating a substantial decrease
in variance as the number of clusters increased up to 20. Beyond this point,
further increases in the number of clusters yielded diminishing returns in
terms of WCSS reduction. Based on this analysis, we determined that 20
clusters provided a suitable balance between parsimony and capturing the
underlying structure within our dataset.

To analyze the diversity of cluster compositions, we employed a co-clustering
approach with the K-means algorithm on the concatenated GenPair, WebLI, and
LTIP datasets. For each of the resulting clusters, we calculated the
normalized cluster sizes for each individual dataset. This allowed us to
visualize the distribution among the datasets within each cluster, as
illustrated in Figure 5.

To calculate the concentration we performed a cumulative sum over the top-5
clusters to determine what percentage of the data points was present there.
The idea is that, the higher the percentage in the top 5 clusters, the less
uniform is the distribution across all clusters, potentially indicating a
lower level of diversity. Here in Table 8 we also report top-3 and the entropy
over the full 20 clusters. Both measures confirm a more uniform distribution
of data across clusters for GenPair versus the WebLi and LTIP. Especially the
entropy complement the other measures as it takes into account all clusters,
even if they are smaller.

Table 8: Zero shot image captioning results when using synthetically generated
caption and image embedding pairs.

Real | Synth | #Real Data | #Synth Data | CIDEr-COCO ($\uparrow$) | Concentration Top-3 ($\downarrow$) | Concentration Top-5 ($\downarrow$) | Entropy ($\uparrow$)
---|---|---|---|---|---|---|---
CCv2+WebLI | - | 10.1M+1M | - | 24.4 | 58.2% | 69.8% | 3.43
CCv2+LTIP | - | 10.1M+1M | - | 23.4 | 76.5% | 83.0% | 2.92
CCv2 | GenPair | 10.1M | 1M | 25.9 | 40.5% | 57.7% | 3.81