# Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion
Model

Deepanway Ghosal‡, Navonil Majumder‡, Ambuj Mehrish‡, Soujanya Poria‡
‡ DeCLaRe Lab, Singapore University of Technology and Design, Singapore
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

The immense scale of the recent large language models (LLM) allows many
interesting properties, such as, instruction- and chain-of-thought-based fine-
tuning, that has significantly improved zero- and few-shot performance in many
natural language processing (NLP) tasks. Inspired by such successes, we adopt
such an instruction-tuned LLM Flan-T5 as the text encoder for text-to-audio
(TTA) generation—a task where the goal is to generate an audio from its
textual description. The prior works on TTA either pre-trained a joint text-
audio encoder or used a non-instruction-tuned model, such as, T5.
Consequently, our latent diffusion model (LDM)-based approach (Tango)
outperforms the state-of-the-art AudioLDM on most metrics and stays comparable
on the rest on AudioCaps test set, despite training the LDM on a 63 times
smaller dataset and keeping the text encoder frozen. This improvement might
also be attributed to the adoption of audio pressure level-based sound mixing
for training set augmentation, whereas the prior methods take a random mix.

: https://github.com/declare-lab/tango

: https://tango-web.github.io/

##  1 Introduction

Following the success of automatic text-to-image (TTI) generation [28, 29,
30], many researchers have also succeeded in text-to-audio (TTA) generation
[16, 17, 38] by employing similar techniques as the former. Such models may
have strong potential use cases in the media production where the creators are
always looking for novel sounds that fit their creations. This could be
especially useful in prototyping or small-scale projects where producing the
exact sound could be infeasible. Beyond this, these techniques also pave the
path toward general-purpose multimodal AI that can simultaneously recognize
and generate multiple modalities.

To this end, the existing works use a large text encoder, such as, RoBERTa
[18] and T5 [27], to encode the textual description of the audio to be
generated. Subsequently, a large transformer decoder or a diffusion model
generates the audio prior, which is subsequently decoded by a pre-trained VAE,
followed by a vocoder. We instead assume that replacing the text encoder with
an instruction-tuned large language model (LLM) would improve text
understanding and overall audio generation without any fine-tuning, due to its
recently discovered gradient-descent mimicking property [3]. To augment
training samples, the existing methods take a randomly generated combination
of audio pairs, along with the concatenation of their descriptions. Such a
mixture does not account for the overall pressure level of the source audios,
potentially leading to a louder audio overwhelming the quieter one. Thus, we
employ a pressure level-based mixing method, as suggested by Tokozume et al.
[35].

Our model (Tango) 111The acronym Tango stands for Text-to-Audio using
iNstruction Guided diffusiOn and was suggested by ChatGPT. The word Tango is
often associated with music [37] and dance [36]. According to Wikipedia [36],
“Tango is a partner dance and social dance that originated in the 1880s along
the Río de la Plata, the natural border between Argentina and Uruguay.” The
image above resembles the Tango dance form and was generated by prompting
Dalle-V2 with ‘‘A couple dancing tango with musical notes in the background’’
is inspired by latent diffusion model (LDM) [30] and AudioLDM [17] models.
However, instead of using CLAP-based embeddings, we used a large language
model (LLM) due to its powerful representational ability and fine-tuning
mechanism, which can help learn complex concepts in the textual description.
Our experimental results show that using an LLM greatly improves text-to-audio
generation and outperforms state-of-the-art models, even when using a
significantly smaller dataset. In the image generation literature, the effects
of LLM has been studied before by Saharia et al. [32]. However, they
considered T5 as the text encoder which is not pre-trained on instruction-
based datasets. Flan-T5 [2] is initialized with a T5 checkpoint and fine-tuned
on a dataset of 1.8K NLP tasks in terms of instructions and chain-of-thought
reasoning. By leveraging instruction-based tuning, Flan-T5 has achieved state-
of-the-art performance on several NLP tasks, matching the performance of LLMs
with billions of parameters.

In Section 3, we empirically show that Tango outperforms AudioLDM and other
baseline approaches on most of the metrics on AudioCaps test set under both
objective and subjective evaluations, despite training the LDM on a $63$ times
smaller dataset. We believe that if Tango is trained on a larger dataset such
as AudioSet (as Liu et al. [17] did), it would be able to provide even better
results and improve its ability to recognize a wider range of sounds.

The overall contribution of this paper is threefold:

  1. 1.

We do not use any joint text-audio encoder—such as CLAP—for guidance. Liu et
al. [17] claim that CLAP-based audio guidance is necessary during training for
better performance. We instead use a frozen instruction-tuned pre-trained LLM
Flan-T5 with strong text representation capacity for text guidance in both
training and inference.

  2. 2.

AudioLDM needed to fine-tune RoBERTa [18] text encoder to pre-train CLAP. We,
however, keep Flan-T5 text encoder frozen during LDM training. Thus, we find
that LDM itself is capable of learning text-to-audio concept mapping and
composition from a 63 times smaller training set, as compared to AudioLDM,
given an instruction-tuned LLM.

  3. 3.

To mix audio pairs for data augmentation, inspired by Tokozume et al. [35], we
consider the pressure levels of the audio pairs, instead of taking a random
combination as the prior works like AudioLDM. This ensures good
representations of both source audios in the fused audio.

##  2 Method

Figure 1: Overall architecture of Tango.

Tango, as depicted in Fig. 1, has three major components: i) textual-prompt
encoder, ii) latent diffusion model (LDM), and iii) mel-spectogram/audio VAE.
The textual-prompt encoder encodes the input description of the audio.
Subsequently, the textual representation is used to construct a latent
representation of the audio or audio prior from standard Gaussian noise, using
reverse diffusion. Thereafter the decoder of the mel-spectogram VAE constructs
a mel-spectogram from the latent audio representation. This mel-spectogram is
fed to a vocoder to generate the final audio.

###  2.1 Textual-Prompt Encoder

We use the pre-trained LLM Flan-T5-Large (780M) [2] as the text encoder
($E_{text}$) to obtain text encoding $\tau\in\mathbb{R}^{L\times d_{text}}$,
where $L$ and $d_{text}$ are the token count and token-embedding size,
respectively. Due to the pre-training of Flan-T5 models on a large-scale
chain-of-thought- (CoT) and instruction-based dataset, Dai et al. [3] posit
that they are able to learn a new task very well from the in-context
information by mimicking gradient descent through attention weights. This
property is missing in the older large models, such as RoBERTa [18] (used by
Liu et al. [17]) and T5 [27] (used by Kreuk et al. [16]). Considering each
input sample a distinct task, it might be reasonable to assume that the
gradient-descent mimicking property could be pivotal in learning the mapping
between textual and acoustic concepts without fine-tuning the text encoder.
The richer pre-training may also allow the encoder to better emphasize the key
details with less noise and enriched context. This again may lead to the
better transformation of the relevant textual concepts into their acoustics
counterparts. Consequently, we keep the text encoder frozen, assuming the
subsequent reverse diffusion process (see Section 2.2) would be able to learn
the inter-modality mapping well for audio prior to construction. We also
suspect that fine-tuning $E_{text}$ may degrade its in-context learning
ability due to gradients from the audio modality that is out of distribution
to the pre-training dataset. This is in contrast with Liu et al. [17] that
fine-tunes the pre-trained text encoder as a part of the text-audio joint-
representation learning (CLAP) to allow audio prior reconstruction from text.
In Section 3, we empirically show that such joint-representation learning may
not be necessary for text-to-audio transformation.

###  2.2 Latent Diffusion Model for Text-Guided Generation

The latent diffusion model (LDM) [30] is adapted from Liu et al. [17], with
the goal to construct the audio prior $z_{0}$ (see Section 2.5) with the
guidance of text encoding $\tau$. This essentially reduces to approximating
the true prior $q(z_{0}|\tau)$ with parameterized $p_{\theta}(z_{0}|\tau)$.

LDM can achieve the above through forward and reverse diffusion processes. The
forward diffusion is a Markov chain of Gaussian distributions with scheduled
noise parameters $0<\beta_{1}<\beta_{2}<\cdots<\beta_{N}<1$ to sample noisier
versions of $z_{0}$:

$\displaystyle q(z_{n}|z_{n-1})$
$\displaystyle=\mathcal{N}(\sqrt{1-\beta_{n}}z_{n-1},\beta_{n}\mathbf{I}),$
(1) $\displaystyle q(z_{n}|z_{0})$
$\displaystyle=\mathcal{N}(\sqrt{\overline{\alpha}_{n}}z_{0},(1-\overline{\alpha}_{n})\mathbf{I}),$
(2)

where $N$ is the number of forward diffusion steps, $\alpha_{n}=1-\beta_{n}$,
and $\overline{\alpha}_{n}=\prod_{i=1}^{n}\alpha_{n}$. Song et al. [34] show
that Eq. 2 conveniently follows from Eq. 1 through reparametrization trick
that allows direct sampling of any $z_{n}$ from $z_{0}$ via a non-Markovian
process:

$z_{n}=\sqrt{\overline{\alpha}_{n}}z_{0}+(1-\overline{\alpha}_{n})\epsilon,$
(3)

where the noise term $\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$. The
final step of the forward process yields
$z_{N}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$.

The reverse process denoises and reconstructs $z_{0}$ through text-guided
noise estimation ($\hat{\epsilon}_{\theta}$) using loss

$\displaystyle\mathcal{L}_{DM}=\sum_{n=1}^{N}\gamma_{n}\mathbb{E}_{\epsilon_{n}\sim\mathcal{N}(\mathbf{0},\mathbf{I}),z_{0}}||\epsilon_{n}-\hat{\epsilon}_{\theta}^{(n)}(z_{n},\tau)||_{2}^{2},$
(4)

where $z_{n}$ is sampled from Eq. 3 using standard normal noise
$\epsilon_{n}$, $\tau$ is the text encoding (see Section 2.1) for guidance,
and $\gamma_{n}$ is the weight of reverse step $n$ [5], taken to be a measure
of signal-to-noise ratio (SNR) in terms of $\alpha_{1:N}$. The estimated noise
is used to reconstruct $z_{0}$:

$\displaystyle p_{\theta}(z_{0:N}|\tau)$
$\displaystyle=p(z_{N})\prod_{n=1}^{N}p_{\theta}(z_{n-1}|z_{n},\tau),$ (5)
$\displaystyle p_{\theta}(z_{n-1}|z_{n},\tau)$
$\displaystyle=\mathcal{N}(\mu^{(n)}_{\theta}(z_{n},\tau),\tilde{\beta}^{(n)}),$
(6) $\displaystyle\mu_{\theta}^{(n)}(z_{n},\tau)$
$\displaystyle=\frac{1}{\sqrt{\alpha_{n}}}[z_{n}-\frac{1-\alpha_{n}}{\sqrt{1-\overline{\alpha}_{n}}}\hat{\epsilon}_{\theta}^{(n)}(z_{n},\tau)],$
(7) $\displaystyle\tilde{\beta}^{(n)}$
$\displaystyle=\frac{1-\bar{\alpha}_{n-1}}{1-\bar{\alpha}_{n}}\beta_{n}.$ (8)

The noise estimation $\hat{\epsilon}_{\theta}$ is parameterized with U-Net
[31] with a cross-attention component to include the text guidance $\tau$. In
contrast, AudioLDM [17] uses audio as the guidance during training. During
inference, they switch back to text guidance, as this is facilitated by pre-
trained joint text-audio embedding (CLAP). We did not find audio-guided
training and pre-training CLAP to be necessary, as argued in Section 2.1.

###  2.3 Augmentation

Many text-to-image [25] and text-to-audio [16] works have shown the efficacy
of training with fusion-based augmented samples to improve cross-modal
concept-composition abilities of the diffusion network. Therefore, we
synthesize additional text-audio pairs by superimposing existing audio pairs
on each other and concatenating their captions.

Unlike Liu et al. [17] and Kreuk et al. [16], to mix audio pairs, we do not
take a random combination of them. Following Tokozume et al. [35], we instead
consider the human auditory perception for fusion. Specifically, the audio
pressure level $G$ is taken into account to ensure that a sample with high
pressure level do not overwhelm the sample with low pressure level. The weight
of an audio sample ($x_{1}$) is calculated as a relative pressure level (see
Fig. 2 in the appendix for its distribution)

$\displaystyle p=(1+10^{\frac{G_{1}-G_{2}}{20}})^{-1},$ (9)

where $G_{1}$ and $G_{2}$ are pressure levels of two audio samples $x_{1}$ and
$x_{2}$, respectively. This ensures good representation of both audio samples,
post mixing.

Furthermore, as pointed out by Tokozume et al. [35], the energy of a sound
wave is proportional to the square of its amplitude. Thus, we mix $x_{1}$ and
$x_{2}$ as

$\displaystyle\text{mix}(x_{1},x_{2})=\frac{px_{1}+(1-p)x_{2}}{\sqrt{p^{2}+(1-p)^{2}}}.$
(10)

###  2.4 Classifier-Free Guidance

To guide the reverse diffusion process to reconstruct the audio prior $z_{0}$,
we employ a classifier-free guidance [6] of text input $\tau$. During
inference, a guidance scale $w$ controls the contribution of text guidance to
the noise estimation $\hat{\epsilon}_{\theta}$, with respect to unguided
estimation, where empty text is passed:

$\displaystyle\hat{\epsilon}_{\theta}^{(n)}(z_{n},\tau)=w\epsilon_{\theta}^{(n)}(z_{n},\tau)+(1-w)\epsilon_{\theta}^{(n)}(z_{n}).$
(11)

We also trained a model for which the text guidance was randomly dropped for
10% of the samples during training. We found this model to perform
equivalently to a model for which text guidance was always used for all
samples.

###  2.5 Audio VAE and Vocoder

Audio variational auto-encoder (VAE) [12] compresses the mel-spectogram of an
audio sample, $m\in\mathbb{R}^{T\times F}$, into an audio prior
$z_{0}\in\mathbb{R}^{C\times T/r\times F/r}$, where $C$, $T$, $F$, $r$ are the
number of channels, number of time-slots, number of frequency-slots, and
compression level, respectively. The LDM (see Section 2.2) reconstructs the
audio prior $\hat{z}_{0}$ using input-text guidance $\tau$. The encoder and
decoder are composed of ResUNet blocks [14] and are trained by maximizing
evidence lower-bound (ELBO) [12] and minimizing adversarial loss [8]. We adopt
the checkpoint of audio VAE provided by Liu et al. [17]. Thus, we use their
best reported setting, where $C$ and $r$ are set to $8$ and $4$, respectively.

As a vocoder to turn the audio-VAE decoder-generated mel-spectogram into an
audio, we also use HiFi-GAN [13] as Liu et al. [17].

##  3 Experiments

###  3.1 Datasets and Training

#### Text-to-Audio Generation.

We perform our main text-to-audio generation experiments on the AudioCaps
dataset [11]. The dataset contains 45,438 audio clips paired with human-
written captions for training. The validation set contains 2,240 instances.
The audio clips are ten seconds long and were collected from YouTube videos.
The clips were originally crowd-sourced as part of the significantly larger
AudioSet dataset [4] for the audio classification task.

We train our LDM using only the paired (text, audio) instances from the
AudioCaps dataset. We use the AudioCaps test set as our evaluation data. The
test set contains five human-written captions for each audio clip. We use one
caption for each clip chosen at random following Liu et al. [17] for
consistent evaluation with their work. The randomly chosen caption is used as
the text prompt, using which we generate the audio signal from our model.

#### Audio VAE and Vocoder.

We use the audio VAE model from Liu et al. [17]. This VAE network was trained
on the AudioSet, AudioCaps, Freesound222https://freesound.org/, and BBC Sound
Effect Library333https://sound-effects.bbcrewind.co.uk (SFX) datasets. Longer
audio clips in Freesound and BBC SFX were truncated to the first thirty
seconds and then segmented into three parts of ten seconds each. All audio
clips were resampled in 16KHz frequency for training the VAE network. We used
a compression level of 4 with 8 latent channels for the VAE network.

We also use the vocoder from Liu et al. [17] for audio waveform generation
from the mel spectrogram generated by the VAE decoder. The vocoder is a HiFi-
GAN [13] network trained on the AudioSet dataset. All audio clips were
resampled at 16KHz for training the vocoder network.

#### Model, Hyperparameters, and Training Details

We freeze the Flan-T5-Large text encoder in Tango and only train the
parameters of the latent diffusion model. The diffusion model is based on the
Stable Diffusion U-Net architecture [30, 31] and has a total of 866M
parameters. We use 8 channels and a cross-attention dimension of 1024 in the
U-Net model.

We use the AdamW optimizer [19] with a learning rate of 3e-5 and a linear
learning rate scheduler for training. We train the model for 40 epochs on the
AudioCaps dataset and report results for the checkpoint with the best
validation loss, which we obtained at epoch 39. We use four A6000 GPUs for
training Tango, where it takes a total of 52 hours to train 40 epochs, with
validation at the end of every epoch. We use a per GPU batch size of 3 (2
original + 1 augmented instance) with 4 gradient accumulation steps. The
effective batch size for training is $3$ (instance) $*4$ (accumulation) $*4$
(GPU) $=48$.

###  3.2 Baseline Models

In our study, we examine three existing models: DiffSound by Yang et al. [38],
AudioGen by Kreuk et al. [16], and AudioLDM by Liu et al. [17]. AudioGen and
DiffSound use text embeddings for conditional generative training, while
AudioLDM employs audio embeddings to avoid potential noise from weak textual
descriptions in the paired text-audio data. AudioLDM uses audio embeddings
from CLAP and asserts that they are effective in capturing cross-modal
information. The models were pre-trained on large datasets, including
AudioSet, and fine-tuned on the AudioCaps dataset, before evaluation, for
enhanced performance. Thus, comparing them to our model Tango would not be
entirely fair.

Despite being trained on a much smaller dataset, our model Tango outperformed
the baselines that were trained on significantly larger datasets. We may
largely attribute this to the use of LLM Flan-T5. Therefore, our model Tango
sets itself apart from the three existing models, making it an exciting
addition to the current research in this area.

It is important to note that the AudioLDM-L-Full-FT checkpoint from Liu et al.
[17] was not available for our study. Therefore, we used the AudioLDM-M-Full-
FT checkpoint, which was released by the authors and has $416$M parameters.
This checkpoint was fine-tuned on both the AudioCaps and MusicCaps datasets.
We performed a subjective evaluation using this checkpoint in our study. We
attempted to fine-tune the AudioLDM-L-Full checkpoint on the AudioCaps
dataset. However, we were unable to reproduce the results reported in Liu et
al. [17] due to a lack of information on the hyperparameters used.

Our model can be compared directly to AudioLDM-L since it has almost the same
number of parameters and was trained solely on the AudioCaps dataset. However,
it is worth noting that Liu et al. [17] did not release this checkpoint, which
made it impossible for us to conduct a subjective evaluation of its generated
samples.

###  3.3 Evaluation Metrics

#### Objective Evaluation.

In this work, we used two commonly used objective metrics: Frechet Audio
Distance (FAD) and KL divergence. FAD [10] is a perceptual metric that is
adapted from Fechet Inception Distance (FID) for the audio domain. Unlike
reference-based metrics, it measures the distance between the generated audio
distribution and the real audio distribution without using any reference audio
samples. On the other hand, KL divergence [38, 16] is a reference-dependent
metric that computes the divergence between the distributions of the original
and generated audio samples based on the labels generated by a pre-trained
classifier. While FAD is more related to human perception, KL divergence
captures the similarities between the original and generated audio signals
based on broad concepts present in them. In addition to FAD, we also used
Frechet Distance (FD) [17] as an objective metric. FD is similar to FAD, but
it replaces the VGGish classifier with PANN. The use of different classifiers
in FAD and FD allows us to evaluate the performance of the generated audio
using different feature representations.

#### Subjective Evaluation.

Following Liu et al. [17] and Kreuk et al. [16], we ask six human evaluators
to assess two aspects –– overall audio quality (OVL) and relevance to the
input text (REL) – of 30 randomly-selected baseline- and Tango-generated audio
samples on a scale from 1 to 100. The evaluators were proficient in the
English language and instructed well to make a fair assessment.

###  3.4 Results and Analysis

#### Main Results.

We report our main comparative study in Table 1. We comapre our proposed
method Tango with DiffSound [38], AudioGen [16] and various configurations of
AudioLDM [17]. AudioLDM obtained best results with 200 sampling steps from the
LDM during inference. For a fair comparison, we also use 200 inference steps
in Tango and in our additional AudioLDM experiments. We used a classifier-free
guidance scale of 3 for Tango. AudioLDM used a guidance scale among {2, 2.5,
3} in their various experiments.

Tango achieves new state-of-the-art results for objective metrics when trained
only on the AudioCaps dataset, with scores of 24.52 FD, 1.37 KL, and 1.59 FAD.
This is significantly better than the most direct baseline AudioLDM-L, which
also used only the AudioCaps dataset for LDM training. We attribute this to
the use of Flan-T5 as text encoder in Tango. We also note that Tango matches
or beats the performance of AudioLDM-*-FT models, which used significantly
($\sim$ 65 times) larger datasets for LDM training. The AudioLDM-*-FT models
used two phases of LDM training – first on the collection of the four
datasets, and then only on AudioCaps. Tango is thus far more sample efficient
as compared to the AudioLDM-*-FT model family.

Tango also shows very promising results for subjective evaluation, with an
overall audio quality score of 85.94 and a relevance score of 80.36,
indicating its significantly better audio generation ability compared to
AudioLDM and other baseline text-to-audio generation approaches.

Table 1: The comparison between Tango and baseline TTA models. _FT_ indicates
the model is fine-tuned on the Audiocaps (AC) dataset. The AS and AC stand for
AudioSet and AudiocCaps datasets respectively. We borrowed all the results
from [17] except for AudioLDM-L-Full which was evaluated using the model
released by the authors on Huggingface. Despite the LDM being trained on a
much smaller dataset, Tango outperforms AudioLDM and other baseline TTA models
as per both objective and subjective metrics. ‡ indicates the results are
obtained using the checkpoints released by Liu et al. [17].

Model | Datasets | Text | #Params | Objective Metrics | Subjective Metrics
---|---|---|---|---|---
FD $\downarrow$ |  KL $\downarrow$ |  FAD $\downarrow$ |  OVL $\uparrow$ |  REL $\uparrow$
Ground truth | $-$ | $-$ | $-$ | $-$ | $-$ | $-$ | $91.61$ | $86.78$
DiffSound [38] | AS+AC | ✓ |  $400$M | $47.68$ | $2.52$ | $7.75$ | $-$ | $-$
AudioGen [16] | AS+AC+8 others | ✓ |  $285$M | $-$ | $2.09$ | $3.13$ | $-$ | $-$
AudioLDM-S | AC | ✗ |  $181$M | $29.48$ | $1.97$ | $2.43$ | $-$ | $-$
AudioLDM-L | AC | ✗ |  $739$M | $27.12$ | $1.86$ | $2.08$ | $-$ | $-$
AudioLDM-M-Full-FT‡ | AS+AC+2 others | ✗ |  $416$M | ${26.12}$ | $\mathbf{1.26}$ | ${2.57}$ | ${79.85}$ | ${76.84}$
AudioLDM-L-Full‡ | AS+AC+2 others | ✗ |  $739$M | ${32.46}$ | ${1.76}$ | ${4.18}$ | ${78.63}$ | ${62.69}$
AudioLDM-L-Full-FT | AS+AC+2 others | ✗ |  $739$M | $\mathbf{23.31}$ | ${1.59}$ | ${1.96}$ | ${-}$ | ${-}$
Tango | AC | ✓ |  $866$M | $24.52$ | 1.37 | $\mathbf{1.59}$ | $\mathbf{85.94}$ | $\mathbf{80.36}$

#### Effect of Inference Steps and Classifier-Free Guidance.

The number of inference steps and the classifier-free guidance scale are of
crucial importance for sampling from latent diffusion models [34, 6]. We
report the effect of varying number of steps and varying guidance scale for
audio generation in AudioCaps in Table 2. We found that a guidance scale of 3
provides the best results for Tango. In the left part of Table 2, we fix the
guidance scale of 3 and vary the number of steps from 10 to 200. The generated
audio quality and resultant objective metrics consistently become better with
more steps. Liu et al. [17] reported that the performance for AudioLDM
plateaus at around 100 steps, with 200 steps providing only marginally better
performance. However, we notice a substantial improvement in performance when
going from 100 to 200 inference steps for Tango, suggesting that there could
be further gain in performance with more inference steps.

Table 2: Effect on the objective evaluation metrics with a varying number of
inference steps and classifier-free guidance.

Model | Varying Steps | Varying Guidance
---|---|---
Guidance | Steps | FD $\downarrow$ | KL $\downarrow$ | FAD $\downarrow$ | Steps | Guidance | FD $\downarrow$ | KL $\downarrow$ | FAD $\downarrow$
Tango | $3$ | $10$ | $45.12$ | $1.66$ | $11.38$ | $100$ | - | 35.76 | 2.02 | 6.22
$20$ | $31.38$ | $1.39$ | $4.52$ | $2.5$ | $26.32$ | $1.39$ | $1.97$
$50$ | $25.33$ | $\mathbf{1.27}$ | $2.13$ | 3 | $26.13$ | $1.37$ | $\mathbf{1.87}$
$100$ | $26.13$ | $1.37$ | $1.87$ | $5$ | $\mathbf{24.28}$ | $\mathbf{1.28}$ | $2.32$
$200$ | $\mathbf{24.52}$ | $1.37$ | $\mathbf{1.59}$ | $10$ | $26.10$ | $1.31$ | $3.30$

We report the effect of varying guidance scale with a fixed 100 steps in the
right half of Table 2. The first row uses a guidance scale of 1, thus
effectively not applying classifier-free guidance at all during inference. Not
surprisingly, the performance of this configuration is poor, lagging far
behind the classifier-free guided models across all the objective measures. We
obtain almost similar results with a guidance scale of 2.5 and better FD and
KL with a guidance scale of 5. We obtain the best FAD metric at a guidance
scale of 3 and the metric becomes poorer with larger guidance.

#### Temporal Sequence Modelling.

We analyze how Tango and AudioLDM models perform audio generation when the
text prompt contains multiple sequential events. Consider the following
examples: A toy train running as a young boy talks followed by plastic
clanking then a child laughing contains three separate sequential events,
whereas Rolling thunder with lightning strikes contains only one. We segregate
the AudioCaps test set using the presence of temporal identifiers – while,
before, after, then, followed – into two subsets, one with multiple events and
the other with single event. We show the objective evaluation results for
audio generation on these subsets in Table 3. Tango achieves the best FD and
FAD scores for both multiple events and single event instances. The best KL
divergence score is achieved by the AudioLDM-M-Full-FT model. We conjecture
that the larger corpus from the four training datasets in AudioLDM could be
more helpful in improving the reference-based KL metric, unlike the reference-
free FD and FAD metrics.

Table 3: Objective evaluation results for audio generation in the presence of
multiple events or a single event in the text prompt in the AudioCaps test
set. The multiple events and single event subsets collectively constitute the
entire AudioCaps test set. It should be noted that FD and FAD are corpus-level
non-linear metrics, and hence the FD and FAD scores reported in Table 1 are
not average of the subset scores reported in this table.

Model | Datasets | Multiple Events | Single Event
---|---|---|---
FD $\downarrow$ | KL $\downarrow$ | FAD $\downarrow$ | FD $\downarrow$ | KL $\downarrow$ | FAD $\downarrow$
AudioLDM-L-Full | AS+AC+2 others | $43.65$ | $1.90$ | $3.77$ | $35.39$ | $1.66$ | $5.24$
AudioLDM-M-Full-FT | $34.57$ | $\mathbf{1.32}$ | $2.45$ | $29.40$ | $\mathbf{1.21}$ | $3.27$
Tango | AC | $\mathbf{33.36}$ | $1.45$ | $\mathbf{1.75}$ | $\mathbf{28.59}$ | $1.30$ | $\mathbf{2.04}$

#### Performance against Number of Labels.

Recall that the AudioCaps dataset was curated from the annotations of the
audio classification task in the AudioSet dataset. The text prompts in
AudioCaps can thus be paired with the discrete class labels of AudioSet. The
AudioSet dataset contains a total of 632 audio event classes. For instance, A
woman and a baby are having a conversation and its corresponding audio clip
has the following three labels: Speech, Child speech kid speaking, Inside
small room. We group instances having one label, two labels, and multiple (two
or more) labels in AudioCaps and evaluate the generated audios across the
objective metrics. We report the result of the experiment in Table 4. Tango
outperforms AudioLDM models across all the objective metrics for audio
generation from texts with one label or two labels. For texts with multiple
labels, AudioLDM achieves a better KL divergence score and Tango achieves
better FD and FAD scores. Interestingly, all the models achieve consistently
better FD and KL scores with progressively more labels, suggesting that such
textual prompts are more effectively processed by the diffusion models.

Table 4: Performance of audio generation in AudioCaps for texts containing
one, two, or multiple (two or more) labels. Each text in AudioCaps has its
corresponding multi-category labels from AudioSet. We use these labels to
segregate the AudioCaps dataset into three subsets.

Model | Datasets | One Label | Two Labels | Multiple Labels
---|---|---|---|---
FD $\downarrow$ | KL $\downarrow$ | FAD $\downarrow$ | FD $\downarrow$ | KL $\downarrow$ | FAD $\downarrow$ | FD $\downarrow$ | KL $\downarrow$ | FAD $\downarrow$
AudioLDM-L-Full | AS+AC+2 others | $48.11$ | $2.07$ | $4.71$ | $44.93$ | $1.90$ | $4.09$ | 34.94 | 1.68 | 4.59
AudioLDM-M-Full-FT | 46.44 | $1.85$ | $3.77$ | $39.01$ | $1.29$ | $3.52$ | $26.74$ | $\mathbf{1.10}$ | $2.62$
Tango | AC | $\mathbf{40.81}$ | $\mathbf{1.84}$ | $\mathbf{1.79}$ | $\mathbf{35.09}$ | $\mathbf{1.56}$ | $\mathbf{2.53}$ | $\mathbf{26.05}$ | $1.24$ | $\mathbf{1.96}$

Table 5: Performance of AudioLDM-M-Full FT and Tango for the most frequently
occurring categories in AudioCaps dataset. CEB indicates the Channel,
environment, and background sounds category.

Human | Animal | Natural | Things | CEB | FD $\downarrow$ | KL $\downarrow$ | FAD $\downarrow$
---|---|---|---|---|---|---|---
AudioLDM | Tango | AudioLDM | Tango | AudioLDM | Tango
✓ | ✗ | ✗ | ✗ | ✗ | $38.15$ | $\mathbf{34.06}$ | $1.01$ | $\mathbf{0.99}$ | $2.81$ | $\mathbf{2.13}$
✗ | ✓ | ✗ | ✗ | ✗ | $78.62$ | $\mathbf{77.78}$ | $\mathbf{1.82}$ | $1.92$ | $\mathbf{4.28}$ | $4.62$
✓ | ✓ | ✗ | ✗ | ✗ | $\mathbf{61.91}$ | $70.32$ | $\mathbf{0.89}$ | $1.29$ | $6.32$ | $\mathbf{5.19}$
✗ | ✗ | ✓ | ✗ | ✗ | $\mathbf{51.61}$ | $57.75$ | $\mathbf{1.89}$ | $1.96$ | $6.75$ | $\mathbf{5.15}$
✗ | ✗ | ✗ | ✓ | ✗ | $35.60$ | $\mathbf{33.13}$ | $\mathbf{1.35}$ | $1.43$ | $5.42$ | $\mathbf{3.40}$
✗ | ✗ | ✓ | ✓ | ✗ | $55.06$ | $\mathbf{42.00}$ | $1.46$ | $\mathbf{1.12}$ | $6.57$ | $\mathbf{3.89}$
✓ | ✗ | ✗ | ✓ | ✗ | $\mathbf{37.57}$ | $39.22$ | $\mathbf{1.11}$ | $1.34$ | $3.26$ | $\mathbf{3.18}$
✗ | ✗ | ✗ | ✓ | ✓ | $54.25$ | $\mathbf{52.77}$ | $1.43$ | $\mathbf{1.33}$ | $11.49$ | $\mathbf{9.26}$

#### Categorical Modelling.

The class labels in AudioSet can be arranged hierarchically to obtain the
following top-level categories: i) Human sounds, ii) Animal sounds, iii)
Natural sounds, iv) Sounds of Things, v) Channel, environment, background
sounds, vi) Source-ambiguous sounds, and vii) Music. We map the class labels
in AudioCaps to the seven main categories listed above. The Music category is
very rare in AudioCaps and the rest either appear on their own or in various
combinations with others. We select the most frequently occurring category
combinations and analyze the performance of various models within the
constituting AudioCaps instances in Table 5. The performance of the two models
is pretty balanced across the FD and KL metrics, with Tango being better in
some, and AudioLDM in others. However, Tango achieves better FAD scores in all
but one group, with large improvements in (human, animal), (natural),
(things), and (natural, things) categories.

##  4 Related Works

#### Diffusion Models.

Recent years have seen a surge in diffusion models as a leading approach for
generating high-quality speech [1, 15, 24, 25, 9, 7]. These models utilize a
fixed number of Markov chain steps to transform white noise signals into
structured waveforms. Among them, FastDiff has achieved remarkable results in
high-quality speech synthesis [7]. By leveraging a stack of time-aware
diffusion processes, FastDiff can generate speech samples of exceptional
quality at an impressive speed, 58 times faster than real-time on a V100 GPU,
making it practical for speech synthesis deployment. It surpasses other
existing methods in end-to-end text-to-speech synthesis. Another noteworthy
probabilistic model for audio synthesis is DiffWave [15], which is non-
autoregressive and generates high-fidelity audio for various waveform
generation tasks, including neural vocoding conditioned on mel spectrogram,
class-conditional generation, and unconditional generation. DiffWave delivers
speech quality that is on par with the powerful WaveNet vocoder [23] while
synthesizing audio much faster. Diffusion models have emerged as a promising
approach for speech processing, particularly in speech enhancement [20, 33,
26, 21]. Recent advancements in diffusion probabilistic models have led to the
development of a new speech enhancement algorithm that incorporates the
characteristics of noisy speech signals into the forward and reverse diffusion
processes [22]. This new algorithm is a generalized form of the probabilistic
diffusion model, known as the conditional diffusion probabilistic model.
During its reverse process, it can adapt to non-Gaussian real noises in the
estimated speech signal, making it highly effective in improving speech
quality. In addition, Qiu et al. [26] propose SRTNet, a novel method for
speech enhancement that incorporates the diffusion model as a module for
stochastic refinement. The proposed method comprises a joint network of
deterministic and stochastic modules, forming the “enhance-and-refine”
paradigm. The paper also includes a theoretical demonstration of the proposed
method’s feasibility and presents experimental results to support its
effectiveness, highlighting its potential in improving speech quality.

#### Text-to-Audio Generation.

The field of text-to-audio generation has received limited attention until
recently [16, 38]. In Yang et al. [38], a text encoder is used to obtain text
features, which are then processed by a non-autoregressive decoder to generate
spectrogram tokens. These tokens are fed to a vector quantized VAE (VQ-VAE) to
generate mel spectrograms that are used by a vocoder to generate audio. The
non-autoregressive decoder is a probabilistic diffusion model. In addition,
Yang et al. [38] introduced a novel data augmentation technique called the
mask-based text generation strategy (MBTG), which masks out portions of input
text that do not represent any event, such as those indicating temporality.
The aim of MBTG is to learn augmented text descriptions from audio during
training. Although this approach seems promising, its fundamental limitation
is the lack of diversity in the generated data, as it fails to mix different
audio samples. Later, Kreuk et al. [16] proposed a correction to this method,
mixing audio signals according to random signal-to-noise ratios and
concatenating the corresponding textual descriptions. This approach allows for
the generation of new (text, audio) pairs and mitigates the limitations of
Yang et al. [38]. Unlike Yang et al. [38], the architecture proposed in Kreuk
et al. [16] uses a transformer encoder and decoder network to autoregressively
generate audio tokens from text input.

Recently, Liu et al. [17] proposed AudioLDM, which translates the Latent
Diffusion Model of text-to-visual to text-to-audio generation. They pre-
trained VAE-based encoder-decoder networks to learn a compressed latent
representation of audio, which was then used to guide a diffusion model to
generate audio tokens from text input. They found that using audio embeddings
instead of text embeddings during the backward diffusion process improved
conditional audio generation. During inference time, they used text embeddings
for text-to-audio generation. Audio and text embeddings were obtained using
pre-trained CLAP, which is the audio counterpart of CLIP embeddings used in
the original LDM model.

##  5 Limitations

Tango is not always able to finely control its generations over textual
control prompts as it is trained only on the small AudioCaps dataset. For
example, the generations from Tango for prompts Chopping tomatoes on a wooden
table and Chopping potatoes on a metal table are very similar. Chopping
vegetables on a table also produces similar audio samples. Training text-to-
audio generation models on larger datasets is thus required for the model to
learn the composition of textual concepts and varied text-audio mappings. In
the future, we plan to improve Tango by training it on larger datasets and
enhancing its compositional and controllable generation ability.

##  6 Conclusion

In this work, we investigate the effectiveness of the instruction-tuned model,
Flan-T5, for text-to-audio generation. Specifically, we use the textual
embeddings produced by Flan-T5 in the latent diffusion model to generate mel-
spectrogram tokens. These tokens are then fed to a pre-trained variational
auto-encoder (VAE) to generate mel-spectrograms, which are later used by a
pre-trained vocoder to generate audio. Our model achieved superior performance
under both objective and subjective evaluations compared to the state-of-the-
art text-to-audio model, AudioLDM, despite using only 63 times less training
data. We primarily attribute this performance improvement to the
representational power of Flan-T5, which is due to its instruction-based
tuning in the pre-training stage. In the future, we plan to investigate the
effectiveness of Flan-T5 in other audio tasks, such as, audio super-resolution
and inpainting.

## References

  * Chen et al. [2020] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan.  Wavegrad: Estimating gradients for waveform generation.  _arXiv preprint arXiv:2009.00713_ , 2020.
  * Chung et al. [2022] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei.  Scaling instruction-finetuned language models, 2022.  URL https://arxiv.org/abs/2210.11416.
  * Dai et al. [2022] Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei.  Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers.  _ArXiv_ , abs/2212.10559, 2022.
  * Gemmeke et al. [2017] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter.  Audio set: An ontology and human-labeled dataset for audio events.  In _2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 776–780. IEEE, 2017.
  * Hang et al. [2023] Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo.  Efficient diffusion training via min-snr weighting strategy, 2023.
  * Ho and Salimans [2021] Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications_ , 2021.
  * Huang et al. [2022] Rongjie Huang, Max WY Lam, Jun Wang, Dan Su, Dong Yu, Yi Ren, and Zhou Zhao.  Fastdiff: A fast conditional diffusion model for high-quality speech synthesis.  _arXiv preprint arXiv:2204.09934_ , 2022.
  * Isola et al. [2016] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros.  Image-to-image translation with conditional adversarial networks.  _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5967–5976, 2016.
  * Jeong et al. [2021] Myeonghun Jeong, Hyeongju Kim, Sung Jun Cheon, Byoung Jin Choi, and Nam Soo Kim.  Diff-tts: A denoising diffusion model for text-to-speech.  _arXiv preprint arXiv:2104.01409_ , 2021.
  * Kilgour et al. [2019] Kevin Kilgour, Mauricio Zuluaga, Dominik Roblek, and Matthew Sharifi.  Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms.  In _INTERSPEECH_ , pages 2350–2354, 2019.
  * Kim et al. [2019] Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim.  Audiocaps: Generating captions for audios in the wild.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 119–132, 2019.
  * Kingma and Welling [2013] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  _CoRR_ , abs/1312.6114, 2013.
  * Kong et al. [2020a] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae.  Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.  _Advances in Neural Information Processing Systems_ , 33:17022–17033, 2020a.
  * Kong et al. [2021] Qiuqiang Kong, Yin Cao, Haohe Liu, Keunwoo Choi, and Yuxuan Wang.  Decoupling magnitude and phase estimation with deep resunet for music source separation.  In _International Society for Music Information Retrieval Conference_ , 2021.
  * Kong et al. [2020b] Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro.  Diffwave: A versatile diffusion model for audio synthesis.  _arXiv preprint arXiv:2009.09761_ , 2020b.
  * Kreuk et al. [2022] Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre D’efossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi.  Audiogen: Textually guided audio generation.  _ArXiv_ , abs/2209.15352, 2022.
  * Liu et al. [2023] Haohe Liu, Zehua Chen, Yiitan Yuan, Xinhao Mei, Xubo Liu, Danilo P. Mandic, Wenwu Wang, and MarkD . Plumbley.  Audioldm: Text-to-audio generation with latent diffusion models.  _ArXiv_ , abs/2301.12503, 2023.
  * Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692, 2019.
  * Loshchilov and Hutter [2017] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Lu et al. [2021] Yen-Ju Lu, Yu Tsao, and Shinji Watanabe.  A study on speech enhancement based on diffusion probabilistic model.  In _2021 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_ , pages 659–666, 2021.
  * Lu et al. [2022a] Yen-Ju Lu, Zhong-Qiu Wang, Shinji Watanabe, Alexander Richard, Cheng Yu, and Yu Tsao.  Conditional diffusion probabilistic model for speech enhancement.  In _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7402–7406, 2022a.  doi: 10.1109/ICASSP43922.2022.9746901.
  * Lu et al. [2022b] Yen-Ju Lu, Zhong-Qiu Wang, Shinji Watanabe, Alexander Richard, Cheng Yu, and Yu Tsao.  Conditional diffusion probabilistic model for speech enhancement.  In _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pages 7402–7406. IEEE, 2022b.
  * Oord et al. [2016] Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu.  Wavenet: A generative model for raw audio.  _arXiv preprint arXiv:1609.03499_ , 2016.
  * Popov et al. [2021a] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov.  Grad-tts: A diffusion probabilistic model for text-to-speech.  In _International Conference on Machine Learning_ , pages 8599–8608. PMLR, 2021a.
  * Popov et al. [2021b] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, Mikhail Kudinov, and Jiansheng Wei.  Diffusion-based voice conversion with fast maximum likelihood sampling scheme.  _arXiv preprint arXiv:2109.13821_ , 2021b.
  * Qiu et al. [2022] Zhibin Qiu, Mengfan Fu, Yinfeng Yu, LiLi Yin, Fuchun Sun, and Hao Huang.  Srtnet: Time domain speech enhancement via stochastic refinement.  _arXiv preprint arXiv:2210.16805_ , 2022.
  * Raffel et al. [2020] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67, 2020.  URL http://jmlr.org/papers/v21/20-074.html.
  * Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever.  Zero-shot text-to-image generation.  _ArXiv_ , abs/2102.12092, 2021.
  * Ramesh et al. [2022] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _ArXiv_ , abs/2204.06125, 2022.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10684–10695, 2022.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, _Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015_ , pages 234–241, Cham, 2015. Springer International Publishing.  ISBN 978-3-319-24574-4.
  * Saharia et al. [2022] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Serrà et al. [2022] Joan Serrà, Santiago Pascual, Jordi Pons, R Oguz Araz, and Davide Scaini.  Universal speech enhancement with score-based diffusion.  _arXiv preprint arXiv:2206.03065_ , 2022.
  * Song et al. [2020] Jiaming Song, Chenlin Meng, and Stefano Ermon.  Denoising diffusion implicit models.  _ArXiv_ , abs/2010.02502, 2020.
  * Tokozume et al. [2017] Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada.  Learning from between-class examples for deep sound recognition.  _CoRR_ , abs/1711.10282, 2017.  URL http://arxiv.org/abs/1711.10282.
  * Wikipedia [2021a] Wikipedia.  Tango.  https://en.wikipedia.org/wiki/Tango, 2021a.  [Online; accessed 21-April-2023].
  * Wikipedia [2021b] Wikipedia.  Tango music.  https://en.wikipedia.org/wiki/Tango_music, 2021b.  [Online; accessed 21-April-2023].
  * Yang et al. [2022] Dongchao Yang, Jianwei Yu, Helin Wang, Wen Wang, Chao Weng, Yuexian Zou, and Dong Yu.  Diffsound: Discrete diffusion model for text-to-sound generation.  _arXiv preprint arXiv:2207.09983_ , 2022.

## Appendix

Figure 2: Distribution of relative pressure level (see Eq. 9) across the
augmented samples.


# What GPT Knows About Who is Who

Xiaohan Yang, Eduardo Peynetti, Vasco Meerman  Chris Tanner
Institute for Applied Computational Science
Harvard University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Coreference resolution – which is a crucial task for understanding discourse
and language at large – has yet to witness widespread benefits from large
language models (LLMs). Moreover, coreference resolution systems largely rely
on supervised labels, which are highly expensive and difficult to annotate,
thus making it ripe for prompt engineering. In this paper, we introduce a QA-
based prompt-engineering method and discern generative, pre-trained LLMs’
abilities and limitations toward the task of coreference resolution. Our
experiments show that GPT-2 and GPT-Neo can return valid answers, but that
their capabilities to identify coreferent mentions are limited and prompt-
sensitive, leading to inconsistent results.

##  1 Introduction

Coreference resolution (CR) aims to identify and cluster all words (i.e.,
mentions) that refer to the same entity or event. Solving this task is
essential for natural language understanding, as mismatched references will
lead to bias. Recent improvements in CR have been incremental (Lee et al.,
2017; Joshi et al., 2020; Cattan et al., 2020), compared to other NLP tasks
that have demonstrated more real-world impact. One reason is the limited
training corpora. For example, one of the primary datasets, ECB+ (Cybulska and
Vossen, 2014), contains only 984 documents, including 6,833 mentions and 2,741
clusters. Moreover, this dataset was built around 43 news topics ten years
ago, potentially leading to generalization problems for the state-of-the-art
(SOTA) models.

When dealing with low-resource tasks, there is an emerging trend to perform
prompt engineering with pre-trained LMs. Unlike fine-tuning (Brown et al.,
2020; Wei et al., 2022), prompt engineering does not update the pre-trained
model’s weights when completing the downstream task. Instead, one transforms
the downstream task to match the original task of the pre-trained model (Liu
et al., 2021). For example, for machine translation, one can create prompts
such as “English: I love bread. French:” and input them to a generative LM
(e.g., GPT-2). If the pre-trained model encountered similar patterns during
training, it should be able to generate the translated French sentence.
Nevertheless, to the best of our knowledge, there is scarce research on
applying this approach to coreference resolution (Sanh et al., 2021).

To better understand if pre-trained LMs can help resolve coreferences, we
construct a QA-based prompting method and experiment with both GPT-2 (Radford
et al., 2019) and GPT-Neo (Gao et al., 2020). By using this prompting
methodology, we measure if these models can predict whether two mentions are
coreferent. For evaluation, we use the ECB+ dataset, which provides gold
mentions and clustering labels. We compare the results with unsupervised and
supervised coreference resolution models, including a classic rule-based
system (Lee et al., 2011), the seminal end-to-end neural model (Lee et al.,
2017), and a recent SOTA model (Cattan et al., 2020).

Figure 1: An example of prompt-based learning for CR. The green block
represents the prefix, which serves as the description of the CR task and
remains unchanged throughout an experiment for all inputs $x$. The purple
block is the unfilled prompt, which changes for each input $x$ and serves as
the prediction. Moreover, in each block, the yellow part is the prompting
function while the blue and red parts are the original data $x$ and $y$,
respectively.

##  2 Related Work

### Prompt-based learning

Prompt-based learning is a fast-growing area in NLP, as it can reduce the need
to fine-tune models and rely on supervised labels. According to the survey by
Liu et al., over 120 papers have been published since 2019, which collectively
demonstrates effectiveness toward many different tasks: text classification
(Tam et al., 2021; Holtzman et al., 2021), factual probing (Perez et al.,
2021), question-answering (Tsimpoukelli et al., 2021), and more. Nevertheless,
to the best of our knowledge, only one prompt-based learning paper concerned
CR. Specifically, Sanh et al. introduces T0, a zero-shot generalization of T5
(Raffel et al., 2020). The authors convert various supervised datasets into
task-specific prompts, including CR. Using the WSC dataset (Levesque et al.,
2012), they achieve over 60% accuracy. Although this result is not comparable
with supervised state-of-the-art (SOTA) models, it still offers compelling
results and suggests the model might contain CR knowledge without requiring
supervised training on the task. However, since the WSC dataset only focuses
on highly ambiguous pronouns, it is not as complete as the standard CR task
that involves named and nominal mentions.

### Traditional CR Models

Similar to other NLP tasks, most CR models can be categorized as being either
unsupervised or supervised. A commonly used unsupervised model is the Multi-
Pass Sieve model (Lee et al., 2011). This rule-based system extracts entity
mentions and clusters them by applying 13 “filters” in successive manner.
Amongst supervised models, e2e-coref (Lee et al., 2017) is the seminal end-to-
end neural model. This model performs within-document CR and was trained on
the OntoNotes (CoNLL-2012) dataset. Building on this architecture, Cattan et
al. (2020) performs cross-document CR for entities and events by training on
the ECB+ dataset and using RoBERTa (Liu et al., 2019) as an encoder. Although
supervised models offer significant improvements over unsupervised models,
they are expensive to train; most SOTA models have $O(n^{4})$ complexity,
where $n$ is the length of each document.

##  3 Methodology

This section introduces our prompt-based learning method for CR. Typically, CR
models can be broken down into three sub-tasks: (1) detecting mentions; (2)
predicting whether two given mentions are coreferent or not; (3) and
clustering mentions accordingly. The crux of CR research centers around the
second part, which is also our focus.

Building on the approach introduced by Sanh et al. (2021), we define our input
$x$ as $[text,m_{1},m_{2}]$ and output $y$ as a binary label. Specifically,
$m_{1}$ and $m_{2}$ are a pair of gold mentions in a document, and the $text$
are the sentences containing those mentions. For example, in Figure 1, within
each green box, the successive blue parts are $text$, $m_{1}$, $m_{2}$,
respectively. We define a prompting function $f$, which takes $x$ as input and
produces a question prompt $q_{x}$ (Equation 1). Further details about $f$ are
in Appendix A.

$q_{x}=f(x)$ (1)

Moreover, to allow the model to understand the task, we use few-shot learning
(Triantafillou et al., 2017) by constructing a filled prefix. In particular,
we select $k$ examples, $A$, from the training dataset and feed these examples
into the same prompting function $f$. Then, we append the true label (‘Yes’ or
‘No’) to the outputs, yielding the filled prefix $q_{A}$ (Equation 2). To be
clear, each individual prefix $q_{i\in k}$ constitutes a single green box in
Figure 1.

$q_{A}=f(A)$ (2)

Last, adding the unfilled prompt $q_{x}$ to the filled prefix $q_{A}$ will
give us the full prompt for data point $x$. This allows us to get a prediction
$z$ without updating any parameters $\theta$ in the pre-trained LM.

$z=P(q_{A}+q_{x};\theta)\\\ $ (3)

Since we use pre-trained LMs directly, without fine-tuning, we do not have
control over its output; the model can generate invalid answers beyond our
desired outputs, ‘Yes’ or ‘No’. Therefore, we repeat the process $m$ times to
get a more robust prediction $\bar{z}$. To mitigate the bias of one specific
$f$, we average the output of $n$ different prompt formulas to get the final
prediction (Equation 4).

$y=\frac{\sum_{i=1}^{n}\bar{z}_{i}}{n}$ (4)

##  4 Experimental Setup

### Datasets

We use the ECB+ dataset (Cybulska and Vossen, 2014) as our input source, which
contains both within- and cross-document coreference information for both
event and entity mentions. This dataset consists of 984 documents around 43
news topics, among which 196 documents are in the development set. After
preprocessing the data, as described in Appendix B, our development set
consists of 172 documents.

To generate a prefix $x_{0}$, we experiment with three data sources: the
training sets of WSC (Levesque et al., 2012) and ECB+ (Cybulska and Vossen,
2014), and a simple dataset that we manually generated. The WSC dataset was
used in the research most similar to ours, T0 (Sanh et al., 2021), which we
compare against while using much smaller pretrained LMs (i.e., GPT-2 and GPT-
Neo). As mentioned, ECB+ provides more natural and comprehensive references
than WSC. Our manually generated dataset uses 10 very simple examples –
allowing one to discern the impact on performance.

When using the ECB+ dataset, we only considered pairs of mentions that are
within the same or successive sentences. When evaluating our model, we
considered all mention-pair combinations, $[m_{1},m_{2}]$, within said
sentences. Relying on the gold mentions, we obtain a dataset with $17832$
candidate mention pairs, among which $7.86\%$ are positive samples. Finally,
we apply 5 prompt functions from Sanh et al. to generate the full prompts.

### Models

We used three traditional CR models as baselines: Multi-Pass Sieve (Lee et
al., 2011), the seminal end-to-end neural model (e2e-coref) (Lee et al.,
2017), and a SOTA extension (the Streamlining model) (Cattan et al., 2020).
Respectively, these models represent three categories: a rules-based model, a
supervised model trained on a different dataset, and a supervised model
trained on the same dataset. In terms of implementations, we use the CoreNLP
toolkit for the Multi-Pass Sieve model (Manning et al., 2014) and AllenNLP
(Gardner et al., 2018) for e2e-coref. Since there is no publicly available
pre-trained Streamlining model (Cattan et al., 2020), we fully train the model
from scratch using a V100 GPU on Google Colab. To fairly compare with other
models, we set a $0.5$ threshold for the pairwise scorer in the Streamlining
model. We evaluate all models by their mention pairwise scorers, not their
clustering decisions.

Limited by our computational resources, we choose GPT-2 and GPT-Neo-125M as
our pre-trained LMs 111Our code can be found at
https://github.com/AwesomeCoref/prompt-coref. During inference, the output
token length is set to 1, since our expected output is one word (i.e., ‘Yes’
or ‘No’). To generate more robust results, the repetition parameter $m$ is set
to $5$. We ran our text generative models with multiple temperature settings
ranging from 0 to 1, none of which produced significant changes. We settled on
using a value of $0.7$, to limit the greediness of the generated responses. In
terms of few-shot learning, we experimented with $k\in\\{0,2,4,10\\}$ and
display the results from the 4-shot setting since it produces the best
accuracy. To reduce bias introduced by prefixes, we ensure each prefix has
equally-balanced samples. For example, for the 4-shot setting, the filled
prefix will have 2 positive examples and 2 negative examples.

##  5 Results and Analysis

| Yes/No Predictions
---|---
0-shot | 5%
2-shot | 93.7%
4-shot | 96.2%
10-shot | 98%
Table 1:  Percentage of Yes/No predictions by GPT-2

We first question if GPT-based models can produce valid answers. In Figure 1,
we observe that GPT-2 predicts ‘Yes’ or ‘No’ for over $93.7\%$ samples when at
least 2 filled prefixes are provided.

Figure 2: Distribution of predicted values

However, although the answers are valid, they are inaccurate. In Figure 2, we
plot the distribution of predicted labels for each model, where the red bars
denote the distribution of positive examples (ground truth), and the blue bars
denote negative ones (ground truth). Traditional CR models generally predict
low values for negative examples, indicated by blue bars being concentrated at
$0$. As for positive examples, e2e-coref shows better precision since more
positive examples are classified correctly at $1$. Yet, GPT-2 seems to be both
sensitive to prompts and unstable over the repetitions of each prompt.
Furthermore, GPT-Neo’s predictions are inaccurate and no better than random,
even though it predicts consistent results for multiple runs with the same
prompt.

Similar conclusions can be drawn from Table 2, where GPT-based models have the
lowest AUC and F1 scores. Specifically, the extremely low precision causes the
bad results. Since the ECB+ dataset is highly imbalanced, random predictions
from GPT-based models will lead to a low precision, reflecting the proportion
of positive samples. For completeness, we also perform an experiment on the
WSC dataset (see GPT-2wsc), which is a test dataset used by Sanh et al.
(2021). GPT-2 also fails on this task, as its mean prediction averaged across
different prompts is always “Yes” .

| Acc | Prec | Rec | F1 | AUC
---|---|---|---|---|---
Multi Sieve | 0.93 | 0.39 | 0.20 | 0.27 | 0.59
e2e-coref | 0.95 | 0.62 | 0.46 | 0.53 | 0.72
Streamline | 0.93 | 0.87 | 0.19 | 0.31 | 0.59
GPT-2 | 0.50 | 0.08 | 0.53 | 0.14 | 0.51
GPT-NEO | 0.38 | 0.08 | 0.68 | 0.15 | 0.52
GPT-2wsc | 0.37 | 0.37 | 1.00 | 0.54 | 0.50
Table 2:  Performance of different models.

### POS and Entity Types

While the overall performance indicates that GPT models are comparable to a
random model, we hypothesize that for some subset of mention pairs, GPT might
perform better. To investigate, we conducted experiments based on part-of-
speech (POS) tags and named-entity types. Figure 3 shows that both GPT-2 and
GPT-Neo can capture coreferent relationships relatively better when the second
mention is a pronoun. Moreover, this trend is stronger when the first mention
is a pronoun or a proper noun. Nonetheless, e2e-coref performs better than
both GPT models across all POS tags, and the gap is widest when the second
mention is a nominal noun phrase.

Figure 3: Model’s precision over various types of noun phrases, including
pronouns, proper nouns and nominal nouns. Each bar’s hue intensity denotes the
data density.

As for named entities, Figure 4 shows that both GPT-2 and GPT-Neo perform
better in precision when one mention is of type PERSON. Moreover, GPT-Neo can
identify coreferent relationships more precisely if the second mention is Non-
GPE locations (i.e., LOC). However, their precision scores are far lower than
the scores from classical CR models. In particular, both the multi-pass sieve
model and e2e-coref model reach the highest precision when a mention is a
PRODUCT object (e.g., vehicle, food) or a NORP object (e.g., nationality,
religious or political group).

(a) GPT-2

(b) GPT-Neo

(c) Multi-pass Sieve

(d) e2e-coref

Figure 4: GPT-2’s performance on different named-entity types. We use colors
to denote performance and the text to show data density in each category.
Figure 5: Different models’ F1 score over various level of mention
similarities based on BERT embedding.

### Mention Similarity

In addition to inspecting how performance varies with mention types, we also
considered how performance is affected by mentions’ similarity. Using pre-
trained BERT (Devlin et al., 2019), we encode each mention into span
representations by averaging its tokens’ last hidden states. Then, we measure
cosine similarity between mention pairs.

Figure 5 shows that F1 scores generally improve as the semantic similarity
increases. Although, the multi-pass sieve model maintains a low F1 because it
is a rule-based model that tends to predict False for most samples – which
yields a high accuracy for unbalanced datasets. The e2e-coref model performs
well on mentions that are not so similar, while the performance of
Streamlining model improves drastically as similarity is greater than 50%.
However, both GPT-2 and GPT-NEO yield low F1 (approximately 0.2) for mention
pairs with less than 70% similarity. When considering mentions of higher
similarity, GPT-based models can achieve over 0.4 F1 score.

##  6 Conclusion

In this paper, we rely on prompt-based learning to analyze how much GPT-like
models know about coreference resolution. Despite the popularity of prompting
in recent NLP research, we find that LLMs perform poorly on this task without
fine-tuning. Nonetheless, these models achieve relatively better performance
for specific types of mentions, including pronouns and person objects, and
mention pairs with high similarity.

## References

  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Cattan et al. (2020) Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, and Ido Dagan. 2020.  Streamlining cross-document coreference resolution: Evaluation and modeling.  _arXiv preprint arXiv:2009.11032_.
  * Cybulska and Vossen (2014) Agata Cybulska and Piek Vossen. 2014.  Using a sledgehammer to crack a nut? lexical diversity and event coreference resolution.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 4545–4552, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gao et al. (2020) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020.  The pile: An 800gb dataset of diverse text for language modeling.  _arXiv preprint arXiv:2101.00027_.
  * Gardner et al. (2018) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew E. Peters, Michael Schmitz, and Luke Zettlemoyer. 2018\.  Allennlp: A deep semantic natural language processing platform.  _CoRR_ , abs/1803.07640.
  * Holtzman et al. (2021) Ari Holtzman, Peter West, Vered Schwartz, Yejin Choi, and Luke Zettlemoyer. 2021\.  Surface form competition: Why the highest probability answer isn’t always right.  In _EMNLP_.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020.  SpanBERT: Improving Pre-training by Representing and Predicting Spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77.
  * Lee et al. (2011) Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011.  Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task.  In _Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task_ , pages 28–34, Portland, Oregon, USA. Association for Computational Linguistics.
  * Lee et al. (2017) Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017.  End-to-end neural coreference resolution.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics.
  * Levesque et al. (2012) Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012.  The Winograd Schema Challenge.  In _Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning_ , KR’12, page 552–561. AAAI Press.
  * Liu et al. (2021) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ArXiv_ , abs/2107.13586.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A robustly optimized BERT pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Manning et al. (2014) Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014.  The Stanford CoreNLP natural language processing toolkit.  In _Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations_ , pages 55–60.
  * Perez et al. (2021) Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021.  True few-shot learning with language models.  In _Advances in Neural Information Processing Systems_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021.  Multitask prompted training enables zero-shot task generalization.
  * Tam et al. (2021) Derek Tam, Rakesh R. Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. 2021.  Improving and simplifying pattern exploiting training.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4980–4991, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Triantafillou et al. (2017) Eleni Triantafillou, Richard Zemel, and Raquel Urtasun. 2017.  Few-shot learning through an information retrieval lens.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 2252–2262, Red Hook, NY, USA. Curran Associates Inc.
  * Tsimpoukelli et al. (2021) Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Ali Eslami, Oriol Vinyals, and Felix Hill. 2021.  Multimodal few-shot learning with frozen language models.  In _Thirty-Fifth Conference on Neural Information Processing Systems_.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.

##  Appendix A Prompt formulas

Figure 6: Prompt Formulas. We experiment with these 5 prompt formulas
mentioned in Sanh et al. (2021). Here, each block is one formula and the parts
highlighted in blue are $[text,m_{1},m_{2}]$ respectively.

##  Appendix B Data Preprocessing

The original ECB+ dataset is in XML format, where everything is tokenized.
Moreover, the information about gold mentions and gold clusters is related to
token ids. However, we cannot easily get the plain text by joining tokens with
a space character. If we do so, we will get strange looking text as shown
below.

http : / / www . accesshollywood . com / lindsay - lohan - leaves - betty -
ford - checks - into - malibu - rehab _ article _ 80744 Lindsay Lohan Leaves
Betty Ford , Checks Into Malibu Rehab First Published : June 13 , 2013 4 : 59
PM EDT Lindsay Lohan has left the Betty Ford Center and is moving to a rehab
facility in Malibu , Calif . , Access Hollywood has confirmed .

In this example, we can see objects like urls, datetime and punctuation are
not in the right format. Since we are using the text as an input to the prompt
function, we need to properly format them to align with normal text that GPTs
are trained on. Moreover, as gold mention and gold clusters are based on
original token ids in ECB+, when we parsed and re-formatted the data, we could
match these ids again. Continuing with the previous example, our parsing
algorithm cleans up the previous text to be something as follows.

http://www.accesshollywood.com/lindsay-lohan-leaves-betty-ford-checks-into-
malibu-rehab_article_80744 [EOS] Lindsay Lohan Leaves Betty Ford, Checks Into
Malibu Rehab First Published: June 13, 2013 4: 59 PM EDT [EOS] Lindsay Lohan
has left the Betty Ford Center and is moving to a rehab facility in Malibu,
Calif., Access Hollywood has confirmed. [EOS]

##  Appendix C Additional Results

Here are additional results for our experiments.

### Experiments on Prefix

The aggregate results from few shot learning are displayed in Table 3. Our
results show that 4-shots learning performs the best for both GPT-2 and GPT-
NEO in terms of accuracy. Unexpectedly, as we increase the size of examples,
the result does not improve accordingly. Given 10 examples in prefix, the
model tend to predict “yes” more easily. One possible explanation might be
that we have balanced examples in prefix while the actual querying data only
have around 8% positive samples.

| Acc | Prec | Recall | F1 | AUC
---|---|---|---|---|---
2-shot | 0.39 | 0.08 | 0.64 | 0.14 | 0.50
4-shot | 0.51 | 0.08 | 0.51 | 0.14 | 0.51
10-shot | 0.19 | 0.08 | 0.90 | 0.15 | 0.51
Table 3:  n-shot performance from the text generative models | Acc | Prec | Recall | F1 | AUC
---|---|---|---|---|---
simple | 0.61 | 0.08 | 0.36 | 0.13 | 0.50
WSC | 0.08 | 0.08 | 1.00 | 0.15 | 0.50
ECB+ | 0.54 | 0.08 | 0.48 | 0.14 | 0.51
Table 4:  Average results from each dataset that is used for the experiments

Moreover, we experiment with various datasets for prefix as discussed in
section 4. The results in Table 4 shows that prefix does have an impact on the
results. The prefix generated from ECB+ dataset performs slightly better than
others regarding to AUC. This is understandable because we evaluate on the
ECB+ development set. Beyond our expectation, WSC-prefix result in a perfect
recall and a super bad accuracy, which means that this prefix lead models to
generate “yes” regardless of the context. This result further proves that
GPT-2 is very sensitive to prompts.