# You Can Generate It Again: Data-to-text Generation with Verification and
Correction Prompting

Xuan Ren
<EMAIL_ADDRESS>
&Lingqiao Liu
<EMAIL_ADDRESS>

###### Abstract

Despite significant advancements in existing models, generating text
descriptions from structured data input, known as data-to-text generation,
remains a challenging task. In this paper, we propose a novel approach that
goes beyond traditional one-shot generation methods by introducing a multi-
step process consisting of generation, verification, and correction stages.
Our approach, VCP(Verification and Correction Prompting), begins with the
model generating an initial output. We then proceed to verify the correctness
of different aspects of the generated text. The observations from the
verification step are converted into a specialized error-indication prompt,
which instructs the model to regenerate the output while considering the
identified errors. To enhance the model’s correction ability, we have
developed a carefully designed training procedure. This procedure enables the
model to incorporate feedback from the error-indication prompt, resulting in
improved output generation. Through experimental results, we demonstrate that
our approach effectively reduces slot error rates while maintaining the
overall quality of the generated text.

##  1 Introduction

Data-to-text generation aims to converting structured data into coherent,
human-readable text. As shown in Figure 1, the input of data-to-text
generation contains information in a well-defined format but missing necessary
linguistic structure.

Figure 1: Data-to-text examples from the ViGGO dataset Juraska et al. (2019).

Data-to-text generation has a broad range of practical applications, including
report generation, automated journalism, data visualization, and dialogue
systems. In these applications, the input can consist of various types of
data, such as tables, graphs, or raw data. It is worth noting that data-to-
text generation is a controlled form of text generation, where the output must
be coherent with the input and maintain semantic accuracy as a crucial aspect.

One popular approach for data-to-text generation involves fine-tuning pre-
trained models, such as T5 Raffel et al. (2020), on specific training data
designed for data-to-text generation Kale and Rastogi (2020b); Yin and Wan
(2022). This strategy has shown promising performance and can even surpass the
capabilities of general-purpose large language models like ChatGPT or GPT-4
(for more details, refer to Comparing to LLMs). Although current methods
generally produce correct and fluent natural language outputs, they may
occasionally suffer from semantic errors, particularly omissions that result
in the absence of crucial keywords Yin and Wan (2022). In this paper, we
introduce the ’slot error rate (SER),’ which quantifies the rate of missing
keywords.

Several research works aim to reduce SER, such as copy mechanism Rebuffel et
al. (2019); Puduppully et al. (2019), template-based generation Kale and
Rastogi (2020a); Mehta et al. (2022), planning then generate Xu et al. (2021);
Su et al. (2021); Kasner and Dusek (2022). These methods are effective in
terms of reducing the SER since they have strict generation process that force
the model to include every key words. A more recent work SEA-GUIDE Juraska and
Walker (2021) identifies attention patterns related to slot errors, then
automatically adjusts the beam search score according to the attention
patterns in the decoding process. While this method maintains the quality of
text generation and partially reduces SER, the attention patterns need to be
designed through a non-trivial manual design. More details are in the Related
Works section (see Section Related Works).

Step1: Initial Prediction
---
Input |  recommand(name(Tom Clancy], release_year[1999], has_linux_release[yes])
T5 predictions |  Since you’re into Linux games, you heard of Tom Clancy?
Step2: Verification
Find slot errors |  1999 $\times$
Tom Clancy ✓
Linux ✓
Label missing slots |  recommand(release_year[<a1><token2><token3>1999], name(Tom Clancy], has_linux_release[no])
Step3: Regeneration
Prompted Input |  recommand(release_year[<token1><token2><token3>1999], name(Tom Clancy], has_linux_release[no])
Send to T5 to generate prediction |  Since you’re into Linux games, have you heard of Tom Clancy which is released in 1999?
Table 1: The inference process comprises three steps: 1. We utilize the fine-
tuned T5 model to generate an initial prediction. 2. The slot error checker is
then deployed to ascertain the presence of any slot errors. If such errors are
detected, we label the error-correcting prompts to highlight the location of
potential slot errors. 3. Lastly, we reintroduce the prompted input to the
fine-tuned T5 model for regeneration. The tokens(error-correcting prompts)
have the capacity to alter the regenerated outputs, ensuring the inclusion of
previously missed slots.

In order to generate texts with low SER while maintaining text quality, we
propose a prompt-based test time correction pipeline that encourages the model
to include missed slots identified by a slot error checker. The overall
inference process is illustrated in Figure 2 (see the inference section).
Initially, the fine-tuned T5 model receives the input slots and generates the
initial predictions. The slot error checker then verifies if any slot is
missing from the output. If a slot is found to be missing, we label the
corresponding error-correcting prompts along with the missed input value.
During the regeneration process, these error-correcting prompts prompt the
fine-tuned T5 model to include the omitted slot value in its subsequent
prediction. An example is provided in Table 1, where the text mentions the
name ’Tom Clancy’ and the presence of a ’Linux’ game, but omits the release
year ’1999’. To indicate potential slot omissions to the T5 model, we label
the input slot value ’1999’ using three error-correcting prompts: <token1>,
<token2>, and <token3>. Subsequently, we feed the prompted input to the T5
model for re-prediction, which has a higher probability of incorporating the
previously missed slot.

Figure 2: The workflow is comprised of train section and inference section.
When training, we first fine tune a T5 model, then use data generator to
generate prompt training dataset. Lastly use prompt tuning to teach error-
correcting prompts how to improve the semantic coverage in T5’s prediction.
The inference section illustrates the overview of the initial prediction,
verification, and regeneration process. Please refer to Table 1 for more
detailed insights.

To enable the aforementioned error-correcting re-generation process, it is
necessary to train error-correcting prompts that encourage the model to
include slots it had previously missed. The training process for prompts is
outlined in the training section of Figure 2. Specifically, in the Data
Generation process (see Data Generation), a data generator is utilized to
create unseen prompted inputs along with their corresponding ground truth,
which is used to construct the prompt initialization training dataset and the
prompt training dataset. During training, we first train the prompt
initialization and then fine-tune the prompt embedding based on the
initialized prompt. These error-correcting prompts are designed to instruct
the fine-tuned T5 model to include the labeled slot values it had previously
missed in its predictions.

##  2 Related works

In data-to-text generation task, the slot error problem has been a persistent
challenge. Several methods have been developed to address this issue, each
with its unique advantages and limitations. The related work on this subject
can be expanded and elaborated as follows.

Copy Mechanism: Researchers have proposed the use of a copy mechanism to
mitigate slot errors Rebuffel et al. (2019); Puduppully et al. (2019). This
mechanism enables the model to directly copy words from the input when
generating the output. Although it effectively reduces slot errors, it may
inadvertently introduce addition errors Yin and Wan (2022).

Template-based Methods: (Kale and Rastogi, 2020a) and (Mehta et al., 2022)
suggest using templates for data generation to address the slot error problem.
By leveraging predefined sentence structures, these methods exhibit reduced
slot errors and enhanced zero-shot and few-shot generalization capabilities.
However, their reliance on fixed templates limits their flexibility, leading
to lower performance in fully trained in-domain settings.

Plan-based Generation: Another approach to reducing slot errors involves
planning the output structure before generating sentences Xu et al. (2021); Su
et al. (2021); Kasner and Dusek (2022). While this method effectively
minimizes slot errors, it suffers from reduced in text fluency due to its
inherent inflexibility. Its primary contribution lies in improving
generalization in unseen domains.

Semantically Attention-Guided Decoding Juraska and Walker (2021) manually
identifies 3 attention patterns related to slot errors. During test time,
these patterns are automatically detected and used to adjust the beam search
results by multiplying the beam search probability with a hyperparameter
associated with each pattern. This approach partially reduce SER, according to
section, because it has trouble reducing certain types of slot errors, such as
boolean ones. This limitation arises from the challenges humans encounter when
trying to discern the attention patterns associated with these specific
errors. Furthermore, the attention pattern detection process is time-
consuming, thereby reducing the overall efficiency of the system.

By adding dynamic memory module to the attention-based network, DM-NLG
(Seifossadat and Sameti, 2023) can store the previous generated words thus
better guide the generation to include key information. They significantly
reduce SER while further improve the text generation quality using
postprocessing network.

##  3 Methodology

Figure 2 illustrates our method, including both the inference and training
stages. During the inference stage, our method follows the generation, error
verification, and regeneration procedures to achieve data-to-text generation.
The regeneration capability is trained by adjusting the error correction
special prompts using a self-generated dataset. This training process occurs
after the normal fine-tuning of the network on a standard data-to-text
training set. Further details are provided as follows.

###  3.1 Inference

The detailed inference procedure is presented in Table 1, encompassing three
steps: initial generation, verification, and regeneration. In the initial
generation step, we input the testing samples into the fine-tuned T5 model,
which was trained in the earlier stage. This model provides us with the
initial predictions. The verification step utilizes the Slot Error Checker
(see Section 3.1), which employs simple rules to examine whether any slots
from the input are missing in the output, thereby identifying slot errors. If
any potential errors are detected, we introduce error-correcting prompts111For
more details about error-correcting prompts, please refer to Section 3.2.
adjacent to the positions of the unmentioned slots in the input. Lastly, in
the regeneration step, the prompted inputs are fed to the fine-tuned T5 model.
These error-correcting prompts guide the T5 model to incorporate the
previously omitted slots during the regeneration process.

Slot Error Checking is a process that verify if there are slot errors. For
non-boolean slot value pairs, the check is straightforward - we simply verify
that all slot values are included in the prediction. However, for boolean slot
value pairs, the process is more complex. We do not examine whether the actual
slot value, that is, yes or no, is present in the prediction. Instead, we
focus on whether the noun part(found by POS tag) of the slot names, such as
linux from has_linux_released, mac from has_mac_released, or steam from
available_on_steam, is mentioned in the predictions. Refer Slot Error Checking
Examples for more details.

###  3.2 Training

Figure 3: The workflow for generating datasets for training error-correcting
prompts

The training procedure, illustrated in Figure 2, begins by fine-tuning a T5
model using the training dataset. Subsequently, a prompt training dataset is
created for learning error-correcting prompts. Finally, the error-correcting
prompts are trained on the prompt training dataset to enhance their ability to
guide the language model in integrating slot values that were previously
missed.

Fine-tune T5: We simply fine-tune the T5 model on the data-to-text generation
training set. This step provides a good initial model.

Data Generation: As the training data has already been exposed to the T5 model
in the previous step for fine-tuning the initial T5 model, we cannot use the
training dataset to learn the error-correction special prompt. Therefore, we
will generate a new training set for prompt learning. Figure 3 illustrates
such a process. Specifically, the data generation process (data generator)
creates input-output pairs. The candidate input is generated by replacing the
slot values of an input in the original training set with other values
randomly sampled from the possible values for the slot. For example,
"recommend(name[Tom Clancy], release_year[1999], has_linux_release[yes])"
comprises the intention recommend, slot names release_year, name, and
has_linux_release, and slot values 1999, Tom Clancy, and yes. After
replacement, we may obtain recommend(name[Tom Clancy], release_year[1999],
has_linux_release[yes]), upon undergoing slot value replacement, yields an
unseen input such as recommend(name[RollerCoaster Tycoon], release_year[2001],
has_linux_release[no]). The candidate inputs are then fed into the fine-tuned
T5 model to generate initial predictions. Following the application of the
Slot Error Checker, those inputs containing slot errors are marked with error-
correcting prompt. For example, in the case shown in Figure 3, we append
’<token1> <token2> <token3>’ next to ’2001’, as it is omitted from the initial
prediction. The prompted inputs will be the final inputs.

Then we need to generate ground-truth output for the newly generated inputs.
To do so, We pass the collected prompted unseen input into the fine-tuned T5
model to produce 10 predictions using beam search methods. These predictions
are then subjected to Slot error checking. The prediction that is free of slot
errors is selected as the ground truth. In scenarios where beam search is
employed and yields more than one output without slot errors, the output with
the highest posterior probability according to the beam search is chosen.

Prompt Tuning. Once sufficient input-output pairs are generated, we then fine-
tuning the error-correcting prompts while keeping T5 fixed. The training
process will learn error correcting prompts that can steer the T5 to produce
correct output.

In our design, we choose deep prompting tuning strategy. Specifically, we use
3 error-correcting prompts for T5-base and 6 error-correcting prompts for
T5-small and add the prompt to each block in T5, and both the word embeddings
of the error-correcting prompts and the key-value embeddings in each layer are
trainable.

Prompt Initialization In addition to the workflow shown in Figure 2, we
empirically found that it is better to introduce an initial training stage for
the error-correcting prompt. The training objective is to ensure that adding
the prompt will not change the output of the fine-tuned T5 model. This can be
implemented by first following the input generation step as in data generation
but choose the output as the current prediction obtained from the fine-tuned
T5. Performing prompt tuning on such a initialized prompt instead of the
random initialized prompt is demonstrated to have better performance as shown
in prompt initialization ablation study.

###  3.3 Dataset and Evaluation Metrics

This study utilizes two datasets for training and evaluating: the E2E and the
Viggo datasets.

The E2E dataset Novikova et al. (2017), specifically designed for the
restaurant domain, offers a data-driven approach for end-to-end natural
language generation system training. The Viggo dataset Juraska et al. (2019)
targets open-domain dialogue systems in video game topics. The utterances
cover 9 generalizable and conversational dialogue act types.

Our system’s performance is assessed using a comprehensive set of metrics:
BLEU Papineni et al. (2002), METEOR Lavie and Agarwal (2007), ROUGE Lin
(2004), CIDEr Vedantam et al. (2015), and SER. SER (Slot Error Rate) measures
the error rate of slot values in the generated text, which is especially
important in task-oriented dialogue systems.

Model | BLEU | MET. | ROUGE | CIDEr | SER ↓
---|---|---|---|---|---
T5-small${}_{\text{beam search}}$ baseline | $53.2\pm 0.54$ | 0.392 | 0.637 | 2.652 | $0.89\pm 0.096\%$
T5-base${}_{\text{beam search}}$ baseline | $53.1\pm 0.28$ | 0.393 | 0.635 | 2.655 | $0.60\pm 0.13\%$
S2SJuraska et al. (2019) | 51.9 | 0.388 | 0.631 | 2.531 | 2.55%
DTHarkous et al. (2020) | 53.6 | 0.394 | 0.640 | 2.700 | 1.68%
K&MKedzie and McKeown (2020) | 48.5 | 0.380 | 0.592 | 2.454 | 0.46%
SEA-GUIDET5-smallJuraska and Walker (2021) |  $53.2\pm$ 0.53 | 0.392 | 0.637 | 2.693 | $0.7\pm 0.097\%$
SEA-GUIDET5-baseJuraska and Walker (2021) | $53.2\pm 0.30$ | 0.393 | 0.635 | 2.658 | $0.51\pm 0.10\%$
VCP${}_{\text{T5 small}}$ | $52.6\pm 0.51$ | 0.392 | 0.632 | 2.628 | $\textbf{0.41}\pm 0.085\%$
VCP${}_{\text{T5 base}}$ | $52.4\pm 0.19$ | 0.391 | 0.627 | 2.620 |  $\textbf{0.33}\pm 0.19$%

Table 2: Comparing of our approach, VCP, to other methods and T5 baseline on
ViGGO dataset

##  4 Experiment

We compare our VCP method to a variable of other methods on the ViGGO and E2E
datasets. In particular, for non-SER evaluation, we use BLEU, METEOR, ROUGE-L,
and CIDEr, which are assessed using the E2E evaluation
script222https://github.com/tuetschek/e2e-metrics. For SER evaluation, we
apply the same auto slot evaluation script in the SEA-GUIDE
peoject333https://github.com/jjuraska/data2text-nlg, which encompasses
hundreds of evaluation rules. The auto evaluation script exhibits 94%
agreement aligning with human judgment.

###  4.1 Setup

We have observed a substantial variance in the SER of T5 baseline, SEA-GUIDE,
and our method when applied to the ViGGO dataset. To ensure a fair comparison,
we train 5 instances each of the T5-small and T5-base models, each instance
for 20 epochs. We then select the optimized model based on validation loss,
resulting in 5 baseline models each for T5-small and T5-base. We then fine-
tune these same T5 baseline models using the SEA-GUIDE code available on
GitHub, as well as our own method. Our testing conditions utilized a batch
size of 10 and a maximum sentence length of 128 tokens. All tests were
conducted on a single RTX 3090 GPU, with the learning rate decreasing linearly
over the course of the training epochs. More comprehensive information
regarding the prompt training hyperparameters is provided in Appendix A.4. For
the E2E datasets, we retrained 5 T5-small and T5-base baseline models using
the same pipeline as with the ViGGO dataset. The experimental results for
other methods, for which standard deviations are not reported, are as
presented in Table 3, and have been taken directly from the original paper.

Our method involves the use of training datasets constructed via a data
generator, and the prompt training examples are produced through a beam search
with a beam size of 10, as shown in Figure 3. The experiment is designed to
compare the effectiveness of our method, VCP, to other methods. We implemented
T5-small and T5-base models, where we inserted 6 and 3 deep error-correcting
prompts for each slot error, respectively. The word and key-value embeddings
of these error-correcting prompts are trainable across all layers, while the
fine-tuned T5 models are kept frozen.

###  4.2 Performance Comparison

Model | BLEU | MET. | ROUGE. | SER
---|---|---|---|---
T5-small${}_{\text{greedy search}}$ baseline | 67.0 | 0.454 | 0.692 | 1.60%
T5-small${}_{\text{beam search}}$ baseline | 66.7 | 0.453 | 0.694 | 2.85%
T5-base${}_{\text{greedy search}}$ baseline | 66.8 | 0.459 | 2.282 | 1.85%
T5-base${}_{\text{beam search}}$ baseline | 66.7 | 0.453 | 0.697 | 3.94%
S2SJuraska et al. (2019) | 66.2 | 0.445 | 0.677 | 0.91%
K&MKedzie and McKeown (2020) | 66.3 | 0.453 | 0.693 | 0
SEA-GUIDE${}_{\text{T5-small}}$ Juraska and Walker (2021) | 67.5 | 0.453 | 0.690 | 0.04%
SEA-GUIDE${}_{\text{T5-base}}$Juraska and Walker (2021) | 68.2 | 0.454 | 0.691 | 0.05%
DM-NLG no postprocessSeifossadat and Sameti (2023) | 66.7 | 0.456 | 0.691 | 0.03%
DM-NLG postprocess: GPT-2Seifossadat and Sameti (2023) | 68.6 | 0.482 | 0.713 | 0.03%
VCP${}_{\text{T5small}}$ | $67.0\pm 0.18$ | 0.451 | 0.690 | 0.015%

Table 3: Comparison of our method VCP to T5 baseline and other methods on E2E
dataset

As demonstrated in Table 2, we contrast our methodology with the T5 baseline
and other methods using the ViGGO dataset. Our method exhibits a notable
advantage in terms of reducing SER while maintaining comparable non-SER scores
to the T5 baseline. The SER result reported by K&M cannot be directly compared
with our methods, as we utilize different SER evaluation techniques. Our
method, VCP, reduces SER from 0.89% to 0.41% on T5-small and from 0.60% to
0.33% on T5-base.

As shown in Table 3, for the E2E dataset, our VCP method retains the text
generation quality on non-SER evaluation matrices, while reducing SER from
over 2.5% for the T5 beam search baseline to almost 0. This is lower than
other methods, except for K&M. K&M performs well on the E2E dataset but
struggles to maintain text generation quality, achieving a 48.5 BLEU score on
the ViGGO dataset. DM-NLG Seifossadat and Sameti (2023), with post-processing
using GPT-2, not only reduces SER to 0.03%, but also improves text fluency on
non-SER evaluation matrices. However, their method incorporates a post-
processing stage that employs a significantly larger language model, GPT-2.
Our method achieves a higher BLEU score compared to DM-NLG without post-
processing, while making half as many slot errors, reaching a SER of 0.015%.

###  4.3 Comparing to LLMs

| BLEU | SER
---|---|---
T5-small${}_{\text{beam search}}$ baseline | 53.2 | 0.89%
VCP${}_{\text{T5small}}$ | 52.6 | 0.41%
ChatGPT examples | 25.1 | 7.3%
ChatGPT selected examples | 22.3 | 6.72%
GPT4 examples | 30.7 | 0.90%
GPT4 selected examples | 27.4 | 0
Table 4: Performance comparison between large language models, T5 baseline and
our method

The findings are summarized in Table 4. It is critical to note that the
performance of ChatGPT, given five in-context examples of the input intent
type (as shown in Appendix A.2), significantly underperforms when compared to
the T5 baseline in terms of the BLEU score. This also holds true when ChatGPT
is given one in-context example for every intent from the ViGGO dataset
(further details are illustrated in Appendix A.1). From the experimental
results, we observe that ChatGPT displays a significant number of slot errors,
with an SER exceeding 6%. The same prompting strategy was also applied to
GPT4. Although GPT4 does not achieve a high BLEU score, it does attain
considerably lower SER scores. Remarkably, GPT4 achieved a 0% SER with
selected examples, showcasing its exceptional capability to accurately follow
instructions.

We believe the low predictions quality generated by ChatGPT and GPT4 primarily
due to the complexities involved in expressing the relationship between the
input and the output through in-context examples and prompts. For example,
despite supplying five distinct request attribute examples, complemented by
clear prompt explanations as detailed in Appendix A, ChatGPT falls short in
accurately replicating the desired tone. Furthermore, it misconstrues the
intended meaning of the request attribute intent in the input data. For
example, in the specific dadata shown in Appendix A, the intent of the request
attribute suggests that the user is seeking to ascertain whether their
feelings are average. Regrettably, ChatGPT misinterprets this, inferring that
the input data is attempting to verify all the available information. There
are myriad ways to interpret how an AI model should convert input data into
text, yet the true relationship can be more effectively understood through
training a language model on thousands of training examples, rather than
presenting it with a limited number of in-context examples accompanied by
descriptions. Consequently, supervised training continues to play an essential
role in data-to-text generation models.

##  5 Ablation Study

| BLEU | SER
---|---|---
T5-small${}_{\text{beam search}}$ baseline | 53.2 | 0.89%
VCP${}_{\text{T5small}}$ | 52.6 | 0.41%
Remove position information | 52.4 | 0.72%
Fine-tune T5(not use prompt) | 51.4 | 0.70%
No prompt initilization | 52.6 | 0.65%
Directly sampling the output | 50.6 | 0.62%
Table 5: Ablation study.

We conduct a comprehensive ablation study, summarized in Table 5. In the first
experiment, we underscore the significance of using error-correcting prompts
to label positions information of the potential slot errors(placing prompt
besides the slot errors). In the second experiment, we establish the advantage
of utilizing the error-correcting prompts over fine-tuning the entire T5 model
with prompt training datasets created in the Data Generation section. The
third experiment omits the prompt initialization step, offering an evaluation
of their necessity for text generation quality. In the final experiment, we
sample 10 outputs directly, selecting the most suitable ones based on the SER
calculated by our slot error checker. This experiment evaluates the advantage
of using error-correcting prompts over directly sampled outputs. All
experiments employ the T5-small model and are evaluated using the ViGGO
dataset, with the BLEU and SER metrics as evaluation standards.

###  5.1 Removing Position Information

In this experiment, we evaluate the importance of placing the error-correcting
prompts besides the slots errors. To this end, We remove the slot error
position information by always position the error-correcting prompts in the
front of the inputs. Table 5 illustrates that slot error rates rise when we
remove the information regarding error locations, which underscores the
advantage of highlight the probable sites of errors.

###  5.2 Fine-tuning the entire models(T5 no prompt)

In this experiment, we evaluate if using error-correcting prompts results in
better performance than finetuning the entire model on the generated training
data. After train a T5 model on the training dataset, we further fine-tune it
with learning rate of 5ê-3 for 10 epochs using the prompt training
dataset(with error-correcting prompts removed) created in Data Generation. We
compare its performance to VCP which only train error-correcting prompts. As
shown in Table 5, the BLEU score and the SER generated by the ’Fine-tune T5’
are noticeably lower than those achieved by prompt tuning methods, which
demonstrates the importance of using prompts. The reason lies in the fact that
prompts not only label the error locations, but also allow the original model
to remain frozen. The original model, trained on ground-truth data labeled by
humans(unlike ground-truth in the prompt training datasets created by T5
models which may contains errors), excels at producing high-quality texts.
Utilizing prompts enables the affect the well-trained T5 model as less as
possible, thereby yielding better-quality text outputs(high BLEU score) and
learning more generalizable ability guiding language models for reducing slot
errors.

###  5.3 Remove the prompt initialization training process

During the training process, we first train prompt initialization, then fine-
tune the initialization embedding instead of directly fine-tune the random
initialized embedding. In this experiment, we aim to evaluate the significance
of prompt initialization by comparing the performance of VCP before and after
using the prompt initialization(Not initial.) step. As shown in Table 5, there
is a decline in the BLEU score and a rise in SER score after the removal of
prompt initialization, thereby emphasizing its vital role in maintaining the
quality of text generation.

###  5.4 Sampling the best prediction directly

Our project uses error-correcting prompts to guide fine-tuned T5 models in
correcting their slot errors. We compare our method to directly sample 10
outputs and select the best prediction using a SER score. The results
demonstrated that while direct sampling method reduced the SER, the quality of
generated texts diminished compared to the prompt-based method.

We propose 2 reasons for this. Firstly, the slot error checker not always
accurately recognize when generated texts use different words to mention slot
information. This can result in the original prediction being incorrectly
identified as having errors, and consequently selecting the alternative
predictions which may have lower text fluency. Secondly, the error-correcting
prompts learn generalizable knowledge, allowing them to guide the T5 model
beyond the sampling search range.

##  6 Discussion and Conclusion

Our method achieves the lowest SER compared to other methods, while preserving
a comparable level of text generation quality. The primary reason for this is
our use of deep learning to learn generalizable error-correcting prompt
representations, as opposed to relying on strict rules that may only be
effective in certain scenarios.

The latest feedback mechanisms primarily depend on the zero-shot text
comprehension capabilities of Large Language Models (LLMs), which inevitably
restricts the selection of suitable models. Additionally, these feedback
networks usually employ an LLM as well. Obtaining precise responses from these
LLMs often necessitates a lengthy prompt and the use of more costly techniques
such as majority voting. These characteristics render the current feedback
mechanisms relatively expensive to utilize.

Our proposed method, on the other hand, can be applied to smaller models and
the prompted feedback are effective even if the rules of placing error-
correcting prompts are inaccurate or naive. This unique feature makes our
approach more cost-effective. Merging our pipeline with the capabilities of
LLMs presents the potential to develop superior solutions. We believe our VCP
technique can be applied to other problems where posterior checking is
valuable, such as image-to-text generation, text style transfer, embodied AI
manipulation or automated code generation.

##  7 Limitations

Our method effectively reduces slot errors; however, it also slightly
decreases text fluency. This happens because we train the prompt tokens using
the ground truth generated by the fine-tuned language model itself, which can
sometimes be inaccurate or sound unnatural. The introduction of a filter to
eliminate low-quality text, or the use of post-processing tools to enhance the
quality of the generated text, may improve text fluency.

Another limitation is that the knowledge imparted by the prompt training
datasets is limited to the capabilities of the posterior checker (such as our
slot error checker). If we cannot check for certain types of errors, then we
are unable to construct the prompt training datasets that impart that specific
kind of knowledge. This issue could potentially be resolved by training
specific error-detection models or using more powerful language models that
can generally check most error types. In this way, we would not need to rely
on hard-to-design rules when the knowledge we are trying to impart is complex.

## References

  * Harkous et al. (2020) Hamza Harkous, Isabel Groves, and Amir Saffari. 2020.  Have your text and use it too! end-to-end neural data-to-text generation with semantic fidelity.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Juraska et al. (2019) Juraj Juraska, Kevin Bowden, and Marilyn Walker. 2019.  ViGGO: A video game corpus for data-to-text generation in open-domain conversation.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , Tokyo, Japan. Association for Computational Linguistics.
  * Juraska and Walker (2021) Juraj Juraska and Marilyn Walker. 2021.  Attention is indeed all you need: Semantically attention-guided decoding for data-to-text NLG.  In _Proceedings of the 14th International Conference on Natural Language Generation_ , pages 416–431, Aberdeen, Scotland, UK. Association for Computational Linguistics.
  * Kale and Rastogi (2020a) Mihir Kale and Abhinav Rastogi. 2020a.  Template guided text generation for task-oriented dialogue.  _arXiv preprint arXiv:2004.15006_.
  * Kale and Rastogi (2020b) Mihir Kale and Abhinav Rastogi. 2020b.  Text-to-text pre-training for data-to-text tasks.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , Dublin, Ireland. Association for Computational Linguistics.
  * Kasner and Dusek (2022) Zdeněk Kasner and Ondrej Dusek. 2022.  Neural pipeline for zero-shot data-to-text generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Dublin, Ireland. Association for Computational Linguistics.
  * Kedzie and McKeown (2020) Chris Kedzie and Kathleen McKeown. 2020.  Controllable meaning representation to text generation: Linearization and data augmentation strategies.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5160–5185, Online. Association for Computational Linguistics.
  * Lavie and Agarwal (2007) Alon Lavie and Abhaya Agarwal. 2007.  METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments.  In _Proceedings of the Second Workshop on Statistical Machine Translation_ , pages 228–231, Prague, Czech Republic. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Mehta et al. (2022) Sanket Vaibhav Mehta, Jinfeng Rao, Yi Tay, Mihir Kale, Ankur Parikh, and Emma Strubell. 2022.  Improving compositional generalization with self-training for data-to-text generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Dublin, Ireland. Association for Computational Linguistics.
  * Novikova et al. (2017) Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017.  The E2E dataset: New challenges for end-to-end generation.  In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 201–206, Saarbrücken, Germany. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Puduppully et al. (2019) Ratish Puduppully, Li Dong, and Mirella Lapata. 2019.  Data-to-text generation with content selection and planning.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 6908–6915.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  21(1).
  * Rebuffel et al. (2019) Clément Rebuffel, Laure Soulier, Geoffrey Scoutheeten, and Patrick Gallinari. 2019\.  A hierarchical model for data-to-text generation.
  * Seifossadat and Sameti (2023) Elham Seifossadat and Hossein Sameti. 2023.  Improving semantic coverage of data-to-text generation model using dynamic memory networks.  _Natural Language Engineering_ , pages 1–26.
  * Su et al. (2021) Yixuan Su, David Vandyke, Sihui Wang, Yimai Fang, and Nigel Collier. 2021.  Plan-then-generate: Controlled data-to-text generation via planning.  _arXiv preprint arXiv:2108.13740_.
  * Vedantam et al. (2015) Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation.
  * Xu et al. (2021) Xinnuo Xu, Ondřej Dušek, Verena Rieser, and Ioannis Konstas. 2021.  AggGen: Ordering and aggregating while generating.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , Online. Association for Computational Linguistics.
  * Yin and Wan (2022) Xunjian Yin and Xiaojun Wan. 2022.  How do Seq2Seq models perform on end-to-end data-to-text generation? In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , Dublin, Ireland. Association for Computational Linguistics.

##  Appendix A Appendix

We demonstrate two example of the prompt we use for ChatGPT. In the first
example appendix A, we demonstrate the prompt for random select the example
from training dataset with the intent that is the same as the intent in the
test example (request attribute). In the second example appendix B, we
demonstrate the prompt we use for all of the test cases. The prompt include
one example for each intent.

###  A.1 Prompt with Selected Examples

please perform data-to-text generation for me. Domain is video game. The words
before the bracket are intentions.

For example, when the intention is give opinion, then the output should be a
sentence that asks for opinion.

when the intention is verify attribute, then the output should be a sentence
that try to verify the attribute.

Example: Input: request attribute(esrb[]) Output: Are there any ESRB content
ratings which you give preference to when picking a game to play?

Input: request attribute(release year[]) Output: Can you think of a year, in
which video games were particularly good?

Input: request attribute(esrb[]) Output: Are there any ESRB content ratings
which you give preference to when picking a game to play?

Input: request attribute(esrb[]) Output: Are there any ESRB content ratings
which you give preference to when picking a game to play?

Input: request attribute(developer[]) Output: Which game developer do you
think is the best?

Question: Input: verify attribute(name[little big adventure], rating[average],
has multiplayer[no], platforms[playstation])

Output:

Answer by ChatGPT: Can you confirm that Little Big Adventure has an average
rating and does not have multiplayer? Also, is it available on PlayStation?

GroundTruth: [’I remember you saying you found Little Big Adventure to be
average. Are you not usually that into single-player games on PlayStation?’,
"Earlier, you stated that you didn’t have strong feelings about PlayStation’s
Little Big Adventure. Is your opinion true for all games which don’t have
multiplayer?", ’I recall that you were not that fond of Little Big Adventure.
Does single-player gaming on the PlayStation quickly get boring for you?’]

###  A.2 Prompt with One Example for Each Intent

please perform data-to-text generation for me. Domain is video game. the words
before the bracket are intentions. For example, when the intention is give
opinion, then the output should be a sentence that asks for opinion. when the
intention is verify attribute, then the output should be a sentence that try
to verify the attribute.

Example for each intention: Input: give opinion(NAME [SpellForce 3], RATING
[poor], GENRES [real-time strategy, role-playing], PLAYER PERSPECTIVE [bird
view]) Output: I think that SpellForce 3 is one of the worst games I’ve ever
played. Trying to combine the real-time strategy and role-playing genres just
doesn’t work, and the bird’s eye view makes it near impossible to play.

Input: verify attribute(NAME [Little Big Adventure], RATING [average], HAS
MULTIPLAYER [no], PLATFORMS [PlayStation]) Output: I recall that you were not
that fond of Little Big Adventure. Does single-player gaming on the
PlayStation quickly get boring for you?

Input: confirm(NAME [Hellblade: Senua’s Sacrifice], RELEASE YEAR [2017],
DEVELOPER [Ninja Theory]) Output: Oh, do you mean the 2017 game from Ninja
Theory, Hellblade: Senua’s Sacrifice?

Input: request(SPECIFIER [interesting]) Output: Have you played any
interesting games lately?

Input: suggest(NAME [Half-Life 2], GENRES [shooter], PLAYER PERSPECTIVE [first
person]) Output: Do you also enjoy playing first-person shooters, such as
Half-Life 2?

Input: request explanation(RATING [poor], HAS MAC RELEASE [yes]) Output: What
is it about Mac games that you find so disappointing?

Input: inform(NAME [Max Payne 3], RELEASE YEAR [2012], GENRES [action-
adventure; shooter], MULTIPLAYER [yes]) Output: Max Payne 3 is a multiplayer
action-adventure shooter from 2012.

Input: request attribute(AVAILABLE ON STEAM []) Output: Do you prefer playing
games that you can get on Steam?

Question: Input: YOUR INPUT QUESTION Output:

###  A.3 Slot Error Checking Examples

For instance, has_linux_released[yes], the first step is to see if linux
appears in the prediction. If it does not, error-correcting prompts are
positioned beside the slot value. If it does appear, we employ simple
dependency parsing rules and POS tags to ascertain if any negation words are
linked to linux. If negation words are found, the slot value is marked with a
error-correcting prompts. If no negation words are present, we infer that
there are no slot errors concerning has_linux_released[yes]. Conversely, if
the slot value is no, such as in has_linux_released[no], the initial step is
to check for the mention of linux in the prediction. If linux is not
mentioned, it is assumed that no slot errors exist. However, if linux is
mentioned, we look for any associated negation words. If none are found,
error-correcting prompts are placed beside the no slot value, indicating a
potential slot error. If negation words are present, we presume the absence of
slot errors.

###  A.4 Training parameters

We use the learning rate begins at 0.01 and reduces gradually over 20 epochs
for prompt embedding initialization. We use 3 error-correcting prompts for
T5-base and 6 error-correcting prompts for T5-small. More details can be seen
in Table 6 and 7.

Model | initial. lr | train. lr | epochs | prompt token num
---|---|---|---|---
VCP${}_{\text{T5-small}}$ | 0.01 | 0.005 | 5 | 6
VCP${}_{\text{T5-base}}$ | 0.01 | 0.01 | 2 | 3

Table 6: The following details pertain to the training process of our VCP
method for experiments on the ViGGO datasets. ’Initial. lr’ stands for the
initial learning rate used for prompt initialization, while ’Train. lr’
represents the learning rate used for prompt tuning. ’Epochs’ refers to the
number of epochs for prompt tuning.

Model | initial. lr | train. lr | epochs | prompt token num
---|---|---|---|---
VCP${}_{\text{T5-small}}$ | 0.01 | 0.01 | 10 | 6

Table 7: The following details pertain to the training process of our VCP
method for experiments on the E2E datasets. ’Initial. lr’ stands for the
initial learning rate used for prompt initialization, while ’Train. lr’
represents the learning rate used for prompt tuning. ’Epochs’ refers to the
number of epochs for prompt tuning.

###  A.5 Experiment details

On the ViGGO dataset, we run the T5 baseline, SEA-GUIDE project, and our VCP
for 5 times. We report the mean and variance in Table 2 and reproduce the
experiment results of S2S, DT, and K&M from the SEA-GUIDE paper. We also run
the ablation study once and report the results in Tables 4 and 5.

For the E2E dataset, we run our VCP and reported the mean and variance. We
report the experiment results from the DM-NLG paper and the SEA-GUIDE paper in
Table 3.

We use the SER auto-evaluation script from the SEA-GUIDE Github project to
evaluate SER on both the ViGGO and E2E datasets. However, when evaluating the
model on the E2E dataset, the SER evaluation script is not accurate. As a
result, we manually check every prediction labeled as having slot errors by
the SER evaluation script. When applying ChatGPT and GPT4 to the ViGGO
dataset, we also manually check every prediction labeled as incorrect by the
SER evaluation script.