# Parameter Efficient Diverse Paraphrase Generation Using Sequence-Level
Knowledge Distillation

Lasal Jayawardena  School of Computing
Informatics Institute of Technology
Colombo, Sri Lanka
<EMAIL_ADDRESS>Prasan Yapa  School of Computing
Informatics Institute of Technology
Colombo, Sri Lanka
<EMAIL_ADDRESS>
###### Abstract

Over the past year, the field of Natural Language Generation (NLG) has
experienced an exponential surge, largely due to the introduction of Large
Language Models (LLMs). These models have exhibited the most effective
performance in a range of domains within the Natural Language Processing and
Generation domains. However, their application in domain-specific tasks, such
as paraphrasing, presents significant challenges. The extensive number of
parameters makes them difficult to operate on commercial hardware, and they
require substantial time for inference, leading to high costs in a production
setting. In this study, we tackle these obstacles by employing LLMs to develop
three distinct models for the paraphrasing field, applying a method referred
to as sequence-level knowledge distillation. These distilled models are
capable of maintaining the quality of paraphrases generated by the LLM. They
demonstrate faster inference times and the ability to generate diverse
paraphrases of comparable quality. A notable characteristic of these models is
their ability to exhibit syntactic diversity while also preserving lexical
diversity, features previously uncommon due to existing data quality issues in
datasets and not typically observed in neural-based approaches. Human
evaluation of our models shows that there is only a 4% drop in performance
compared to the LLM teacher model used in the distillation process, despite
being 1000 times smaller. This research provides a significant contribution to
the NLG field, offering a more efficient and cost-effective solution for
paraphrasing tasks.

###### Index Terms:

paraphrase generation, natural language processing, knowledge distillation,
deep learning, large language models

##  I Introduction

Paraphrase Generation, alternatively known as Question Paraphrase Generation,
occupies a key role as a fundamental task within the field of Natural Language
Processing (NLP). For several decades, this section of the field has been
explored consistently and the derived results have found immense applications
in enhancing data augmentation processes[1]. The influence of Paraphrase
Generation on data augmentation is pivotal for optimizing the output of
numerous NLP operations, which leads to a substantial enrichment of training
data [2]. Throughout the years, extensive research has gone into this division
of NLP, generating key strategies for optimizing the process [3].

Traditional methods of paraphrase generation, such as rule-based [4],
thesaurus-based [5], and SMT-based methods [6], have been widely used in the
past. However, they are inherently limited in their capacity to generate
diverse and contextually accurate paraphrases, therefore recent research has
been mainly based on neural-based approaches.

The advent of Large Language Models (LLMs) has significantly transformed the
landscape of NLP, including the domain of paraphrase generation. These models,
underpinned by advanced neural networks such as Transformers, have shown
unparalleled performance in a spectrum of NLP applications, covering
everything from categorizing texts and assessing sentiments to translating
languages and textual content generation. [7]. LLMs undergo training using
extensive text corpora and web resources, which equips them with the
capability to decipher complex structures and patterns inherent in human
language. Advanced techniques such as Reinforcement Learning with Human
Feedback (RLHF) [8] have drastically improved its capabilities. This extensive
training allows them to generate human-like writing, which is not just
syntactically accurate but also pertinent and logically consistent within the
context. Moreover, LLMs can understand and generate text in a context-
dependent manner, an essential element in the creation of paraphrases.

Despite the remarkable proficiency of LLMs, several challenges remain. One of
the most significant issues is the high number of parameters in these models.
The sheer size of LLMs makes them difficult to run on consumer hardware,
limiting their accessibility for many users and applications. Additionally,
the large number of parameters also leads to longer inference times, which can
be a bottleneck in real-time applications. These challenges highlight the
urgent need for domain-specific models that maintain the high performance of
LLMs while being more efficient and accessible. In the case of paraphrase
generation, there is a need for models that can generate high-quality
paraphrases quickly and efficiently, without the need for high-end hardware.

In response to this need, we have leveraged the power of LLMs to distill and
build models that are significantly smaller than the original teacher models.
Our models are approximately a thousand times smaller in terms of the number
of parameters, making them much more efficient and easier to run on consumer
hardware. Despite their smaller size, these models maintain the high-quality
paraphrase generation capabilities of their larger counterparts, being a
testament to the effectiveness of our approach. As shown in Section I, we have
utilized a gold standard evaluation strategy which is not commonly seen in
most paraphrase generation research [9]. In addition, results seen in Section
IV-B1, qualitative results obtained by employing human evaluators, illustrate
that the distilled models were indeed capable of maintaining the quality and
diversity of the output despite being drastically smaller than the LLM it is
trained on.

The results of our research not only contribute to the field of paraphrase
generation but also demonstrate the potential of knowledge distillation as a
strategy for leveraging the power of LLMs in a more efficient and accessible
manner. We believe that our work lays the groundwork for upcoming studies
within the field of paraphrase generation, opening up new possibilities for
the application of LLMs effectively.

##  II Related Work

###  II-A Paraphrase Datasets

To support the exploration and creation of models for generating paraphrases,
a variety of datasets have been established. The Paraphrase Database (PPDB)
[10] is a comprehensive resource that contains over 220 million paraphrase
pairs. However, its utility has been questioned due to its exclusive focus on
phrasal and lexical paraphrases, neglecting sentence paraphrases.

The Twitter URL dataset [11] is a large-scale collection of sentential
paraphrases sourced from Twitter. However, due to the noisiness of the labels,
this dataset is not widely used. The Wiki Answer dataset [12] contains an
estimated 18 million pairs of paraphrased questions, but it is limited in
scope as all the sentences provided are in the form of questions.

The MSCOCO dataset [13] was primarily characterized as a comprehensive object
detection dataset. It comprises over 120,000 images, each of which is
accompanied by five distinct captions, contributed by five separate
annotators. The Microsoft Research Paraphrase Corpus (MRPC) Dataset [14]
comprises 5800 sentence pairs derived from online news sources. It also
includes human annotations that denote whether each pair represents a
paraphrase or semantic equivalent.

The Quora Dataset or Quora Question Pair Dataset [15] contains 150,000
question pairs that are annotated as paraphrases. The ParaNMT dataset [16]
comprises over 50 million pairs of English sentential paraphrases. These pairs
were independently created by utilizing bidirectional translation for
converting the non-English elements within a significant Czech-to-English
equivalent dataset.

In the development of the ParaBank datasets, a Czech-English Neural Machine
Translation (NMT) system was employed to create new variations of English
reference sentences [17, 18]. The PAWS Dataset [19] contains sentences with
high bag-of-words (BOW) overlap but having different word order.

Despite the variety of datasets available, there are still challenges in using
them for paraphrase generation. The noise introduced in these datasets due to
utilizing techniques such as back-translation can lead to error propagation,
which cannot be mitigated even by improving the architecture of the model.
Therefore, a data-centric approach is needed to handle these issues.

###  II-B Paraphrase Generation

Paraphrase generation has seen significant advancements with the introduction
of sophisticated techniques. These include multi-round generation [20], which
involves generating multiple iterations of paraphrases, and reinforcement
learning-based paraphrasing [21], which employs reinforcement learning
algorithms to optimize the paraphrasing process. Another noteworthy approach
is prompt-tuning [22], which fine-tunes the prompts used in the generation
process to yield better paraphrases. There is also a subset of research
dedicated to enhancing the syntactic diversity of the generated paraphrases.
This is achieved through various methods such as sampling from latent spaces
[23], which generates diverse paraphrases by sampling different points in the
latent space, and controlling the word order [24], which manipulates the
arrangement of words to create diverse paraphrases. However, these methods
often focus on one aspect of diversity, typically neglecting lexical
diversity. Furthermore, the quality of data used in these approaches often
leaves room for improvement.

Generative adversarial networks (GANs) [25] have also been utilized in
paraphrase creation. The distinct characteristics of textual content
generation pose an obstacle to the conventional training techniques applied in
GANs. To overcome this, the concept of policy gradient is employed [26]. Word-
level paraphrasing is another technique that focuses on generating paraphrases
by substituting original words with synonyms. Some researchers have leveraged
external linguistic knowledge to achieve this [27], while others have proposed
unique mechanisms to learn synonym mappings [28]. These mechanisms can
significantly enhance lexical diversity.

In the search for more sophisticated paraphrase generation, experts have
investigated techniques to manage the syntactic structure of paraphrased text,
integrating various levels of detail. Studies in this field typically fall
into two groups based on their approach to syntax control: implicit and
explicit. In explicit control strategies, the sentence’s syntactic tree is
transformed into vector forms, which are then incorporated into the decoder at
each step of decoding [29]. Implicit control methods, alternatively, acquire
knowledge of syntactic information distribution through a Variational
Autoencoder (VAE). In this approach, a syntax variable, drawn from the learned
distribution, is integrated into the decoder at every step of decoding. [30].

Building on these syntax-focused methods, multi-level paraphrasing techniques
combine multiple granularity levels, enabling their models to generate
synonyms, substitute phrases, and rearrange sentential structures [31]. These
techniques aim to create a more comprehensive and nuanced approach to
paraphrase generation.

However, a common challenge faced by most of these approaches stems from the
scarcity of extensive corpora containing significant quality paraphrases. A
high-quality paraphrase should be lexically diverse, syntactically diverse,
grammatically correct, and semantically similar. Balancing all these aspects
in paraphrase generation remains a significant challenge.

###  II-C Knowledge Distillation

The technique of knowledge distillation is utilized to educate compact models,
frequently termed as the student, through leveraging the “knowledge” from a
more expansive model, designated as the teacher [32]. A common method of
knowledge distillation is to train the student with an additional goal of
aligning with the teacher’s representation, such as logits, output
probability, or intermediate activation [33].

In the context of sequence-to-sequence or generative models, [34] introduced
the idea of sequence-level knowledge distillation. This method involves the
generation of a synthetic output by conducting inference with the teacher
model, which is subsequently used to train the learner/student model. The
efficiency of sequence-level distillation lies in the fact that it only
necessitates running the typically large teacher model once. The efficacy of
sequence-level distillation has been demonstrated in previous studies [35].
Recent research has adopted this technique, particularly with the use of LLMs
[36]. Further extensions of this technique have been explored, such as the use
of reverse Kullback-Leibler divergence (KLD) objectives to enhance the
distillation process [37]. This indicates a growing interest in the field of
NLP in utilizing sequence-level distillation to develop smaller, yet effective
models, especially with the recent research involving LLMs.

##  III Methodology

Figure 1: High-Level Architecture Diagram for Training and Inference Phases.

We utilize a data-centric sequence-level knowledge distillation technique
where we utilize the ChatGPT (gpt-3.5-turbo) LLM to create smaller models
which are capable of producing diverse paraphrases to a quality similar but
with a significantly less number of parameters. The detailed methodology will
be broken down and explained in the subsequent sections.

###  III-A Dataset Creation

To create the full dataset we used multiple data sources. We incorporated a
subset of the Quora Dataset [14], specifically choosing sentence pairs labeled
as paraphrases. Next, we utilized PAWSWiki, a segment within the PAWS
(Paraphrase Adversaries from Word Scrambling) Dataset[19]. However, we
deliberately avoided using PAWSQQP as it contained source sentences identical
to those in the Quora Dataset. These two are the main corpora employed in the
training process. For evaluation purposes, the Microsoft Research Paraphrase
Corpus (MRPC) Dataset [14], the MSCOCO dataset [13], the Wiki Answer dataset
[12] and the Twitter URL dataset [11] was incorporated.

For the paraphrase pair generation phase, we initially combined the
aforementioned data sources, selecting approximately 750,000 source sentences
for paraphrase generation. An initial pass was conducted to filter out
offensive content, using OpenAI’s Moderation Endpoint. This process flagged
any source sentence that fell under the categories of offensive content,
allowing us to filter out approximately source sentences containing such
content. In addition to this, data sources created using back-translation
often contain a significant amount of noise, including non-English source
sentences. To tackle this, we developed a new prompt for the gpt-3.5-turbo
model, instructing it to identify English sentences and generate paraphrases,
or output ”Error” for non-English sentences. This method, coupled with a rule-
based approach to filter out certain responses from the model, proved
efficient in significantly reducing noise.

We then used the ChatGPT (gpt-3.5-turbo) LLM to augment the source sentences.
The prompt used for each dataset varied slightly. For paraphrase generation,
we adjusted the temperature parameter value to zero. The model output was a
string containing augmented, diverse sentence paraphrases in a numbered list
format. This string was subsequently processed to generate a list of strings,
which was then used to construct a pool of paraphrases. By utilizing the
paraphrase pool, we successfully generated nearly 2 million unique sentence
paraphrase pairs. This was then used for model training and evaluation.

###  III-B Model Training

The models selected for distillation were T5-small [38], Flant5-small[39], and
BART-base[40], each chosen for their unique strengths and capabilities. These
models are built on an encoder-decoder architecture, a design that is
particularly effective for tasks involving conditional generation, such as
paraphrase generation.

The T5-small model is known for its efficiency and high performance in text
generation tasks. The model is crafted to adeptly deal with a wide range of
NLP tasks, making it a versatile choice for our project. Its small size,
compared to larger versions of T5, makes it more computationally efficient
while still delivering strong performance. The T5-small model has only around
60 million parameters.

The Flant5-small model is a multilingual version of T5. It has been trained on
multiple languages, which could potentially enhance the quality of paraphrase
generation by leveraging the syntactic and semantic similarities across
languages. By including this model in our study, we aimed to explore the
potential benefits of multilingual capabilities in paraphrase generation. The
Flant5-small is slightly bigger than the T5-small model with around 80 million
parameters.

Lastly, the BART-base model was included because of its impressive
capabilities in generating text and summarizing information. BART stands out
in its pretraining methodology, being designed to restore the original content
after specific tokens are obscured or complete sentences are rearranged. This
makes BART particularly effective at tasks that require understanding the
broader context of a sentence, such as paraphrase generation. This model is
slightly bigger with around 140 million parameters which is still a thousand
times smaller than ChatGPT.

By distilling these three models, we aimed to leverage their unique strengths
and capabilities to create a robust and efficient model for paraphrase
generation.

For the training process, we utilized the Quora Question Pairs and PAWS
datasets. To make the training more effective we lower cased the data and
trained the models to further simplify the learning process so that models
could learn on the core task effectively. The training method employed was
Low-Rank Adaptation (LoRA) [41]. This method preserves the weights of the pre-
trained model and integrates trainable rank decomposition matrices into every
layer of the Transformer structure. Consequently, this strategy considerably
lowers the number of parameters needed for subsequent tasks, resulting in
reduced GPU memory usage and enhanced training throughput. The LoRA technique
was applied through the use of a library specifically designed for Parameter-
Efficient Fine-Tuning (PEFT), developed by Hugging
Face111https://github.com/huggingface/peft. The training of each model was
conducted using RTX A100 40GB GPU, using approximately 1.4 million rows of
data. This approach is very effective in knowledge distillation when working
with such a large volume of data.

The hyperparameters for the three models were set as follows: The models were
programmed to accommodate a diverse range of sentence lengths by setting the
maximum sequence length at 256. The models were trained for 10 epochs,
providing a balance between training time and model performance. To ensure
numerical stability, the Adam optimizer was utilized, accompanied by an
epsilon value fixed at 1e-08. A learning rate of 0.0003 was determined, along
with a maximum gradient norm limited to 1.0, which serves to inhibit
excessively large gradients. The LoRA configuration for these models was set
with a rank (r) of 8 and an alpha value of 32, providing a balance between
model complexity and performance. To avert the risk of overfitting, the
dropout rate was established at 0.1. In all models, “paraphrase” was used as
the prefix for the input sentences during training. The training times for the
models were as follows: T5-small took 24 hours, flant5-small took 30 hours,
and BART-base took 50 hours to train for 10 epochs.

###  III-C Model Inference

Model Inference is a bit more tricky than anticipated since hyperparameters
need to be explored to gain the optimal output. The hyperparameters for
inference were set differently for each model to optimize their performance.

For BART, the maximum amount of new tokens for a generation was limited to
256, whilst early stopping was enabled to prevent the generation of
unnecessary tokens. Sampling was enabled with 100 beams. The top-p value was
set to 0.35 to control the nucleus sampling, and the no-repeat parameter for
n-gram size was set to 2 so that it could prevent the model from generating
repetitive phrases. The temperature was set to 2.5 to control the randomness
of the output.

For the T5 model, the restriction on repeating n-grams was configured with a
limit of 2. Moreover, the top-k parameter had a configuration value of 50,000
for its sampling method. The setting for temperature was adjusted to 0.7, and
for nucleus sampling, the top-p parameter was calibrated to 0.75.
Additionally, the model incorporated early stopping.

In the case of flan T5, early stopping was integrated. The configuration for
sampling included a limitation on the n-gram repetition, which was established
at 2. The beam search was conducted with 200 beams, and the temperature
parameter was adjusted to a level of 1.5.

Some post-processing was done to refine the output of the models. Since the
output from the models was in lowercase, the first letter of each sentence was
capitalized to ensure proper sentence structure. Next, we performed entity
case correction. For this, we used the Named Entity Recognition (NER) model
from the SpaCy library222https://github.com/explosion/spaCy. The NER model
identifies various entities within the text, including the names of people,
organizations, or locations. We preserved the case capitalization for these
entities, ensuring that proper nouns were correctly capitalized in the output.
This step is crucial as it enhances the readability and accuracy of the
generated sentences. Finally, if multiple sequences were output, we removed
any duplicates to ensure the uniqueness of the generated paraphrases. This
step helps to maintain the diversity of the output, providing a wider range of
paraphrases for each input sentence.

##  IV Evaluation

TABLE I: Comparison of Semantic Similarity of the original dataset, distilled models and teacher model. Model | ADA Score (↑) | SimCSE Score (↑) | PromCSE Score (↑) | Roberta Score (↑) | Mpnet Score (↑)
---|---|---|---|---|---
Original Dataset | 90.88% | 71.90% | 98.68% | 72.10% | 69.79%
ChatGPT | 95.60% | 91.25% | 99.41% | 88.23% | 87.03%
T5 Small (Ours) | 97.28% | 94.59% | 99.67% | 92.77% | 92.60%
Flan T5 Small (Ours) | 97.75% | 95.42% | 99.71% | 93.71% | 93.69%
BART Base (Ours) | 98.07% | 95.77% | 99.72% | 94.04% | 93.77%

The evaluation was done using both quantitative and qualitative evaluation
techniques. For the quantitative analysis, we utilized 100,000 paraphrase
pairs extracted from the MRPC dataset, the MSCOCO evaluation subset, the
Twitter URL dataset, and the Wiki Answer dataset. For the qualitative
analysis, we incorporated human evaluations and a novel evaluation technique
known as LLM evaluations. The subsequent section will outline our findings in
the evaluation process.

###  IV-A Quantitative Analysis

For the quantitative analysis, we will be evaluating whether the paraphrases
are high quality and diverse. We will assess three main characteristics:
semantic similarity, syntactic diversity, and lexical diversity.

####  IV-A1 Semantic Simialrity

In our research, we evaluate semantic similarity by utilizing a range of
models to produce embeddings of sentences from both the original text and its
paraphrased form. Following this, we measure the degree of similarity using
the cosine similarity method between these embeddings. The resulting scores
are derived by taking one and subtracting the cosine similarity metric of the
base and paraphrased text.

  * •

The “Ada Score” is computed using OpenAI’s text-embedding-ada-002 model [42].

  * •

The “SimCSE Score” is derived from the sup-simcse-roberta-large model by
SimCSE [43].

  * •

The “PromCSE Score” is based on the sup-promcse-roberta-large model from
PromCSE [44].

We also make use of several models from the sentence-transformers library
[45]:

  * •

The “Mpnet Score” is derived by employing the model known as all-mpnet-
base-v1.

  * •

The “Roberta Score” is determined through the utilization of the all-roberta-
large-v1 model.

The data in Table I show that despite the distillation process the models have
retained the semantic similarity compared to the teacher model. The difference
in similarities is negligible likely caused due to different lexical phrases
and sentence structures. The original dataset indicates that there is noise in
the datasets despite being used in other research for training models.

####  IV-A2 Syntactic Diversity

Syntactic diversity is a measure of the range and intricacy of sentence
structures in a paraphrase, given an original sentence. A high level of
syntactic diversity suggests that the paraphrased sentences are varied and
linguistically sophisticated, meaning that it is a characteristic of a high-
quality paraphrase. We evaluate the diversity using metrics that take into
account the sentence syntax trees.

  * •

The “Ted-F” encompasses the complete Tree Edit Distance measurement. This is
achieved through constructing the constituency parse trees for the original
and paraphrased sentences using Stanza [46], transforming the trees to bracket
notation with the NLTK library [47] and regex, and then employing the APTED
library [48] to compute the entire Tree Edit Distance.

  * •

The “Ted-3” represents Tree Edit Distance of the first three layers. It is
calculated in a similar manner to “Ted-F”, but rather than calculating the
comprehensive Tree Edit Distance, it focuses on the Tree Edit Distance for the
first three strata.

  * •

The “Kermit Score” is computed by finding the cosine measurement of similarity
between the original and the paraphrase syntactic vectors using the Kermit
library [49], and then subtracting this similarity from one. The syntactic
embeddings are obtained by feeding the syntax trees of the two sentences.

  * •

The “Subtree K Score” is the Subtree Kernel diversity. It is calculated by
initially constructing constituency parse trees for the original and
paraphrased sentences using Stanza, transforming the trees to an NLTK Tree,
and then identifying all the subtrees. The kernel similarity is calculated by
the ratio of unique common subtrees to the total count of unique subtrees.
This figure is then subtracted from one to yield the diversity score.

  * •

The “Node Pair K Score” is the Subtree Node Pair Kernel diversity. It is
calculated similarly to the “Subtree K Score”. The only difference is instead
of subtrees this uses node pairs for the calculation.

Results in Table II show the full results of syntactic diversity. The data
suggests that the distilled models have successfully retained the ability to
generate syntactically diverse paraphrases similar to the teacher. This
characteristic is unseen in most neural-based approaches which is a
significant improvement.

TABLE II: Comparison of Syntactic Diversity of the original dataset, distilled models and teacher model. Model | Ted-F (↑) | Ted-3 (↑) | Kermit Score (↑) | Subtree K Score (↑) | Node Pair K Score (↑)
---|---|---|---|---|---
Original Dataset | 17.38 | 4.02 | 74.81% | 95.36% | 84.14%
ChatGPT | 21.24 | 4.53 | 66.94% | 92.77% | 79.20%
T5 Small (Ours) | 17.29 | 3.99 | 54.89% | 83.36% | 66.58%
Flan T5 Small (Ours) | 18.38 | 4.40 | 54.96% | 83.23% | 65.45%
BART Base (Ours) | 23.45 | 5.06 | 61.98% | 88.97% | 72.30%
TABLE III: Comparison of lexical diversity of the original dataset, distilled models and teacher model. Model | BOW Overlap | Corpus BLEU | Corpus BLEU2 | METOER | ROUGE 1 | ROUGE 2
---|---|---|---|---|---|---
| Score (↑) | Score (↑) | Score(↑) | Score (↑) | Score(↑) | Score(↑)
Original Dataset | 58.59% | 99.40% | 85.12% | 59.28% | 53.64% | 77.02%
ChatGPT | 50.55% | 99.54% | 84.01% | 43.20% | 42.56% | 70.15%
T5 Small (Ours) | 35.42% | 99.46% | 65.79% | 29.50% | 28.19% | 50.78%
Flan T5 Small (Ours) | 34.06% | 99.48% | 63.93% | 27.80% | 26.23% | 47.05%
BART Base (Ours) | 39.49% | 99.50% | 75.71% | 36.27% | 33.51% | 59.79%
Model | ROUGE L | Token $\cap/\cup$ | TER | WER | CharacTER | Google BLEU
| Score (↑) | Score (↑) | Score (↑) | Score (↑) | Score (↑)) | Score (↑)
Original Dataset | 58.49% | 69.73% | 80.45 | 85.44 | 69.27 | 80.88%
ChatGPT | 54.17% | 62.93% | 63.05 | 77.44 | 77.90 | 77.89%
T5 Small (Ours) | 42.17% | 43.52% | 49.05 | 66.83 | 54.80 | 59.55%
Flan T5 Small (Ours) | 41.98% | 40.94% | 48.77 | 69.97 | 56.02 | 57.91%
BART Base (Ours) | 52.11% | 49.68% | 59.23 | 82.47 | 68.00 | 68.15%

####  IV-A3 Lexical Diversity

Lexical diversity is a concept that signifies the array and diversity of words
incorporated in a text. It serves as an indicator of the width of vocabulary
and the application of synonyms. In the sphere of paraphrasing, it becomes
essential to gauge lexical diversity to grasp the level of vocabulary
fluctuation. We utilized an array of metrics to gauge lexical diversity.

  * •

“BOW Overlap Score” is calculated by identifying the shared tokens between the
original and the paraphrased text, and dividing by the total count of tokens.
This value is then subtracted by one.

  * •

“Corpus BLEU Score” is evaluated using the SacreBLEU Library [50]. This score
is then subtracted by one.

  * •

“Corpus BLEU2 Score” is constructed using the SacreBLEU Library with the
“method1” smoothing function. This score is then subtracted by one.

  * •

“Sentence BLEU Score” is calculated similarly to the Corpus BLEU score using
the SacreBLEU Library but at the sentence level. This score is also subtracted
by one.

  * •

“METEOR Score” is evaluated using the NLTK library. This score is then
subtracted by one.

  * •

“ROUGE 1 Score” is constructed using the Google Research
library333https://github.com/google-research/google-research. This score is
then subtracted by one.

  * •

“ROUGE 2 Score” is evaluated using the Google Research library. This score is
then subtracted by one.

  * •

“ROUGE L Score” is calculated using the Google Research library. This score is
then subtracted by one.

  * •

“Token $\cap/\cup$ Score” is similar to the BOW Overlap score but with a minor
variation. It is calculated using the shared tokens between the original and
the paraphrased text, divided by the total unique tokens. This value is then
subtracted by one.

  * •

“Google BLEU Score” is evaluated using Huggingface’s Evaluate library. This
score is then subtracted by one.444https://github.com/huggingface/evaluate

  * •

“TER Score” is the Translation Error Rate score which is constructed using
Huggingface’s Evaluate library.

  * •

“WER Score” is the Word Error Rate score evaluated using Huggingface’s
Evaluate library.

  * •

“CharacTER Score” is the Character Error Rate score calculated using
Huggingface’s Evaluate library.

Results in Table III show the distilled models were able to generate lexically
diverse paraphrases on par with the teacher model. Even though the teacher
model is better, models still exhibit significant lexical diversity similar to
that of the teacher.

TABLE IV: Human evaluation Results. Model | Semantic Similarity (↑) | Lexical Diversity (↑) | Syntactic Diversity (↑) | Grammatical Correctness (↑)
---|---|---|---|---
Original Dataset | 3.47 | 3.08 | 2.93 | 4.53
ChatGPT | 4.33 | 3.43 | 3.26 | 4.97
T5 Small (Ours) | 4.21 | 3.23 | 2.95 | 4.83
Flan T5 Small (Ours) | 4.05 | 3.19 | 2.83 | 4.74
BART Base (Ours) | 4.20 | 3.25 | 3.16 | 4.89
TABLE V: LLM evaluation results. Model | Semantic Similarity (↑) | Lexical Diversity (↑) | Syntactic Diversity (↑) | Grammatical Correctness (↑)
---|---|---|---|---
Original Dataset | 3.40 | 2.83 | 2.88 | 4.37
ChatGPT | 4.82 | 3.01 | 3.85 | 4.89
T5 Small (Ours) | 4.51 | 2.48 | 3.20 | 4.68
Flan T5 Small (Ours) | 4.26 | 2.20 | 2.85 | 4.41
BART Base (Ours) | 4.75 | 2.63 | 3.41 | 4.69

###  IV-B Qualitative Analysis

In order to gain an in-depth insight of the performance of our model, we
conducted a qualitative analysis using two distinct methods: human evaluation
and Large Language Model (LLM) evaluation.

####  IV-B1 Human Evaluation

We recruited a group of five independent evaluators who are proficient in
English for the human evaluation phase. The evaluation process involved the
selection of thousand paraphrase pairs from four different data sources: the
MRPC dataset, the MSCOCO dataset evaluation subset, the Twitter URL dataset,
and the Wiki Answer dataset. From each of these data sources, 250 paraphrase
pairs were selected, ensuring a balanced representation. The selection process
was carefully conducted to ensure that the lengths of the sentences from each
data source were accurately represented. For each of the thousand paraphrase
pairs, we obtained model outputs from four different models: ChatGPT, and the
three trained models. This resulted in a total annotation set of 5000 pairs
for evaluation, which includes both the original dataset paraphrase pairs and
the model outputs. The extensive set of evaluations facilitates a
comprehensive and reliable assessment of each model’s performance.

In our assessment, we employed a 5-point Likert scale [51] for the evaluation
of Semantic Similarity, Lexical Diversity, Syntactic Diversity, and
Grammatical Correctness. The breakdown of the Likert scale is as follows:

  * •

For Semantic Similarity, a score of 5 implies that the text’s meaning aligns
perfectly or almost perfectly with the source text. This could suggest that
the text expresses the same ideas, reaches the same conclusions, or presents
the same arguments. Conversely, a score of 1 in semantic similarity would
suggest that the text’s meaning is entirely different or unrelated to the
source text, indicating differing ideas, conclusions, or arguments.

  * •

For Lexical Diversity, a score of 5 signifies a broad and rich vocabulary.
This could suggest that the text employs a diverse array of words, synonyms,
and phrases, and avoids repetition. In contrast, a score of 1 in lexical
diversity would suggest a limited vocabulary range, indicating that the text
uses a small set of words, heavily relies on a few key phrases, or frequently
repeats the same words or phrases.

  * •

For Syntactic Diversity, a score of 5 denotes a high degree of variation in
sentence structure. This could suggest that the text employs a variety of
sentence types, lengths, and structures, and avoids repetition. Conversely, a
score of 1 in syntactic diversity would suggest minimal variation in sentence
structure, indicating that the text uses a limited number of sentence types,
heavily relies on a few key structures, or frequently repeats the same
sentence structures.

  * •

For Grammatical Correctness, a score of 5 signifies perfect grammar. This
could suggest that the text employs correct punctuation, spelling, and syntax,
and avoids grammatical errors. Conversely, a score of 1 in grammatical
correctness would suggest significant errors that affect comprehension,
indicating that the text contains frequent spelling, punctuation, or syntax
errors, or that these errors hinder its comprehensibility.

The evaluation instruction given to the human evaluators can be seen in Fig.
A2. Grammatical correctness is one aspect that is normally evaluated by other
research work but is crucial to identifying the effectiveness of a paraphrase.
The final data obtained from the human evaluation are given in Table IV

####  IV-B2 LLM Evaluation

Over the recent year, the application of Language Model Metrics (LLMs) for
evaluation in NLP has seen some rise. This surge in popularity is primarily
attributed to the superior performance of LLMs over existing reference-free
metrics [52]. Recognizing this potential, our research also embarked on an
evaluation strategy using LLMs, utilizing OpenAI’s gpt-4 model, which was the
state-of-the-art (SOTA) LLM at the time of conducting this study [53].

The data used for this evaluation was identical to the one provided to our
human annotators. We crafted the prompt in alignment with the instructions
given to the human annotators, as depicted in Fig. A1. This approach ensured a
fair evaluation strategy to that of the human evaluation.

This innovative methodology employed in our research could serve as a valuable
reference for future studies in this field. It not only provides a new
perspective on the use of LLMs in NLP but also opens up possibilities for
further exploration and development of more advanced and efficient evaluation
techniques. The final results of the LLM evaluation are given in Table V

##  V Discussion

Our evaluation process, which combined both quantitative and qualitative
methods, provided a comprehensive understanding of the performance of our
models. The quantitative analysis focused on three main characteristics:
semantic similarity, syntactic diversity, and lexical diversity. The results
indicated that, despite the distillation process, our models were able to
maintain semantic similarity, syntactic diversity, and lexical diversity in
comparison to the teacher model (ChatGPT).

Our comprehensive qualitative evaluation, which incorporated human evaluation,
provided an accurate and in-depth understanding of our research outcomes. The
results distinctly demonstrated that, despite our models being a thousand
times smaller than ChatGPT, they were able to maintain comparable performance
levels. This is a significant achievement, highlighting the efficiency and
effectiveness of our models in generating high-quality paraphrases. Moreover,
the use of LLM evaluation in our study introduced a novel approach to
performance assessment in our research domain. This innovative strategy, which
leverages the capabilities of SOTA language models for evaluation, offers a
promising avenue for future research. With further refinement and development,
this LLM evaluation approach has the potential to serve as a robust
benchmarking tool for assessing the performance of paraphrasing models and
other natural language processing tasks.

In our evaluation, we employed a wide array of metrics to assess various
aspects of the generated paraphrases. This approach was adopted in recognition
of the fact that a single metric may not fully capture the effectiveness of a
paraphrase, as it tends to focus on a specific characteristic. This highlights
an active area of research that warrants further exploration. The use of
inappropriate or insufficient metrics can lead to a skewed understanding of
the models’ performance. Therefore, future research should emphasize the
development of a unified metric that can holistically evaluate the quality of
paraphrases. Such a metric would not only provide a more precise assessment of
the model performance but also contribute to the advancement of the field of
paraphrase generation.

In our methodology even though the distilled models were able to retain the
quality of the distilled knowledge. One area where the ChatGPT is superior is
the ability to generate paraphrases that are diverse from each other. This
means that there was a notable variation among the paraphrases generated by
ChatGPT. Our models, however, did not guarantee the same level of diversity,
indicating a need for further research to optimize the inference
hyperparameters to enhance the diversity of the generated paraphrases. Random
Sampling could be a potential starting point for this approach, but more
extensive work is required to fully address this issue.

Since our models were trained using ChatGPT, they may inherit potential risks
associated with it. This could include the propagation of biases inherent in
the train dataset. Therefore, it is crucial for future research to address
these issues and develop strategies to mitigate the potential risks associated
with the use of LLMs with knowledge distillation for paraphrase generation.

##  VI Conclusion

Our research offers a more efficient and cost-effective solution for
paraphrase generation, making it more accessible for various applications. The
distilled models provide a viable alternative to using large LLMs, opening up
new possibilities for their application in real-world scenarios. A notable
result of our models is the ability to produce both lexically and
syntactically diverse paraphrases. Our models were at least a thousand times
smaller than the original LLM and were still able to perform on par with it.

Overall, our work sets the foundation for future advancements in parameter-
efficient and diverse paraphrase generation. It underscores the opportunities
for additional exploration and innovation in this domain, paving the way for
more efficient and accessible solutions in the field of NLG and paraphrase
generation.

##  VII Acknowledgment

We wish to extend our appreciation and thanks to the research participants who
participated in the human evaluation of the models. Their insights and
feedback have been instrumental in assessing the quality and effectiveness of
our research.

## References

  * [1] K. R. McKeown, “Paraphrasing using given and new information in a question-answer system,” in _Proceedings of the 17th annual meeting on Association for Computational Linguistics -_. La Jolla, California: Association for Computational Linguistics, 1979, p. 67. [Online]. Available: http://portal.acm.org/citation.cfm?doid=982163.982182
  * [2] M. Meteer and V. Shaked, “Strategies for effective paraphrasing,” in _Proceedings of the 12th conference on Computational linguistics -_ , vol. 2. Budapest, Hungry: Association for Computational Linguistics, 1988, pp. 431–436. [Online]. Available: http://portal.acm.org/citation.cfm?doid=991719.991724
  * [3] R. Kozlowski, K. F. McCoy, and K. Vijay-Shanker, “Generation of single-sentence paraphrases from predicate/argument structure using lexico-grammatical resources,” in _Proceedings of the second international workshop on Paraphrasing -_ , vol. 16. Sapporo, Japan: Association for Computational Linguistics, 2003, pp. 1–8. [Online]. Available: http://portal.acm.org/citation.cfm?doid=1118984.1118985
  * [4] D. Lin and P. Pantel, “Discovery of inference rules for question-answering,” _Natural Language Engineering_ , vol. 7, no. 4, pp. 343–360, Dec. 2001. [Online]. Available: https://www.cambridge.org/core/product/identifier/S1351324901002765/type/journal_article
  * [5] D. Kauchak and R. Barzilay, “Paraphrasing for Automatic Evaluation,” in _Proceedings of the Human Language Technology Conference of the NAACL, Main Conference_. New York City, USA: Association for Computational Linguistics, Jun. 2006, pp. 455–462. [Online]. Available: https://aclanthology.org/N06-1058
  * [6] S. Wubben, A. van den Bosch, and E. Krahmer, “Paraphrase Generation as Monolingual Translation: Data and Evaluation,” in _Proceedings of the 6th International Natural Language Generation Conference_. Association for Computational Linguistics, Jul. 2010. [Online]. Available: https://aclanthology.org/W10-4223
  * [7] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, Y. Du, C. Yang, Y. Chen, Z. Chen, J. Jiang, R. Ren, Y. Li, X. Tang, Z. Liu, P. Liu, J.-Y. Nie, and J.-R. Wen, “A Survey of Large Language Models,” Sep. 2023, arXiv:2303.18223 [cs] version: 12. [Online]. Available: http://arxiv.org/abs/2303.18223
  * [8] Z. Li, Z. Yang, and M. Wang, “Reinforcement Learning with Human Feedback: Learning Dynamic Choices via Pessimism,” 2023. [Online]. Available: https://arxiv.org/abs/2305.18438
  * [9] J. Zhou and S. Bhat, “Paraphrase Generation: A Survey of the State of the Art,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 5075–5086. [Online]. Available: https://aclanthology.org/2021.emnlp-main.414
  * [10] J. Ganitkevitch, B. Van Durme, and C. Callison-Burch, “PPDB: The Paraphrase Database,” in _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Atlanta, Georgia: Association for Computational Linguistics, Jun. 2013, pp. 758–764. [Online]. Available: https://aclanthology.org/N13-1092
  * [11] W. Lan, S. Qiu, H. He, and W. Xu, “A Continuously Growing Dataset of Sentential Paraphrases,” in _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Copenhagen, Denmark: Association for Computational Linguistics, 2017, pp. 1224–1234. [Online]. Available: http://aclweb.org/anthology/D17-1126
  * [12] A. Fader, L. Zettlemoyer, and O. Etzioni, “Paraphrase-Driven Learning for Open Question Answering,” in _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Sofia, Bulgaria: Association for Computational Linguistics, Aug. 2013, pp. 1608–1618. [Online]. Available: https://aclanthology.org/P13-1158
  * [13] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common Objects in Context,” in _Computer Vision – ECCV 2014_ , D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds. Cham: Springer International Publishing, 2014, vol. 8693, pp. 740–755, series Title: Lecture Notes in Computer Science. [Online]. Available: http://link.springer.com/10.1007/978-3-319-10602-1_48
  * [14] W. B. Dolan and C. Brockett, “Automatically Constructing a Corpus of Sentential Paraphrases,” in _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_ , 2005. [Online]. Available: https://aclanthology.org/I05-5002
  * [15] S. Iyer, N. Dandeka, and K. Csernai, “First Quora Dataset Release: Question Pairs,” 2017. [Online]. Available: https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs
  * [16] J. Wieting and K. Gimpel, “ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Melbourne, Australia: Association for Computational Linguistics, 2018, pp. 451–462. [Online]. Available: http://aclweb.org/anthology/P18-1042
  * [17] J. E. Hu, R. Rudinger, M. Post, and B. Van Durme, “ParaBank: Monolingual Bitext Generation and Sentential Paraphrasing via Lexically-constrained Neural Machine Translation,” Jan. 2019, arXiv:1901.03644 [cs]. [Online]. Available: http://arxiv.org/abs/1901.03644
  * [18] J. E. Hu, A. Singh, N. Holzenberger, M. Post, and B. Van Durme, “Large-Scale, Diverse, Paraphrastic Bitexts via Sampling and Clustering,” in _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_. Hong Kong, China: Association for Computational Linguistics, 2019, pp. 44–54. [Online]. Available: https://www.aclweb.org/anthology/K19-1005
  * [19] Y. Zhang, J. Baldridge, and L. He, “PAWS: Paraphrase Adversaries from Word Scrambling,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 1298–1308. [Online]. Available: https://aclanthology.org/N19-1131
  * [20] Z. Lin and X. Wan, “Pushing Paraphrase Away from Original Sentence: A Multi-Round Paraphrase Generation Approach,” 2021. [Online]. Available: https://arxiv.org/abs/2109.01862
  * [21] M. Liu, E. Yang, D. Xiong, Y. Zhang, Y. Meng, C. Hu, J. Xu, and Y. Chen, “A Learning-Exploring Method to Generate Diverse Paraphrases with Multi-Objective Deep Reinforcement Learning,” in _Proceedings of the 28th International Conference on Computational Linguistics_ , D. Scott, N. Bel, and C. Zong, Eds. Barcelona, Spain (Online): International Committee on Computational Linguistics, Dec. 2020, pp. 2310–2321. [Online]. Available: https://aclanthology.org/2020.coling-main.209
  * [22] J. R. Chowdhury, Y. Zhuang, and S. Wang, “Novelty Controlled Paraphrase Generation with Retrieval Augmented Conditional Prompt Tuning,” 2022. [Online]. Available: https://arxiv.org/abs/2202.00535
  * [23] Y. Cao and X. Wan, “DivGAN: Towards Diverse Paraphrase Generation via Diversified Generative Adversarial Network,” in _Findings of the Association for Computational Linguistics: EMNLP 2020_ , T. Cohn, Y. He, and Y. Liu, Eds. Online: Association for Computational Linguistics, Nov. 2020, pp. 2411–2421. [Online]. Available: https://aclanthology.org/2020.findings-emnlp.218
  * [24] T. Goyal and G. Durrett, “Neural Syntactic Preordering for Controlled Paraphrase Generation,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds. Online: Association for Computational Linguistics, Jul. 2020, pp. 238–252. [Online]. Available: https://aclanthology.org/2020.acl-main.22
  * [25] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative Adversarial Networks,” 2014. [Online]. Available: https://arxiv.org/abs/1406.2661
  * [26] L. Yu, W. Zhang, J. Wang, and Y. Yu, “SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient,” Aug. 2017, arXiv:1609.05473 [cs]. [Online]. Available: http://arxiv.org/abs/1609.05473
  * [27] Z. Lin, Z. Li, N. Ding, H.-T. Zheng, Y. Shen, W. Wang, and C.-Z. Zhao, “Integrating Linguistic Knowledge to Sentence Paraphrase Generation,” _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 34, no. 05, pp. 8368–8375, Apr. 2020. [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/6354
  * [28] Y. Fu, Y. Feng, and J. P. Cunningham, “Paraphrase Generation with Latent Bag of Words,” Jan. 2020, arXiv:2001.01941 [cs]. [Online]. Available: http://arxiv.org/abs/2001.01941
  * [29] A. Kumar, K. Ahuja, R. Vadapalli, and P. Talukdar, “Syntax-Guided Controlled Generation of Paraphrases,” _Transactions of the Association for Computational Linguistics_ , vol. 8, pp. 329–345, 2020. [Online]. Available: https://aclanthology.org/2020.tacl-1.22
  * [30] W. Chen, J. Tian, L. Xiao, H. He, and Y. Jin, “A Semantically Consistent and Syntactically Variational Encoder-Decoder Framework for Paraphrase Generation,” in _Proceedings of the 28th International Conference on Computational Linguistics_ , D. Scott, N. Bel, and C. Zong, Eds. Barcelona, Spain (Online): International Committee on Computational Linguistics, Dec. 2020, pp. 1186–1198. [Online]. Available: https://aclanthology.org/2020.coling-main.102
  * [31] A. Kazemnejad, M. Salehi, and M. Soleymani Baghshah, “Paraphrase Generation by Learning How to Edit from Samples,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, Eds. Online: Association for Computational Linguistics, Jul. 2020, pp. 6010–6021. [Online]. Available: https://aclanthology.org/2020.acl-main.535
  * [32] G. Hinton, O. Vinyals, and J. Dean, “Distilling the Knowledge in a Neural Network,” 2015. [Online]. Available: https://arxiv.org/abs/1503.02531
  * [33] X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu, “TinyBERT: Distilling BERT for Natural Language Understanding,” in _Findings of the Association for Computational Linguistics: EMNLP 2020_ , T. Cohn, Y. He, and Y. Liu, Eds. Online: Association for Computational Linguistics, Nov. 2020, pp. 4163–4174. [Online]. Available: https://aclanthology.org/2020.findings-emnlp.372
  * [34] Y. Kim and A. M. Rush, “Sequence-Level Knowledge Distillation,” in _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , J. Su, K. Duh, and X. Carreras, Eds. Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. 1317–1327. [Online]. Available: https://aclanthology.org/D16-1139
  * [35] N. Bogoychev, R. Grundkiewicz, A. F. Aji, M. Behnke, K. Heafield, S. Kashyap, E.-I. Farsarakis, and M. Chudyk, “Edinburgh’s Submissions to the 2020 Machine Translation Efficiency Task,” in _Proceedings of the Fourth Workshop on Neural Generation and Translation_ , A. Birch, A. Finch, H. Hayashi, K. Heafield, M. Junczys-Dowmunt, I. Konstas, X. Li, G. Neubig, and Y. Oda, Eds. Online: Association for Computational Linguistics, Jul. 2020, pp. 218–224. [Online]. Available: https://aclanthology.org/2020.ngt-1.26
  * [36] M. Wu, A. Waheed, C. Zhang, M. Abdul-Mageed, and A. F. Aji, “LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions,” May 2023, arXiv:2304.14402 [cs]. [Online]. Available: http://arxiv.org/abs/2304.14402
  * [37] Y. Gu, L. Dong, F. Wei, and M. Huang, “Knowledge Distillation of Large Language Models,” Jun. 2023, arXiv:2306.08543 [cs]. [Online]. Available: http://arxiv.org/abs/2306.08543
  * [38] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” 2019. [Online]. Available: https://arxiv.org/abs/1910.10683
  * [39] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, A. Castro-Ros, M. Pellat, K. Robinson, D. Valter, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei, “Scaling Instruction-Finetuned Language Models,” 2022. [Online]. Available: https://arxiv.org/abs/2210.11416
  * [40] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension,” 2019. [Online]. Available: https://arxiv.org/abs/1910.13461
  * [41] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” 2021. [Online]. Available: https://arxiv.org/abs/2106.09685
  * [42] OpenAI, “New and Improved Embedding Model,” 2023, publisher: OpenAI. [Online]. Available: https://openai.com/blog/new-and-improved-embedding-model
  * [43] T. Gao, X. Yao, and D. Chen, “SimCSE: Simple Contrastive Learning of Sentence Embeddings,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 6894–6910. [Online]. Available: https://aclanthology.org/2021.emnlp-main.552
  * [44] Y. Jiang, L. Zhang, and W. Wang, “Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning,” in _Findings of the Association for Computational Linguistics: EMNLP 2022_. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 3021–3035. [Online]. Available: https://aclanthology.org/2022.findings-emnlp.220
  * [45] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Nov. 2019. [Online]. Available: http://arxiv.org/abs/1908.10084
  * [46] P. Qi, Y. Zhang, Y. Zhang, J. Bolton, and C. D. Manning, “Stanza: A Python Natural Language Processing Toolkit for Many Human Languages,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , 2020.
  * [47] S. Bird, E. Loper, and E. Klein, _Natural Language Processing with Python_. O’Reilly Media Inc., 2009.
  * [48] M. Pawlik and N. Augsten, “Efficient Computation of the Tree Edit Distance,” _ACM Transactions on Database Systems_ , vol. 40, no. 1, pp. 1–40, Mar. 2015. [Online]. Available: https://dl.acm.org/doi/10.1145/2699485
  * [49] F. M. Zanzotto, A. Santilli, L. Ranaldi, D. Onorati, P. Tommasino, and F. Fallucchi, “KERMIT: Complementing Transformer Architectures with Encoders of Explicit Syntactic Interpretations,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Online: Association for Computational Linguistics, Nov. 2020, pp. 256–267. [Online]. Available: https://www.aclweb.org/anthology/2020.emnlp-main.18
  * [50] M. Post, “A Call for Clarity in Reporting BLEU Scores,” in _Proceedings of the Third Conference on Machine Translation: Research Papers_. Belgium, Brussels: Association for Computational Linguistics, Oct. 2018, pp. 186–191. [Online]. Available: https://www.aclweb.org/anthology/W18-6319
  * [51] C. Van Der Lee, A. Gatt, E. Van Miltenburg, S. Wubben, and E. Krahmer, “Best practices for the human evaluation of automatically generated text,” in _Proceedings of the 12th International Conference on Natural Language Generation_. Tokyo, Japan: Association for Computational Linguistics, 2019, pp. 355–368. [Online]. Available: https://www.aclweb.org/anthology/W19-8643
  * [52] Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment,” 2023, publisher: arXiv Version Number: 3. [Online]. Available: https://arxiv.org/abs/2303.16634
  * [53] OpenAI, “GPT-4 Technical Report,” 2023, publisher: arXiv Version Number: 3. [Online]. Available: https://arxiv.org/abs/2303.08774

##  Appendix A Appendix A

Source Text: $source_text Paraphrase: $paraphrase Please evaluate the
following aspects of the paraphrase in comparison to its source text on a
likert scale of 1 to 5, where: Semantic Similarity: This refers to how closely
the meaning of the paraphrase matches the meaning of the source text. Rating
Scale for Semantic Similairty 1: The paraphrase has a completely different
meaning or is unrelated to the source text. 2: The paraphrase has a somewhat
different meaning from the source text 3: The paraphrase captures the general
idea of the source text, but some details or nuances are missing. 4: The
paraphrase largely captures the meaning of the source text but may have slight
differences in wording or expression. 5: The paraphrase has an identical or
nearly identical meaning to the source text. Lexical Diversity: This aspect
evaluates the range and richness of vocabulary used in the paraphrase,
considering its comparison to the source text. Rating Scale for Lexical
Diversity 1: The paraphrase shows a limited use of words and lacks diversity
when compared to the source text. 2: The paraphrase exhibits some variation in
word choice but heavily relies on a few specific terms, which may not reflect
the lexical diversity of the source text. 3: The paraphrase demonstrates
moderate diversity in vocabulary, but there is room for improvement in terms
of incorporating more varied word choices from the source text. 4: The
paraphrase displays a good range of vocabulary, utilizing several different
words and expressions that align with the lexical diversity of the source
text. 5: The paraphrase showcases an extensive array of vocabulary,
demonstrating excellent lexical diversity that closely matches or surpasses
the richness of the source text. Syntactic Diversity: This aspect assesses the
structural variations in the paraphrase compared to the source text. Rating
Scale for Syntactic Diversity 1: The paraphrase closely mirrors the sentence
structure of the source text with minimal variation. 2: The paraphrase shows
some minor changes in sentence structure but largely follows the same pattern
as the source text. 3: The paraphrase introduces moderate variations in
sentence structure, deviating from the structure of the source text in certain
aspects. 4: The paraphrase exhibits significant syntactic diversity, using
different sentence structures while still conveying the same meaning as the
source text. 5: The paraphrase displays a high level of syntactic diversity,
employing various sentence structures creatively while maintaining the meaning
of the source text. Grammatical Correctness: This evaluates the grammatical
accuracy of the paraphrase. Rating Scale for Grammatical Correctness 1: The
paraphrase contains numerous grammatical errors that significantly impact
comprehension. 2: The paraphrase has several grammatical errors that
occasionally affect understanding. 3: The paraphrase includes some grammatical
errors, but they do not hinder overall comprehension. 4: The paraphrase
demonstrates good grammatical correctness with only occasional minor errors.
5: The paraphrase is grammatically flawless, with no errors or inaccuracies.
Please provide your ratings for each aspect using the following json format:
{”Semantic Similarity”: [Rating from 1 to 5], ”Lexical Diversity”: [Rating
from 1 to 5], ”Syntactic Diversity”: [Rating from 1 to 5], ”Grammatical
Correctness”: [Rating from 1 to 5]}

Figure A1: This figure illustrates the prompt fed to the gpt-4 model for
evaluation.

##  Appendix B Appendix B

Figure A2: Instructions given to Human Evaluators.