# Domain-Adaptive Text Classification with
Structured Knowledge from Unlabeled Data

Tian Li
Peking University
<EMAIL_ADDRESS>
&Xiang Chen∗
Peking University
<EMAIL_ADDRESS>
&Zhen Dong
University of California, Berkeley
<EMAIL_ADDRESS>
&Weijiang Yu
Sun Yat-sen University
<EMAIL_ADDRESS>&Yijun Yan
University of California, Berkeley
<EMAIL_ADDRESS>
&Kurt Keutzer
University of California, Berkeley
<EMAIL_ADDRESS>
&Shanghang Zhang
Peking University
<EMAIL_ADDRESS>Equal contribution.Corresponding author. (E-mail:
<EMAIL_ADDRESS>
###### Abstract

Domain adaptive text classification is a challenging problem for the large-
scale pretrained language models because they often require expensive
additional labeled data to adapt to new domains. Existing works usually fails
to leverage the implicit relationships among words across domains. In this
paper, we propose a novel method, called Domain Adaptation with Structured
Knowledge (DASK), to enhance domain adaptation by exploiting word-level
semantic relationships. DASK first builds a knowledge graph to capture the
relationship between pivot terms (domain-independent words) and non-pivot
terms in the target domain. Then during training, DASK injects pivot-related
knowledge graph information into source domain texts. For the downstream task,
these knowledge-injected texts are fed into a BERT variant capable of
processing knowledge-injected textual data. Thanks to the knowledge injection,
our model learns domain-invariant features for non-pivots according to their
relationships with pivots. DASK ensures the pivots to have domain-invariant
behaviors by dynamically inferring via the polarity scores of candidate pivots
during training with pseudo-labels. We validate DASK on a wide range of cross-
domain sentiment classification tasks and observe up to 2.9% absolute
performance improvement over baselines for 20 different domain pairs. Code
will be made available at https://github.com/hikaru-nara/DASK.

##  1 Introduction

Figure 1: Language models perform worse when domain shift is present. The
figure shows the cross-validation results of BERT baseline models trained on 5
domains. The prediction accuracy of the models tested on the trained domain is
5-15% higher than those tested on distant domains.

Domain shift is common in many natural language processing (NLP) applications.
For example, the word “rechargeable” is much more common in electronics
product reviews than in book reviews, while the word “readable” is much more
common in book reviews. Existing language models devlin2018bert ;
liu2019roberta  have exhibited outstanding performance in text classification
tasks, but they fail to generalize to new domains without expensive labeling
and retraining (Figure 1). To break out the data constraint, some methods with
unlabeled data have been proposed as follows.

Existing unsupervised domain adaptation methods for text classification can be
grouped into two categories: task-agnostic methods Long2015Learning ;
ganin2016domain ; li2020cross ; glorot2011domain  and pivot-based methods.
Task-agnostic methods generally ignore the correlation among words across
domains, which can contain rich semantic information in an NLP context. In
contrast, pivot-based methods use domain-independent words (pivots) to bridge
the domain gap by leveraging the correlations between pivots and non-pivots to
learn domain-invariant features. Therefore, we would like to marry pivot-based
method and pretrained language models to adapt them to novel domains.

The most prominent pivot-based methods are Structure Correspondence Learning
(SCL) and its variants blitzer2006domain ; yu2016learning ; ziser-
reichart-2018-pivot . In SCL, the pivots are defined as the words that occur
frequently on both source and target domains and behave in similar ways that
are discriminable for the classification task111In the existing literature,
the definition of pivots often uses the term “behavior”. To put it simply, the
pivots should be highly correlated to one of the labels across domains.. The
model can effectively learn domain-invariant features for pivots, but it is
more challenging for the non-pivots as they have domain-specific meanings.
Therefore a self-supervised auxiliary task is applied to predict the pivots
from the non-pivots. As a result, SCL implicitly captures the relationships
between words by recognizing co-occurrence patterns between pivots and non-
pivots and uses these relationships to infer domain-invariant features for the
non-pivots.

However, SCL is limited in that it uses all non-pivots to predict the pivot
terms, which leads to a noisy inference problem as very few non-pivots have a
real relationship with the pivots. As a result, false correlations often occur
for frequently used words such as pronouns (see Figure 2(a)). Alternatively,
Knowledge Graphs (KG) is an effective way to represent complex relationships
between concepts in a structured format and do not solely rely on noisy co-
occurrence information. Therefore, in this paper, we present a pivot-based
domain adaptation method from the KG perspective. Our method, called Domain
Adaptation with Structured Knowledge (DASK), follows a 2-step approach as
illustrated in Figure 2(b). In contrast to SCL, DASK filters out false
correlations by building a knowledge graph to explicitly capture the
relationships between pivots and non-pivots on the target domain. Then during
training, DASK employs a novel knowledge injection (KI) mechanism for the
model to learn domain-invariant features of the non-pivots.

Another critical drawback of SCL is that the pivots are pre-defined only on
labeled source domain texts and _unlabeled_ target domain texts. There is
little to ensure that the pre-defined pivots actually have consistent behavior
across domains. To tackle this problem, DASK dynamically learns the pivots
during training with pseudo-labels. A memory bank is maintained to keep track
of the polarity scores of the candidate pivots on the source domain and the
target domain, respectively. The words that have consistently high scores in
the memory banks are collected as the pivots at the beginning of each training
epoch. As we show, DASK using learned pivots outperforms the static, pre-
defined pivots. To summarize, our major contributions are as follows:

  * •

We propose DASK for text classification, which injects knowledge graph facts
to better leverage the relationship between words for adaptation across
domains.

  * •

We construct a novel knowledge graph with attention scores from BERT to
explicitly capture the relationship between pivots and non-pivots on the
target domain.

  * •

We design the pivot-induced cross-domain knowledge injection mechanism to
learn domain-invariant features for non-pivots using their relationship with
pivots.

  * •

We propose to maintain memory banks to learn domain-invariant pivots.

  * •

We evaluate our method on the task of cross-domain sentiment classification
for 20 domain pairs, where our method outperforms strong baselines by up to
2.9%.

(a) Structure Correspondence Learning.

(b) Two-step approach of DASK: extract and inject

Figure 2:  a) The example shows a pair of texts from the source domain (top)
and target domain (bottom) respectively. Due to the frequency of “it” co-
occurring with “great”, the model tends to capture this false correlation. b)
In contrast, DASK extracts a fact, represented by a triplet (great, make,
simple), from the target domain text to filter false correlations. We inject
the target domain fact into the source domain text.

##  2 Related Work

###  2.1 Cross-Domain Text Classification

Cross-domain text classification glorot2011domain ; ganin2016domain ; ziser-
reichart-2018-pivot ; du2020adversarial  is a fundamental problem in domain
adaptation for NLP. Before applying domain adaptation methods to complex
problems, like machine translation and question answering, it was extensively
researched as a simpler problem. Among text classification tasks, sentiment
classification is most representative and established in terms of domain
adaptation because there are standard datasets and evaluation protocols
blitzer-etal-2007-biographies ; ziser-reichart-2018-pivot . Therefore, we use
sentiment classification to evaluate our method. It is important to note that
our method can be easily generalized to other tasks such as topic labeling,
news classification and so forth.

Methods for cross-domain text classification can be roughly categorized into
two classes: task-agnostic methods, and pivot-based methods. The former
includes divergence minimization Long2015Learning ; sun2016return ;
he2018adaptive , stacked denoising auto-encoders glorot2011domain ,
discriminative adversarial training ganin2016domain ; tzeng2017adversarial ,
instance reweighting chen2021wind  and so forth. There are also some works
that combine task-agnostic methods with NLP-specific approaches or models
ghosal2020kingdom ; du2020adversarial . In contrast, pivot-based methods are
different from them in two aspects: 1) they make use of the relationship
between pivots and non-pivots to help learn discriminative feature, 2) they
focus on learning word-level features instead of instance-level features,
which align with the nature of language. These two points make pivot-based
methods stand out on NLP tasks.

###  2.2 Knowledge Graph

##### Knowledge Graph Construction.

KG can be constructed in a supervised manner, such as DBPedia
lehmann2015dbpedia  and Wikidata vrandevcic2014wikidata ; semi-supervised
manner such as Google’s Knowledge Vault dong2014knowledge ; or unsupervised
manner, such as MAMA wang2020language . Among them, we are most interested in
MAMA because it uses learned knowledge stored in pre-trained language models
without human supervision to construct a KG. With a forward pass of BERT, it
collects the words or phrases in the input that have high attention scores
with each other to form a candidate fact, which is represented by a triplet
_(head, relation, tail)_. Hence it is an easy fit for our knowledge-injected
language model P-BERT. We can initialize P-BERT with the same weights from
BERT used in MAMA, so that it becomes easier for P-BERT to interpret the
knowledge graph facts that are constructed by MAMA.

##### Knowledge-Injected Transformers.

Since the emergence of pre-trained language models like BERT devlin2018bert ;
liu2019roberta , many works have followed up to incorporate external knowledge
into them wang2020k ; zhang2019ernie ; wang2021kepler ; liu2020k . Typically,
these works either fuse textual features and factual features at the end of
the corresponding encoders to enhance language understanding wang2021kepler ;
zhang2019ernie , or use adapters for injection of multiple knowledge sources
with the original model fixed wang2020k . Although most of them tried to
utilize open-domain knowledge graphs or knowledge bases, K-BERT liu2020k
takes domain-specific knowledge as input along with text, and thus is
particularly suitable for our task of domain adaptation. Also, by using a
flattened sentence tree and the visible matrix, it is compatible to BERT and
free from additional pre-training. Therefore, to promote domain transfer with
pivots, we develop K-BERT into P-BERT by replacing its KI module with our
Pivot-induced Cross-domain Knowledge Injection.

Beyond knowledge-injected transformers, another line of related works is tree-
based transformers for processing tree-structured data in general
Shiv2019NovelPE ; sun2020treegen . Despite their power in learning from
arbitrary tree-structured data, they are not directly compatible to BERT.

Figure 3: Illustration of DASK. DASK consists of two steps. In step 1 we
construct a knowledge graph from target domain unlabeled data. In step 2 we
finetune the model on knowledge-injected data and learn the pivots with memory
bank.

##  3 Method

DASK aims to facilitate domain adaptation with structured knowledge from the
target domain. To structure the knowledge in the target domain, we build a
knowledge graph to capture the relationship among pivots and non-pivots and
inject them into source domain data. This is helpful because the model is able
to learn domain-invariant feature for non-pivots (domain-specific) by
inferring from their relationship with domain-shared pivots. Specifically,
DASK follows a 2-step approach as follows (see Figure 3):

##### Step 1.

We construct a knowledge graph (KG) from the target domain texts to model the
relationship between pivots and non-pivots. Specifically, it involves 1)
extracting candidate facts from sentences, 2) filtering low-confidence facts.
Section 3.1 shows details of how we construct the KG.

##### Step 2.

After constructing the knowledge graph, we start knowledge-injected finetuning
on the source and target domains jointly. At each training step, we obtain
labeled texts from the source domain, unlabeled texts from the source domain
and target domain. The inputs are fed into P-BERT where all of them are
injected with pivot-relevant facts from the knowledge graph (section 3.2), and
then forwarded to the transformer encoder and the linear classifier
sequentially. Finally, the predictions of labeled data are used to compute the
cross-entropy loss with the ground truth labels. Meanwhile, the predictions of
unlabeled data are leveraged to generate pseudo-labels for pivot learning
(section 3.3). The pseudo-labels of unlabeled data and ground truth labels of
labeled data give updates to the polarity scores in the memory banks. Note
that during inference the memory bank is not updated.

###  3.1 Knowledge Graph Construction

Prior to training, we build a KG on the target domain corpus to capture the
relationship between pivots and non-pivots.

##### Candidate Fact Extraction.

We extract candidate facts on a sentence level. An input text is decomposed to
a list of sentences. We feed each sentence into BERT to get the attention
matrix in the last transformer encoder layer. As a pre-processing step, the
multi-head attention is averaged into single-head so that one pair of words
only corresponds to one scalar attention value (if a word consists of multiple
tokens, the attentions of the tokens are also averaged). Denote the pre-
processed attention matrix as $M$. For each pivot $p$ in the sentence, we
search for the words $w_{1},w_{2}$ that have the highest and second highest
attention with $p$. Then the fact is formed as the triplet of $w_{1},w_{2},p$
in their original order in the sentence. In addition, each fact is assigned a
confidence score $M[p][w_{1}]+M[p][w_{2}]$.

##### Filtering.

The candidate facts are filtered according to their confidence scores. Those
whose confidence scores are under a threshold are removed from the knowledge
graph.

Note that although we learn pivots dynamically during training, those pivots
all come from a large pivot pool that is constructed before training (see
section 3.3). We construct the knowledge graph according to the pivot pool so
that we do not have to update the knowledge graph as we learn new pivots and
eliminate old ones. In the appendix, we showcase some qualitative results of a
KG that we constructed in this way and compare it with the ConceptNet
subgraph.

###  3.2 Knowledge Injection

In order to utilize the rich semantics in the KG to learn domain-invariant
features for the non-pivots, we propose Pivot-induced Cross-domain Knowledge
Injection (PCKI) to inject knowledge facts into input texts. On the other
hand, we would like to use a pretrained language model as the feature
extractor, so we borrow ideas from K-BERT liu2020k  to let the language model
understand the structure of knowledge injected input. As a whole, our model
P-BERT is composed of the knowledge injection module and the subsequent
transformer encoder, as illustrated in Figure 3.

In the knowledge injection module, for each pivot in the input text, we search
for the facts relevant to it and inject them into the text, forming a tree
structure. Figure 2(b) shows an example for a knowledge injected text. In the
example, “Great” is a pivot in the sentence on the right, and we have the fact
triplet “(great, makes, simple)” in the KG that we extract from the sentence
on the left. The triplet is then appended to “great” as a branch. By injecting
facts extracted from the target domain into labeled source domain texts, we
embed the target domain non-pivots in a labeled context, conditioning on the
related pivots. Therefore, our transformer encoder is able to infer their
features according to their relationships with pivots, under the supervision
of source domain labels.

To feed the knowledge-injected text to the transformer encoder while keeping
the structure information, we flatten it into a token sequence, and use
position embedding to recover its structure liu2020k . Specifically, as
illustrated in Figure 4, the injected words are inserted before or after the
corresponding pivot following their order in the triplet so as to get the
flattened token sequence. Meanwhile, to recover the tree structure, we assign
the index for position embedding of each token as its depth in the tree. In
the example, “makes” and “book” are assigned the same position embedding index
2. In this way, the token sequence is fed into the transformers, with the
position embedding informing its structure. Besides, a visible matrix is
applied to the self-attention modules in the transformers to control knowledge
noise. We refer readers to K-BERT liu2020k  for more details on visible
matrix.

Figure 4:  Continuing the example in Figure 2(b), we inject the fact triplet
_(great, makes, simple)_ to the main sentence forming a tree structure (left).
On the right, we flatten the tree into a sequence and use the depth of the
tokens in the tree as their position embedding index. The highlighted words in
orange are the injected fact.

###  3.3 Pivot Learning

As a pivot-based method, DASK heavily relies on the domain-shared pivots. In
this section, we describe how we select and learn the pivots with memory
banks.

Recall that pivots are defined as the words that behave similarly in the
source and target domains. Following previous works, we represent the behavior
of a word with the labels of the texts in which it appears. The principle is
that the label distribution of a pivot should be low-entropy (biased towards
one label), and consistent across domains. While it is applicable to most
classification tasks, for the binary sentiment classification task we focus
on, we define a polarity score $p(w,D)$ to measure more easily how much the
label distribution of $w$ is biased towards the labels on domain $D$:

$p(w,D)=\frac{\left|\\{l=1|l\in b(w,D)\\}\right|-\left|\\{l=-1|l\in
b(w,D)\\}\right|}{|b(w,D)|}$ (1)

where $b(w,D)$ is the label set of $w$ on $D$. And thus the behaviour on the
domain pair $(S,T)$ can be characterized as:

$\bar{p}(w,(S,T))=\frac{|p(w,S)+p(w,T)|}{2}$ (2)

Taking the absolute average ensures that a word of high score must be biased
towards the same label on both domains.

Prior to training, we collect a large pool of candidate pivots from the
vocabulary which contains words that appear frequently on both domains. Other
words are not taken into account because the precondition is that pivots
should be frequent on both domains. To track the polarity scores of the
candidate pivots, we build memory banks for the source domain and the target
domain respectively. Both of them are initialized by scores computed with
equation 1 constrained on source domain labeled data. Then we compute the
absolute mean polarity score of the candidate pivots following equation 2 and
take the $K$ candidate pivots with the highest score as the initial pivots.

During training, at each training step, we acquire pseudo-labels for the
unlabeled texts if the prediction confidence in the softmax logits is over a
threshold. The pseudo-labels of the unlabeled source domain inputs, and the
ground truth labels of the labeled source domain inputs are used to update the
source memory bank, while the pseudo-labels of the target domain inputs are
used to update the target memory bank. The update is carried out in a temporal
difference style: For each candidate pivot, if it is in the text from domain
$D$ labeled as $l\in\\{1,-1\\}$, then

$p(w,D)\leftarrow\alpha\cdot p(w,D)+(1-\alpha)\cdot l$ (3)

where $\alpha$ is the update rate. In this way, we manage to estimate the
behavior of words more accurately on the domains, so that domain-invariant
pivots are acquired. At the beginning of each epoch, we compute the absolute
mean scores of the candidate pivots following equation 2 and take the top-$K$
candidate pivots as learnt pivots. The pivots are kept fixed in the middle of
an epoch to avoid overly frequent pivot changes.

##  4 Experiments

To evaluate the efficacy of our proposed method, we extensively experiment on
two cross-domain sentiment classification datasets. We compare our method with
multiple baselines on 20 domain pairs and provide justification for the
improvement of our method. Moreover, we carry out ablation study to
respectively assess the effect of knowledge injection and dynamic pivot
learning.

###  4.1 Experiment Settings

##### Datasets.

We use the standard Amazon-product-review222Dataset can be found at
http://www.cs.jhu.edu/~mdredze/datasets/sentiment/index2.html dataset blitzer-
etal-2007-biographies . It contains four types of product reviews: books(B),
dvd(D), electronics(E), kitchen(K), which form the four domains in the
dataset. Besides, following PBLM ziser-reichart-2018-pivot , we also introduce
the Airlines dataset333Dataset and process procedures can be found at
https://github.com/quankiquanki/skytrax-reviews-dataset.

S$\rightarrow$ T | BERT
---|---
Base | HATN | DANN | DAAT | DASK | DASK+SCL
B$\to$ E | 90.50 | 87.21 | 91.67 | 89.57 | 91.95 | 92.30
B$\to$ D | 90.45 | 89.36 | 89.93 | 89.70 | 90.55 | 90.90
B$\to$ K | 92.46 | 89.41 | 92.80 | 90.75 | 92.85 | 92.75
E$\to$ B | 89.85 | 87.10 | 89.19 | 88.91 | 89.70 | 90.00
E$\to$ D | 88.30 | 88.81 | 88.49 | 90.13 | 88.65 | 89.20
E$\to$ K | 94.20 | 92.01 | 94.54 | 93.18 | 94.35 | 94.65
D$\to$ B | 90.75 | 89.81 | 91.37 | 90.86 | 91.20 | 91.85
D$\to$ E | 91.30 | 86.99 | 91.52 | 89.30 | 88.70 | 92.40
D$\to$ K | 90.85 | 87.59 | 92.16 | 90.50 | 91.80 | 92.35
K$\to$ B | 88.50 | 89.36 | 89.38 | 87.98 | 90.15 | 89.75
K$\to$ E | 93.34 | 90.31 | 93.15 | 91.72 | 92.80 | 93.35
K$\to$ D | 87.90 | 87.89 | 88.89 | 88.81 | 88.40 | 89.45
_Average_ | 90.70 | 88.69 | 91.09 | 90.12 | 90.92 | 91.59
Table 1: Cross-domain sentiment classification accuracy on 12 domain pairs
from Amazon-product-review dataset. Our method is able to outperform all the
strong baselines on all domain pairs with the only exception of E$\to$D. For
BERT-HATN and BERT-DAAT we use numbers reported by du2020adversarial .

S$\rightarrow$ T | BERT
---|---
Base | DANN | DASK | DASK+SCL
A$\to$ B | 81.65 | 81.50 | 82.10 | 84.15
A$\to$ E | 88.85 | 89.53 | 89.35 | 89.10
A$\to$ D | 80.60 | 82.74 | 83.15 | 82.85
A$\to$ K | 89.50 | 89.53 | 89.90 | 90.00
B$\to$ A | 86.18 | 86.66 | 86.30 | 86.70
E$\to$ A | 87.60 | 87.90 | 87.30 | 87.90
D$\to$ A | 84.55 | 86.71 | 84.85 | 86.75
K$\to$ A | 86.30 | 86.56 | 86.50 | 86.80
_Average_ | 85.65 | 86.39 | 86.12 | 86.78

(a)

PCKI | SCL | Dynamic | Accuracy
---|---|---|---
A$\to$ B | B$\to$E | E$\to$K | K$\to$D | D$\to$A
|  |  | 81.65 | 90.50 | 94.20 | 87.90 | 84.55
| ✓ |  | 81.85 | 90.95 | 94.40 | 87.95 | 84.90
✓ |  |  | 82.10 | 91.90 | 94.25 | 88.20 | 84.75
✓ | ✓ |  | 82.95 | 92.05 | 94.60 | 88.40 | 86.05
✓ | ✓ | ✓ | 84.15 | 92.30 | 94.65 | 89.45 | 86.75

(b)

Table 2: a) Cross-domain sentiment classification accuracy on the 8 domain
pairs between airlines dataset and 4 domains from Amazon-product-review
dataset. b) Ablation study on PCKI, SCL and dynamic memory bank. We did
experiments on 5 domain pairs.

We evaluate our method on all 20 domain pairs involving the five domains. Each
time before training, we randomly sample 400 labeled source domain data for
dev set, and the rest 1600 along with all unlabeled data from source and
target domain are used for training. During testing on the 2000 target labeled
data, we stop the memory bank update, and use the pivots learnt at the best
training epoch.

##### Baselines.

We use BERT devlin2018bert  as the feature extractor. Apart from directly
finetuning BERT on the source domain, we compare our method with multiple
strong baselines incorporating methods proposed by previous works. The
baselines are listed as follows:

  * •

Base: BERT trained on the source labeled data and directly test on the target
labeled data.

  * •

HATN: BERT combined with HATN proposed by li2018hierarchical . It is a
prominent variant of Structure Correspondence Learning (SCL).

  * •

DANN: BERT combined with the popular adversarial training method, DANN
ganin2016domain .

  * •

DAAT: BERT-DAAT proposed by du2020adversarial . It combines target domain
post-training along with domain adversarial training to boost domain
adaptation.

To show the compatibility of DASK with existing methods, we stack SCL on top
of our method. Since there was no existing work applying SCL on transformers,
we manufactured a training scheme for SCL that mimics MLM in BERT
devlin2018bert  pretraining, that is, to replace the pivots as [MASK] tokens
and ask the model to recover the masked tokens given the context.

##### Hyperparameter Tuning.

For all the methods in our experiments, we set the learning rate to 2e-5,
warmup $0.1$, batch size 32, and select weight decay in $\\{\text{1e-4, 2e-4,
3e-4}\\}$. For adversarial training, we select $\gamma$ in the gradient
reversal layer from $\\{0.15,0.25,0.5,0.75,1.0\\}$. For KG fact filtering, we
set the confidence threshold in the interval $[0.1,0.45]$, according to the
confidence distribution. Besides, we leverage the stopwords in the NLTK
library and discard the facts that include the stopwords. For SCL, we apply a
balance factor $\lambda$ to the pivot-prediction loss, and choose its value
from $[0.1,0.5]$. To make the learning progress smoother, we also use a linear
warmup to $\lambda$ and set the warmup rate to $0.1$. In addition, since our
sentiment classifier is trained on the source labeled data, and the size of
labeled data is 6-20 times smaller than that of unlabeled data, our model see
each labeled data for 6-20 times in one epoch. In order to prevent
overfitting, we update the sentiment classifier once in 5-11 training steps.
For the dynamic memory bank, we update the pivots every $10$ training steps,
and the top 500 words are selected as pivots.The learning rate of word
sentiment score is set to 1e-4 or 2e-4, and the pseudo-labeling confidence
threshold to $0.9$.

###  4.2 Experimental Results

Table 1 shows the performance of our method on the 12 domain pairs from
Amazon-product-review dataset, compared with multiple baselines.

Among all those methods, we observe that DASK+SCL gives the best performance
on average. It outperforms the Base model by 0.89%, and beats the other strong
baselines by 0.50%-2.90% on an average basis over all domain pairs. On the
other hand, DASK also has competitive performance. It is better than DAAT and
HATN by 0.8% and 1.43%, outperforms the Base Model by 0.22%. Although its
average accuracy is slightly lower than DANN, it gives the best performance on
the B$\to$ K and K$\to$B settings.

Moreover, we conduct experiments on 8 domain pairs involving the Airlines
dataset. Table 2(a) shows the performance of our method on those settings.
Although this setting is more difficult than domain pairs within Amazon-
product-review dataset, we are still able to outperform Base on all of the
domain pairs, by 1.13% on average. This shows that, by exploiting correlation
among words, our method can help the model capture relationship between pivots
and non-pivots even from distant domains. Besides, our method betters DANN by
0.39%. Although this is not a thick margin, it is non-trivial because 1) DANN
itself is a very competitive method, 2) we do not expect to beat DANN by any
larger margin, but to propose a strong baseline for domain adaption from a KG
perspective, 3) our method is orthogonal to DANN and thus can be coupled with
DANN to achieve even better performance.

Note that our Base model performs better than the previous state-of-the-art
BERT-DAAT. This owes to hyperparameter tuning where BERT-DAAT uses 1e-2 for
weight decay while we use 1e-4. It is not open-sourced so we cannot reproduce
their results for better comparison. Nevertheless, it does not mean our
performance gain comes from hyperparameter tuning as we analyze in the
ablation studies below.

KI method | KG | Accuracy
---|---|---
normal | subgraph | 80.45
normal | learnt | 77.45
PCKI | subgraph | 79.50
PCKI | learnt | 82.10
Base | 81.65
Table 3: Ablation studies on knowledge injection mechanism and KG construction
method. All experiments are done on the A$\to$B domain pair.

##### Ablation Study.

As a sanity check, we performed experiments to study the effect of our choices
of KI method and the knowledge graph. We compare PCKI versus normal KI method
which inserts knowledge to every word possible, and our learnt KG versus the
ConceptNet subgraph. From table 3, it can be observe that none of the
configurations works other than PCKI plus learnt KG, our proposed method.

More importantly, to analyze how much each part of our method contribute to
the performance gain, we conduct ablation study on three aspects: PCKI, SCL
and pivot learning (Dynamic). We did experiments on 5 domain pairs as shown in
table 2(b). From the results, we have the following observations: 1) PCKI and
SCL are both able to help DA independently, and PCKI generally works better
than SCL, 2) Jointly applying them can further boost the performance, which
implies that they complement each other in the ability of exploiting the
correlation among pivots and non-pivots, 3) dynamically learning the pivots
improve the performance by a large margin compared to using a set of pre-
defined pivots. This supports that the learnt pivots are more beneficial to
domain adaptation than the pre-defined ones.

###  4.3 Qualitative Results

Entity | ConceptNet Subgraph | Learnt KG
---|---|---
great | (great, related to, good) | (looks, surprisingly, great)
_(great, related to, alexander)_ | (great, is, awesome)
(great, similar to, large) | (great, is, excellent)
(mega, related to, great) | (great, save, $)
(great, related to, awesome) | (really simple, got, great)
_(rocking, related to, great)_ | (easy, seems, great)
_(lies, related to, great)_ | (also, looks, great)
simple | _(simple, related to, unsophisticated)_ | (simple, is, amazing)
_(five needled, similar to, simple)_ | (charging, simple, quick)
(simpler, form of, simple) | (excellent condition, putting, simple)
_(simple, synonym, unsuspecting)_ | (plain, and, simple)
_(cakewalk, related to, simple)_ | (the setup, fairly, simple)
(easy, related to, simple) | (amazon, simple, fast)
(plain, related to, simple) | (makes, it, simple)
Table 4: Visualization of the triplets in learnt knowledge graph compared to
Conceptnet subgraph. Both graphs are extracted for the domain pair B$\to$E.
The bold triplets indicate a relationship between non-pivots on the E domain
and pivots between B$\to$E domain pair. The _italicized_ triplets are
knowledge noise that irrelevant to the target domain. The results show that
our learnt knowledge graph better models the relationships between pivots and
relevant non-pivots and avoids irrelevant knowledge noise.

####  4.3.1 Knowledge Graph

To demonstrate the advantage of the learnt KG over the ConceptNet subgraph in
modeling relationship between pivots and non-pivots for domain adaptation, we
visualize an example of ConceptNet subgraph and our KG in Table 4. Each cell
contains the triplets involving the concept on the left. For example, the top-
left grid contains the triplets in ConceptNet subgraph that involves “great”,
etc. In the table, the ConceptNet subgraph is the one-hop subgraph induced by
words on the electronics (E) domain corpus and the learnt KG is the result of
applying our method described in 3.1 to the B$\to$E domain pair. Here, we
consider the concepts “great” and “simple” because “great” is a pivot that
carries positive sentiment, while “simple” is a positive non-pivot on E domain
that carries positive sentiment (“simple” could mean “unsophisticated” in book
reviews, which is a negative word for stories.). We want to examine how our
learnt KG explicitly models the relationship between pivots and non-pivots and
therefore benefits domain adaptation with PCKI. From table 4, we notice the
following facts:

In both ConceptNet subgraph and our KG, “great” is related to other positive
pivots (good, awesome, large, excellent), and some domain-specific concepts.
However, in ConceptNet subgraph, the related domain-specific concepts is not
necessarily related to electronic products. This is because ConceptNet is a
commonsense KG extracted from open-domain corpus such as Wikipedia. Therefore,
concepts like “alexander”, “rocking”, and “lies” that are irrelevant to
electronics would introduce noise when used in knowledge injection.

In contrast, our KG only contains concepts that occur in electronics domain
corpus, which avoids knowledge noise. Besides, in our KG, “great” is usually
related to other sentiment indicating words or phrases, such as “save $”,
“really simple”, “easy” and so forth. Notably, those words or phrases probably
does not convey consistent sentiment on source domain (Books), and thus the
model cannot learn discriminative features for them without knowledge
injection. Coupled with PCKI, these relationships will enable our model to
understand those target domain-specific concepts and to learn discriminative
features for them.

Similarly, ConceptNet relates “simple” to synonyms like “easy” and various
irrelevant domain-specific concepts such as “cakewalk” and “five needled”. In
contrast, our KG relates it to pivots such as “amazing” and “excellent
condition”. These connections between pivots and non-pivots are able to help
model learn discriminative feature with PCKI.

Moreover, our KG is more flexible and specific than ConceptNet subgraph. The
vast majority of ConceptNet subgraph relations are “related to”, “similar to”,
but the relation in our Bilevel KG has better diversity. Conventionally, the
relation set of a KG is designed by human experts and is thus relatively
limited. In contrast, we extract Bilevel KG with a language model from the
domain corpus. Although some of the extracted relations are not verb, they
actually indicate anonymous relationship with complex semantic meaning that
should not be plausibly marked with “related to”. For example, (great, save,
$) actually indicates anonymous relationship between “great” and “save $”.
Those anonymous relationships greatly enrich the content of the knowledge
graph and avoids using plausible terms like “related to” which could induce
knowledge noise.

In conclusion, the ConceptNet subgraph is noisier than the learnt KG. The
anonymous relationship in the learnt KG taps the potential of an NN language
model to interpret it.

S$\to$T | Initial Pivots | Learned Pivots
---|---|---
B$\to$E | anything, _pages,_ comfortable, got, wanted, left, |  great, best, love, history, personal, lives,
completely, _paper,_ went, began, pick, seems, | involved, larger, enjoyed, trust, eye, also,
trouble, average, add, example, stand, |  thank, definitely, knowledge, honest, means,
fell, effort, self, expectations, virtually, _artist_ | leader, critical, reviewed, draw, sweet, drawing
E$\to$K | back, tried, minutes, different, year, cut, |  well, works, use, good, nice, used, also, easily,
anything, work, _charge,_ goes, service, months, |  like, recommend, quality, hand, need, music,
several, received, products, days, four, imagine, |  heavy, beautiful, far, sound, included, paper,
selling, point, hot, track, happen, something, whole, | including, fairly, taking, watch, operation, home,
went, experience, neither, ok, pass, half, _image_ | become, turning, includes, lot, transfer, connects
K$\to$D | back, money, end, thought, left, trying, second, |  well, one, every, like, good, makes, really, set,
check, coming, stand, help, alone, times, instead, | also, recommend, especially, seen, ever, beautiful,
huge, _sheets, months,_ forget, _temperature,_ count |  always, use, friend, kind, etc, _pans,_ might, _tea,_ want
seconds, _rust,_ saying, plan, _ingredients,_ twice, | mostly, wife, food, almost, let, version, long, heat,
picture, _loaf,_ putting, _delivery, turned, brown,_ went, | much, handles, glasses, party, mix, duty,
behind, directions, fair, correct, elsewhere, sort | green, come

Table 5: Qualitative results of dynamically learnt pivots at the end of
training compared to the initial pivots. To be clear and concise, we do not
show their intersection but only show their difference. Bold ones are the
words we would probably deem as pivots from human instinct. _Italicized_ ones
are the words that bias to the source domain, i.e. source domain-specific
concepts. This visualization demonstrates that the pivots that we learn from
dynamic memory banks are more consistent with human sense and more domain-
general than those pre-defined by rules in previous works.

###  4.4 Dynamically Learned Pivots

To demonstrate the effect of pivot learning, we visualize the initial pivots
and our learnt pivots on B$\to$E, E$\to$K and K$\to$D domain pairs in table 5.
The “initial pivots” are the pivots from the initialized memory banks before
training. Note that they are identical to the pre-defined pivots used by
previous works ziser-reichart-2018-pivot . The “learnt pivots” are the pivots
from the memory banks at the end of training (see section 3.3). We evaluate
the quality of a set of pivots by two criteria. Firstly, we expect the pivots
are sentiment indicating, which means that we expect the pivots to have
significant sentiment polarities. Secondly, to promote the domain transfer, we
aim to prevent the pivots from being biased to the source domain. We report
the difference of the two sets of pivots in table 5 without cherry picking.
From table 5, we have the following observations:

  * •

Bold words are some typical pivots that we judge from human instinct. The
number of this kind of words, such as “great”, “best” and “love”, is greatly
increased by dynamically learning the pivots with the memory bank.

  * •

Italicized words are some source domain-specific concepts which are biased to
the source domain and may degrade the performance in the target domain. We
observe less source domain-specific concepts after pivot learning. This shows
that our method is able to learn domain-general pivots in order to better
promote the domain transfer.

These observations owe to our dynamic memory banks, which maintain the
polarity scores of the common words between the two domains. When computing
the initial polarity scores of the words, only labeled data on the source
domain is available, which makes the induced pivots prone to bias to the
source domain. In contrast, our memory bank is dynamically updated during
training not only with pseudo-labels of the unlabeled text on both source and
target domains, but also with the ground-truth labels of the labeled text.
Therefore, our memory banks obtain comprehensive understandings of the words
during training, and thus the learnt pivots will be more accurate and less
biased to the source domain.

##  5 Conclusion

In this paper, we proposed Domain-Adaptive text classification with Structured
Knowledge (DASK) which elegantly marries pivot-based methods, knowledge graph
and pretrained language models. It builds a knowledge graph from the target
domain data to model the relationship between pivots and non-pivots. During
training, DASK injects knowledge into the input sentence by means of pivot-
induced cross-domain knowledge injection, and P-BERT is proposed to encode the
knowledge-injected sentence. In this way, DASK learns domain invariant
features by capturing the relationship between pivots and target domain-
specific non-pivots. We conducted experiments on cross-domain sentiment
classification and showed that DASK outperformed strong baselines in
prediction accuracy by up to 2.9% on 20 domain pairs. Despite that our
experiments are focused on sentiment classification, DASK generalizes to other
cross-domain text classification tasks, as long as pivots are well-defined for
the task. We reserve it as future work to extend DASK to other tasks.

## References

  * [1] John Blitzer, Mark Dredze, and Fernando Pereira.  Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification.  In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, 2007.
  * [2] John Blitzer, Ryan McDonald, and Fernando Pereira.  Domain adaptation with structural correspondence learning.  In Proceedings of the 2006 conference on empirical methods in natural language processing, 2006.
  * [3] Xiang Chen, Yue Cao, and Xiaojun Wan.  Wind: Weighting instances differentially for model-agnostic domain adaptation.  In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2366–2376, 2021.
  * [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [5] Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang.  Knowledge vault: A web-scale approach to probabilistic knowledge fusion.  In SIGKDD, 2014.
  * [6] Chunning Du, Haifeng Sun, Jingyu Wang, Qi Qi, and Jianxin Liao.  Adversarial and domain-aware bert for cross-domain sentiment analysis.  In ACL, 2020.
  * [7] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky.  Domain-adversarial training of neural networks.  The journal of machine learning research, 2016.
  * [8] Deepanway Ghosal, Devamanyu Hazarika, Abhinaba Roy, Navonil Majumder, Rada Mihalcea, and Soujanya Poria.  Kingdom: Knowledge-guided domain adaptation for sentiment analysis.  In ACL, 2020.
  * [9] Xavier Glorot, Antoine Bordes, and Yoshua Bengio.  Domain adaptation for large-scale sentiment classification: A deep learning approach.  In ICML, 2011.
  * [10] Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier.  Adaptive semi-supervised learning for cross-domain sentiment classification.  arXiv preprint arXiv:1809.00530, 2018.
  * [11] Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, et al.  Dbpedia–a large-scale, multilingual knowledge base extracted from wikipedia.  Semantic web, 2015.
  * [12] Tian Li, Xiang Chen, Shanghang Zhang, Zhen Dong, and Kurt Keutzer.  Cross-domain sentiment classification with contrastive learning and mutual information maximization.  In ICASSP, 2021.
  * [13] Zheng Li, Ying Wei, Yu Zhang, and Qiang Yang.  Hierarchical attention transfer network for cross-domain sentiment classification.  In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
  * [14] Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang.  K-bert: Enabling language representation with knowledge graph.  In Proc. of AAAI, 2020.
  * [15] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [16] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan.  Learning transferable features with deep adaptation networks.  In ICML, 2015.
  * [17] Vighnesh Leonardo Shiv and Chris Quirk.  Novel positional encodings to enable tree-based transformers.  In NeurIPS, 2019.
  * [18] Baochen Sun, Jiashi Feng, and Kate Saenko.  Return of frustratingly easy domain adaptation.  In AAAI, 2016.
  * [19] Zeyu Sun, Qihao Zhu, Yingfei Xiong, Yican Sun, Lili Mou, and Lu Zhang.  Treegen: A tree-based transformer architecture for code generation.  In AAAI, 2020.
  * [20] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell.  Adversarial discriminative domain adaptation.  In CVPR, 2017.
  * [21] Denny Vrandečić and Markus Krötzsch.  Wikidata: a free collaborative knowledgebase.  Communications of the ACM, 2014.
  * [22] Chenguang Wang, Xiao Liu, and Dawn Song.  Language models are open knowledge graphs.  arXiv preprint arXiv:2010.11967, 2020.
  * [23] Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Cuihong Cao, Daxin Jiang, Ming Zhou, et al.  K-adapter: Infusing knowledge into pre-trained models with adapters.  arXiv preprint arXiv:2002.01808, 2020.
  * [24] Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang.  Kepler: A unified model for knowledge embedding and pre-trained language representation.  ACL, 2021.
  * [25] Jianfei Yu and Jing Jiang.  Learning sentence embeddings with auxiliary tasks for cross-domain sentiment classification.  In EMNLP, 2016.
  * [26] Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu.  Ernie: Enhanced language representation with informative entities.  arXiv preprint arXiv:1905.07129, 2019.
  * [27] Yftah Ziser and Roi Reichart.  Pivot based language modeling for improved neural domain adaptation.  In Proc. of ACL, 2018.


11institutetext: School of Computing, Informatics Institute of Technology,
Colombo 00600, Sri Lanka

#  ParaFusion: A Large-Scale LLM-Driven English Paraphrase Dataset Infused
with High-Quality Lexical and Syntactic Diversity

Lasal Jayawardena and Prasan Yapa

###### Abstract

Paraphrase generation is a pivotal task in natural language processing (NLP).
Existing datasets in the domain lack syntactic and lexical diversity,
resulting in paraphrases that closely resemble the source sentences. Moreover,
these datasets often contain hate speech and noise, and may unintentionally
include non-English language sentences. This research introduces ParaFusion, a
large-scale, high-quality English paraphrase dataset developed using Large
Language Models (LLM) to address these challenges. ParaFusion augments
existing datasets with high-quality data, significantly enhancing both lexical
and syntactic diversity while maintaining close semantic similarity. It also
mitigates the presence of hate speech and reduces noise, ensuring a cleaner
and more focused English dataset. Results show that ParaFusion offers at least
a 25% improvement in both syntactic and lexical diversity, measured across
several metrics for each data source. The paper also aims to set a gold
standard for paraphrase evaluation as it contains one of the most
comprehensive evaluation strategies to date. The results underscore the
potential of ParaFusion as a valuable resource for improving NLP applications.

###### Keywords:

Paraphrase Generation, Natural Language Generation, Deep Learning, Large
Language Models, Data Centric AI.

##  1 Introduction

Paraphrase Generation, also known as Question Paraphrase Generation, is a
fundamental task and a significant area of focus in NLP. This field has been
the subject of research for several decades. Paraphrase Generation plays a
critical role in data augmentation, a process that is vital for enhancing the
performance of numerous NLP tasks. By generating diverse expressions of
identical information, it significantly enriches the training data, thereby
improving the robustness and generalization capabilities of NLP models
[1][2][3][4].

In recent years, neural-based approaches, such as sequence-to-sequence models,
have been increasingly used for paraphrase generation due to their ability to
learn complex patterns and generate fluent text. However, they require large
amounts of high-quality annotated data for training, which can be difficult
and costly to obtain. Data quality determines the capability of the model to
generate diverse paraphrases. Existing models often struggle with maintaining
the semantic equivalence between the original text and the generated
paraphrase, especially for longer and more complex sentences. These
limitations stem from the issue that the current datasets that are available
do not have high-quality paraphrases. A paraphrase to be considered a high-
quality paraphrase needs to be lexically diverse, syntactically diverse,
grammatically correct, and semantically similar. Section 4 outlines an
analysis of existing data sources that highlight this issue. Apart from
dataset quality, there is no proper evaluation strategy employed by
researchers that assesses the quality of diverse paraphrases. Most existing
work primarily use lexical metrics to determine model quality whereas the
other three components of high-quality paraphrases are often ignored [5].

In light of these challenges, ParaFusion is introduced as a more precise
English Dataset to address these issues. LLMs have gained a lot of attraction
in recent years, significantly outperforming several state-of-the-art models
(SOTA) in several domains [6]. In this paper, we employ existing datasets and
augment the data to generate high-quality paraphrases using the ChatGPT
(gpt-3.5-turbo) LLM to create ParaFusion. This paper provides a comprehensive
analysis of ParaFusion, investigating it using a range of evaluation metrics
to explore various facets of the dataset’s quality, as shown in Section 4, to
demonstrate that the paraphrases generated by ParaFusion are more diverse in
terms of syntax and lexical compared to existing datasets, while
simultaneously maintaining strong semantic similarity between paraphrased
sentences. We utilize a rigorous framework for dataset evaluation in hopes of
setting a gold standard for future paraphrase evaluation research. Such a
comprehensive strategy is needed to precisely evaluate paraphrases which is
drastically different from other Sequence-to-Sequence NLP Taks. Moreover, our
human evaluation results Section 4.3 corroborate that ParaFusion indeed offers
higher-quality paraphrases compared to previous datasets. The results suggest
a notable potential for realizing enhancements in paraphrase generation tasks,
underlining ParaFusion’s ability to shepherd future advancements in NLP.

##  2 Related Work

The landscape of paraphrase generation datasets is critical to understanding
the research context and challenges in this domain. This section presents a
review of noteworthy paraphrase datasets, highlighting their strengths and
limitations.

Paraphrase Database (PPDB) PPDB is a comprehensive resource that houses over
220 million paraphrase pairs [7]. The PPDB is compiled through a technique
known as bilingual pivoting. The rationale behind this approach is that if two
English phrases are translated into the same foreign language phrase, they can
be inferred to have identical meanings. Each pair within the PPDB is
accompanied by a range of scores, such as paraphrase probabilities and
monolingual distributional similarity scores. However, despite its extensive
content and detailed scoring system, the PPDB’s utility has been questioned
recently due to its exclusive focus on phrasal and lexical paraphrases,
neglecting sentence paraphrases.

Twitter URL The Twitter URL dataset [8] is a comprehensive collection of
large-scale sentential paraphrases sourced from Twitter and connected through
shared URLs. This dataset is bifurcated into two subsets, each encompassing
both paraphrases and non-paraphrases. The labeling of one subset is performed
by human annotators, while the other subset is labeled automatically. It
should be noted that the annotation does contain some noise due to the
automatic labeling of sentence pairs. Due to the noisiness of the labels, this
dataset is not widely used.

Wiki Answer The Wiki Answer dataset [9] encompasses an estimated 18 million
pairs of questions that are paraphrased. The dataset was constructed by
mapping open-domain questions to queries over a database of web extractions.
The dataset also includes word alignments that connect synonyms within the
paraphrased sentences. The dataset is limited in scope as all the sentences
provided are in the form of questions, thereby confining the paraphrases to
question format only. The dataset is also noisy such that paraphrases do not
have the needed semantic similarity of a high-quality dataset.

MSCOCO The MSCOCO dataset [10] was primarily characterized as a comprehensive
object detection dataset. It comprises over 120,000 images, each of which is
accompanied by five distinct captions, contributed by five separate
annotators. Typically, the annotators focus on detailing the most conspicuous
object or action within an image, rendering this dataset particularly useful
for tasks related to paraphrasing.

Microsoft Research Paraphrase Corpus The Microsoft Research Paraphrase Corpus
(MRPC) Dataset [11] comprises 5800 sentence pairs derived from online news
sources. It also includes human annotations that denote whether each pair
represents a paraphrase or semantic equivalence relationship. This was one of
the oldest datasets which is still being used for model evaluation but its
only downside is that there are very few sentences in the corpora. The
sentence length is quite longer compared to other phrase heavy datasets making
it a valuable addition.

Quora The Quora Dataset or Quora Question Pair Dataset [12] which is
predominantly used for training and evaluation, contains 150,000 question
pairs that are annotated as paraphrases. These validated paraphrase question
pairs were specifically employed for the training and testing phases of the
paraphrase generation task. The dataset is similar to WikiAnswer in its
limitation of having only questions.

ParaNMT The ParaNMT dataset [13] comprises over 50 million pairs of English
sentential paraphrases. These pairs were autonomously generated through the
application of back-translation to translate the non-English component of a
substantial Czech-English parallel corpus. A Czech-English Neural Machine
Translation (NMT) system was employed to translate Czech sentences from the
training data into English. These translations were then paired with the
English references to form English-English paraphrase pairs. This is the first
paraphrase dataset that utilized back-translation. Upon analysis, one downside
is the inclusion of improperly formed paraphrases and non-English sentences.

ParaBank The ParaBank datasets [14, 15] were developed using a Czech-English
Neural Machine Translation (NMT) system to generate new paraphrases of English
reference sentences. The first version, ParaBank1, introduced lexical
constraints to the NMT decoding process, allowing for the generation of
multiple high-quality sentential paraphrases for each source sentence. This
resulted in an English paraphrase resource that exhibits a higher degree of
lexical diversity. Its successor, ParaBank2, addressed the issue of syntactic
diversity by providing multiple diverse sentential paraphrases. These
paraphrases were generated from a bilingual corpus using negative constraints,
inference sampling, and clustering. Even with the improvements both datasets
still lack syntactic diversity.

PAWS (Paraphrase Adversaries from Word Scrambling) The PAWS Dataset [16]
contains sentences with high bag-of-words (BOW) overlap but having different
word order. The PAWS dataset creation involved a two-step process. Initially,
a language model was used to generate sentence pairs with high lexical overlap
through word swapping, ensuring naturalness and well-formedness. Subsequently,
back translation was employed to create paraphrases with high bag-of-words
overlap but distinct word order. The PAWS dataset is further divided into two
subsets: PAWSQQP and PAWSWiki. The PAWSQQP subset is derived from the Quora
Question Pairs (QQP) corpus, while the PAWSWiki subset is derived from
Wikipedia. Subsets of the dataset were then subjected to human review for
sentence correction and paraphrase identification. PAWSWiki has significantly
better quality paraphrases than all the other datasets, yet improvements can
be made for syntactic diversity. There is a considerable portion of PAWS that
is noisy and the other portion that is labeled exhibit high-quality.

##  3 ParaFusion

Figure 1: High-level diagram outlining the dataset creation process.

###  3.1 Data Sources

ParaFusion is a comprehensive tool constructed on the foundation of several
datasets, each contributing unique elements to the overall structure. The
first dataset we utilized was the MRPC Dataset [11], which provided a solid
base for our work. Following this, we incorporated a subset of the Quora
Dataset [12], specifically selecting sentences that were labeled as
paraphrases to enrich our data pool. To further diversify our data, we
included PAWSWiki, a component of the PAWS Dataset [16]. However, we
consciously decided against using PAWSQQP as it contained source sentences
identical to those in the Quora Dataset, which would have introduced
unnecessary redundancy into our data. We also considered the use of three
additional datasets: ParaNMT [13], Parabank1 [14], and Parabank2 [15].
However, due to financial constraints, we were unable to utilize these
datasets in their entirety. Instead, we strategically selected all the common
source sentences from these datasets and supplemented this with an additional
250,000 source sentences.

The decision to incorporate multiple datasets into our research methodology
was a strategic one, aimed at enhancing the comprehensiveness and diversity of
the dataset. Different datasets inherently possess varying sentence lengths
and unique content, style, and context. By amalgamating several datasets, we
ensured that ParaFusion captured a broad spectrum of sentence lengths, and
provided an extensive range of topics, writing styles, and contexts for
training. This approach also helped us to mitigate the risk of potential data
bias, ensuring a more balanced and representative dataset.

It should be emphasized that the method we suggest is not confined to these
particular datasets. Instead, it has the potential to be utilized with any
general text to generate paraphrases.

###  3.2 Base Dataset Creation

In the construction of ParaFusion, we initially amalgamated the aforementioned
data sources, selecting approximately 750,000 source sentences for paraphrase
generation. An initial pass was conducted to filter out offensive content,
utilizing OpenAI’s Moderation Endpoint [17]. This process flagged any source
sentence that fell under the categories of ”sexual”, ”hate”, ”harassment”,
”self-harm”, ”sexual/minors”, “hate/threatening”, ”violence/graphic”, ”self-
harm/instructions”, ”self-harm/intent”, ”harassment/threatening”, or
”violence”. This enabled us to filter out approximately 50,000 source
sentences containing offensive content.

Subsequently, we employed the ChatGPT (gpt-3.5-turbo) LLM to augment the
source sentences. The prompt used for each dataset varied slightly, and the
reasons for these variations are discussed in the Additional Processing
Section 3.3. See Figure 2 for an example illustration of this.

Given a Source Sentence: “‘$Source Sentence“‘, generate 5 diverse paraphrases.
Try to generate paraphrases that are both lexical and syntactically diverse
from the Source Sentence. Give the output as a numbered list.

Figure 2: This figure illustrates a sample prompt fed to the gpt-3.5-turbo
model for generating diverse paraphrases.

We adopted an iterative prompt engineering approach to construct an effective
prompt. This prompt was then input into the gpt-3.5-turbo model, along with
the source sentences. For paraphrase generation, we set the temperature
parameter to 0. The model output was a string containing augmented, diverse
sentence paraphrases in a numbered list format. This string was subsequently
processed to generate a list of strings, which was then used to construct the
dataset. By utilizing the source sentences, we successfully generated nearly
3.5 million sentence paraphrases.

###  3.3 Additional Processing

Our study revealed that datasets constructed using back-translation
[13][14][15] often contained a significant amount of noise, including non-
English source sentences. To tackle this, we iteratively developed a new
prompt for the gpt-3.5-turbo model, instructing it to identify English
sentences and generate paraphrases, or output ”Error” for non-English
sentences. This method, coupled with a rule-based approach to filter out
certain responses from the model, proved efficient in significantly reducing
noise, and eliminating approximately 10,000 non-English source sentences
during generation.

In the final stage of dataset creation, we didn’t merely use the source
sentence and generated paraphrase pairs as they were. Instead, we treated the
source sentence and generated paraphrase as a pool of sentences, from which we
created unique paraphrase pairs. This approach diversified the paraphrase
pairs in our dataset, ensuring a wider range of sentence structures and
expressions.

This method offers several benefits. Firstly, it enhances the diversity of our
dataset, which is crucial for training robust and generalizable models.
Secondly, it helps to mitigate the risk of overfitting by exposing our models
to a more representative sample of the data they will encounter in the real
world. Lastly, it helps to reduce the impact of any noise in the source
sentence. By creating unique paraphrase pairs, we can ensure that any noise in
the source sentences or generated paraphrases are not consistently paired with
the same sentences, thereby reducing the likelihood that our models will learn
to associate this noise with specific inputs or outputs.

The final dataset comprises around 2 million unique paraphrase sentence pairs.
These techniques significantly reduced noise in ParaFusion, contributing to
the construction of a higher-quality dataset, comparable to human annotation
or the next best alternative.

##  4 Evaluation

In this section, we evaluate ParaFusion, which was created using several
source datasets: the Microsoft Research Paraphrase Corpus (MRPC), a subset of
the Quora Dataset, PAWSWiki, ParaNMT, Parabank1, and Parabank2. For a fair
evaluation, we only consider source sentences common to both ParaFusion and
these source datasets.

We use ”Para-Common Subset” to denote paraphrases in ParaFusion common to
ParaNMT, Parabank1, and Parabank2. ”MRPC Subset”, ”PAWS Subset”, and ”QQP
Subset” refer to paraphrases in ParaFusion common to MRPC, Quora Dataset, and
PAWSWiki Dataset, respectively. Each subset is analyzed separately for
fairness.

###  4.1 Quantitative Analysis

We adopt a comprehensive quantitative evaluation methodology to assess the
data sources and sentence pairs in ParaFusion. To guarantee a fair evaluation,
the dataset is partitioned into four segments based on the source sentences.
Our evaluation focuses on three crucial characteristics: semantic similarity,
syntactic diversity, and lexical diversity.

####  4.1.1 Semantic Similarity

Semantic similarity is a measure of the degree to which two pieces of text are
related in terms of their meaning. In our research, we quantify this
similarity by leveraging various models to obtain sentence embeddings of the
source and the paraphrase, and then calculating the cosine similarity between
these embeddings.

For instance, we use the ”Ada Score” which is derived from OpenAI’s text-
embedding-ada-002 model [18]. Similarly, the ”SimCSE Score” is calculated
using SimCSE’s sup-simcse-roberta-large model [19], and the ”PromCSE Score” is
based on PromCSE’s sup-promcse-roberta-large model [20].

We also utilize several models from the sentence-transformers library [21].
The ”Mpnet Score” is calculated using the all-mpnet-base-v2 model, while the
”Mpnet-qa Similarity Score” is derived from the multi-qa-mpnet-base-dot-v1
model. The ”Roberta Score” is based on the all-distilroberta-v1 model, and the
”Mini Score” and ”Mini Score2” are calculated using the all-MiniLM-L12-v2 and
all-MiniLM-L6-v2 models, respectively.

The comprehensive evaluation of semantic similarity is presented in Table 1.
It is evident that ParaFusion not only maintains semantic similarity but also,
in some cases, surpasses the quality of the original data source. The
perceived low similarity in less sophisticated models can be attributed to the
lack of lexical or syntactic diversity in the original source sentences, which
results in highly similar sentences. However, in ParaFusion, the complexity of
the sentences allows for a more accurate measurement of semantic similarity
using advanced models such as the text-embedding-ada-002 or SIMCSE.

Data Source |  Ada Score (↑) |  SimCSE Score (↑) |  PromCSE Score (↑) |  Mpnet Score (↑) |  Mpnet-qa Score (↑) |  Roberta Score (↑) |  Mini Score (↑) |  Mini Score2 (↑)
---|---|---|---|---|---|---|---|---
MSR Original |  95.53% |  86.50% |  99.22% |  84.27% |  86.23% |  82.80% |  83.27% |  82.47%
MSR Subset (Ours) |  96.59% |  93.30% |  99.56% |  88.33% |  90.80% |  86.45% |  86.82% |  86.01%
QQP Original |  95.52% |  89.61% |  99.65% |  88.74% |  91.30% |  87.00% |  88.75% |  88.37%
QQP Subset (Ours) |  94.56% |  89.87% |  99.65% |  86.86% |  89.33% |  84.29% |  85.12% |  83.85%
PAWS Original |  98.90% |  97.42% |  99.90% |  96.96% |  97.06% |  97.20% |  97.24% |  97.13%
PAWS Subset (Ours) |  96.22% |  92.20% |  99.34% |  91.74% |  91.57% |  91.72% |  92.08% |  91.33%
ParaNMT Original |  94.83% |  87.50% |  99.88% |  87.00% |  90.67% |  82.05% |  87.69% |  87.58%
ParaBank1 Original |  95.58% |  89.85% |  99.91% |  89.00% |  92.09% |  85.31% |  89.60% |  89.51%
ParaBank2 Original |  94.49% |  85.52% |  98.87% |  82.89% |  87.60% |  79.76% |  83.63% |  83.26%
Para-Common Subset (Ours) |  95.04% |  74.32% |  98.60% |  67.16% |  74.11% |  63.00% |  67.19% |  65.80%
Table 1: Comparison of Semantic Similarity of the original data sources and
corresponding ParaFusion data subsets.

####  4.1.2 Syntactic Diversity

Syntactic diversity refers to the variety and complexity of sentence
structures of a paraphrase given a source sentence. High syntactic diversity
indicates that the paraphrase sentences are diverse and linguistically rich.
We assess this diversity using several metrics.

The ”Ted-F Score” and ”Ted-3 Score” are calculated by building constituency
parse trees for the source and paraphrase sentences using Stanza [22],
converting the trees to bracket notation using the NLTK library [23] and
regex, and then using the APTED library [24] to calculate the Full Tree Edit
Distance and the Tree Edit Distance of the first three layers, respectively.

The ”Kermit Score” is calculated by obtaining the cosine similarity of the
source and the paraphrase syntactic embeddings using the Kermit library [25],
and then subtracting this similarity from one.

The ”ST Kernel Score” and ”NP Kernel Score” are calculated by first building
constituency parse trees for the source and paraphrase sentences using Stanza,
converting the trees to an NLTK Tree, and then finding all the subtrees or
node pairs, respectively. Then the kernel similarity is calculated using the
number of unique common subtrees or node pairs over the number of unique total
subtrees or node pairs, and then subtracting this similarity from one. ”ST
Kernel Score” is the score calculated related to the subtrees in the
constituency parse trees whereas ”NP Kernel Score” is the score calculated
using the node pair similarity.

The full syntactic diversity evaluation is shown in Table 2. We can see that
ParaFusion has a significant improvement over all the original data sources.
This is because it was able to generate more syntactically rich paraphrases.

Data Source |  Ted-F Score (↑) |  Ted-3 Score (↑) |  Kermit Score (↑) |  ST Kernel Score (↑) |  NP Kernel Score (↑)
---|---|---|---|---|---
MSR Original |  18.65 |  3.76 |  55.23% |  65.18% |  82.87%
MSR Subset (Ours) |  29.09 |  4.92 |  64.37% |  73.55% |  89.34%
QQP Original |  9.30 |  2.02 |  57.59% |  70.14% |  88.29%
QQP Subset (Ours) |  16.91 |  3.35 |  71.62% |  81.13% |  93.64%
PAWS Original |  9.46 |  1.70 |  37.74% |  47.44% |  73.47%
PAWS Subset (Ours) |  27.02 |  4.67 |  61.06% |  69.48% |  87.35%
ParaNMT Original |  3.12 |  2.01 |  54.06% |  73.99% |  82.51%
ParaBank1 Original |  2.06 |  1.48 |  45.85% |  63.22% |  70.37%
ParaBank2 Original |  3.76 |  2.07 |  61.94% |  85.01% |  96.78%
Para-Common Subset (Ours) |  9.33 |  3.99 |  80.78% |  91.73% |  97.94%
Table 2: Comparison of Syntactic Diversity of the original data sources and
corresponding ParaFusion data subsets.

####  4.1.3 Lexical Diversity

Lexical diversity refers to the range and variety of words used in a text. It
is a measure of the breadth of vocabulary and the use of synonyms. In the
context of paraphrasing, assessing lexical diversity is crucial to understand
the extent of vocabulary variation. We used several metrics to assess lexical
diversity.

The ”BOW Overlap” is calculated by determining the intersection of tokens
between the source and the paraphrase, divided by the total number of tokens.
This value is then subtracted from one.

The ”Corpus BLEU” and ”Corpus BLEU2” scores are calculated using the SacreBLEU
Library [26]. The Corpus BLEU2 score uses the “method1” smoothing function in
the SacreBLEU library. Both scores are then subtracted from one.

The ”Sentence BLEU” score is calculated in a similar manner to the Corpus BLEU
score using the SacreBLEU Library but at the sentence level. This score is
also subtracted from one.

The ”METEOR” score is calculated using the NLTK library and then subtracted
from one.

The ”ROUGE 1”, ”ROUGE 2”, and ”ROUGE L” scores are calculated using the Google
Research library [27] and then subtracted from one.

The ”Token $\cap/\cup$” score is similar to the BOW Overlap score, but with a
small difference. It is calculated using the intersection of tokens between
the source and the paraphrase, divided by the total number of unique tokens.
This value is then subtracted from one.

The ”Google BLEU” score is calculated using Huggingface’s Evaluate library and
then subtracted from one.111https://github.com/huggingface/evaluate

The ”TER (Translation Error Rate)”, ”WER (Word Error Rate)”, and ”CharacTER
(Character Error Rate)” scores are calculated using Huggingface’s Evaluate
library.1

The full lexical diversity evaluation is shown in Table 3 and Table 4. We can
see that ParaFusion has a significant improvement over all the original data
sources.

Data Source |  1 - BOW Overlap (↑) |  1 - Corpus BLEU (↑) |  1 - Corpus BLEU2 (↑) |  1 - Sentence BLEU (↑) |  1 - METEOR (↑) |  1 - ROUGE 1 (↑)
---|---|---|---|---|---|---
MSR Original |  35.37% |  93.92% |  99.62% |  59.75% |  31.43% |  29.49%
MSR Subset (Ours) |  43.72% |  96.53% |  99.65% |  75.93% |  38.96% |  36.81%
QQP Original |  36.69% |  95.44% |  99.15% |  70.80% |  35.33% |  33.46%
QQP Subset (Ours) |  53.04% |  82.21% |  99.46% |  84.33% |  50.20% |  51.86%
PAWS Original |  19.33% |  94.21% |  99.60% |  34.96% |  8.58% |  5.96%
PAWS Subset (Ours) |  40.26% |  98.15% |  99.65% |  72.39% |  34.59% |  30.22%
ParaNMT Original |  53.56% |  62.19% |  97.75% |  58.26% |  27.33% |  18.37%
ParaBank1 Original |  46.50% |  62.47% |  97.73% |  51.62% |  24.54% |  16.08%
ParaBank2 Original |  55.36% |  63.06% |  97.86% |  66.85% |  39.24% |  30.43%
Para-Common Subset (Ours) |  74.31% |  82.24% |  98.72% |  86.45% |  69.03% |  64.82%
Table 3: Comparison of Lexical Diversity of the original data sources and corresponding ParaFusion Data Subsets. Data Source |  1 - ROUGE 2 (↑) |  1 - ROUGE L (↑) |  1 - Token $\cap/\cup$ (↑) |  TER (↑) |  WER (↑) |  CER (↑) |  1 - Google BLEU (↑)
---|---|---|---|---|---|---|---
MSR Original |  47.55% |  33.97% |  44.95% |  70.34 |  76.07 |  42.98% |  55.96%
MSR Subset (Ours) |  63.22% |  51.86% |  54.79% |  85.35 |  98.47 |  64.95% |  69.81%
QQP Original |  57.72% |  36.85% |  49.04% |  56.56 |  62.77 |  44.74% |  64.96%
QQP Subset (Ours) |  76.95% |  57.89% |  66.33% |  69.92 |  75.34 |  76.94% |  79.12%
PAWS Original |  21.44% |  12.78% |  15.74% |  14.81 |  23.63 |  14.19% |  31.80%
PAWS Subset (Ours) |  56.97% |  47.10% |  66.33% |  56.10 |  85.66 |  62.38% |  65.92%
ParaNMT Original |  33.20% |  19.25% |  61.39% |  30.10 |  66.10 |  25.74% |  44.99%
ParaBank1 Original |  31.69% |  16.90% |  52.51% |  23.71 |  50.47 |  21.61% |  37.37%
ParaBank2 Original |  55.24% |  32.97% |  67.69% |  46.84 |  68.01 |  35.39% |  66.87%
Para-Common Subset (Ours) |  89.12% |  69.68% |  82.23% |  82.46 |  89.31 |  85.10% |  81.30%
Table 4: Comparison of Lexical Diversity of the original data sources and
corresponding ParaFusion Data Subsets.

###  4.2 Qualitative Evaluation

During the qualitative analysis, we uncovered intriguing information. A
prevalent issue in existing datasets is that most paraphrases are merely
sentences with substituted synonyms. Figure 3 illustrates an example of this,
where ”A” is replaced by ”One”, leaving the rest of the sentence identical to
the source. This does not constitute an effective paraphrase.

Another problem we observed is that existing data sources, particularly those
relying on back-translation, often contain paraphrases with altered meanings
due to word choice. Figure 4 exemplifies a situation where the meaning of a
paraphrase has deviated from the original source datasets. This occurs when
inappropriate words are used without considering the context. For instance,
replacing ”culverts” with ”driers” is not suitable in this context.

In contrast, paraphrases in ParaFusion demonstrate superior lexical diversity
and more syntactic changes, while preserving the original meaning.

Source Sentence: A poetic example of early modern philosophical thought can be
found in the surprising works of the renowned intellectual Stoyan Mihaylovski.
Original Paraphrase: One poetic example of early modern philosophical thought
can be found in the surprising works of the renowned intellectual Stoyan
Mihaylovski.
ParaFusion Paraphrase: Stoyan Mihaylovski’s works are a remarkable
representation of early modern philosophical thought, expressed in a poetic
manner.

Figure 3: This figure illustrates an instance where the paraphrase in a source
dataset has only word substitutions.

Source Sentence: The water is moved by the gravity and is controlled by huge
valves in the driers .
Original Paraphrase: The water is moved by gravity and is controlled by huge
valves in the culverts .
ParaFusion Paraphrase: Huge valves in the driers regulate the movement of
water, which is facilitated by the force of gravity.

Figure 4: This figure illustrates an instance where the paraphrase in a
previous dataset has a different meaning.

Furthermore, we observed that the original data sources contained a
substantial number of paraphrase pairs where the source and the paraphrase
were identical. This presents a significant problem because training a model
on such datasets could encourage the model to simply reproduce the input,
thereby defeating the purpose of paraphrasing. This qualitative analysis
further underscores the effectiveness and reliability of ParaFusion in these
scenarios.

###  4.3 Human Evaluation

In our research, we conducted human evaluations using four annotators who
assessed approximately 7000 paraphrase pairs across various datasets,
including the source and ParaFusion. The source sentences were selected as
follows: 200 from the MRPC Dataset, 250 from the Quora Dataset, 250 from the
PAWSWiki, and 300 common source sentences from ParaNMT, Parabank1, and
Parabank2. The sentences were sampled to ensure that the lengths of the data
source sentences were properly represented. The corresponding paraphrases from
the source datasets and ParaFusion were then selected, resulting in a total of
7000 paraphrase pairs for evaluation.

For the evaluation, we used a 5-point Likert scale [28] to assess key metrics,
including Semantic Similarity, Lexical Diversity, Syntactic Diversity, and
Grammatical Correctness. The full breakdown of the Likert scale can be seen in
Figure 6. In this scale, 5 represents the highest level of similarity,
diversity, or correctness, while 1 indicates the lowest.

Specifically, semantic similarity ratings ranged from 5 for identical meaning
to the source text, to 1 for completely different or unrelated meaning.
Lexical diversity was evaluated based on vocabulary range and richness, with 5
indicating excellent diversity and 1 signifying limited diversity. Syntactic
diversity was assessed by structural variations, with 5 denoting high
diversity and 1 signifying minimal variation. Lastly, grammatical correctness
was evaluated, with 5 indicating flawless grammar and 1 representing
significant errors impacting comprehension.

Table 5 provides a summary of the human evaluation, and a full breakdown can
be seen in Table 8. The results clearly indicate that ParaFusion is more
lexically and syntactically diverse than the original data sources.

|  Original |  ParaFusion
---|---|---
Semantic Similarity |  4.36 |  4.46
Lexical Diversity |  2.29 |  3.09
Syntactic Diversity |  2.44 |  3.40
Grammatical
Correctness |  4.37 |  4.79

Table 5: Comparison of Semantic Similarity, Lexical Diversity, Syntactic
Diversity, and Grammatical Correctness between two data sets in Human
Evaluation.

###  4.4 LLM Evaluation

In the past year, the use of LLMs for evaluation in NLP has gained traction.
The primary reason for this trend is the ability of LLMs to outperform
existing reference-free metrics [29]. In light of this, we conducted an LLM
evaluation using OpenAI’s gpt-4 model on the same data provided to our human
annotators. The gpt-4 model was selected due to its status as the SOTA LLM at
the time of writing this paper [6]. We designed the prompt using the same
instructions given to the human annotators, as illustrated in Figure 5.

The summary of the results is presented in Table 6, while a full breakdown is
provided in Table 7. The observations clearly indicate that ParaFusion is more
lexically and syntactically diverse than the original data sources, mirroring
the results of the Human Evaluation.

|  Original |  ParaFusion
---|---|---
Semantic Similarity |  4.49 |  4.94
Lexical Diversity |  1.75 |  3.34
Syntactic Diversity |  2.02 |  3.84
Grammatical
Correctness |  4.75 |  4.99

Table 6: Comparison of Semantic Similarity, Lexical Diversity, Syntactic
Diversity, and Grammatical Correctness between two data sets in LLM
Evaluation.

##  5 Conclusion

This research paper introduces ParaFusion, a large-scale, high-quality English
paraphrase dataset developed using LLMs. The dataset is designed to address
the limitations of the lack of syntactic and lexical diversity in existing
datasets. Additionally, it shows potential as a very high-quality alternative
to human-annotated paraphrase pairs which are costly to obtain. ParaFusion
augmented existing datasets to generate high-quality data, significantly
enhancing both lexical and syntactic diversity while maintaining semantic
similarity which was seen in the evaluation section. It also mitigates the
presence of hate speech and reduces noise, ensuring a cleaner, more focused
English dataset. The evaluation of ParaFusion demonstrated its potential as a
valuable resource for improving NLP applications.

## Limitations

While ParaFusion addresses several challenges in paraphrase generation, there
are a few limitations to be considered. Firstly, the dataset is focused on
English paraphrases, which may limit its applicability to other languages.
Future research could explore the development of similar datasets for other
languages to enhance the diversity and inclusivity of NLP applications. This
could be accomplished using multi-lingual LLMs or language-specific LLMs.

Secondly, although efforts were made to ensure the quality and reliability of
the dataset, there may still be instances of noise, inaccuracies, or in rare
cases, offensive language. The use of LLMs for paraphrase generation
introduces the possibility of generating incorrect paraphrases. Researchers
and practitioners should exercise caution and conduct thorough evaluations
when using the ParaFusion dataset.

Another point to be considered is the issue of error propagation. Our dataset
was generated using gpt-3.5-turbo, thus inheriting all potential risks
associated with it. This can also include phenomena like quality drift where
the model output can change as the model adapts, if expansions were to be
done. This could potentially introduce additional inaccuracies into the
paraphrase generation process, and users should be aware of this when using
the ParaFusion dataset and expanding it.

Lastly, the evaluation metrics used in this research provide valuable insights
into the quality and diversity of the dataset. However, they may not capture
all aspects of paraphrase generation. Future research could explore additional
evaluation metrics or approaches to further assess the effectiveness and
performance of paraphrase generation models using our ParaFusion dataset.

## Ethics Statement

In conducting this research and developing the ParaFusion dataset, we took
several ethical considerations into account. Firstly, we ensured that the
dataset contained minimal hate speech or offensive language by implementing a
moderation filtering process. This was done to ensure that the dataset is safe
and suitable for using in NLP applications.

Secondly, we made efforts to reduce noise in the dataset, such as removing
non-English sentences and filtering out responses that did not meet the
criteria for high-quality paraphrases. This was done to ensure that the
dataset is of high quality and reliable for training NLP models.

Additionally, we considered the potential impact of our research on the
broader NLP community. By providing a large-scale, high-quality paraphrase
dataset, we aim to contribute to the advancement of NLP applications. This
dataset can be used to improve the performance and robustness of NLP models,
leading to more accurate and reliable NLP.

## References

  * [1] K. R. McKeown, “Paraphrasing using given and new information in a question-answer system,” in _Proceedings of the 17th annual meeting on Association for Computational Linguistics -_. La Jolla, California: Association for Computational Linguistics, 1979, p. 67. [Online]. Available: http://portal.acm.org/citation.cfm?doid=982163.982182
  * [2] M. Meteer and V. Shaked, “Strategies for effective paraphrasing,” in _Proceedings of the 12th conference on Computational linguistics -_ , vol. 2. Budapest, Hungry: Association for Computational Linguistics, 1988, pp. 431–436. [Online]. Available: http://portal.acm.org/citation.cfm?doid=991719.991724
  * [3] R. Kozlowski, K. F. McCoy, and K. Vijay-Shanker, “Generation of single-sentence paraphrases from predicate/argument structure using lexico-grammatical resources,” in _Proceedings of the second international workshop on Paraphrasing -_ , vol. 16. Sapporo, Japan: Association for Computational Linguistics, 2003, pp. 1–8. [Online]. Available: http://portal.acm.org/citation.cfm?doid=1118984.1118985
  * [4] I. A. Bolshakov and A. Gelbukh, “Synonymous Paraphrasing Using WordNet and Internet,” in _Natural Language Processing and Information Systems_ , D. Hutchison, T. Kanade, J. Kittler, J. M. Kleinberg, F. Mattern, J. C. Mitchell, M. Naor, O. Nierstrasz, C. Pandu Rangan, B. Steffen, M. Sudan, D. Terzopoulos, D. Tygar, M. Y. Vardi, G. Weikum, F. Meziane, and E. Métais, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2004, vol. 3136, pp. 312–323, series Title: Lecture Notes in Computer Science. [Online]. Available: http://link.springer.com/10.1007/978-3-540-27779-8_27
  * [5] J. Zhou and S. Bhat, “Paraphrase Generation: A Survey of the State of the Art,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, Eds. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 5075–5086. [Online]. Available: https://aclanthology.org/2021.emnlp-main.414
  * [6] OpenAI, “GPT-4 Technical Report,” 2023, publisher: arXiv Version Number: 3. [Online]. Available: https://arxiv.org/abs/2303.08774
  * [7] J. Ganitkevitch, B. Van Durme, and C. Callison-Burch, “PPDB: The Paraphrase Database,” in _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Atlanta, Georgia: Association for Computational Linguistics, Jun. 2013, pp. 758–764. [Online]. Available: https://aclanthology.org/N13-1092
  * [8] W. Lan, S. Qiu, H. He, and W. Xu, “A Continuously Growing Dataset of Sentential Paraphrases,” in _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Copenhagen, Denmark: Association for Computational Linguistics, 2017, pp. 1224–1234. [Online]. Available: http://aclweb.org/anthology/D17-1126
  * [9] A. Fader, L. Zettlemoyer, and O. Etzioni, “Paraphrase-Driven Learning for Open Question Answering,” in _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Sofia, Bulgaria: Association for Computational Linguistics, Aug. 2013, pp. 1608–1618. [Online]. Available: https://aclanthology.org/P13-1158
  * [10] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common Objects in Context,” in _Computer Vision – ECCV 2014_ , D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, Eds. Cham: Springer International Publishing, 2014, vol. 8693, pp. 740–755, series Title: Lecture Notes in Computer Science. [Online]. Available: http://link.springer.com/10.1007/978-3-319-10602-1_48
  * [11] W. B. Dolan and C. Brockett, “Automatically Constructing a Corpus of Sentential Paraphrases,” in _Proceedings of the Third International Workshop on Paraphrasing (IWP2005)_ , 2005. [Online]. Available: https://aclanthology.org/I05-5002
  * [12] S. Iyer, N. Dandeka, and K. Csernai, “First Quora Dataset Release: Question Pairs,” 2017. [Online]. Available: https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs
  * [13] J. Wieting and K. Gimpel, “ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations,” in _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Melbourne, Australia: Association for Computational Linguistics, 2018, pp. 451–462. [Online]. Available: http://aclweb.org/anthology/P18-1042
  * [14] J. E. Hu, R. Rudinger, M. Post, and B. Van Durme, “ParaBank: Monolingual Bitext Generation and Sentential Paraphrasing via Lexically-constrained Neural Machine Translation,” Jan. 2019, arXiv:1901.03644 [cs]. [Online]. Available: http://arxiv.org/abs/1901.03644
  * [15] J. E. Hu, A. Singh, N. Holzenberger, M. Post, and B. Van Durme, “Large-Scale, Diverse, Paraphrastic Bitexts via Sampling and Clustering,” in _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_. Hong Kong, China: Association for Computational Linguistics, 2019, pp. 44–54. [Online]. Available: https://www.aclweb.org/anthology/K19-1005
  * [16] Y. Zhang, J. Baldridge, and L. He, “PAWS: Paraphrase Adversaries from Word Scrambling,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 1298–1308. [Online]. Available: https://aclanthology.org/N19-1131
  * [17] OpenAI, “Moderation,” 2023. [Online]. Available: https://platform.openai.com/docs/guides/moderation
  * [18] OpenAI, “New and Improved Embedding Model,” 2023, publisher: OpenAI. [Online]. Available: https://openai.com/blog/new-and-improved-embedding-model
  * [19] T. Gao, X. Yao, and D. Chen, “SimCSE: Simple Contrastive Learning of Sentence Embeddings,” in _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. 2021, pp. 6894–6910. [Online]. Available: https://aclanthology.org/2021.emnlp-main.552
  * [20] Y. Jiang, L. Zhang, and W. Wang, “Improved Universal Sentence Embeddings with Prompt-based Contrastive Learning and Energy-based Learning,” in _Findings of the Association for Computational Linguistics: EMNLP 2022_. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 3021–3035. [Online]. Available: https://aclanthology.org/2022.findings-emnlp.220
  * [21] N. Reimers and I. Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” in _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Nov. 2019. [Online]. Available: http://arxiv.org/abs/1908.10084
  * [22] P. Qi, Y. Zhang, Y. Zhang, J. Bolton, and C. D. Manning, “Stanza: A Python Natural Language Processing Toolkit for Many Human Languages,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , 2020.
  * [23] S. Bird, E. Loper, and E. Klein, _Natural Language Processing with Python_. O’Reilly Media Inc., 2009.
  * [24] M. Pawlik and N. Augsten, “Efficient Computation of the Tree Edit Distance,” _ACM Transactions on Database Systems_ , vol. 40, no. 1, pp. 1–40, Mar. 2015. [Online]. Available: https://dl.acm.org/doi/10.1145/2699485
  * [25] F. M. Zanzotto, A. Santilli, L. Ranaldi, D. Onorati, P. Tommasino, and F. Fallucchi, “KERMIT: Complementing Transformer Architectures with Encoders of Explicit Syntactic Interpretations,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Online: Association for Computational Linguistics, Nov. 2020, pp. 256–267. [Online]. Available: https://www.aclweb.org/anthology/2020.emnlp-main.18
  * [26] M. Post, “A Call for Clarity in Reporting BLEU Scores,” in _Proceedings of the Third Conference on Machine Translation: Research Papers_. Belgium, Brussels: Association for Computational Linguistics, Oct. 2018, pp. 186–191. [Online]. Available: https://www.aclweb.org/anthology/W18-6319
  * [27] Google, “Google Research,” 2023. [Online]. Available: https://github.com/google-research/google-research/tree/master
  * [28] C. Van Der Lee, A. Gatt, E. Van Miltenburg, S. Wubben, and E. Krahmer, “Best practices for the human evaluation of automatically generated text,” in _Proceedings of the 12th International Conference on Natural Language Generation_. Tokyo, Japan: Association for Computational Linguistics, 2019, pp. 355–368. [Online]. Available: https://www.aclweb.org/anthology/W19-8643
  * [29] Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment,” 2023, publisher: arXiv Version Number: 3. [Online]. Available: https://arxiv.org/abs/2303.16634

## Authors

Lasal Jayawardena is a fourth-year undergraduate student at the Informatic
Institute of Technology (IIT) Sri Lanka, affiliated with Robert Gordon
University in Aberdeen, Scotland. He is currently pursuing a Bachelor of
Science honors degree in Artificial Intelligence and Data Science. Lasal’s
research interests primarily revolve around the domain of Natural Language
Processing (NLP), with a strong focus on Large Language Models and Deep
Learning techniques.

Prasan Yapa received Master of Computer Science (Research Major) and Bachelor
of Science in Information Technology and Management from University of
Moratuwa, Sri Lanka. Currently, he is pursuing his PhD in Computer Science &
Engineering from Kyoto University of Advanced Science in Japan. His research
interests include Natural Language Processing, Deep Learning, Affective
Computing, Health Informatics and Computer Vision.

##  Appendix 0.A Human and LLM Evaluation

Data Source |  Semantic Similarity: Scale from 1 to 5 |  Lexical Diversity: Scale from 1 to 5 |  Syntactic Diversity: Scale from 1 to 5 |  Grammatical Correctness: Scale from 1 to 5
---|---|---|---|---
MSR Original |  4.61 |  2.61 |  2.98 |  4.99
MSR Subset (Ours) |  4.99 |  3.56 |  3.91 |  4.99
QQP Original |  4.35 |  2.14 |  2.63 |  4.96
QQP Subset (Ours) |  4.97 |  3.52 |  3.82 |  4.99
PAWS Original |  4.87 |  1.30 |  1.82 |  4.86
PAWS Subset (Ours) |  4.95 |  3.20 |  3.88 |  4.98
ParaNMT Original |  4.49 |  1.34 |  1.41 |  4.78
ParaBank1 Original |  4.66 |  1.34 |  1.40 |  4.81
ParaBank2 Original |  4.07 |  2.07 |  2.26 |  4.27
Para-Common Subset (Ours) |  4.84 |  3.08 |  3.78 |  5.00
Table 7: Full Breakdown of the LLM Evaluation using gpt-4 Data Source |  Semantic Similarity: Scale from 1 to 5 |  Lexical Diversity: Scale from 1 to 5 |  Syntactic Diversity: Scale from 1 to 5 |  Grammatical Correctness: Scale from 1 to 5
---|---|---|---|---
MSR Original |  4.26 |  2.66 |  2.96 |  4.80
MSR Subset (Ours) |  4.53 |  3.10 |  3.46 |  4.88
QQP Original |  4.25 |  2.38 |  2.62 |  4.92
QQP Subset (Ours) |  4.49 |  3.15 |  3.38 |  4.96
PAWS Original |  4.79 |  2.07 |  2.25 |  4.13
PAWS Subset (Ours) |  4.42 |  3.02 |  3.39 |  4.69
ParaNMT Original |  4.51 |  2.00 |  2.10 |  4.36
ParaBank1 Original |  4.66 |  1.97 |  2.04 |  4.24
ParaBank2 Original) |  4.11 |  2.37 |  2.59 |  4.04
Para-Common Subset (Ours) |  4.42 |  3.02 |  3.45 |  4.76
Table 8: Full Breakdown of the Human Evaluation

Source Text: $source_text Paraphrase: $paraphrase Please evaluate the
following aspects of the paraphrase in comparison to its source text on a
likert scale of 1 to 5, where: Semantic Similarity: This refers to how closely
the meaning of the paraphrase matches the meaning of the source text. Rating
Scale for Semantic Similairty 1: The paraphrase has a completely different
meaning or is unrelated to the source text. 2: The paraphrase has a somewhat
different meaning from the source text 3: The paraphrase captures the general
idea of the source text, but some details or nuances are missing. 4: The
paraphrase largely captures the meaning of the source text but may have slight
differences in wording or expression. 5: The paraphrase has an identical or
nearly identical meaning to the source text. Lexical Diversity: This aspect
evaluates the range and richness of vocabulary used in the paraphrase,
considering its comparison to the source text. Rating Scale for Lexical
Diversity 1: The paraphrase shows a limited use of words and lacks diversity
when compared to the source text. 2: The paraphrase exhibits some variation in
word choice but heavily relies on a few specific terms, which may not reflect
the lexical diversity of the source text. 3: The paraphrase demonstrates
moderate diversity in vocabulary, but there is room for improvement in terms
of incorporating more varied word choices from the source text. 4: The
paraphrase displays a good range of vocabulary, utilizing several different
words and expressions that align with the lexical diversity of the source
text. 5: The paraphrase showcases an extensive array of vocabulary,
demonstrating excellent lexical diversity that closely matches or surpasses
the richness of the source text. Syntactic Diversity: This aspect assesses the
structural variations in the paraphrase compared to the source text. Rating
Scale for Syntactic Diversity 1: The paraphrase closely mirrors the sentence
structure of the source text with minimal variation. 2: The paraphrase shows
some minor changes in sentence structure but largely follows the same pattern
as the source text. 3: The paraphrase introduces moderate variations in
sentence structure, deviating from the structure of the source text in certain
aspects. 4: The paraphrase exhibits significant syntactic diversity, using
different sentence structures while still conveying the same meaning as the
source text. 5: The paraphrase displays a high level of syntactic diversity,
employing various sentence structures creatively while maintaining the meaning
of the source text. Grammatical Correctness: This evaluates the grammatical
accuracy of the paraphrase. Rating Scale for Grammatical Correctness 1: The
paraphrase contains numerous grammatical errors that significantly impact
comprehension. 2: The paraphrase has several grammatical errors that
occasionally affect understanding. 3: The paraphrase includes some grammatical
errors, but they do not hinder overall comprehension. 4: The paraphrase
demonstrates good grammatical correctness with only occasional minor errors.
5: The paraphrase is grammatically flawless, with no errors or inaccuracies.
Please provide your ratings for each aspect using the following json format:
{”Semantic Similarity”: [Rating from 1 to 5], ”Lexical Diversity”: [Rating
from 1 to 5], ”Syntactic Diversity”: [Rating from 1 to 5], ”Grammatical
Correctness”: [Rating from 1 to 5]}

Figure 5: This figure illustrates the prompt fed to the gpt-4 model for
evaluation.

Figure 6: Instructions given to Human Annotators and breakdown of the 5 point
Likert Scale.