# A comparative analysis of embedding models for patent similarity

Grazia Sveva Ascione 1, Valerio Sterzi 1
1 Bordeaux School of Economics (Université de Bordeaux)
{grazia.ascione<EMAIL_ADDRESS>

###### Abstract

This paper makes two contributions to the field of text-based patent
similarity. First, it compares the performance of different kinds of patent-
specific pretrained embedding models, namely static word embeddings (such as
word2vec and doc2vec models) and contextual word embeddings (such as
transformers based models), on the task of patent similarity calculation.
Second, it compares specifically the performance of Sentence Transformers
(SBERT) architectures with different training phases on the patent similarity
task. To assess the models’ performance, we use information about patent
interferences, a phenomenon in which two or more patent claims belonging to
different patent applications are proven to be overlapping by patent
examiners. Therefore, we use these interferences cases as a proxy for maximum
similarity between two patents, treating them as ground-truth to evaluate the
performance of the different embedding models. Our results point out that,
first, Patent SBERT-adapt-ub, the domain adaptation of the pretrained Sentence
Transformer architecture proposed in this research, outperforms the current
state-of-the-art in patent similarity. Second, they show that, in some cases,
large static models performances are still comparable to contextual ones when
trained on extensive data; thus, we believe that the superiority in the
performance of contextual embeddings may not be related to the actual
architecture but rather to the way the training phase is performed.

##  1 Introduction

Patents provide their owners with a legal right to exclude others from making,
using, selling, offering for sale, or importing the patented invention. This
legal right is embodied in semi-structured text documents that describe the
background, inventive concept, as well as the “metes and bounds” of the
invention.

In recent times, a growing body of research started to link patent analytics
to natural language processing (NLP). This is possible because patents have
textual attributes which constitute the patent itself: a title, an abstract,
one or more claims and a description. In the last five years, the number of
studies on information retrieval using patent text had a steep increase,
covering different NLP tasks, such as name entity recognition (see for
instance, the very recent study by Puccetti et al. (2023) on how to extract
technological entities from patent text), technological class classification
(Tran and Kavuluru, 2017; Lee and Hsiang, 2019; Li et al., 2018; Roudsari et
al., 2021) and textual similarity calculation (Arts et al., 2021; Bekamiri et
al., 2021; Hain et al., 2022).

Textual similarity among patent (p2p) or patent portfolios is an essential
tool to map, understand and predict the innovation pattern and dynamics of the
actors involved in the innovation ecosystem. Recently, researchers have been
using patent vector space models based on different NLP embeddings models to
calculate the similarity between pairs of patents to help better understanding
innovations, patent landscaping, technology mapping, and patent quality
evaluation (Bekamiri et al., 2022). Recently, replacing static embeddings with
contextualized representations has yielded significant improvements on a
diverse array of NLP tasks, ranging from question answering to coreference
resolution. In the patent domain, on the one hand, authors previously
underlined the fact that polysemy is especially significant when looking at
patent data, because patent text is characterized by specific, technical,
legal and therefore ambiguous jargon (Bonino et al., 2010; Tseng et al.,
2007). Furthermore, patent text contains novel technologies by definition such
that these vocabulary based methods would need to be retrained to include them
in their dictionary. However, static embeddings have advantages in many
scenarios, such as a much lower training cost (Strubell et al., 2019) and an
easier interpretability (Gupta and Jaggi, 2021). In recent works, both static
(Whalen et al., 2020; Hain et al., 2022) and contextual (Bekamiri et al.,
2021) are used to measure patent similarity. Further, scholars underline the
lack of a ground-truth benchmark dataset to compare and validate those
similarity measures (Whalen et al., 2020; Hain et al., 2022; Bekamiri et al.,
2022) to establish the better approach.

Following the idea that for evaluating domain-specific word embeddings there
is the need of a ground-truth dataset (Betti et al., 2020), this research
brings two original contributions on the textual patent similarity task. For
both contributions, we use of a dataset about patent interferences, which
measure (nearly) simultaneous instances of identical invention by two or more
independent parties: by recording instances of common invention, patent
interferences create a unique record of common knowledge inputs (Ganguli et
al., 2020). In particular, the claims of patents involved in an interference
can be considered as a case of maximum similarity between patent texts,
therefore they can be used as a ground-truth benchmark dataset. First, using
interferences data, we are able to assess the performance of the most relevant
patent-specific deep learning models on the task of p2p similarity, testing
whether pretrained static embeddings or contextual embeddings perform better
on the patent similarity task. Second, we compare the performance of three
different contextualized models, specifically Sentence Transformers (SBERT)
models, following the idea that, among transformers based architectures, SBERT
vectorial representations are best suited for sentence-level downstream tasks
compared to those produced by the vanilla BERT model and its variants (Reimers
and Gurevych, 2019; Nikolaev and Padó, 2023). In particular, we compare the
augmented SBERT proposed by Bekamiri et al. (2021) to two original Sentence
Transformer models developed in this research, which are trained using pairs
of patents which share the same label (technological CPC class):

  1. 1.

Patent sBERT-ub: a finetuned model from a pretrained RoBERTa model (Liu et
al., 2019)

  2. 2.

Patent sBERT-adapt-ub: a domain adaptation of a pretrained SBERT (Reimers and
Gurevych, 2019)

to verify which model leads to the best performance.

The results show that, on the task of patent similarity, the domain adaptation
of Sentence Transformers (Patent sBERT-ub) has the best performance; however,
a static embedding model results’, the word2vec proposed by Hain et al.
(2022), are comparable with transformers based ones. Therefore, we find that
there is not a clear superiority of contextual over static embeddings or
viceversa in the patent similarity task. This might be explained by the fact
that, despite patent specific models would surely benefit of the contextual
embeddings advantages, such as a non-fixed dictionary and polysemy aware
embeddings, an extensive training phase still impacts the results.
Furthermore, when comparing Sentence Transformers’ models, the proposed domain
adaptation which exploits golden labeled data leads to the best performance.

The rest of the paper is structured as follows: Section 2 presents the state-
of-the-art in deep-learning derived patent embeddings, Section 3 presents the
data used, Section 4 depicts the models and the experiments, Section 5
analyzes the results, while Section 6 concludes, acknowledging the limitations
and opening the path for future developments.

##  2 Related work

The use of text attributes to calculate patent similarity started from simple
and straightforward keywords-based approaches, using co-occurrence or some
weighted versions of it, such as point-wise mutual information (PMI) or
n-grams (Arts et al., 2021). However, keyword based approaches suffer of a
number of shortcomings including the lack of consideration of the context to
define the semantic meaning of words or the assessment of type of co-
occurrence (synonymy or antonymy) as well as the computation expense and the
curse of dimensionality due to the large and very sparse matrices.

A first step towards solving this kind of problems involved the creation
static word vectors, which have demonstrated to be able capture meaningful
syntactic and semantic regularities, with an architecture defined as word2vec
(Mikolov et al., 2013) which takes into account the window of words which
surrounds the word of interest. Word2vec architectures link each token to a
vector creating a lookup matrix where there is a correspondence one to one;
however, this kind of model is unable to produce differentiated vectorial
representations for words that have a different meaning in different contexts.
This problem is especially significant when looking at patent data, because
patent text is characterized by specific technical, legal and often new
jargon, where a single technology might be associated to a variety of uses and
a single use to a variety of technologies.

For these reasons, newer attempts of creating meaningful patent vectors to
calculate patent similarity involve the use of more recent neural networks
architectures which are able to produce contextual representations for patent
text, meaning that the vectors change according to the surrounding context.
Table 1 summarizes some of the most relevant attempts of creating vectorial
representation for patent text using neural networks (NN). These works are
different to many respects, both conceptual and technical.

First of all, some of them use the vectorial representation of patent texts to
perform a multilabel classification task (Li et al., 2018; Lee and Hsiang,
2019; Bekamiri et al., 2021; Hain et al., 2022; Roudsari et al., 2021). To
reduce the burden over examiners and taking into account the increasing
complexity of technological classes, many researcher focused on this task
(Chen and Chang, 2012; Tran and Kavuluru, 2017; Grawe et al., 2017; Hepburn,
2018; Li et al., 2018; Lee and Hsiang, 2019; Bekamiri et al., 2021; Roudsari
et al., 2021). To accomplish the task, different authors not only use
different architectures but also different textual parts and different
validation methods. For instance, Li et al. (2018) propose their neural model
DeepPatent which encodes titles and abstracts of around 2 millions of USPTO
patents using skipgram and perform the classification using a convolutional
neural network (CNN) approach. The following work of Lee and Hsiang (2019)
uses DeepPatent as the baseline to benchmark; they propose a new model called
PatentBERT using both the same data as Li et al. (2018) and a larger sample
containing 3’050’615 patents. Further, they first compare their performance to
DeepPatent using title and abstract, and then they make an original
contribution using claims. In patent classification, the recent work of
Roudsari et al. (2021) proved that contextual embeddings perform better
compared to the static ones, while in Bekamiri et al. (2022), the authors show
that models fine-tuned on patent data, such as the word2vec TF-IDF model
presented in Hain et al. (2022), perform better than generic contextual
models.

Other authors create patents’ embeddings which are explicitly aimed at patent
similarity calculation. Whalen et al. (2020) follow this second approach. In
particular, their originality lies in the choice of the text to be used
(description and independent claims) and in the model, as they train a Doc2vec
architecture. Being trained on USPTO granted patent data from 1976 to 2019,
their 300-dimensions embeddings provide document-level representations for
each patent text111Further, their model was computed using the Gensim Python
library, using the distributed bag of words (DBOW) algorithm with 10 epochs..
Their validation procedure consists in checking the correlation between the
obtained measure of patent similarity, measured as cosine among vectors, and
being in the same technological field (CPC) or between citing and citetd
patents. However, this kind of measures might have some drawbacks. First,
patents in the same patent class might cover different technologies which
share a similar purpose; at the same time, often the same patent is attributed
to more than one technological class, creating further confusion. Second, the
similarity relationship between citing and citetd patent is an indeed debated
topic in the economics of innovation literature which relevant scholars state
might be biased (Bacchiocchi and Montobbio, 2010).

The work of Bekamiri et al. (2021) and the one of Hain et al. (2022) combine
the multi-label classification task with the patent similarity calculation.
Bekamiri et al. (2021) use USPTO granted patent data from 2013 to 2017, for a
total of 1’492’294 patents. Their choice involves the use of patent claims, in
particular of the first claim only, as it is usually the fundamental claim
from which the rest of the patent application is derived from. Their technique
is also original, as they propose an augmented SBERT approach, following the
paper of Thakur et al. (2020). The model is created according to the following
steps: first they fine-tune RoBERTa as a cross-encoder over a small STS
benchmark dataset; then, they use the trained RoBERTa to label 1’143 patent
claims; finally, they train SBERT on the labeled target dataset (a small STS +
3’432 pair of claims sentences). Their overall result for the classification
is represented by a F1 score in the multilabel classification task with CPC
codes of 66%. Finally, Hain et al. (2022) use around 48 millions of patents in
English language filed at the EPO in a time range from 1980 to 2017. They use
patents’ abstracts only to produce a word2vec based encoding which is then
multiplied by a TF-IDF weighted bag-of-word representations of the abstracts.
As a result, they obtain a 300-dimensional patent signature vector that they
use for similarity calculations. For the multi-label classification task, they
report a F1 of 52%. Further, to validate their similarity measures, they
investigate the correlation between their generated p2p similarity and the
existing observable measures commonly used to approximate technological
similarity, such as being filed from the same inventor, being in the same
technological class or citing each other. However, Hain et al. (2022)
acknowledge the limitations of this validation approach, very similar to the
one proposed by Whalen et al. (2020), and stress the lack of a ground-truth to
evaluate patent similarity.

Table 1: Most recent contributions related to creation of word embeddings for
patent text using neural networks

Auhtor(s) | Year | Data | Text | Task | Technique |  | Performance/
---
validation
Li et al. | 2018 |  | USPTO 2006-2015
---
(2’000’147)
1.title and abstract |  | multi-label classification
---
(IPC codes)
| skipgram +
---
CNN
| Precision:
---
73.88%
F1:
43% top 5
Lee and Hsiang | 2020 |  | USPTO (two versions: first as in Li et al.
---
second of 3’050’615 obs. )
| 1.title and abstract
---
2.claims
| multi-label classification
---
(IPC and CPC codes)
| fine-tuning of
---
BERT uncased +
CNN
| v1(IPC):
---
Precision:
30.31%
F1:
45% top 5
v2(CPC):
Precision best top 1:
85%
F1 best top 1:
67%
Whalen et al. | 2020 |  | USPTO (utility) granted
---
(1976-2019)222the number of patents has not been specified in the paper
| 1\. description
---
2\. independent claim(s)
text
Patent similarity calculation | Doc2vec (trained) |  | Correlation between patent similarity
---
and being in the same CPC field and
between citing and citetd patents (backward
and forward citations)
Bekamiri et al. | 2021 |  | USPTO 2013-2017
---
(1’492’294)
1.first claim |  | Patent similarity calculation and
---
multi-label classification (CPC
codes)
| augmented approach for
---
fine-tuning SBERT
| F1:
---
66%
Hain et al. | 2022 |  | EPO granted 1980-2017
---
(around 12 millions of patent applications)
1.abstract |  | Patent similarity calculation and
---
multi-label classification (IPC
codes)
| Word2vec TF-IDF weighted
---
BoW
| F1:
---
52%
Correlation between being in the same IPC
class and being similar and between citetd
and citing patents (backward and forward
citations)

##  3 Data

Two datasets are crafted in this paper and both are created using online free
available data only. The former dataset includes the necessary triplets for
creating the two SBERT derived models using data from Patents View; the latter
is the patent interference dataset, which, after some preprocessing operations
explained in Section 3.2, will be used as ground-truth benchmark dataset for
evaluating the different similarities measures, as it reports some couple of
claims which experts at the USPTO judged as covering the exact same invention.
More detail about both datasets is provided in the following subsections.

###  3.1 Triplets creation: anchor, positive and negative

The first dataset used in this paper is the one including the necessary
triplets for creating the two proposed SBERT versions. In particular, the
triplets are created starting from three tables from Patents View. The Patents
View platform is built on a regularly updated database that longitudinally
links inventors, their organizations, locations, and patent characteristics.
The data collected from Patents View include three kinds of information which
are useful for creating the triplets333The data is free and accessible at
https://patentsview.org/download/data-download-tables:

  1. 1.

Information on the technological CPC class(es) assigned to each patent
application (Table g_cpc_current)

  2. 2.

Information on the filing date of the patent (Table g_application)

  3. 3.

Patents’ abstracts (Table g_patent)

In particular, the Table g_cpc_current contains information on 7’469’689
patents, among which we select 940’564 that are assigned only to one CPC code.
We do this as our goal is to pair similar patents and the easiest choice is to
pair those having the same CPC code. Further, we select only those patents
whose CPC code is defined as ”inventional”, considering the other type, the
”additional”, usually identifies additional classes to which the patent can be
assigned and not the main one. The second step is to attach to each patent the
information about the filing date, coming from the Table g_application. Third,
we retrieve the abstract of each patent using the Table g_patent. At this
point we select only utility patents, for a total of 939’517 observations for
which we have patent identifier, filing date, the abstract and the single CPC
class to which it is assigned. For the purpose of this research, we consider
two patents belonging to the same CPC class if their CPC section, CPC class,
CPC group and CPC subgroup are the same. After, we create groups of patents
which share the same CPC class and filing year and delete those groups which
do not contain two patents at least, cutting our observations to 568’086. We
select patent pairs filed close in time as patenting activities are usually
consistent with technological trends. In this sense, patents filed close in
time might be more similar than those filed years apart. Further, considering
that technological trends might drastically change in very short periods of
times (Sobolieva et al., 2021) 444For instance, the history of Kodak is an
example of such dramatic changes. In 1998, the company produced 85% of the
world’s volume of paper for cameras and there were 170 thousands employees. In
fact, three years later digital cameras have replaced the company with its
market and it became bankrupt. However, different kinds of technologies might
have technological cycles of different length., we select couples of
technologies filed in the same civil year. After, doing all possible pairwise
combinations among patents which have the same CPC code and are filed in the
same year, we obtain 1’537’809 pairs; in addition, we exclude those couples
whose abstract is the same.555The phenomenon according to which two different
patents have the same abstract is known as patent continuation, where a new
patent is filed by an applicant who wants to pursue additional claims to an
invention disclosed in an earlier application. More information on this issue
could be found, among other, in Righi and Simcoe (2020). For each of the
created pairs (anchor and positive), we randomly add an abstract (negative)
from a patent which does not belong the same CPC class of the two others in
the pair.

###  3.2 Patent interferences: a dataset for validation

Section 1 quickly introduced the phenomenon of patent interferences. To
recall, an interference takes place when an examiner at the patent office
finds two applications whose claims are partially or totally overlapping and
therefore a panel of judges is needed to decide who put together the idea
first. The recent work by Ganguli et al. (2020) explores this issue and
provides a public dataset which contains information on 1’324 interferences
cases, filed from 1998 to 2014, involving USPTO patents666The dataset is
available in the ”Additional material” section at the following url
https://www.aeaweb.org/articles?id=10.1257/app.20180017. For each
interference, they provide the number of patent applications which are
involved in the cases. However, there is no information on which the exact
claims involved from each application are and the claims themselves are not
provided along with the interferences dataset. For these reasons, we
complement the information in the interferences dataset with data from Patents
View and we create a strategy to select the most likely couple of claims
object of the interference. In particular, for each patent application we
first retrieve all its pre-grant claims.777Patent claims information, divided
by year, is available at https://patentsview.org/download/pg_claims Pre-grant
claims information is available only from 2001 onwards, therefore we have to
limit our dataset to interferences cases which involved patents filed between
2001 and 2014 included, leaving us with 219 interferences cases linked to 338
patent applications. Furthermore, we exclude those interferences where we do
not have information about the claims for both the patents involved and those
including more than two patents at a time; in that way, we obtain a dataset
which contains 133 interferences linked to 260 unique patent
applications.888The same patent application can be involved in more than one
interference. After this step, for each of the patent applications we consider
the independent claims999Usually patents have a mixture of independent and
dependent claims; the main difference among the two is that, while the former
can ”stand alone”, the latter are linked to the former for both technological
and semantic content. only, for a total, after performing some preprocessing
operations.101010For example, we deleted those claims which included the word
”canceled/cancelled” as they were not informative about the content of the
patent application., of 1’447 total unique independent claims. With those
claims, we generate all the possible combinations belonging to two distinct
patent applications in the same interference, obtaining a total of 5’537
couples. Among those, for each interference we select the couple with the
greatest patent similarity, measured using the encoding of SBERT (Reimers and
Gurevych, 2019). At the end, we retain 133 pairs of claims, one for each of
the interference cases. Table 2 reports some of the couples that were selected
as an example. The interference data in this form will be used as ground-truth
benchmark dataset to evaluate the performance of the different models proposed
in literature to estimate patent similarity and compare them with those
originally proposed in this research.111111It would be useful to compare this
dataset to the one used for fine tuning, as presented in Section 3.1. However,
the dataset presented in this section only has 133 pairs of sentences which we
then use for validation of the different models; therefore, an analysis
comparing the two would not be helpful. This is a limitation of this research
which is further discussed in Section 6.

Table 2: Example of patent interferences claims’ couples No. |  Claim 1 |  Claim 2
---|---|---
105075 |  A method for the preparation of citalopram wherein the aldehyde of formula 16 is converted to the corresponding 5-cyano compound of formula (i) 17 which is alkylated to form citalopram, which is isolated in the form of the base or an acid addition salt thereof |  A production method of citalopram represented by the formula which comprises reacting a compound of the formula with 3-(dimethylamino) propyl chloride in the presence of a condensing agent and at least one member selected from tetramethylethylenediamine and 1,3-dimethyl-2-imidazolidinone.
105099 |  An antibody that binds to mouse flt-3 ligand, wherein said mouse flt-3 ligand comprises amino acids 28-163. |  An antibody that is immunoreactive with a flt3-l polypeptide.
105224 |  A glass door for a refrigerated display case, the door comprising: a first glass panel having in inside and an outside surface. A low emissivity coating on the inside surface of the first glass panel; a second glass panel having in inside and an outside surface; a low emissivity coating on the inside surface of the second glass panel; an intermediate glass panel between the first and second glass panels; a first spacer assembly between the first and intermediate glass panels and a second spacer assembly between the intermediate and second glass panels wherein the first and second spacer assemblies are formed from warm edge spacer assemblies; and a frame extending about and supporting at least one of the glass panels. |  A refrigeration door having an outer surface and adapted to be mounted on a refrigerating compartment, said door comprising: a first sheet of glass; a second sheet of glass; a first sealant assembly disposed around the periphery of said first sheet of glass and said second sheet of glass for maintaining said first sheet and said second sheet in spaced-apart relationship from each other; a first low emissivity coating adjacent a surface of said first sheet or said second sheet of glass; said first sheet and second sheets of glass, said first sealant assembly, and said first low emissivity coating forming an insulating glass unit having a u value substantially equal to or less than 0.2 btu/hr-sq ft-f; and a frame secured around the periphery of said insulating glass unit.
105236 |  A juvenile seat assembly is provided for use with both a vehicle seat and anchor mounts provided in or near the vehicle seat, the assembly comprising a juvenile seat having a first pair of openings and a second pair of openings, an anchor belt including a strap having a central portion and opposite end portions, and a connector coupled to each end portion and adapted to be coupled one of the anchor mounts, the anchor belt being threaded through the first openings to position the seat in a first, rearwardly facing position and the anchor belt being threaded through the second openings to position the seat in a second, forwardly facing position, and a leash coupled to the juvenile seat and to the central portion of the anchor belt, the leash remaining coupled to the seat and to the anchor belt when the anchor belt is moved between the first and second openings. |  An improved child car seat having a plurality of belt restraining guides for securing the child car seat to an automobile seat, the improvement comprising: a. an installation belt having a belt length adjuster, a belt first end having a fastener, a belt second end having a fastener, and a belt intermediate region between the ends, and; b. a tether having a tether first end attached to an anchor point on the child car seat and a tether second end attached to an anchor point on the installation belt, wherein the installation belt is anchored to the child car seat only by the tether.
105252 |  An implantable device for insertion into a cavity in a vertebral body comprising: a flexible container having a wall membrane: said wall membrane defining an interior and an exterior of said container; said wall having at least one hole connecting the interior with the exterior; a fill tube coupled to said container at a location proximate an edge of said container for injecting a flowable or fluid bone filler material into said container such that said bone filler leaves said interior and enters said vertebral body. |  An implantable device for insertion into a cavity in a vertebral body comprising: a container including; an upper wall member; a lower wall member; a circumferential wall member; said wall members together defining a single chamber.

_Notes:_ The table reports 5 examples of couples of claims belonging to the
two patent applications involved in a certain interference; column 1 reports
the interference number, while column 2 and column 3 reports the couple of
most similar claims from the two patent applications.

##  4 Models and experiments

This research first tests whether static or contextual embeddings perform
better on the patent similarity task. In particular, we consider the following
models from the literature:

  * •

Word2vec TF-IDF model by Hain et al. (2022)

  * •

Doc2vec model by Whalen et al. (2020)

  * •

Augmented SBERT model (Patent sBERTa) by Bekamiri et al. (2021)

Furthermore, this research proposes two original models to be compared with
the others, both deriving from the Sentence Transformers architecture. To
fine-tune both architectures, we exploit the triplets dataset as explained in
Section 3.1. :

  1. 1.

Patent SBERT-ub: this model is a fine-tuning of the RoBERTa base model by Liu
et al. (2019) which uses a random 10% of our triplets dataset121212The RoBERTa
model used is available at https://huggingface.co/roberta-base, we trained
this model with a batch size of 8 and 1 epoch.

  2. 2.

Patent SBERT-adapt-ub: this model is a domain adaptation of the SBERT model
presented in Reimers and Gurevych (2019) which uses a random 10% of our
triplets dataset 131313The SBERT model used is available at
https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2, we trained this
model with a batch size of 8 and 1 epoch.

Those two models have been trained using the architecture proposed by Reimers
and Gurevych (2019). The dataset described in Section 3.1 has been split in
traning and validation using respectively 70% and 30 % of the data. Further,
in both cases we used mean as pooling strategy. As for the loss function, we
used the TripletLoss function141414 For further detail please refer to the
sentence-transformers package documentation
https://www.sbert.net/docs/package_reference/losses.html. introduced by
Schroff et al. (2015), where the distance is calculated using euclidean
distance. The minimum margin for the distance between the anchor and the
negative has been set to 5. Finally, we used the validation step each 1’000
training steps and in both cases we used 1 epoch for the training.

Second we test, among transformers based models, which architecture performs
best in the patent similarity task; therefore, for the second analysis, we
limit the analysis to Patent sBERTa, Patent SBERT-ub and Patent SBERT-adapt-
ub.

To evaluate our results, we exploit the dataset of Ganguli et al. (2020) about
patent interferences, whose preparation steps are explained in the Section 3.
In particular, knowing that the pairs of claims in the interference shall
represent a case of maximum similarity, we define the best performing model
the one which predicts in the majority of the cases the highest cosine
similarity among those sentences. Furthermore, to prevent the case in which a
certain model might overestimate the closeness among claims in general, we
compare the results of the maximum similarity with those of the same dataset
in which we create random couples of claims belonging to different
interferences. In this case, we define the best performing model as the one
which predicts the lower cosine similarity among the randomly generated pairs
in most of the cases.

##  5 Analysis

Table 3 reports the results of the comparison among the five models described
in Section 4 on the task of patent similarity on the interferences dataset,
veryfing whether static or contextual embeddings perform better on this
specific task. The first column reports the name of the considered model, the
second column the percentage of cases in which it reported the highest cosine
similarity for each interference and the third one the percentage of cases in
which a certain model reported the minimum value for cosine similarity among
randomly generated pairs of claims. The model which has the greatest figures
in both cases is Patent-SBERT-ub-adapt. Interestingly, the word2vec TF-IDF
model by Hain et al. (2022) performs better than the doc2vec by Whalen et al.
(2020) and the SBERT by Bekamiri et al. (2021). This might be due to the fact
that the word2vec model has been trained on a very large number of abstracts
(48 millions), therefore the representation for each patent abstract is quite
accurate. On the other hand, Patent SBERTa has been fine tuned using 3’432
pairs of claims only. This highlights that static embeddings could also have
good performances on specific tasks when their training is large enough even
when comparing with more recent architectures such as transformers.

Table 3: Percentage of cases of greatest and lowest similarity by model (all)

Model |  | Max similarity
---
(%)
| Min similarity
---
(%)
Patent SBERT_ub_adapt | 52 | 40
Patent SBERT_ub | 32 | 26
Word2vec TF-IDF | 11 | 15
Doc2vec | 4 | 6
Patent SBERTa | 1 | 13

_Notes:_ The table reports the results of the comparison among the five models
(Word2vec TF-IDF, Doc2vec and Patent SBERTa from previous works plus the two
original models). In particular, the second column shows the no. of times, in
percentage, in which each of the models assigned the maximum similarity score
to each of the 133 couples of the interferences dataset; the third column
reports in fact the no. of times, in percentage, in which each of the models
assigned the lowest similarity score for the randomly generated pairs of
patent claims belonging to the interferences dataset.

Table 4: Percentage of cases of greatest and lowest similarity by model (SBERT
only)

Model |  | Percentage with
---
max similarity
| Percentage with
---
min similarity
Patent SBERT_ub_adapt | 60 | 47
Patent SBERT_ub | 37 | 30
Patent SBERTa | 3 | 23

_Notes:_ The table reports the results of the comparison among the three SBERT
models (Patent SBERTa from Bekamiri et al. (2021) and 2 original). In
particular, the second column shows the no. of times, in percentage, in which
each of the models assigned the maximum similarity score to each of the 133
couples of the interferences dataset; the third column reports in fact the no.
of times, in percentage, in which each of the models assigned the lowest
similarity score for the randomly generated pairs of patent claims belonging
to the interferences dataset.

Table 4 reports the results for the comparison among SBERT models only. From
the table, we acknowledge that, among the three SBERT architectures, the best
performing one is the domain adaptation of the SBERT by Reimers and Gurevych
(2019). The better performance of the domain adaptation of SBERT is consistent
with the idea that it is valuable to complement work on large LMs and use
domain and task relevant corpora to specialize models (Gururangan et al.,
2020).

##  6 Conclusion

This paper enriched the discussion about textual patent similarity in several
ways: first of all, it analyzed the differences among performance of
pretrained static versus contextual embeddings on the task of patent
similarity and proposed a new SOTA contextual model (Patent-SBERT-ub-adapt);
second, it compared different architectures of SBERT models to identify which
one better performs for the patent similarity task. Consistently, from the
results we can derive two main contributions: first of all, a contextual model
is the best performer on the patent similarity task; however, we demonstrated
that not always the more recent contextual embeddings perform better than
static embeddings as well. Therefore, we claim that the performance is also
linked to the task itself and to the ways in which the training of the models
took place. Second, among SBERT architectures, Patent-SBERT-ub-adapt is the
one which leads to the best results, confirming the benefits of domain
adaptation for pretrained models (Gururangan et al., 2020) .

However, this research is not free from limitations. For instance, the domain
adaptation of SBERT might be less general than the original architecture by
Reimers and Gurevych (2019). Another important limitation is related to the
fact that we do not have a ground-truth labeled dataset for very dissimilar
patents, therefore we just create random couples of claims. The best option
would be in fact to have another manually labeled dataset. At the same time, a
bigger ground-truth dataset could improve the overall reliability of this
analysis and allow allineation between the dataset using for fine-tuning and
the validation dataset.

Eventually, future work could improve the current creation process of the
triplets dataset. In particular, the triplets could be created using not only
the patents’ asbtracts but also one or more claims and exploit their hierarchy
as well.

## Acknowledgments

This research was funded by CNRS as part of the 2022-2023 ”Programme
Prématuration”.

## References

  * Arts et al. (2021) Sam Arts, Jianan Hou, and Juan Carlos Gomez. 2021.  Natural language processing to identify the creation and impact of new technologies in patent text: Code, data, and new measures.  _Research Policy_ , 50(2):104144.
  * Bacchiocchi and Montobbio (2010) Emanuele Bacchiocchi and Fabio Montobbio. 2010.  International knowledge diffusion and home-bias effect: Do uspto and epo patent citations tell the same story?  _Scandinavian Journal of Economics_ , 112(3):441–470.
  * Bekamiri et al. (2021) Hamid Bekamiri, Daniel S Hain, and Roman Jurowetzki. 2021.  Patentsberta: a deep nlp based hybrid model for patent distance and classification using augmented sbert.  _arXiv preprint arXiv:2103.11933_.
  * Bekamiri et al. (2022) Hamid Bekamiri, Daniel S Hain, and Roman Jurowetzki. 2022.  A survey on sentence embedding models performance for patent analysis.  _arXiv preprint arXiv:2206.02690_.
  * Betti et al. (2020) Arianna Betti, Martin Reynaert, Thijs Ossenkoppele, Yvette Oortwijn, Andrew Salway, and Jelke Bloem. 2020.  Expert concept-modeling ground truth construction for word embeddings evaluation in concept-focused domains.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6690–6702.
  * Bonino et al. (2010) Dario Bonino, Alberto Ciaramella, and Fulvio Corno. 2010.  Review of the state-of-the-art in patent information and forthcoming evolutions in intelligent patent informatics.  _World Patent Information_ , 32(1):30–38.
  * Chen and Chang (2012) Yen-Liang Chen and Yuan-Che Chang. 2012.  A three-phase method for patent classification.  _Information Processing & Management_, 48(6):1017–1030.
  * Ganguli et al. (2020) Ina Ganguli, Jeffrey Lin, and Nicholas Reynolds. 2020.  The paper trail of knowledge spillovers: Evidence from patent interferences.  _American Economic Journal: Applied Economics_ , 12(2):278–302.
  * Grawe et al. (2017) Mattyws F Grawe, Claudia A Martins, and Andreia G Bonfante. 2017.  Automated patent classification using word embedding.  In _2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA)_ , pages 408–411. IEEE.
  * Gupta and Jaggi (2021) Prakhar Gupta and Martin Jaggi. 2021.  Obtaining better static word embeddings using contextual embedding models.  _arXiv preprint arXiv:2106.04302_.
  * Gururangan et al. (2020) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020.  Don’t stop pretraining: Adapt language models to domains and tasks.  _arXiv preprint arXiv:2004.10964_.
  * Hain et al. (2022) Daniel S Hain, Roman Jurowetzki, Tobias Buchmann, and Patrick Wolf. 2022.  A text-embedding-based approach to measuring patent-to-patent technological similarity.  _Technological Forecasting and Social Change_ , 177:121559.
  * Hepburn (2018) Jason Hepburn. 2018.  Universal language model fine-tuning for patent classification.  In _Proceedings of the Australasian Language Technology Association Workshop 2018_ , pages 93–96.
  * Lee and Hsiang (2019) Jieh-Sheng Lee and Jieh Hsiang. 2019.  Patentbert: Patent classification with fine-tuning a pre-trained bert model.  _arXiv preprint arXiv:1906.02124_.
  * Li et al. (2018) Shaobo Li, Jie Hu, Yuxin Cui, and Jianjun Hu. 2018.  Deeppatent: patent classification with convolutional neural networks and word embedding.  _Scientometrics_ , 117:721–744.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  _Advances in neural information processing systems_ , 26.
  * Nikolaev and Padó (2023) Dmitry Nikolaev and Sebastian Padó. 2023.  Representation biases in sentence transformers.  _arXiv preprint arXiv:2301.13039_.
  * Puccetti et al. (2023) Giovanni Puccetti, Vito Giordano, Irene Spada, Filippo Chiarello, and Gualtiero Fantoni. 2023.  Technology identification from patent texts: A novel named entity recognition method.  _Technological Forecasting and Social Change_ , 186:122160.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_.
  * Righi and Simcoe (2020) Cesare Righi and Timothy Simcoe. 2020.  _Patenting Inventions Or Inventing Patents?: Strategic Use of Continuations at the USPTO_.  National Bureau of Economic Research.
  * Roudsari et al. (2021) Arousha Haghighian Roudsari, Jafar Afshar, Suan Lee, and Wookey Lee. 2021.  Comparison and analysis of embedding methods for patent documents.  In _2021 IEEE International Conference on Big Data and Smart Computing (BigComp)_ , pages 152–155. IEEE.
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015.  Facenet: A unified embedding for face recognition and clustering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 815–823.
  * Sobolieva et al. (2021) TO Sobolieva, NG Holionko, LP Batenko, and TI Reshetniak. 2021.  Global technology trends through patent data analysis.  In _IOP Conference Series: Materials Science and Engineering_ , volume 1037, page 012059. IOP Publishing.
  * Strubell et al. (2019) Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019.  Energy and policy considerations for deep learning in nlp.  _arXiv preprint arXiv:1906.02243_.
  * Thakur et al. (2020) Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2020.  Augmented sbert: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks.  _arXiv preprint arXiv:2010.08240_.
  * Tran and Kavuluru (2017) Tung Tran and Ramakanth Kavuluru. 2017.  Supervised approaches to assign cooperative patent classification (cpc) codes to patents.  In _Mining Intelligence and Knowledge Exploration: 5th International Conference, MIKE 2017, Hyderabad, India, December 13–15, 2017, Proceedings 5_ , pages 22–34. Springer.
  * Tseng et al. (2007) Yuen-Hsien Tseng, Chi-Jen Lin, and Yu-I Lin. 2007.  Text mining techniques for patent analysis.  _Information processing & management_, 43(5):1216–1247.
  * Whalen et al. (2020) Ryan Whalen, Alina Lungeanu, Leslie DeChurch, and Noshir Contractor. 2020.  Patent similarity data and innovation metrics.  _Journal of Empirical Legal Studies_ , 17(3):615–639.


# Quality Assurance of A GPT-based Sentiment Analysis System:
Adversarial Review Data Generation and Detection

Tinghui Ouyang1, Hoang-Quoc Nguyen-Son1, Huy H. Nguyen1, Isao Echizen1, and
Yoshiki Seo2
1National Institute of Informatics
2Digital Architecture Research Center, National Institute of Advanced
Industrial Science and Technology, Japan
E-mail: {thouyang, nhhuy<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs) have been garnering significant attention of AI
researchers, especially following the widespread popularity of ChatGPT.
However, due to LLMs’ intricate architecture and vast parameters, several
concerns and challenges regarding their quality assurance require to be
addressed. In this paper, a fine-tuned GPT-based sentiment analysis model is
first constructed and studied as the reference in AI quality analysis. Then,
the quality analysis related to data adequacy is implemented, including
employing the content-based approach to generate reasonable adversarial review
comments as the wrongly-annotated data, and developing surprise adequacy
(SA)-based techniques to detect these abnormal data. Experiments based on
Amazon.com review data and a fine-tuned GPT model were implemented. Results
were thoroughly discussed from the perspective of AI quality assurance to
present the quality analysis of an LLM model on generated adversarial textual
data and the effectiveness of using SA on anomaly detection in data quality
assurance.

###### Index Terms:

Large Language Models (LLMs), sentiment analysis, quality assurance,
adversarial examples, surprise adequacy.

##  I Introduction

ChatGPT, developed and released by OpenAI [1], has emerged as the hottest
“super-star” in the field of artificial intelligence (AI) currently. It is
primarily based on LLMs and a variant of the Generative Pre-trained
Transformers (GPT) [2]. Due to its friendly interface and tremendous in-
context learning capability, ChatGPT has achieved great success in various
natural language process (NLP) tasks, like question answering, chatting,
grammar checking, and so on. For example, ChatGPT was utilized as a translator
in [3]. In [4], it was also used to check the grammar errors in writing with a
comparative performance with commercial products. Moreover, another success of
ChatGPT usage is in the education field [5]-[6], like exam testing or
plagiarism.

However, as a kind of LLM-based product, its vast parameters and complex
architecture bring several challenges regarding AI quality management (AIQM)
in the usage of ChatGPT, such as concerns related to security, privacy
protection, attacks, and the risks of misuse. For instance, using ChatGPT for
plagiarism is a severe issue of scientific integrity. Therefore, adhering to
the standard guideline of AI quality assurance [7], it becomes imperative to
thoroughly investigate the quality issues associated with ChatGPT and other
LLMs based on the GPT framework. Currently, the quality issues of ChatGPT and
LLMs have attracted many scholars’ attention to research. For example, the
coverage of ChatGPT was tested on high-resource and low-resource languages,
respectively [3]. The ability of ChatGPT on code and mathematics was also
investigated [8]. Moreover, by taking ChatGPT as an AI product, its qualities
of reliability and robustness were studied in [9]-[10].

In this paper, a specific NLP topic focusing on sentiment analysis is selected
as a reference for the AIQM study on LLMs. Sentiment analysis is the simplest
NLP task, which aims to classify a piece of text into different sentiment
categories, such as positive, negative, and neutral. Companies will use these
comments to evaluate customers’ satisfaction with products/services, guide
manufacturing and production, and predict market movements. However, as
reported in [11]-[13], it is super easy for customers to generate fake
comments in the real world when reviewing products on e-commercial platforms,
especially with the help of ChatGPT. If these fake comments are consistent
with the rating customers gave, it is not a big issue. If the comments are
randomly generated with mismatched rating levels, it will bring a high risk of
economic loss to product manufacturers. A lot of such cases were reported in
[14]. For example, in the context of an app store, if the rating level of a
particular app does not align with the review comment, it can have a
detrimental impact on its future downloads. This discrepancy can lead to a
loss in profits, particularly for upcoming apps with fewer comments.
Furthermore, the prevalence of dishonest reviews and unfair ratings has
emerged as a significant problem in practical commercial sales [15].
Therefore, it is seen that detecting fake comments with unjustifiable labels
has become an intriguing area of research in AIQM, especially data quality
assurance.

In AIQM, data quality analysis suggests that one straightforward approach to
detect these wrongly-annotated review data is through examining data
distribution [16]. However, textual data, being non-numeric and having complex
properties, present challenges in distribution analysis. This is also the
reason why NLP studies usually use some techniques to transform text into
numeric data first, like TF-IDF, bag-of-words, word2vec, or other tokenization
processes [17]-[19]. Moreover, the transformed numerical vectors may have
sparsity and high dimensionality, so it is not directly feasible for data
distribution analysis. With the above considerations, the other type of method
to detect these wrongly-annotated data is through misclassification, which
assumes the given sentiment analysis system is confident to make correct
decisions. For example, a deep learning model based on a recurrent neural
network (RNN) with a gate recurrent unit was constructed for sentiment
analysis, and instances of incorrect decisions are attributed to the wrongly-
annotated data [20]. In [21], three systems using different ML/DL algorithms
were developed to detect the wrongly-annotated data in Android app reviews.
Moreover, review data were categorized into unfair sentiment comments based on
the model’s misclassification in [22]. However, the related question is how to
guarantee the trained AI model is truly confident in sentiment analysis. If
wrongly-annotated data exists in the training dataset without notification,
the practical training process and decision patterns of AI will also be
influenced by those mismatched data. Then, it is hard to guarantee the
confidence and quality assurance of the trained AI model.

Based on the statement and discussion above, this paper proposes to
investigate the wrongly-annotated data and their influence on a given LLM
model for AIQM. First, we orient to the sentiment analysis task and build an
LLM model. Aiming at the reference study, a basic GPT model released by OpenAI
is fine-tuned on the Amazon.com review data which is publicly available and
widely used as the reference in sentiment analysis. Then, we implement some
possible AI quality assurance studies based on the constructed LLM model,
e.g., correctness, robustness and data adequacy analysis. One study is to
acquire the wrongly-annotated review comments. This paper proposes to apply
the content-based method to generate adversarial textual data, which is
required to have similar data qualities with the original ones but leads to
incorrect decisions. The other study is to develop methods for detecting these
abnormal data. This paper develops a new metric based on surprise adequacy
(SA) [23], and evaluates its performance on abnormal data detection and data
quality assurance.

The rest of this paper is organized as follows. Section II presents the
general definition of adversarial samples in deep neural networks (DNN).
Section III introduces the methods for generating adversarial textual data,
especially the wrongly-annotated review comments in this paper. Section IV
proposed advanced techniques to detect those adversarial data for quality
analysis on data adequacy. Experiments and numerical comparisons on GPT-based
sentiment analysis are implemented in Section V. Finally, the results and
conclusions are summarized in the Section VI.

##  II Definition of DNN’s adversarial data

According to the above description, it is easily seen that the wrongly-
annotated review comments share a similar concept with adversarial data. In AI
quality assurance [24]-[25], adversarial data have been widely studied as
special data samples fooling a neural network. Typically, in the context of
recognition and classification, the fooling behaviors can result in
misclassification, so they tightly correlate to data adequacy and data
security in AIQM.

To formulate the adversarial examples in mathematics, we can first assume a
deep neural network (DNN) for classification or recognition as
$F(\cdot):R^{d}\rightarrow[0,1]^{m}$. This model aims to map a $d$-dimensional
input $x$ to the probability vector $F(x)$ with a dimensionality of $m$, where
$m$ is the number of output labels. The input can be numeric data, image data,
or textual data based on the studied scenarios, and the output is the
probability distribution of all possible classification labels. Let the output
of the last layer of DNN be denoted as $Z(\cdot)$, then the label probability
of DNN is usually calculated by the softmax activation function of $Z$,
expressed below.

$F(x)=softmax(Z(x))$ (1)

The predicted class label of $x$ can be determined by the maximum probability
in $F(x)$, namely

$C(x)={\operatorname{argmax}}\ F(x)$ (2)

Generally, to train this classification model, supervised learning is
implemented in the training process along with a set of training samples
consisting of $(x,y)$ where $y$ is the target class label.

Then, based on the definition of adversarial samples, the following equation
can formulate the adversarial samples.

$\left\\{\begin{array}[]{l}\hat{x}=x+\delta\\\ s.t.\ C(\hat{x})\neq C(x)\\\
\end{array}\right.$ (3)

This equation illustrates that adversarial samples are actually produced by
original samples with additional small perturbation $\delta$. This
perturbation is usually thought imperceptible to human beings. However, they
can lead the trained DNN $F(\cdot)$ to an incorrect label.

##  III Adversarial textual comments generation

Based on the definition and formulation of adversarial samples, three
categories of adversarial samples generation methods can be summarized by
considering the information of trained DNN model [26], such as zero-knowledge,
limited- knowledge and perfect-knowledge methods. The first type aims to
generate adversarial samples on an unsecured model $F(\cdot)$ and ignore
whether an attack detector exists or not. The second one is trained to
generate adversarial samples fooling the model $F(\cdot)$ but with limited
knowledge. For example, it is known that there exists a detector securing the
model $F(\cdot)$ against a given attacker, but no details about the detector.
Unlike the second type, the final one has the knowledge of both the detector’s
existence and its architecture and parameters. Therefore, it can use “perfect
knowledge” to generate adversarial samples to bypass the given detector.

Currently, in adversarial data generation study, the popular way is to use the
gradient to optimize the perturbation $\delta$, e.g., using $L_{0-2}$ norm in
objective functions. Some famous methods, like FGSM/BIM/PGD and DeepFool
[27]-[30], have been widely applied in image-based recognition systems. For
textual data, a gradient-based method is also applicable [31], though
gradient-based methods are commonly implemented based on numeric data. While
NLP studies usually involve tokenization and reverse-tokenization processes to
realize the transformation between text and numeric data, it is not completely
feasible to guarantee the same semantic information of original data in
gradient-based adversarial text generation. That means the perturbation
generated by gradient-based methods may not be imperceptible to humans. With
this consideration, this paper proposes to adopt another type of adversarial
data generation method, namely the content-based method [32]. This method
leverages the input content and adds restrained perturbations required to
provide semantically consistent with simulating real-world context. For
example, concerning image data collected from cameras, it is possible to have
some black spots in an image due to a dirty camera lens. Therefore, attackers
can easily add black spots to images to generate adversarial samples instead
of using gradient-based methods. Some examples of this kind of attack were
reported in literature [33]-[34].

Considering the target of this paper is to study the AIQM in an LLM-based
model, especially the data quality assurance in sentiment analysis, the
targeted objective is textual review comments from the real world. Therefore,
if using the content-based method for adversarial data generation, it is
possible to have some typos in the text due to spelling errors or have
different contractions on expression, e.g., using ”hasn’t” to replace ”has
not”, as presented below.

These cases reasonably exist in the real world with high possibility, and they
can surely preserve the semantic information of original data. Therefore, we
can use the content-based method to generate some adversarial textual data
fooling a given AI system, e.g., the GPT-based sentiment analysis model. To
make use of the content-based method in adversarial text generation, this
paper proposed a framework as Fig. 1

Figure 1: Framework of adversarial data generation in GPT-based model

According to the proposed framework, it is seen that there are mainly two
parts:

(1) Fine-tuning a GPT-based model for sentiment analysis

In this paper, we first need to obtain a sentiment analysis model as the
reference in AIQM. Therefore, to evaluate the quality of LLMs, we choose to
fine-tune a model based on the GPT-3 released by OpenAI. Here, the specific
scenarios can be chosen as the sentiment analysis on Amazon.com review
comments, so the public Amazon review dataset is chosen as the training
dataset.

(2) Generating perturbed textual data

To add perturbations to text according to the content-based method, we can use
the useful NLP package ”CHECKLIST” [35], which is easily used to add a
different number of typos and contraction operations to textual data. Then, we
can feed these perturbed data to the fine-tuned GPT model and compare the
classification results. If a mismatched classification label is outputted, we
can collect it as an adversarial candidate for the quality study in the next
steps.

##  IV Adversarial review comments detection

After generating adversarial review comments, we can detect these data for
data adequacy analysis. Considering that ChatGPT API or GPT model provided by
OpenAI seems like a black box, this paper proposed using surprise adequacy as
the tool to develop adversarial data detection methods. Surprise adequacy (SA)
was initially proposed [23] to measure the novelty between testing data and
training data, and it was also verified helpful in AI quality assurance [36]
as well as in NLP study [16]. The detailed processes are described as follows
to calculate the value of SA.

1) Neuron activation status

It is known that the easy way of comparing data differences is through their
numeric values. According to the definition of adversarial samples, we see a
significant difference in adversarial data happening on their behavior with
respect to a given AI model. In SA calculation, the neuron activation status
is usually used to describe the behavior of data, as shown in Fig. 2(a).
Assuming an AI system consisting of a set of neurons
$N=\\{n_{1},n_{2},\cdots\\}$, then the output value or status of a given
neuron $n\in N$ with respect to a testing data $x$ can be directly regarded as
the data’s behavior, denoted as $\varphi(x)$. Then, for a sequence of neurons
in an organized architecture, like a layer or graph, the neuron activation
status of the testing data is meaningful in AI’s quality analysis. Assuming a
subset $N_{S}\subseteq N$, its neuron behaviour on $x$ can be easily expressed
as

$\Phi_{N_{S}}(x)=[\varphi_{n_{1}}(x),\varphi_{n_{2}}(x),\cdots,\varphi_{n_{k}}(x)]^{T},n_{k}\in
N_{S}$ (4)

Then, based on the behaviors on a given subset $N_{S}$, the difference between
testing data and training data could be calculated, namely the surprise
studied in SA. Therefore, SA can be expressed in the following form.

$SA=sp(\Phi_{N_{S}}(x_{t}),\Phi_{N_{S}}(D_{T}))$ (5)

where, $sp(\cdot)$ is the surprise function. $x_{t}$ and $D_{T}$ represent the
testing and training data, respectively. For the consideration of simplifying
calculation, the subset $N_{S}$ can be chosen as neurons of a given layer.
Then, it is easily understood that SA is namely to compare the AI model’s
reaction to testing data and that of training data, so it is useful in
software testing and quality assurance study.

2) Distance-based SA (DSA)

According to the above definition, there are two types of SA defined in [23],
such as likelihood-based SA (LSA) and distance-based SA (DSA). While
considering DSA is more suitable in classification problems, e.g., sentiment
classification, so here DSA is introduced in this paper. The definition of DSA
is expressed in (6) and shown in Figure 2(b).

$DSA(x)=\frac{dist_{a}}{dist_{b}}$ (6)

It is seen that DSA is defined as the ratio between two distances, namely
meaning that the function $sp(\cdot)$ is selected as a distance function. In
[23], these two distances are defined as the following equations

$\left\\{\begin{array}[]{l}dist_{a}=\|\Phi(x_{t})-\Phi(x_{a})\|\\\
dist_{b}=\|\Phi(x_{a})-\Phi(x_{b})\|\end{array}\right.$ (7)

where, $x_{t}$ is a testing data point in the class $y_{t}$. $x_{a}\in D_{T}$
is the nearest data point of $x_{t}$ in the same class $y_{t}$ of training
dataset, defined as

$x_{a}=\underset{C({x_{i}})=y_{t}}{\operatorname{argmin}}\|\Phi(x_{t})-\Phi(x_{i})\|$
(8)

In (7), $x_{b}$ is the nearest data point of $x_{a}$ in another classes of
$D_{T}$, expressed as

$x_{b}=\underset{C({x_{i}})\in\\{C-y_{t}\\}}{\operatorname{argmin}}\|\Phi(x_{a})-\Phi(x_{i})\|$
(9)

where, $\\{C-c_{t}\\}$ represents the set of all classes different with
$c_{t}$. According to the above definitions, it is seen that the DSA value
will be high when a large value of numerator and a small value of denominator
in (6). That implies the testing data $x_{t}$ is distant to its labelled class
$c_{t}$ and close to data of other classes. In this case, it is reasonable to
say it’s surprised to normal data of its belonging class, so having high risk
of being anomaly. Based on this idea, DSA was verified useful to describe
anomaly data’s behaviours respect to a given DL model, and to detect corner
case data [37].

3) DSA modification

While from the original definition of DSA, we see that SA is actually to
compare the surprise of $x_{a}$ (the nearest training data point of $x_{t}$ in
the same class) with that of other classes. Therefore, it can be simplified as
the comparison between $x_{t}$’s novelty in its labeled class and its novelty
in other classes. This definition is helpful to describe the surprise of
testing data to the training dataset. However, there are some shortages in the
calculation of $dist_{b}$ found. For example, if a data point belongs to
corner case data [37], its novelty with respect to all classes seems more
critical. Moreover, considering the distances of (8)-(9) are processing pair-
wise rare data points, if the testing data $x_{t}$ is an extreme point and
happens to have a neighbor $x_{a}$ very close to it, $x\approx x_{a}$, then
the calculation of DSA will be a low value $dist_{a}\rightarrow 0$ which will
mislead the surprise measurement, especially on abnormal data detection.
Therefore, with the above consideration, this paper proposed to improve the
calculation of DSA, especially modifying the measure of $dist_{a}$ and
$dist_{b}$ to improve the generability of DSA, as expressed below.

$\left\\{\begin{array}[]{l}dist_{a}=\left\|\Phi\left(x_{t}\right)-\Phi\left(x_{a}\right)\right\|\\\
dist_{b}=\left\|\Phi\left(x_{t}\right)-\Phi\left(x_{b}\right)\right\|\end{array}\right.$
(10)

where the calculation of $x_{a}$ and $x_{b}$ will be modified as below

$\displaystyle x_{a}=\frac{1}{|X_{a}|}\sum_{x_{i}\in X_{a}}{x_{i}},$
$\displaystyle\ X_{a}=\\{x_{i}|\ C({x_{i}})=y_{t}\ \&\ {x_{i}\in
Nb(x_{t})}\\}$ (11) $\displaystyle x_{b}=\frac{1}{|X_{b}|}\sum_{x_{i}\in
X_{b}}{x_{i}},$ $\displaystyle\ X_{b}=\\{x_{i}|\ C({x_{i}})\in\\{C-y_{t}\\}\
\&\ {x_{i}\in Nb(x_{t})}\\}$

where $Nb(x_{t})$ is defined as a function of finding the neighborhood of the
testing data $x_{t}$ with a given condition; $|X|$ means the cardinality of
the set $X$. Here, the modified definition of $x_{a}$ is to find the center of
$x_{t}$’s neighborhood in the same class $y_{t}$. Meanwhile, $x_{b}$ is the
center of $x_{t}$’s neighborhood in the other class, as shown in Fig. 2(c).

Figure 2: Calculation of distance-based surprise adequacy. (a) activation
trace; (b) original DSA; (c) improved DSA.

With the help of the above modification, some drawbacks of the original DSA
can be eliminated. Moreover, in terms of some special cases, more variants of
DSAs can also be developed based on the description in Figure 2(c), e.g.,
using global or local neighborhood in the calculation of $Nb(\cdot)$. Then,
using these DSAs, we can realize the abnormal data detection in AI quality
assurance, like adversarial data detection for data adequacy study.

##  V Experiments

To study the AI quality issues of LLM, this paper chooses the simplest task of
sentiment analysis as an example. In accordance with the framework in Fig. 1,
the AI model studied in this paper is a GPT-3 model from OpenAI and fine-tuned
on the well-known Amazon.com review dataset. Then, we can generate some
adversarial review comments and evaluate the performance of GPT-based LLM on
data adequacy analysis. Aiming at the mentioned task, this paper mainly
addresses three questions listed as below.

1) Correctness study of the fine-tuned GPT model

2) Adversarial textual data generation and robustness analysis

3) Adversarial data detection for data quality assurance

###  V-A Correctness analysis

Currently, it seems that ChatGPT can help humans to do any NLP jobs under its
popularity. With the help of the foundation model, OpenAI also provides
developers with commercial ChatGPT API for research. Therefore, it is easy to
use ChatGPT API directly to realize the sentiment analysis on Amazon.com
review data. However, as a general model, the ChatGPT API has no defined
boundary for specific sentiment categories. To achieve accurate classification
on a particular task, OpenAI also provides the other option to developers,
namely using their existing GPT model for fine-tuning on a given task. In this
paper, both of these two options are considered in modeling the sentiment
analysis task on Amazon.com review data. Their performances are shown below.

Figure 3: Performance of the fine-tuned GPT model on sentiment analysis

Fig. 3 shows the performance of the fine-tuned GPT model on both the training
and testing process. It is seen that the GPT model can obtain good performance
on sentiment analysis after fine-tuning the Amazon.com review dataset, seeing
the decreasing loss and increasing accuracy in Fig. 3(a)-(b), respectively.
Moreover, to quantitatively investigate the performance of studied models,
Table I presents the testing accuracy of both using ChatGPT API and fine-tuned
GPT model in sentiment analysis.

TABLE I: Accuracy of sentiment analysis models | ChatGPT API | Fine-tuned GPT
---|---|---
Accuracy | 84.54% | 94.91%

Through the comparative analysis, it is seen that the fine-tuned model has
great improvement in accuracy compared with the method using ChatGPT API
directly for sentiment analysis. Therefore, it illustrates the necessity of
fine-tuning when making use of LLM to develop some specific NLP tasks in real
engineering. Moreover, it is reasonable for us to take the fine-tuned GPT
model as the reference in AI quality analysis in sentiment analysis.

###  V-B Adversarial data generation and robustness analysis

Considering that wrongly-annotated data is a realistic data quality problem in
AIQM, these data usually harm modeling and detection. However, the knowledge
of these data is yet to be discovered by developers, and it usually requires
enormous manual efforts for annotation. For example, regarding the Amazon.com
review data, the raw data are obtained from realistic customer reviews.
Sometimes customers make these wrongly-annotated data by mistakes, so it is
unaware to pick them up in data preparation. The other issue is about
labeling. This process is subjective since the sentiment labels of review
comments are usually determined by the rating levels customers give, e.g., the
rating stars. With the above considerations, this paper proposes to generate
some adversarial review comments instead of the realistic wrongly-annotated
data for data quality analysis in Section III. Then, aiming at the given
reference AI model, namely the fine-tuned GPT-based sentiment analysis model,
this paper adopts the content-based method to generate adversarial review
comments. In this paper, using the “CHECKLIST” package, different numbers of
typos are added to the original review comments to generate adversarial
samples. The results are shown below.

TABLE II: attack success rate | 1 typo | 2 typos | 3 typos | 4 typos | 5 typos
---|---|---|---|---|---
ASR | 0.0702 | 0.0814 | 0.0804 | 0.0732 | 0.1068

Table II shows the successful attack rate (ASR) under different numbers of
typos in perturbation. It is seen that the GPT-based model is generally robust
again the content-based attack, like the typo perturbation, since the attacked
samples have a proportion of around 10% in Table II. To further study these
perturbed data’s properties, the text length of successfully-attacked review
comments is summarized in the following figure.

Figure 4: The statistics of text length under successful attacks

From the above figures, it is seen that successful attacks usually happen on
data with short text length, implying long review comments are robust to typo
perturbation. Moreover, Fig. 4(f) shows the average text length of successful
attacks under different numbers of typos. It is seen, as the number of typos
increases in perturbation, the average text length of attached review comments
decreases. It implies that more short review comments are attacked when more
typos are added. It further demonstrates the conclusion that short review
comments are less robust than long review comments, which is reasonable with
human beings’ knowledge in the real world.

###  V-C Adversarial data detection for data quality assurance

Moreover, considering the importance of data adequacy quality analysis, this
paper also proposed advanced anomaly detection methods oriented to the
generated adversarial review comments. Section IV presents the original and
modified distance-based surprise adequacy (DSA) metrics for anomaly detection.
In this paper, to evaluate the AI model’s quality thoroughly, several variants
of DSAs are adopted in the experiment for comparative study. For example, the
original DSA is selected as the reference in adversarial textual data
detection, denoted as DSA0. The modified DSA can be represented as DSA1, which
is set up with the simplest parameters, namely using the nearest point in the
calculation $x_{a}$ and $x_{b}$ in (11). Moreover, considering class centers
are common helpful global descriptors [38], we can also modify the calculation
in (11) with the whole class of data as $X_{a}$ and $X_{b}$, and denote the
DSA variant as DSA2. For comparison, a version of using a local descriptor,
like utilizing a neighborhood consisting of 10-nearest points as $X_{a}$ and
$X_{b}$ in (11), is also applied in this paper, denoted as DSA3. Then, by
extracting the hidden layer’s output as data’s behaviors, these four DSAs are
calculated. Their performance on adversarial review comments detection is
shown via ROC curves, as shown in Fig. 5.

Figure 5: Performance of different DSA metrics on adversarial text detection

To quantitative analyse their performance, numerical results are computed via
AUC-ROC values are shown in the following table.

TABLE III: Numerical results of adversarial text detection | 1 typo | 2 typos | 3 typos | 4 typos | 5 typos | combined
---|---|---|---|---|---|---
DSA0 | 0.9931 | 0.9939 | 0.9879 | 0.9857 | 0.9852 | 0.9683
DSA1 | 0.9927 | 0.9973 | 0.9945 | 0.9894 | 0.9893 | 0.9653
DSA2 | 0.9345 | 0.9311 | 0.9270 | 0.9256 | 0.9188 | 0.9277
DSA3 | 0.9994 | 0.9980 | 0.9964 | 0.9978 | 0.9966 | 0.9707

In Fig. 5 and Table III, four DSA variants are measured based on adversarial
data generated with different numbers of typos. Moreover, a comprehensive
study combining these generated adversarial review comments together is also
implemented in this section. It is seen from these results that DSA3
outperforms the other DSAs, and DSA2 performs the worst via comparison. These
results illustrate that a local descriptor in DSA measurement is better than a
global one. Moreover, the modified DSA can outperform the original one on
adversarial data detection via optimized parameters. Therefore, in the data
adequacy study on GPT-based sentiment analysis, we could select optimized DSA
for abnormal data detection and extract the possible wrongly-annotated review
comments for the GPT-based model’s correctness and robustness analysis.

##  VI Conclusions

In this paper, to investigate the AI quality assurance issue of LLMs, this
paper proposes to fine-tune a GPT model as the reference. Orienting to
classify the sentiment categories of Amazon.com review comments, a GPT-3 model
provided by OpenAI is fine-tuned. Then, reasonable adversarial data based on
the content-based method are generated for data adequacy analysis. These
adversarial texts well preserved the semantic information of the original text
by using typo perturbations. The reference GPT model is demonstrated to be
robust in these adversarial attacks. Moreover, using SA to describe the data’s
novelty and surprise on the model’s behaviors, some variants of DSA were
developed in this paper and used to detect the generated adversarial review
comments. Finally, numerical results show that modified DSA with optimized
local descriptors can achieve good performance in detecting adversarial data
and possibly detect wrongly-annotated review comments in sentiment analysis.
Summarizing all of these results, it is seen that the fine-tuned GPT model
performs well on sentiment analysis, and it is robust to natural adversarial
review comments generated by the content-based method. Moreover, the proposed
DSAs in this paper are feasible and effective in detecting the adversarial
data in the AIQM study of GPT-based models and especially useful for the data
quality assurance quality. Furthermore, based on the results of this paper, it
is possible to study some extra AI qualities with the detected adversarial
textual data in the future.

## Acknowledgement

This research is supported by the project ’JPNP20006’, commissioned by the New
Energy and Industrial Technology Development Organization (NEDO), and partly
supported by JSPS Grant-in-Aid for Early-Career Scientists (Grant Number
22K17961)

## References

  * [1] OpenAI. https://chat.openai.com.chat, 2023.
  * [2] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022.
  * [3] Jiao, W. X., W. X. Wang, J. T. Huang, Xing Wang, and Z. P. Tu. Is ChatGPT a good translator? Yes with GPT-4 as the engine, 2023. arXiv preprint arXiv:2301.08745.
  * [4] Wu, Haoran, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael Lyu. ChatGPT or Grammarly? Evaluating ChatGPT on Grammatical Error Correction Benchmark, 2023. arXiv preprint arXiv:2303.13648.
  * [5] Jalil, Sajed, Suzzana Rafi, Thomas D. LaToza, Kevin Moran, and Wing Lam. Chatgpt and software testing education: Promises & perils, 2023. arXiv preprint arXiv:2302.03287.
  * [6] Fitria, T. N. Artificial intelligence (AI) technology in OpenAI ChatGPT application: A review of ChatGPT in writing English essay. In ELT Forum: Journal of English Language Teaching 12(1), pp. 44-58, 2023.
  * [7] Machine Learning Quality Management Guideline (https://www.digiarc.aist.go.jp/en/publication/aiqm/)
  * [8] Frieder, Simon, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, Alexis Chevalier, and Julius Berner. Mathematical capabilities of ChatGPT, 2023. arXiv preprint arXiv:2301.13867.
  * [9] Reiss, Michael V. Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark, 2023. arXiv preprint arXiv:2304.11085 .
  * [10] Wang, Jindong, Xixu Hu, Wenxin Hou, Hao Chen, Runkai Zheng, Yidong Wang, Linyi Yang et al. On the robustness of chatgpt: An adversarial and out-of-distribution perspective, 2023. arXiv preprint arXiv:2302.12095.
  * [11] Mitrović, Sandra, Davide Andreoletti, and Omran Ayoub. Chatgpt or human? detect and explain. explaining decisions of machine learning model for detecting short chatgpt-generated text, 2023. arXiv preprint arXiv:2301.13852.
  * [12] Dai, Haixing, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Zihao Wu, Lin Zhao, Wei Liu et al. Chataug: Leveraging chatgpt for text data augmentation, 2023. arXiv preprint arXiv:2302.13007.
  * [13] Adelani, David Ifeoluwa, Haotian Mai, Fuming Fang, Huy H. Nguyen, Junichi Yamagishi, and Isao Echizen. Generating sentiment-preserving fake online reviews using neural language models and their human-and machine-based detection. In Advanced Information Networking and Applications: Proceedings of the 34th International Conference on Advanced Information Networking and Applications (AINA-2020), pp. 1341-1354. Springer International Publishing, 2020.
  * [14] Kushal Dave, Steve Lawrence, and David M. Pennock. Mining the Peanut Gallery: Opinion Extraction and Semantic Classification of Product Reviews. In Proceedings of the 12th International Conference on World Wide Web (WWW ’03). ACM, New York, NY, USA, 519–528. 2003. https://doi.org/10.1145/775152.775226
  * [15] Diekmann, A., B. Jann, W. Przepiorka and S. Wehrli, Reputation formation and the evolution of cooperation in anonymous online markets. Am. Sociol. Rev., 79: 65-85, 2014.
  * [16] Ouyang, T., Seo, Y., and Oiwa, Y. Quality assurance study with mismatched data in sentiment analysis. In 2022 29th Asia-Pacific Software Engineering Conference (APSEC) (pp. 442-446). IEEE. 2022, December.
  * [17] Qaiser, Shahzad, and Ramsha Ali. Text mining: use of TF-IDF to examine the relevance of words to documents. International Journal of Computer Applications 181, no. 1: 25-29, 2018.
  * [18] Zhang, Yin, Rong Jin, and Zhi-Hua Zhou. Understanding bag-of-words model: a statistical framework. International journal of machine learning and cybernetics 1, no. 1: 43-52, 2010.
  * [19] Church, Kenneth Ward. Word2Vec. Natural Language Engineering 23, no. 1: 155-162, 2017.
  * [20] Aralikatte, Rahul, Giriprasad Sridhara, Neelamadhav Gantayat, and Senthil Mani. Fault in your stars: an analysis of android app reviews. In Proceedings of the acm india joint international conference on data science and management of data, pp. 57-66. 2018.
  * [21] Elmurngi, Elshrif Ibrahim, and Abdelouahed Gherbi. Unfair reviews detection on amazon reviews using sentiment analysis with supervised learning techniques. J. Comput. Sci. 14, no. 5: 714-726, 2018.
  * [22] Patel, Nidhi A., and Rakesh Patel. A survey on fake review detection using machine learning techniques. In 2018 4th International Conference on Computing Communication and Automation (ICCCA), pp. 1-6. IEEE, 2018.
  * [23] Kim, Jinhan, Robert Feldt, and Shin Yoo. Guiding deep learning system testing using surprise adequacy. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE), pp. 1039-1049. IEEE, 2019.
  * [24] Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Srndi/c, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 387–402, 2013.
  * [25] C. Szegedy et al. Intriguing properties of neural networks. in Proc. 2nd Int. Conf. Learn. Representation, pp. 14–16, 2014.
  * [26] N. Carlini and D. A. Wagner, Adversarial examples are not easily detected: Bypassing ten detection methods/ in Proc. 10th ACMWorkshop Artif. Intell. Secur., pp. 3–14, 2017.
  * [27] Ian J. Goodfellow, J. Shlens, and C. Szegedy, Explaining and harnessing adversarial examples. in Proc. 3rd Int. Conf. Learn. Representation, pp. 7–9, 2015.
  * [28] A. Kurakin, Ian J. Goodfellow, and S. Bengio, Adversarial examples in the physical world. in Proc. 5th Int. Conf. Learn. Representation, 2017, arXiv:1607.02533.
  * [29] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, Towards deep learning models resistant to adversarial attacks, in Proc. 6th Int. Conf. Learn. Representat., 2018, arXiv:1706.06083.
  * [30] Moosavi-Dezfooli, S. M., Fawzi, A., and Frossard, P. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2574-2582, 2016.
  * [31] Gao, J., Lanchantin, J., Soffa, M. L., and Qi, Y. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW) (pp. 50-56). IEEE. 2018, May.
  * [32] Ma, S., Liu, Y., Tao, G., Lee, W. C., and Zhang, X. Nic: Detecting adversarial samples with neural network invariant checking. In 26th Annual Network And Distributed System Security Symposium (NDSS 2019), 2019, January.
  * [33] I.Evtimov, K.Eykholt, E.Fernandes, T.Kohno, B.Li, A.Prakash, A.Rahmati, and D. Song, Robust physical-world attacks on deep learning models, 2017. arXiv preprint arXiv:1707.08945.
  * [34] K. Pei, Y.Cao, J.Yang, and S.Jana, Deepxplore:Automated white box testing of deep learning systems. in Proceedings of the 26th Symposium on Operating Systems Principles. ACM,pp.1–18, 2017.
  * [35] Ribeiro, M. T., Wu, T., Guestrin, C., and Singh, S. (2020). Beyond accuracy: Behavioral testing of NLP models with CheckList. arXiv preprint arXiv:2005.04118.
  * [36] Ouyang, T., Isobe, Y., Sultana, S., Seo, Y., and Oiwa, Y. Autonomous driving quality assurance with data uncertainty analysis. In 2022 International Joint Conference on Neural Networks (IJCNN) (pp. 1-7). IEEE. 2022, July.
  * [37] Ouyang, T., Marco, V. S., Isobe, Y., Asoh, H., Oiwa, Y., and Seo, Y. Improved Surprise Adequacy Tools for Corner Case Data Description and Detection. Applied Sciences, 11(15), 6826, 2021.
  * [38] Lee, D., Yu, S., and Yu, H. Multi-class data description for out-of-distribution detection. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1362-1370, 2020, August.