# Sim-GPT: Text Similarity via GPT Annotated Data

Shuhe Wang♠, Beiming Cao◆, Shengyu Zhang♣, Xiaoya Li▼
Jiwei Li♣, Fei Wu♣, Guoyin Wang★, Eduard Hovy▲

###### Abstract

Due to the lack of a large collection of high-quality labeled sentence pairs
with textual similarity scores, existing approaches for Semantic Textual
Similarity (STS) mostly rely on unsupervised techniques or training signals
that are only partially correlated with textual similarity, e.g., NLI-based
datasets.

To tackle this issue, in this paper, we propose the strategy of measuring text
similarity via GPT annotated data (Sim-GPT for short). The core idea of Sim-
GPT is to generate data with STS labels using GPT-4, based on which an STS
model is trained. Sim-GPT framework utilizes LLMs to provide a substantial
amount of reliable annotated data filling the gap of the lack of training
signals for STS. Sim-GPT is trained on a one-time generated dataset using BERT
or RoBERTa as the backbone, which offers long-term savings in cost and speed
compared to repeatedly invoking LLMs for each sentence pair.

Trained on the examples from GPT-4 (371K), Sim-GPT yields SOTA performances on
the widely-used seven STS benchmarks: +0.99 over supervised-SimCSE Gao et al.
(2021), and +0.42 over the current SOTA PromCSE Jiang et al. (2022) model. To
encourage further advancements of the field, we release both models and the
371K annotated examples from GPT-4. Code, models and annotated data are
available at: https://github.com/ShuheWang1998/Sim-GPT.

Sim-GPT: Text Similarity via GPT Annotated Data

Shuhe Wang♠, Beiming Cao◆, Shengyu Zhang♣, Xiaoya Li▼ Jiwei Li♣, Fei Wu♣,
Guoyin Wang★, Eduard Hovy▲

††footnotetext: ♠Peking University, ♣Zhejiang University, ▼Shannon.AI
◆University of Texas at Austin, ▲University of Melbourne,
★Bytedance††footnotetext<EMAIL_ADDRESS><EMAIL_ADDRESS>
{sy_zhang, jiwei_li<EMAIL_ADDRESS>beiming.cao @utexas.edu,
<EMAIL_ADDRESS>eduard.hovy @unimelb.edu.au

##  1 Introduction

Measuring Semantic textual similarity (STS) Devlin et al. (2018); Liu et al.
(2019); Reimers and Gurevych (2019); Lan et al. (2019); Gao et al. (2021); Liu
et al. (2019); Jiang et al. (2022); Seonwoo et al. (2022); Chuang et al.
(2022); Sun et al. (2022); Chen et al. (2023) between sentences is an
important task in natural language processing. A longstanding issue with STS
is the lack of supervised training data: it is infeasible to obtain a large
collection of high-quality labeled sentence pairs with textual similarity
scores. Existing approaches either are in a unsupervised fashion Jiang et al.
(2019); Wang et al. (2019); Lan et al. (2019); Yang et al. (2019); Raffel et
al. (2020); Zaheer et al. (2020) or rely on training signals that are
partially correlated with textual similarityReimers and Gurevych (2019); Yan
et al. (2021); Gao et al. (2021); Jiang et al. (2022), e..g, SNLIBowman et al.
(2015) and MNLIWilliams et al. (2017) datasets whose annotations indicate
whether the entailment or contradiction relationship holds between two
sentences. Both unsupervised approaches and NLI-based approaches suffer from
the lack of direct supervised signals for training STS models.

Large language models (LLMs) Brown et al. (2020); Zhang et al. (2020); Rae et
al. (2021); Chowdhery et al. (2022); Tan et al. (2022); Ouyang et al. (2022);
OpenAI (2023); Touvron et al. (2023) open a new door for the task of STS. With
their impressive language comprehension abilities, these models have the
ability to directly generate accurate semantic similarity scores for pairs of
sentences. However, there are two main concerns when using LLMs to provide STS
scores. Firstly, it can be costly to use LLMs like GPT each time a new
sentence pair is encountered. Secondly, the speed of LLMs is typically
significantly slower compared to models like BERTDevlin et al. (2018) or
RoBERTaLiu et al. (2019).

To better harness LLMs’ ability for the STS task, in this paper, we propose
Sim-GPT. Sim-GPT does not directly ask LLMs (e.g., GPT-4) to provide STS
scores for a newly-encounter sentence pair. But rather, it firstly asks LLMs
to generate a relatively large set of training data; secondly, a smaller model
(e.g., backboned by BERT or RoBERTa) is trained based on the synthesized data
from LLMs; At test time, the trained model is used for inference.

Sim-GPT tackles the aforementioned issues simultaneously. It resolves the
long-standing problem of the lack of training signals for STS by utilizing
LLMs to provide a substantial amount of reliable supervision labels for
training. Additionally, it addresses concerns regarding cost and speed. In
terms of cost, while generating synthesized data using LLMs can still be
expensive for large training sets, this cost is incurred only once. The
trained model can then be repeatedly used by developers and the community
without any additional expenses, unlike the scenario where the LLM needs to be
invoked every time an STS score needs to be computed. As for speed, the
inference process using a BERT or RoBERTa backboned model is significantly
faster.

Trained on the 646K synthesized examples from GPT-4 (371K) and NLI (275K)
data††SNLIBowman et al. (2015) and MNLIWilliams et al. (2017), Sim-GPT yields
SOTA performances on the widely-used seven STS benchmarks: +0.99 over
supervised-SimCSE Gao et al. (2021), and +0.42 over the current SOTA PromCSE
Jiang et al. (2022) model. The contribution of this work can be summarized as
follows:

  1. 1.

We propose Sim-GPT, a general framework that utilizes data produced by LLMs to
train the STS model effectively.

  2. 2.

We collect a dataset consisting of 371K annotated STS examples from GPT-4.
This dataset will be made available to the community to encourage further
advancements.

  3. 3.

Sim-GPT achieves SOTA performances on the STS benchmark, demonstrating the
effectiveness of the proposed framework.

The rest of this paper is organized as follows: in Section 2, we go through
related work. Section 3 describes the data generation process in detail.
Section 4 describes the model training process. Section 5 details experimental
results. Ablations studies are presented in Section 6, followed by a brief
conclusion in Section 7.

Figure 1: An example for the process of Sim-GPT, where the actual part
represents the current input, e.g., the input sentence “A man playing an
electric guitar on stage” in the prompt, while the virtual part represents the
input that are pending. Step 1 GPT-4 Data Annotation: Bottom-up, we instruct
GPT-4 to annotate (similar, dissimilar) format data for several input
sentences. Step 2 Contrastive-based STS: The annotated data is used with
similar sentence as positive and dissimilar sentence as negative to train STS
models under the contrastive framework.

##  2 Related Work

###  2.1 Semantic Textual Similarity

Due to the lack of supervised training data, previous approaches to STS relied
on encoding sentence in high-dimensional spaces and enabling the computation
of semantic similarity based on vector distances Mikolov et al. (2013);
Pennington et al. (2014); Devlin et al. (2018); Liu et al. (2019). Despite
their strong performance, those unsupervised approaches are trained to capture
general linguistic patterns in the language data lacking the guidance of
specific STS information: similar granularity of two sentences.

To fill the gap of annotation data, Bowman et al. (2015) proposed the SNLI
dataset and Williams et al. (2017) proposed MNLI dataset covering a range of
genres of spoken and written text. SNLI and MNLI consist of several hand-
written sentence pairs, and each sentence pair is manually annotated with
three labels according to whether there is a logical relationship between
them: entailment, neutral and contradiction.

With the availability of NLI-data, many supervised approaches emerged
promising to surpass their unsupervised counterparts. Fine-tuning Siamese
networks Neculoiu et al. (2016); Conneau et al. (2017); Pontes et al. (2018);
Ranasinghe et al. (2019); Benajiba et al. (2019) that involve two parallel
neural networks with shared weights, processing each sentence in a pair and
learning sentence embeddings in a way that maximizes similarity for entailment
sentences and minimizes it for contraditcion ones. Sentence-BERT Reimers and
Gurevych (2019) that derives from the BERT Devlin et al. (2018) architecture,
encoding semantically meaningful sentence embeddings and optimizing them with
cosine-similarity. Contrastive-based models Yan et al. (2021); Kim et al.
(2021); Gao et al. (2021); Jiang et al. (2022) that penalize the model for
misclassifying similar sentence pairs as dissimilar, making similar sentences
closer to each other in the embedding space, while dissimilar sentences are
farther apart.

###  2.2 Large Language Models

LLMs are trained on vast volumes of text data and have emerged as a pivotal
element in recent NLP studies. To improve the generation ability, some
researchers explore to scale up LLMs, such as GPT-3 (Brown et al., 2020) with
175B parameters, Gopher (Rae et al., 2021) with 280B parameters and PaLM
(Chowdhery et al., 2022) with 540B parameters. However, with the scaling up,
diminishing margins are becoming more apparent. Thus, some researchers turned
to explore how to efficiently utilize large training data and the learning
ability of large parameters, including Touvron et al. (2023) proposed LLaMA
demonstrating that the performance of a 7B model continues to improve even
after 1T tokens and trained LLaMA-13B with only 13B parameters outperforming
GPT-3 (Brown et al., 2020) (175B) on most benchmarks. Others aims to train
better LLMs with supervised learning or human instruction on each epoch (Wei
et al., 2022; Wang et al., 2022; Mishra et al., 2022; Ouyang et al., 2022;
Zhang et al., 2023a, b).

Given the impressive capability of LLMs to comprehend and produce human-like
content, it’s crucial to investigate their application to various downstream
activities. One line of work aims to directly fine-tune LLMs on specific
tasks, and to minimize fine-tune consumption, some of work focus on efficient
fine-tuning such as partial parameters fine-tuning (Hu et al., 2021; Li and
Liang, 2021; Liu et al., 2022; Sun et al., 2023b, d) and using memory
mechanisms to cache past inputs (Wu et al., 2022; Bulatov et al., 2022).
Rather than fine-tuning, there is a line of work aims to maximize the pre-
trained knowledge of LLMs by prompts without altering the architecture (Brown
et al., 2020; Wang et al., 2023; Sun et al., 2023c; Wan et al., 2023; Qin et
al., 2023; Li et al., 2023). Other lines includes knowledge distillation that
use LLMs to teach smaller task-specific models (Hsieh et al., 2023; Dasgupta
et al., 2023; Gu et al., 2023), view LLMs as encoder embedding input words to
vectors (Ge et al., 2023) and using LLMs as auxiliary parts to activate the
ability of supervised models (Sun et al., 2023a).

Captions:

A statue at a museum that no seems to be looking at.

A man reads the paper in a bar with green lighting.

A woman wearing a ball cap squats down to touch the cracked earth.

Two children re laying on a rug with some wooden bricks laid out in a square
between them.

Questions:

What is the step by step guide to invest in share market in india?

How can I increase the speed of my internet connection while using a VPN?

Motorola (company): Can I hack my Charter Motorolla DCX3400?

Method to find separation of slits using fresnel biprism?

Multi-genre Long Sentences:

Networks also undergo a discrete training phase, as opposed to engaging in
ongoing learning.

Take advantage of the current price and include it in your purchase if it
catches your interest.

Vietnam on the verge of securing its position as the second largest economy in
Southeast Asia…

We would appreciate it if you could share your knowledge about the music scene
in Dallas, TX.

Figure 2: An example of our collected data, which consists of three types of
sources: captions, questions and multi-genre long sentences.

##  3 Sim-GPT: GPT-4 Data Annotation

To address the longstanding issue of the STS: the lack of supervised training
data, we propose Sim-GPT, which can be decomposed into two steps: (1) GPT-4
Data Annotation, utilize LLMs to generate a relatively large set of training
data; and (2) Contrastive-based STS, utilize the annotated data to train
smaller models (backboned by contrastive learning), and then at test time do
inference on the trained model. Illustrations are shown in Figure 3.

In this section, we describe the step “(1) GPT-4 Data Annotation”, and for
step “(2) Contrastive-based STS” we put it in Section 4.

###  3.1 Overview

Formally, GPT-4 is anticipated to generate (similar sentence, dissimilar
sentence) format pair for each given sentence. Thus, below, we sequentially
focus on collecting sentences from multi-genre sources and prompt design in
guiding GPT-4 annotation.

Data | Source/Annotation | Sentence Number | Max Length | Min Length | Mean Tokens
---|---|---|---|---|---
Captions | source | 154933 | 82 | 2 | 13.5
origin | 138100 | 82 | 2 | 13.5
similar | 138100 | 27 | 4 | 10.4
dissimilar | 138100 | 28 | 4 | 9.3
Questions | source | 100000 | 87 | 2 | 11.4
origin | 96751 | 87 | 2 | 11.4
similar | 96751 | 65 | 2 | 12.2
dissimilar | 96751 | 50 | 2 | 10.4
Multi-genre Sentences | source | 150000 | 182 | 1 | 11.5
origin | 136545 | 182 | 1 | 11.6
similar | 136545 | 56 | 2 | 10.5
dissimilar | 136545 | 53 | 2 | 10.2

Table 1: Key statistics for the raw sentence pairs in our collected data. For
the annotated data, we report the triplet (origin, similar, dissimilar)
separately. Noted that to ensure the resulting models are not overly
specialized in understanding or responding to questions alone, we randomly
selected 100K questions from the total pool of 400K for labeling.

###  3.2 Data Sources

In order to ensure that the training data generated by GPT-4 is comprehensive,
we desire a wide range of input sentences. Input sentences come from three
sources, the details of which are shown in Figure 2.

(1) Captions, which refer to hand-written texts describing related images or
videos. We choose the Flickr30k Young et al. (2014) corpus as the caption
source. It consists of 31,783 images collected from everyday scenes containing
people engaged in various activities, and each image in the Flickr30k dataset
is accompanied by five different textual captions, providing a rich set of
linguistic data to complement the visual information.

(2) Questions, which refer to hand-written questions about specific
information or insights. We choose the Quora Question Pairs (QQP) Data et al.
(2017) dataset as the hand-written question source. It is a collection of 400K
user question-answers from Quora website involving a wide range of topics,
e.g, education (“Is it a good idea to go to medical school without a Science
degree?”), math (“If 3x + 9y = 7x + y. then 8y=?”) and nationality (“Why do
Slavs squat?”).

(3) Multi-genre Long Sentences, which ranges over a variety of spoken and
written sentences. We use a subset of
redpajamas††https://github.com/togethercomputer/RedPajama-Data which contains
150K long sentences from our real-world applications as multi-genre sources.

Table 1 shows some key statistics about the collected data. It’s worth noting
that the amount of annotated data is less than the amount of source data. The
main reason lies that GPT-4 has the probability to violate the few-shot format
outputting “weird” results, thus, we write scripts to remove those unqualified
GPT-4 outputs.

Here, we have collected a large amount of sentences from three types of
sources. Next, we will describe the designed prompt to guide GPT-4 to generate
decent annotation data.

This task will involve reading a line from a caption and writing two sentences
that relate to it. Using only this description and what you know about the
world:

Write one sentence that is definitely similar to the situation or event in the
line.

Write one sentence that is definitely dissimilar to the situation or event in
the line.

Note that don’t simply paraphrase or negate the given sentence, and we don’t
allow reuse the same sentence for many different prompts.

Input: A land rover is being driven across a river.

Output:

1\. A vehicle is crossing a river.

2\. A sedan is stuck in the middle of a river.

Input: Two teenage girls conversing next to lockers.

Output:

1\. People talking next to lockers.

2\. Girls talking next to the toilet.

Input:

Figure 3: The 2-shot example of the designed prompt for our caption source
data.

###  3.3 Prompt Design

Given the input sentence, we instruct GPT-4 to generate two sentences: one is
a similar sentence that semantically follows the input sentence; the other is
a dissimilar sentence in semantics, e.g.,

Input Sentence: Dogs run outdoors.

Similar Sentence: Dogs are running.

Dissimilar Sentence: There aren’t any animals.

Shown in Figure 1, the designed prompt consists of three parts: (1) Task
Description; (2) Few-shot Examples; and (3) Input Sentence:

####  3.3.1 Task Description

Task description in the prompts for different sources are a little bit
different. We use the caption for illustration, which is designed as follows:

This task will involve reading a line from a caption and writing two sentences
that relate to it. Using only this description and what you know about the
world:

  * •

Write one sentence that is definitely similar to the situation or event in the
line.

  * •

Write one sentence that is definitely dissimilar to the situation or event in
the line.

Note that don’t simply paraphrase or negate the given sentence, and we don’t
allow reuse the same sentence for many different prompts.

This is the version for caption sources. For the version of question sources
and multi-genre sources, they are akin to the caption version and are put in
Appendix A.

The core of the task description lies in three aspects:

(1) The first aspect,

“This task will involve reading a line from a caption and writing two
sentences that relate to it. Using only this description and what you know
about the world.”

emphasizes that the following task is a STS task, and the annotation shouldn’t
be influenced or biased by the original information contained in the
photograph (e.g., place names or author names about the photograph).

(2) The second aspect,

“Write one sentence that is definitely [similar / dissimilar] to the situation
or event in the line.”

indicates that GPT-4 should generate two sentences: One describes the same
object as the input caption (as the similar sentence), and the other describes
partially or totally different object as the input caption (as the dissimilar
sentence).

(3) The third aspect,

“Note that don’t simply paraphrase or negate the given sentence, and we don’t
allow reuse the same sentence for many different prompts.”

avoids that GPT-4 just alter the syntactic structure of the input sentence,
for example:

Input: A man is in front of the tower.

Output:

1\. The tower is in front of a man.

2\. A man isn’t in front of the tower.

####  3.3.2 Few-shot Examples

Few-shot examples mainly serve as two purposes: (1) it provides a direct
interpretation about the task description as the complexity of the above
description; (2) it regulates the format of the GPT-4 outputs as the feature
that GPT-4 mimics the format of the few-shot.

For the format, we sequentially pack $k$ examples in the few-shot, and each
example consists of one sentence as the input and two sentences
(similar/dissimilar) as the output:

Input: $\displaystyle\text{[Example Sentence]}_{1}$ Output:
$\displaystyle\text{1. [Similar Sentence]}_{1}$ $\displaystyle\text{2.
[Dissimilar Sentence]}_{1}$ $\displaystyle\cdots$ Input:
$\displaystyle\text{[Example Sentence]}_{k}$ Output: $\displaystyle\text{1.
[Similar Sentence]}_{k}$ $\displaystyle\text{2. [Dissimilar Sentence]}_{k}$

For each example, we manually select sentence in our data sources and write
related similar/dissimilar sentences. Here, we display the designed prompt
with 2-shot for our caption source data in Figure 3, and put the full designed
prompt used for other data sources in Appendix A.

####  3.3.3 Input Sentence

Input sentence feeds the current input sentence into the GPT-4 expecting it to
output sequence according to the above defined format:

Input: [Current Input] Output:

where Output is the signal for the GPT-4 to output annotated results.

Here, we have completed instructions for GPT-4 annotating (similar,
dissimilar) format data from various sources, and below we continue to
describe the application of those annotated data in contrastive-based STS
models.

##  4 Sim-GPT: Contrastive-based STS Learning

Here, we describe the model training and inference process based on the
annotated data. In this work, we use the SimCSE framework Gao et al. (2021) as
the backbone, which is a contrastive-based framework aiming to effectively
pull semantically close similar sentences together and push apart dissimilar
sentences.

###  4.1 SimCSE

Formally, supposed that
$\mathcal{D}=\\{(x_{i},x_{i}^{+},x_{i}^{-})\\}_{i=1}^{m}$ represents the
training set, where each sentence $x_{i}$ is packed with one similar sentence
$x_{i}^{+}$ (semantically close) and one dissimilar sentence $x_{i}^{-}$
(semantically opposite), and $m$ denotes the size of the training set. The
similar sentence and the dissimilar sentence can be directly taken from the
generated training dataset. We first encode each triplet $(x,x^{+},x^{-})$
using the RoBERTaLiu et al. (2019) and generating vector representation
triplet $(h,h^{+},h^{-})$. Then, the pre-trained RoBERTa is fine-tuned to
optimize the contrastive-based loss function as:

$\displaystyle\mathcal{L}_{CL}=$ (1)
$\displaystyle-\log\frac{e^{sim(h_{i},h_{i}^{+})/\tau}}{\sum_{j=1}^{N}(e^{sim(h_{j},h_{j}^{+})/\tau}+\alpha*e^{sim(h_{i},h_{j}^{-})/\tau})}$

where $sim(h,h^{+})/\tau$ denotes the cosine similarity between sentence $x$
and $x^{+}$ with $\tau$ as the normalization parameter, and $\alpha$ denotes
the weight of the dissimilar sentence.

During the inference stage, given the test input sentences pair $(x,y)$, we
first send them into the fine-tuned RoBERTa generating the vector
representations pair $(h_{x},h_{y})$. Then, the cosine similarity score is
computed between $h_{x}$ and $h_{y}$ as the final similarity score of the
sentences pair $(x,y)$.

Furthermore, in our evaluations of STS tasks where each sentence pair is
assigned a score from 0 to 5, we will adjust/normalize the produced cosine
similarity scores to fit within the 0 to 5 range.

###  4.2 PromCSE

A follow-up work of SimCSE, which is called PromCSE Jiang et al. (2019),
addresses the bottleneck of SimCSE because of the insufficient ability of the
contrastive-based loss function $\mathcal{L}_{CL}$ in separating hard
negatives and positives by expanding $\mathcal{L}_{CL}$ to:

$\displaystyle\mathcal{L}=\mathcal{L}_{CL}+\lambda\cdot\mathcal{L}_{M},$ (2)
$\displaystyle\mathcal{L}_{M}=\max(0,m+sim(h_{i},h_{i}^{*})-sim(h_{i},h_{i}^{+})),$
$\displaystyle h_{i}^{*}=\mathop{\arg\min}_{1\leq j\leq N,j\neq
i}sim(h_{i},h_{j})$

where $sim$ denotes the cosine similarity, and $m$, $\lambda$ are two
hyperparameters to amplify the differences between negatives and positives.

PromCSE achieves the current SOTA performance on supervised STS tasks, thus,
in this work, we also use PromCSE as the backbone, trained on the generated
data from GPT-4. For the training and inference stage, PromCSE aligns with the
above SimCSE, which is omitted for brevity.

##  5 Experiments

###  5.1 Evaluation Settings

To evaluate the effectiveness of Sim-GPT, we train the supervised-SimCSE and
the current SOTA PromCSE model on the annotated Sim-GPT data, and conduct
evaluations on 7 STS tasks. Parameters and settings are detailed below.

##### GPT Parameters.

We use GPT-4 OpenAI (2023) to generate annotated data. For the accessing
parameters, we set:

model = gpt-4, temperature = 0, top_p = 1, frequency_penalty = 0,
presence_penalty = 0, max_token = 4096

For the part of few-shot, we manually write 8 few-shot examples for each type
of data source, which can be found in Appendix A.

##### Training Details.

We follow default settings in SimCSE Gao et al. (2021) and PromCSE Jiang et
al. (2019) using the pre-trained RoBERTa (cased) Liu et al. (2019) and taking
the $[CLS]$ representation as the sentence embedding. For training parameters,
we adjust to the best as:

| SimCSE | PromCSE
---|---|---
| base | large | base | large
Batch size | 512 | 512 | 512 | 256
Learning rate | 5e-5 | 1e-5 | 1e-2 | 5e-3
Negative weight ($\alpha$) | 1 | 0.8 | 0.8 | 0.8

Model | STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICK-R | Avg
---|---|---|---|---|---|---|---|---
In-Context Learning with GPT-4
8-shot | 61.18 | 83.09 | 71.04 | 69.85 | 74.29 | 80.61 | 67.73 | 72.54
16-shot | 66.33 | 78.04 | 73.20 | 80.02 | 76.07 | 73.25 | 69.90 | 73.83
32-shot | 63.03 | 78.94 | 73.79 | 77.07 | 76.80 | 82.06 | 69.47 | 74.45
Unsupervised Models
Word Embeddings (GloVe) ♠ | 55.14 | 70.66 | 59.73 | 68.25 | 63.66 | 58.02 | 53.76 | 61.32
BERT${}_{\text{base}}^{\bigstar}$ (average word embeddings) | 39.70 | 59.38 | 49.67 | 66.03 | 66.19 | 53.87 | 62.06 | 56.70
$\text{CT-BERT}_{\text{base}}^{\bigstar}$ | 61.63 | 76.80 | 68.47 | 77.50 | 76.48 | 74.31 | 69.19 | 72.05
PromCSE-$\text{BERT}_{\text{base}}^{\blacktriangle}$ | 73.03 | 85.18 | 76.70 | 84.19 | 79.69 | 80.62 | 70.00 | 78.49
SimCSE-$\text{RoBERTa}_{\text{large}}^{\bigstar}$ | 72.86 | 83.99 | 75.62 | 84.77 | 81.80 | 81.98 | 71.26 | 78.90
Supervised Models
InferSent-GloVe♠ | 52.86 | 66.75 | 62.15 | 72.77 | 66.87 | 68.03 | 65.65 | 65.01
Universal Sentence Encoder♠ | 64.49 | 67.80 | 64.61 | 76.83 | 73.18 | 74.92 | 76.69 | 71.22
$\text{SRoBERTa}_{\text{base}}^{\spadesuit}$ | 71.54 | 72.49 | 70.80 | 78.74 | 73.69 | 77.77 | 74.46 | 74.21
$\text{SBERT}_{\text{base}}^{\spadesuit}$ | 70.97 | 76.53 | 73.19 | 79.09 | 74.30 | 77.03 | 72.91 | 74.89
$\text{CT-SBERT}_{\text{base}}^{\bigstar}$ | 74.84 | 83.20 | 78.07 | 83.84 | 77.93 | 81.46 | 76.42 | 79.39
Supervised Models based on Our Sim-GPT Framework
SimCSE-$\text{RoBERTa}_{\text{base}}^{\bigstar}$ | 76.53 | 85.21 | 80.95 | 86.03 | 82.57 | 85.83 | 80.50 | 82.52
Sim-GPT + SimCSE-$\text{RoBERTa}_{\text{base}}$ | 77.65 (+1.12) | 86.15 (+0.94) | 80.58 (-0.37) | 86.47 (+0.44) | 84.08 (+1.51) | 86.20 (+0.37) | 80.88 (+0.38) | 83.14 (+0.62)
SimCSE-$\text{RoBERTa}_{\text{large}}^{\bigstar}$ | 77.46 | 87.27 | 82.36 | 86.66 | 83.93 | 86.70 | 81.95 | 83.76
Sim-GPT + SimCSE-$\text{RoBERTa}_{\text{large}}$ | 78.79 (+1.33) | 88.22 (+0.95) | 83.48 (+1.12) | 88.32 (+1.66) | 85.48 (+1.55) | 87.91 (+1.21) | 81.07 (-0.88) | 84.75 (+0.99)
PromCSE-$\text{RoBERTa}_{\text{base}}^{\blacktriangle}$ | 77.51 | 86.15 | 81.59 | 86.92 | 83.81 | 86.35 | 80.49 | 83.26
Sim-GPT + PromCSE-$\text{RoBERTa}_{\text{base}}$ | 77.74 (+0.23) | 86.82 (+0.77) | 81.36 (-0.23) | 87.01 (+0.09) | 84.58 (+0.77) | 86.98 (+0.63) | 80.48 (-0.01) | 83.57 (+0.31)
PromCSE-$\text{RoBERTa}_{\text{large}}^{\blacktriangle}$ | 79.56 | 88.97 | 83.81 | 88.08 | 84.96 | 87.87 | 82.43 | 85.10
Sim-GPT + PromCSE-$\text{RoBERTa}_{\text{large}}$ | 79.92 (+0.36) | 88.87 (-0.10) | 84.29 (+0.48) | 88.64 (+0.56) | 85.94 (+0.98) | 88.18 (+0.31) | 82.79 (+0.36) | 85.52 (+0.42)

Table 2: Results on STS tasks, and we highlight the highest numbers among
models with the same framework. $\spadesuit$: results from Reimers and
Gurevych (2019), results from Gao et al. (2021), and $\blacktriangle$: results
from Jiang et al. (2022).

##### Evaluation Tasks.

Following the previous work Reimers and Gurevych (2019); Gao et al. (2021);
Chuang et al. (2022); Jiang et al. (2022), we conduct experiments on 7 STS
tasks: STS12-16 Agirre et al. (2012, 2013, 2014, 2015, 2016) , STS Benchmark
Cer et al. (2017) and SICK-Relatedness Marelli et al. (2014).

These tasks consist of a large amount of sentence pairs, each of which
requires the evaluated model to produce a similarity score ranging from 0 to
5. For the metric, the Spearman’s correlation is reported for all experiments.

##### Baselines.

We compare with both unsupervised and supervised models:

For unsupervised models, we choose two methods based on word embeddings: GloVe
embeddings Pennington et al. (2014) and BERT embeddings Devlin et al. (2018);
and three methods based on contrastive learning: CT-BERT Carlsson et al.
(2020), SimCSE Gao et al. (2021) and PromCSE Jiang et al. (2022).

For supervised models, we choose three methods based on siamese neural
networks: InferSent-GloVe Conneau et al. (2017), Universal Sentence Encoder
Cer et al. (2018) and SBERT/SRoBERTa Reimers and Gurevych (2019); and two
methods based on contrastive learning: CT-SBERT Carlsson et al. (2020), SimCSE
Gao et al. (2021) and the current supervised SOTA model PromCSE Jiang et al.
(2022). Only the InferSent-GloVe model is trained on the SNLI Bowman et al.
(2015) dataset, all the others are trained on SNLI+MNLI Williams et al. (2017)
dataset.

Additionally, we conduct experiments to contrast Sim-GPT against leveraging
GPT-4 to directly generate semantic similarity scores for sentence pairs based
on the given prompt. For more details about the accessing prompt, please check
Appendix 9.

###  5.2 Main Results

Table 2 shows evaluation results on 7 STS tasks, where we can observe:

(1) The average score achieved by the in-context learning-based method is
significantly lower compared to that of our Sim-GPT approach, e.g., with
32-shot, GPT-4 achieves an average score of 74.45, while the $\text{SimCSE-
RoBERTa}_{\text{large}}$ model trained on our Sim-GPT annotated data achieves
an average score of 84.75 (+10.3). This is because the number of illustration
examples in in-context learning is limited, while the proposed Sim-GPT is
designed to take the advantage of large amount of training examples.

(2) Focusing on a specific task, as the few-shot changes, the score achieved
by the in-context learning-based method fluctuates dramatically. The dramatic
fluctuations in scores suggest that in-context learning based STS approaches
are highly sensitive to different few-shot selections, and are not stable over
different few-shot settings. In contrast, in Section 6.2, we do the ablation
study by changing few-shot examples of Sim-GPT, showing that our Sim-GPT is
highly stable in performance across different few-shot selections.

(3) There is a significant improvement by training supervised models on the
Sim-GPT annotated data: 83.76 v.s. 84.75 (+0.99) based on the $\text{SimCSE-
RoBERTa}_{\text{large}}$ model, and 85.10 v.s. 85.52 (+0.42) based on the
$\text{PromCSE-RoBERTa}_{\text{large}}$ model. These results not only
illustrate the effectiveness of Sim-GPT in annotating STS data but also
demonstrate the flexibility of Sim-GPT in the selection of backboned models.

(4) Sim-GPT achieves the best average score of 85.52, which +0.42 over the
current SOTA supervised PromCSE Jiang et al. (2019) model. This SOTA
performance further illustrates the effectiveness of the proposed Sim-GPT
framework, encouraging future advancements based on the same idea as Sim-GPT.

##  6 Ablation Study

We do ablation studies on: (1) using different data sources; and (2) changing
few-shot examples. All results are reported based on STS Benchmark development
set using default GPT-4 settings and $\text{SimCSE-RoBERTa}_{\it base}$
parameters.

###  6.1 Data Sources Permutation

It’s crucial to visually compare the contribution of each data source to the
STS task. Thus, we use different combinations of the three data sources: (1)
Captions, (2) Questions, (3) Multi-genre Sentences, (4) Captions+Questions,
(5) Captions+Multi-genre Sentences, (6) Questions+Multi-genre Sentences, and
(7) Captions+Questions+Multi-genre Sentences.

Table 3 shows results for different combinations on the STS Benchmark
development set, and we can clearly observe that more data leads to higher
scores, e.g., combination “Captions + Questions + Multi-genre Sentences” leads
to the score of 85.9, which is higher than the score 83.88 for “Captions +
Questions” and the score 83.54 for “Captions” alone.

Combinations | Results
---|---
Captions | 83.54
Questions | 81.38
Multi-genre Sentences | 83.81
Captions + Questions | 83.88
Captions + Multi-genre Sentences | 84.96
Questions + Multi-genre Sentences | 84.02
Captions + Questions | 85.9
\+ Multi-genre Sentences

Table 3: STS Benchmark development results for different combinations of data
sources.

Few-shot Variation | STS-B Results
---|---
Sim-GPT | 83.54
Changing few-shot number | 83.41
Changing few-shot examples | 83.44

Table 4: STS Benchmark development results with different types of few-shot
examples. Sim-GPT: the used caption 8-shot examples in Sim-GPT.Changing few-
shot number: we manually expand 8-shot examples of Sim-GPT to 16-shot.
Changing few-shot examples: we alter caption 8-shot examples in Sim-GPT with
multi-genre 8-shot examples.

###  6.2 Few-shot Variation in Data Generation

Commonly, the output of LLMs are sensitive to the strategy taken in few-shot
learning. To evaluate the influence of different few-shot setups in terms of
the number of shots and prompts during the data generation stage on the final
result, we explore the following setups: (1) changing the number of shots from
8 to 16, the details of which are shown in Appendix 7; (2) changing few-shot
prompts: we pair prompt originally designed for caption with the multi-genre
examples.

We employ GPT4 to label the data using two strategies. Then, we use the
revised training data to train SimCSE-RoBERTabase models. The results are
presented in Table 4. It can be observed that the final results on STS are not
affected by altering the prompt during the training stage. The reason for this
is as follows: STS-GPT is trained on a large amount of data points generated
by LLMs, making it more resistant to prompt changes. In contrast, the in-
context learning strategy, which directly outputs results from few-shot
prompting, is highly sensitive to prompt modifications. This demonstrates the
superiority of the proposed model over the in-context learning baseline.

##  7 Conclusion

In this paper, we propose Sim-GPT, the strategy of measuring text similarity
via GPT annotated data. Sim-GPT utilizes GPT-4 to provide a substantial amount
of reliable annotated data filling the gap of the lack of training signals for
STS. Furthermore, we conduct experiments and analyses on 7 STS tasks: Sim-GPT
not only achieves an average SOTA performances but also demonstrates
consistent stability with variations in prompts. Additionally, we release the
dataset consisting of 371K annotated STS examples from GPT-4 to encourage
further advancements in semantic textual similarity.

## References

  * Agirre et al. (2015) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, et al. 2015.  Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability.  pages 252–263.
  * Agirre et al. (2014) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014.  Semeval-2014 task 10: Multilingual semantic textual similarity.  pages 81–91.
  * Agirre et al. (2016) Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. 2016.  Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation.
  * Agirre et al. (2012) Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012.  Semeval-2012 task 6: A pilot on semantic textual similarity.  pages 385–393.
  * Agirre et al. (2013) Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013.  * sem 2013 shared task: Semantic textual similarity.  pages 32–43.
  * Benajiba et al. (2019) Yassine Benajiba, Jin Sun, Yong Zhang, Longquan Jiang, Zhiliang Weng, and Or Biran. 2019.  Siamese networks for semantic pattern similarity.  pages 191–194.
  * Bowman et al. (2015) Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015.  A large annotated corpus for learning natural language inference.  _arXiv preprint arXiv:1508.05326_.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Bulatov et al. (2022) Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev. 2022.  Recurrent memory transformer.  _Advances in Neural Information Processing Systems_ , 35:11079–11091.
  * Carlsson et al. (2020) Fredrik Carlsson, Amaru Cuba Gyllensten, Evangelia Gogoulou, Erik Ylipää Hellqvist, and Magnus Sahlgren. 2020.  Semantic re-tuning with contrastive tension.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017.  Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation.  _arXiv preprint arXiv:1708.00055_.
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018.  Universal sentence encoder for english.  pages 169–174.
  * Chen et al. (2023) Sihao Chen, Hongming Zhang, Tong Chen, Ben Zhou, Wenhao Yu, Dian Yu, Baolin Peng, Hongwei Wang, Dan Roth, and Dong Yu. 2023.  Sub-sentence encoder: Contrastive learning of propositional semantic representations.  _arXiv preprint arXiv:2311.04335_.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  Palm: Scaling language modeling with pathways.
  * Chuang et al. (2022) Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljačić, Shang-Wen Li, Wen-tau Yih, Yoon Kim, and James Glass. 2022.  Diffcse: Difference-based contrastive learning for sentence embeddings.  _arXiv preprint arXiv:2204.10298_.
  * Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017.  Supervised learning of universal sentence representations from natural language inference data.  _arXiv preprint arXiv:1705.02364_.
  * Dasgupta et al. (2023) Sayantan Dasgupta, Trevor Cohn, and Timothy Baldwin. 2023.  Cost-effective distillation of large language models.
  * Data et al. (2017) Canary Data, hilfialkaff, Jiang Lili, Risdal Meg, Dandekar Nikhil, and tomtung. 2017.  Quora question pairs.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  Simcse: Simple contrastive learning of sentence embeddings.  _arXiv preprint arXiv:2104.08821_.
  * Ge et al. (2023) Tao Ge, Jing Hu, Xun Wang, Si-Qing Chen, and Furu Wei. 2023.  In-context autoencoder for context compression in a large language model.
  * Gu et al. (2023) Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. 2023.  Knowledge distillation of large language models.  _arXiv preprint arXiv:2306.08543_.
  * Hsieh et al. (2023) Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. 2023.  Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.
  * Jiang et al. (2019) Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2019.  Smart: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization.  _arXiv preprint arXiv:1911.03437_.
  * Jiang et al. (2022) Yuxin Jiang, Linhan Zhang, and Wei Wang. 2022.  Improved universal sentence embeddings with prompt-based contrastive learning and energy-based learning.  _arXiv preprint arXiv:2203.06875_.
  * Kim et al. (2021) Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021.  Self-guided contrastive learning for bert sentence representations.  _arXiv preprint arXiv:2106.07345_.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_.
  * Li et al. (2023) Rui Li, Guoyin Wang, and Jiwei Li. 2023.  Are human-generated demonstrations necessary for in-context learning?  _arXiv preprint arXiv:2309.14681_.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-tuning: Optimizing continuous prompts for generation.
  * Liu et al. (2022) Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014.  A SICK cure for the evaluation of compositional distributional semantic models.  pages 216–223.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  _Advances in neural information processing systems_ , 26.
  * Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022.  Cross-task generalization via natural language crowdsourcing instructions.
  * Neculoiu et al. (2016) Paul Neculoiu, Maarten Versteegh, and Mihai Rotaru. 2016.  Learning text similarity with siamese recurrent networks.  pages 148–157.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation.  pages 1532–1543.
  * Pontes et al. (2018) Elvys Linhares Pontes, Stéphane Huet, Andréa Carneiro Linhares, and Juan-Manuel Torres-Moreno. 2018.  Predicting the semantic textual similarity with siamese cnn and lstm.  _arXiv preprint arXiv:1810.10641_.
  * Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023.  Is chatgpt a general-purpose natural language processing task solver?
  * Rae et al. (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021.  Scaling language models: Methods, analysis & insights from training gopher.  _arXiv preprint arXiv:2112.11446_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Ranasinghe et al. (2019) Tharindu Ranasinghe, Constantin Orašan, and Ruslan Mitkov. 2019.  Semantic textual similarity with siamese neural networks.  pages 1004–1011.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_.
  * Seonwoo et al. (2022) Yeon Seonwoo, Guoyin Wang, Sajal Choudhary, Changmin Seo, Jiwei Li, Xiang Li, Puyang Xu, Sunghyun Park, and Alice Oh. 2022.  Ranking-enhanced unsupervised sentence representation learning.  _arXiv preprint arXiv:2209.04333_.
  * Sun et al. (2023a) Chenxi Sun, Yaliang Li, Hongyan Li, and Shenda Hong. 2023a.  Test: Text prototype aligned embedding to activate llm’s ability for time series.
  * Sun et al. (2023b) Xiaofei Sun, Linfeng Dong, Xiaoya Li, Zhen Wan, Shuhe Wang, Tianwei Zhang, Jiwei Li, Fei Cheng, Lingjuan Lyu, Fei Wu, and Guoyin Wang. 2023b.  Pushing the limits of chatgpt on nlp tasks.
  * Sun et al. (2023c) Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023c.  Text classification via large language models.  _arXiv preprint arXiv:2305.08377_.
  * Sun et al. (2023d) Xiaofei Sun, Xiaoya Li, Shengyu Zhang, Shuhe Wang, Fei Wu, Jiwei Li, Tianwei Zhang, and Guoyin Wang. 2023d.  Sentiment analysis through llm negotiations.  _arXiv preprint arXiv:2311.01876_.
  * Sun et al. (2022) Xiaofei Sun, Yuxian Meng, Xiang Ao, Fei Wu, Tianwei Zhang, Jiwei Li, and Chun Fan. 2022.  Sentence similarity based on contexts.  _Transactions of the Association for Computational Linguistics_ , 10:573–588.
  * Tan et al. (2022) Minghuan Tan, Yong Dai, Duyu Tang, Zhangyin Feng, Guoping Huang, Jing Jiang, Jiwei Li, and Shuming Shi. 2022.  Exploring and adapting chinese gpt to pinyin input method.  _arXiv preprint arXiv:2203.00249_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Wan et al. (2023) Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. 2023.  Gpt-re: In-context learning for relation extraction using large language models.
  * Wang et al. (2022) Haifeng Wang, Jiwei Li, Hua Wu, Eduard Hovy, and Yu Sun. 2022.  Pre-trained language models and their applications.  _Engineering_.
  * Wang et al. (2023) Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023.  Gpt-ner: Named entity recognition via large language models.  _arXiv preprint arXiv:2304.10428_.
  * Wang et al. (2019) Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Jiangnan Xia, Liwei Peng, and Luo Si. 2019.  Structbert: Incorporating language structures into pre-training for deep language understanding.  _arXiv preprint arXiv:1908.04577_.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022.  Finetuned language models are zero-shot learners.
  * Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R Bowman. 2017.  A broad-coverage challenge corpus for sentence understanding through inference.  _arXiv preprint arXiv:1704.05426_.
  * Wu et al. (2022) Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, and Christian Szegedy. 2022.  Memorizing transformers.
  * Yan et al. (2021) Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021.  Consert: A contrastive framework for self-supervised sentence representation transfer.  _arXiv preprint arXiv:2105.11741_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ , 32.
  * Young et al. (2014) Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014.  From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.  _Transactions of the Association for Computational Linguistics_ , 2:67–78.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020.  Big bird: Transformers for longer sequences.  _Advances in neural information processing systems_ , 33:17283–17297.
  * Zhang et al. (2023a) Jiahao Zhang, Haiyang Zhang, Dongmei Zhang, Yong Liu, and Shen Huang. 2023a.  Beam retrieval: General end-to-end retrieval for multi-hop question answering.
  * Zhang et al. (2023b) Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. 2023b.  Instruction tuning for large language models: A survey.  _arXiv preprint arXiv:2308.10792_.
  * Zhang et al. (2020) Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. 2020.  An unsupervised sentence embedding method by mutual information maximization.  _arXiv preprint arXiv:2009.12061_.

##  Appendix A Prompt Design

As describe in Section 3, we utilize GPT-4 to generate a relatively large set
of training data from multi-genre sources. In this section, we clearly show
the prompt used in Sim-GPT, where Figure 4 for caption sources, Figure 5 for
question sources and Figure 6 for multi-genre sentence sources.

This task will involve reading a line from a caption and writing two sentences
that relate to it. Using only this description and what you know about the
world:

  * •

Write one sentence that is definitely similar to the situation or event in the
line.

  * •

Write one sentence that is definitely dissimilar to the situation or event in
the line.

Note that don’t simply paraphrase or negate the given sentence, and we don’t
allow reuse the same sentence for many different prompts.

Input: This church choir sings to the masses as they sing joyous songs from
the book at a church.

Output: 1\. The church is filled with song.

2\. A choir singing at a baseball game.

Input: A woman with a green headscarf, blue shirt and a very big grin.

Output: 1\. The woman is very happy.

2\. The woman has been shot.

Input: An old man with a package poses in front of an advertisement.

Output: 1\. A man poses in front of an ad.

2\. A man walks by an ad.

Input: A statue at a museum that no seems to be looking at.

Output: 1\. There is a statue that not many people seem to be interested in.

2\. Tons of people are gathered around the statue.

Input: A land rover is being driven across a river.

Output: 1\. A vehicle is crossing a river.

2\. A sedan is stuck in the middle of a river.

Input: A man playing an electric guitar on stage.

Output: 1\. A man playing guitar on stage.

2\. A man playing banjo on the floor.

Input: One tan girl with a wool hat is running and leaning over an object,
while another person in a wool hat is sitting on the ground.

Output: 1\. A tan girl runs leans over an object.

2\. A boy runs into a wall.

Input: Two teenage girls conversing next to lockers.

Output: 1\. People talking next to lockers.

2\. Girls talking next to the toilet.

Figure 4: The designed prompt used for caption sources.

This task will involve reading a line from a question and writing two
questions that relate to it. The line will describe a situation or event.
Using only this description and what you know about the world:

  * •

Write one question is definitely similar to the situation or event in the
line.

  * •

Write one question is definitely dissimilar to the situation or event in the
line.

Note that don’t simply paraphrase or negate the given sentence, and we don’t
allow reuse the same sentence for many different prompts.

Input: How did Michel Foucault feel about surveillance society?

Output: 1\. What is Michel Foucaults opinion on surveillance society?

2\. What are the main ingredients in a traditional apple pie?

Input: At what time should I drink green tea to be fit?

Output: 1\. What is the best time to drink green tea for fitness?

2\. Does Lipton green tea Assist in weight loss?

Input: Is Quora becoming the new Facebook?

Output: 1\. Is Quora becoming a new Facebook?

2\. Does Quora have nothing in common with Facebook?

Input: Is 12 Mbps a good internet speed for streaming and gaming?

Output: 1\. Would I be able to stream and game effectively with a 12 Mbps
internet connection?

2\. Is 6 Mbps download and 2 Mbps upload speed good for a 4G LTE connection?

Input: How can I increase my website page rank?

Output: 1\. How do I increase page rank of website?

2\. What techniques can lower my website’s page rank?

Input: 2 states movie download?

Output: 1\. Where can I download the movie "2 States"?

2\. Who controls the U.S president?

Input: What direction are you going when travelling from Malaga to Gibraltar?

Output: 1\. What direction are you travelling from malaga to gibralter?

2\. What direction are you going when travelling from Gibraltar to Malaga?

Input: Do animal cells have cell walls?

Output: 1\. Can you find cell walls in animal cells?

2\. Is anatomy destiny?

Figure 5: The designed prompt used for questions sources.

This task will involve reading a line from a non-fiction article and writing
two sentences that relate to it. The line will describe a situation or event.
Using only this description and what you know about the world:

  * •

Write one sentence that is definitely similar to the situation or event in the
line.

  * •

Write one sentence that is definitely dissimilar to the situation or event in
the line.

Note that don’t simply paraphrase or negate the given sentence, and we don’t
allow reuse the same sentence for many different prompts.

Input: Your contribution helped make it possible for us to provide our
students with a quality education.

Output: 1\. Your support helped us provide students with a quality education.

2\. Your contributions were of no help with our students’ education.

Input: They were promptly executed.

Output: 1\. They were executed.

2\. They were set free, uninjured.

Input: The simplest is for one or more of the members to simply donate one
million dollars to the IGGS Scholarship Fund.

Output: 1\. The simplest is for one million dollars to be donated by one of
the members to the IGGS Scholarship Fund.

2\. The most complicated is to arrange a donation of one million dollars to
the IGGS Scholarship Fund.

Input: In the felled seams of shirts and jeans, for example, the visible
stitches might be of a color designed to decorate the garment.

Output: 1\. The felled seams of shirts are visible.

2\. The felled seams of shirts are not visible.

Input: In this competition for legitimacy, secular regimes had no alternative
to offer.

Output: 1\. Secular regimes had little to off in way of legitimacy.

2\. Secular regimes had many options to offer.

Input: In this period, a number of key technological changes sewing machines
that made many more stitches a minute, long knives instead of shears for
cutting, and pressing machines.

Output: 1\. In this period, various key innovative changes, sewing machines
that made numerous more lines a moment, long blades rather than shears for
cutting, and squeezing machines.

2\. Industry tools and machineries have stayed the same until today.

Input: At the end of the second year, children begin to label their own and
others’ internal states with words, such as want, happy, mad, think, and
pretend.

Output: 1\. Usually by the end of the second year, children will begin to
label their own and other’s internal states with words.

2\. Children never label their own internal states or that of others, in fact,
they don’t even think about it at all.

Input: Eleven, the first guy, he’s heading towards Washington.

Output: 1\. Eleven is DC bound.

2\. The first one was bound for Boston.

Figure 6: The designed prompt used for multi-genre long sentences sources.

This task will involve reading a line from a caption and writing two sentences
that relate to it. Using only this description and what you know about the
world:

  * •

Write one sentence that is definitely similar to the situation or event in the
line.

  * •

Write one sentence that is definitely dissimilar to the situation or event in
the line.

Note that don’t simply paraphrase or negate the given sentence, and we don’t
allow reuse the same sentence for many different prompts.

Input: A biker races.

Output: 1\. A person is riding a bike.

2\. The car is yellow.

Input: A woman with a green headscarf, blue shirt and a very big grin.

Output: 1\. The woman is very happy.

2\. The woman has been shot.

Input: A softball player throws the ball to her teammate.

Output: 1\. Two people are playing softball.

2\. Two softball players are sitting on a bench.

Input: Island native fishermen reeling in their nets after a long day’s work.

Output: 1\. The men are finishing their day of work.

2\. The men did not go to work today but instead played bridge.

Input: Everyone on the street in the city seem to be busy doing their own
thing,

Output: 1\. People are conducting business in the city.

2\. The people are watching what is happening.

Input: People are on an escalator waiting to get to their destination while
looking outside of the glass that makes up the wall.

Output: 1\. People are riding on the escalator.

2\. People are taking the elevator.

Input: These are young adults who seem to be working together to protect the
plants surrounding the white pole.

Output: 1\. The adults are young.

2\. The adults are old.

Input: Children in yellow sports uniforms climbing a tower.

Output: 1\. Children in uniforms climb a tower.

2\. The kids crawl in sand.

Input: This church choir sings to the masses as they sing joyous songs from
the book at a church.

Output: 1\. The church is filled with song.

2\. A choir singing at a baseball game.

Input: A woman with a green headscarf, blue shirt and a very big grin.

Output: 1\. The woman is very happy.

2\. The woman has been shot.

Input: An old man with a package poses in front of an advertisement.

Output: 1\. A man poses in front of an ad.

2\. A man walks by an ad.

Input: A statue at a museum that no seems to be looking at.

Output: 1\. There is a statue that not many people seem to be interested in.

2\. Tons of people are gathered around the statue.

Input: A land rover is being driven across a river.

Output: 1\. A vehicle is crossing a river.

2\. A sedan is stuck in the middle of a river.

Input: A man playing an electric guitar on stage.

Output: 1\. A man playing guitar on stage.

2\. A man playing banjo on the floor.

Input: One tan girl with a wool hat is running and leaning over an object,
while another person in a wool hat is sitting on the ground.

Output: 1\. A tan girl runs leans over an object.

2\. A boy runs into a wall.

Input: Two teenage girls conversing next to lockers.

Output: 1\. People talking next to lockers.

2\. Girls talking next to the toilet.

Figure 7: The designed prompt with 16-shot examples used for caption sources.

This task will involve reading a line from a caption and writing two sentences
that relate to it. Using only this description and what you know about the
world:

  * •

Write one sentence that is definitely similar to the situation or event in the
line.

  * •

Write one sentence that is definitely dissimilar to the situation or event in
the line.

Note that don’t simply paraphrase or negate the given sentence, and we don’t
allow reuse the same sentence for many different prompts.

Input: Your contribution helped make it possible for us to provide our
students with a quality education.

Output: 1\. Your support helped us provide students with a quality education.

2\. Your contributions were of no help with our students’ education.

Input: At the end of the second year, children begin to label their own and
others’ internal states with words, such as want, happy, mad, think, and
pretend.

Output: 1\. Usually by the end of the second year, children will begin to
label their own and other’s internal states with words.

2\. Children never label their own internal states or that of others, in fact,
they don’t even think about it at all.

Input: They were promptly executed.

Output: 1\. They were executed.

2\. They were set free, uninjured.

Input: The simplest is for one or more of the members to simply donate one
million dollars to the IGGS Scholarship Fund.

Output: 1\. The simplest is for one million dollars to be donated by one of
the members to the IGGS Scholarship Fund.

2\. The most complicated is to arrange a donation of one million dollars to
the IGGS Scholarship Fund.

Input: In the felled seams of shirts and jeans, for example, the visible
stitches might be of a color designed to decorate the garment.

Output: 1\. The felled seams of shirts are visible.

2\. The felled seams of shirts are not visible.

Input: In this competition for legitimacy, secular regimes had no alternative
to offer.

Output: 1\. Secular regimes had little to off in way of legitimacy.

2\. Secular regimes had many options to offer.

Input: In this period, a number of key technological changes sewing machines
that made many more stitches a minute, long knives instead of shears for
cutting, and pressing machines.

Output: 1\. In this period, various key innovative changes, sewing machines
that made numerous more lines a moment, long blades rather than shears for
cutting, and squeezing machines.

2\. Industry tools and machineries have stayed the same until today.

Input: Eleven, the first guy, he’s heading towards Washington.

Output: 1\. Eleven is DC bound.

2\. The first one was bound for Boston.

Figure 8: The designed prompt with 8-shot multi-genre examples for caption
sources.

This task is the semantic textual similarity task. You will read two sentences
and write a similarity score of them. The score ranges from 0 to 5, including
decimal values, where 0 indicates no similarity and 5 represents the highest
level of similarity.

Input: a man with a hard hat is dancing .

a man wearing a hard hat is dancing .

Output: 5.0

Input: a woman is playing the guitar .

a man is playing guitar .

Output: 2.4

Input: people are playing cricket .

men are playing cricket .

Output: 3.2

Input: a man is speaking .

a man is spitting .

Output: 0.636

Input: a man is eating food .

a man is eating something .

Output: 4.2

Input: the man is riding a horse .

a woman is using a hoe .

Output: 0

Input: the boy is playing the piano .

a band is playing on stage .

Output: 1.333

Input: a man and a woman walk through the woods .

the man and woman are walking .

Output: 3.0

Figure 9: The designed 8-shot prompt used for in-context learning-based
method.

This task is the semantic textual similarity task. You will read two sentences
and write a similarity score of them. The score ranges from 0 to 5, including
decimal values, where 0 indicates no similarity and 5 represents the highest
level of similarity.

Input: a man with a hard hat is dancing .

a man wearing a hard hat is dancing .

Output: 5.0

Input: a woman is playing the guitar .

a man is playing guitar .

Output: 2.4

Input: people are playing cricket .

men are playing cricket .

Output: 3.2

Input: a man is speaking .

a man is spitting .

Output: 0.636

Input: a man is eating food .

a man is eating something .

Output: 4.2

Input: the man is riding a horse .

a woman is using a hoe .

Output: 0

Input: the boy is playing the piano .

a band is playing on stage .

Output: 1.333

Input: a man and a woman walk through the woods .

the man and woman are walking .

Output: 3.0

Input: the lady stirred up raw eggs in the bowl .

a woman is pouring eyes into a bowl .

Output: 1.0

Input: a cow is eating grass .

a dog is pulling a girl down a hill .

Output: 0.0

Input: the man stirred the sauce for the chicken .

the man is stirring oil .

Output: 2.4

Input: the lady cut the tail and body of a shrimp .

a woman is cleaning a shrimp .

Output: 4.5

Input: a person is peeling shrimp .

a person is preparing shrimp .

Output: 3.6

Input: the man is slicing a potato .

a man is slicing potato .

Output: 5.0

Input: ta man is cutting up a potato .

a man is cutting up carrots .

Output: 2.375

Input: the man is hiking in the woods .

a man is tracking in the wood .

Output: 3.0

Figure 10: The designed 16-shot prompt used for in-context learning-based
method.

This task is the semantic textual similarity task. You will read two sentences
and write a similarity score of them. The score ranges from 0 to 5, including
decimal values, where 0 indicates no similarity and 5 represents the highest
level of similarity.

Input: a man with a hard hat is dancing .

a man wearing a hard hat is dancing .

Output: 5.0

Input: a woman is playing the guitar .

a man is playing guitar .

Output: 2.4

Input: people are playing cricket .

men are playing cricket .

Output: 3.2

Input: a man is speaking .

a man is spitting .

Output: 0.636

Input: a man is eating food .

a man is eating something .

Output: 4.2

Input: the man is riding a horse .

a woman is using a hoe .

Output: 0

Input: the boy is playing the piano .

a band is playing on stage .

Output: 1.333

Input: a man and a woman walk through the woods .

the man and woman are walking .

Output: 3.0

Input: the lady stirred up raw eggs in the bowl .

a woman is pouring eyes into a bowl .

Output: 1.0

Input: a cow is eating grass .

a dog is pulling a girl down a hill .

Output: 0.0

Input: the man stirred the sauce for the chicken .

the man is stirring oil .

Output: 2.4

Input: the lady cut the tail and body of a shrimp .

a woman is cleaning a shrimp .

Output: 4.5

Input: a person is peeling shrimp .

a person is preparing shrimp .

Output: 3.6

Input: the man is slicing a potato .

a man is slicing potato .

Output: 5.0

Input: ta man is cutting up a potato .

a man is cutting up carrots .

Output: 2.375

Input: the man is hiking in the woods .

a man is tracking in the wood .

Output: 3.0

Input: a man is playing guitar .

a man plays a guitar .

Output: 4.857

Input: the man is buttering the bread .

the man is stirring the rice .

Output: 0.4

Input: large silver locomotive engine in a shed .

the silver train is parked in a station .

Output: 2.6

Input: the back of a stop sign with many stickers on it .

the back of a sign with stickers on .

Output: 3.8

Input: a doubly decker red bus driving down the road .

a red double decker bus driving down a street .

Output: 5

Input: the black bird is sitting on the ground .

the back of a pig under a tree with a cow in the background .

Output: 0

Input: a black and white horned cow standing in a field .

a large black and white cow in a field .

Output: 4

Input: a red bird and four other birds sitting in the snow .

five birds stand on the snow .

Output: 2.8

Input: domestic cat looking out window .

a white cat looking out of a window .

Output: 3.6

Input: two kids are playing a game of foosball .

the kids are playing a game with each other .

Output: 3.4

Input: the man is in a deserted field .

the man is outside in the field .

Output: 4.0

Input: a musician is smearing jam on his white guitar at a concert .

trombonist playing the her instrument in a band for a parade .

Output: 0.4

Input: a pair of young boys in t-shirts are hiding in the woods with one
looking aghast .

two smiling little girls playing in a fountain with other people .

Output: 0.0

Input: one football player tries to tackle a player on the opposing team .

a football player attempts a tackle .

Output: 4.6

Input: there are people out on the street .

people are out on the street .

Output: 5.0

Input: all we know is this : distant objects are receding from us at a rate
proportional to their distance from us .

the expansion of space means that objects in cosmological distances are
receding away from each other .

Output: 3.4

Figure 11: The designed 32-shot prompt used for in-context learning-based
method.