# DebCSE: Rethinking Unsupervised Contrastive Sentence Embedding Learning in
the Debiasing Perspective

Pu Miao  0000-0002-9308-5247 Sina WeiboWest 8 Xibeiwangdong RdBeijingChina
<EMAIL_ADDRESS>,  Zeyao Du  0000-0002-9054-3337 China Literature
LimitedBinjiangzhongxin MTShang HaiChina<EMAIL_ADDRESS>and  Junlin Zhang
0000-0002-7982-3194 Sina WeiboWest 8 Xibeiwangdong RdBeijingChina
<EMAIL_ADDRESS>
(2023)

###### Abstract.

Several prior studies have suggested that word frequency biases can cause the
Bert model to learn indistinguishable sentence embeddings. Contrastive
learning schemes such as SimCSE and ConSERT have already been adopted
successfully in unsupervised sentence embedding to improve the quality of
embeddings by reducing this bias. However, these methods still introduce new
biases such as sentence length bias and false negative sample bias, that
hinders model’s ability to learn more fine-grained semantics. In this paper,
we reexamine the challenges of contrastive sentence embedding learning from a
debiasing perspective and argue that effectively eliminating the influence of
various biases is crucial for learning high-quality sentence embeddings. We
think all those biases are introduced by simple rules for constructing
training data in contrastive learning and the key for contrastive learning
sentence embedding is to ”mimic” the distribution of training data in
supervised machine learning in unsupervised way. We propose a novel
contrastive framework for sentence embedding, termed DebCSE, which can
eliminate the impact of these biases by an inverse propensity weighted
sampling method to select high-quality positive and negative pairs according
to both the surface and semantic similarity between sentences. Extensive
experiments on semantic textual similarity (STS) benchmarks reveal that DebCSE
significantly outperforms the latest state-of-the-art models with an average
Spearman’s correlation coefficient of 80.33% on BERTbase.

Sentences Embeddings, Contrastive Learning, Information Extraction, Text
Retrieval, Natural Language Understanding

††journalyear: 2023††copyright: acmlicensed††conference: Proceedings of the
32nd ACM International Conference on Information and Knowledge Management;
October 21–25, 2023; Birmingham, United Kingdom††booktitle: Proceedings of the
32nd ACM International Conference on Information and Knowledge Management
(CIKM ’23), October 21–25, 2023, Birmingham, United Kingdom††price:
15.00††doi: 10.1145/3583780.3614833††isbn:
979-8-4007-0124-5/23/10††submissionid: 989††ccs: Computing methodologies
Information extraction

##  1\. Introduction

Sentence representation learning is one of the most important tasks of natural
language processing (NLP), and high-quality sentence embedding is useful for a
large number of downstream tasks, such as semantic textual similarity measure
and semantic retrieval. Though pre-trained language model (Kenton and
Toutanova, 2019; Yang et al., 2019) achieves great successes in many
supervised tasks, it actually damages the quality of sentence embeddings
(Jiang et al., 2022). Sometimes it even underperforms the traditional
approaches such as GloVe (Pennington et al., 2014).

Some previous work(Li et al., 2020) has attributed this kind of poor
performance to the word frequency bias introduced by masked language modeling
(MLM) pre-training task in BERT(Devlin et al., 2018) or RoBERTa (Liu et al.,
2019). For example, Bert-Flow (Li et al., 2020) find out that the masked
language modeling task makes the embedding of high-frequency words concentrate
densely while low-frequency words disperse sparsely, which leads to an
anisotropy word embedding space and results in a high similarity between any
sentence pair.

Recent works (Gao et al., 2021; Yan et al., 2021) have shown that contrastive
learning helps pre-trained language models to reduce the word frequency bias
and learn good sentence embeddings without any labeled data. Contrastive
learning aims to generate a representation that maximizes the similarity
between positive pairs while minimizing the similarity between negative pairs.
As a representative method, unsupervised SimCSE(Gao et al., 2021) passes the
same input sentence to the pre-trained encoder twice and obtain two embeddings
as “positive pairs”. It randomly selects some other sentences in the same
batch as “negative pairs”. Even though SimCSE is rather simple, sentence
embeddings learned by it have been shown to be better than other more
complicated methods(Gao et al., 2021).

(a) Semantic Similarity distribution of positive pairs

(b) Semantic Similarity distribution of negative pairs

Figure 1. Training Data Distribution Bias(Semantic)

Most of the following works (Wu et al., 2021; Chuang et al., 2022; Wang et
al., 2022a) adopt the similar strategy as SimCSE to construct the positive and
negative pairs. However, we argue that this kind of strategy is too simple and
brings new types of bias, which degrades the quality of sentence embedding.
Some other studies also find the problem and try to solve this problem by
reducing some specific kind of bias. For example, ESimCSE(Wu et al., 2021)
argues SimCSE contains a ”sentence length bias” because positive pair has the
same length information, which has negative effect on sentence representation.
The authors applies a simple word repetition operation to modify sentence
lengths in a positive pair to alleviate this problem. Another known bias is
called ”false negative sample bias”(Chuang et al., 2020), which means sampling
negative pairs at random will inevitably draws some “false negatives”. The
authors propose an instance weighting method to resolve this problem.

Compared with the complex distribution of positive and negative pairs in
supervised machine learning models, the training data constructed by simple
rules in contrastive learning contains various bias, known or unknown. We
think all those biases are introduced by these simple rules. Obviously, there
are huge differences in similarity distribution of training data between
contrastive learning and supervised machine learning, which can be regarded as
a kind of ”training data distribution bias” for constrastive learning. In
order to reduce this bias, we argue that the key for contrastive learning
sentence embedding is to ”mimic” the distribution of training data in
supervised machine learning in unsupervised way, which can be regarded as a
kind of debiasing procedure.

To tackle this problem, in this paper we propose a simple yet effective
framework DebCSE, which regards the contrastive learning for unsupervised
sentence representation as a debiasing process to take both the known biases,
such as sentence length bias and false negative sample bias, and some unknown
biases into account. Specifically, given an input sentence, we generate the
positive candidates and sample the positive pairs from candidates by using an
inverse propensity weighted sampling method, which opts to select the
candidate with low surface similarity and high semantic similarity with input
sentence as the positive example. For negative pairs, we filter out candidates
according to embedding similarity and then also leverage the inverse
propensity weighted sampling method to select high-quality negatives which
have high surface similarity and low semantic similarity with input sentence.
Besides, we also propose an alternative normalization contrastive loss in this
paper to enforce the sentence embedding to be close to its sampled positive
pairs, and far apart from the sampled negative pairs as well as the random in-
batch negative pairs in an alternative way.

Extensive experimental results on semantic textual similarity tasks
demonstrate the effectiveness of our proposed model compared to strong
baselines. For example, DebCSE achieves on average 4.1 points absolute
improvements in terms of Spearman’s correlation on BERTbase compared with
SimCSE. Our experimental results also show that the DebCSE model effectively
alleviates the above-mentioned ”training data distribution bias”.

Our contributions are summarized as follows:

(1) To our knowledge, our approach is the first attempt to reexamine the
challenges of contrastive sentence embedding learning from a debiasing
perspective. We argue that reducing the ”training data distribution bias” is
the key problem for better sentence embedding.

(2) We propose DebCSE, a debiased contrastive learning framework that takes
both the surface similarity and semantic similarity between sentences into
account and incorporates inverse propensity weighting method to sample high-
quality negatives and positives, which guarantee the representation quality of
the sentence embedding.

(3) Experimental results on seven semantic textual similarity tasks show that
DebCSE outperforms the baselines by a substantial margin.

##  2\. Biases in Contrastive Sentence Embedding

In the following, we summarize the biases in contrastive sentence embedding.
First, we describe the known biases mentioned in the literature such as word
frequency bias, sentence length bias and false negative sample bias. Then we
prove the existence of the training data distribution gap between the
contrastive learning and supervised learning, which is called ”training data
distribution bias” in this paper.

###  2.1. Word Frequency Bias in Sentence Embedding

Studies in (Gao et al., 2018) have proved that most of the learnt word
embedding tend to degenerate and be distributed into a narrow cone when
training a model for natural language generation tasks through likelihood
maximization. This representation degeneration problem limits the
representation power of word embedding and the learned model parameters do not
have enough capacity to represent the diverse semantics in natural language.
They also pointed out that this anisotropy is highly relevant to the imbalance
of word frequency.

Studies in (Li et al., 2020) verify this hypothesis in the context of BERT and
indicate that the word embedding can be biased to word frequency. They observe
that high-frequency words concentrates densely and low-frequency words
disperse sparsely in the learned anisotropic embedding space. Therefore, both
the resulted sentence embedding and the induced sentence similarity are
problematic.

###  2.2. Biases in Contrastive Sentence Embedding Learning

Some following methods such as SimCSE(Gao et al., 2021) and ConSERT (Yan et
al., 2021) introduce the contastive learning into the unsupervised sentence
embedding task and alleviate the word frequency bias to some extent.

As a current state-of-the-art method, unsupervised SimCSE feeds each sentence
to the pre-trained BERT twice with two independently sampled dropout masks to
produce positive pair while randomly chose two sentences in a batch as
negative pair. It’s obvious that a positive pair would contain the same length
information because it’s derived from the same sentence while two different
sentences in negative pair usually have different length information.
Therefore, the different length information contained by positive pairs and
negative pairs can act as a strong feature to distinguish them. The model
trained with these pairs contains ”sentence length bias”, which opt to
consider that two sentences with similar lengths have near distance in
semantics space. ESimCSE(Wu et al., 2021) applies a simple word repetition
operation to modify sentence lengths in a positive pare to alleviate the
”sentence length bias”.

While significant improvement is achieved, contrastive learning approaches
generally do not take into account the high-level semantics contained in
images or sentences. For example, most contrastive learning frameworks, both
in CV and NLP, directly regards other samples within the same training batch
as negatives without considering the semantics of the content. Obviously, this
kind of sampling negative pairs at random will inevitably draws some “false
negatives” and learn less effective representation models. To alleviate the
above ”false negative sample bias” problem, some work(Chuang et al., 2020)
proposes an instance weighting method to punish false negatives and generate
noise-based negatives to guarantee the uniformity of the representation space.

###  2.3. Training Data Distribution Bias

Most current contrastive sentence learning methods leverage either same input
sentences with different dropout ratio as positive pairs or random in-batch
samples as negative pairs. So the training data construction procedure of
contrastive sentence learning can be regarded as a data generator consisting
of some simple heuristic rules. However, these simple training data
construction rules bring known or unknown new bias as discussed above. In
contrast to contrastive sentence learning, the training data of supervised
learning is usually produced by human labor and shows high quality to train
better models without bias introduced by simple rules. Many models using
supervised datasets generally outperform self-supervised approach(Gao et al.,
2021; Chuang et al., 2020; Yan et al., 2021) while they need extensive human
labour for the ground truth labels.

In this section, we will empirically prove that there is huge training data
distribution gap between the contrastive sentence learning and supervised
learning methods. By taking both the surface and semantic similarity into
consideration, we extract the training data of two different approaches to
show the distribution gap between them.

The supervised natural language inference (NLI) datasets have been proved to
be effective for learning sentence embeddings, by predicting whether the
relationship between two sentences is entailment, neutral or contradiction. In
NLI datasets, given one premise, annotators manually write one entailment
sentence that is absolutely true , one neutral sentence that might be true,
and one contradiction sentence that is definitely false. Therefore, we use
entailment pairs and its contradiction pairs from the NLI datasets as
supervised positive pairs and negative pairs, respectively.

For the training data of contrastive sentence learning, we follow SimCSE and
use the same sentence augmented by different dropout ratio as positive pairs
and random in-batch examples as negative pairs.

Table 1. Training Data Distribution Bias (Surface) | Supervised | Contrastive
---|---|---
Positive | 0.38 | 1.0
Negative | 0.28 | 0.09

After obtaining the training data, we firstly use the lexical overlap ( the
ratio of shared word number to the longer sentence length between two
sentences ) between the positive pairs or negative pairs as surface similarity
of training data. Table 1 shows the distribution difference between
contrastive and supervised approaches. It can be seen that contrastive method
has a much higher surface similarity for positive pairs and much lower surface
similarity for negative pairs, compared with the training data distribution of
supervised learning. This demonstrates that positive pairs are too similar and
negative pairs have little common words for contrastive learning, which means
the training data is too easy for model to learn and contains little useful
information.

Then, we use a pre-trained SimCSE as sentence encoder and cosine function as
similarity metric to measure semantic similarity of positive and negative
pairs for both contrastive and supervised learning. Figure 1 shows the
results. We can see that contrastive method has higher semantic similarity for
positive pairs and lower semantic similarity for negative pairs, compared with
the training data distribution of supervised learning. This also indicates the
training data of contrastive sentence learning is too easy and uninformative
even in semantic perspective.

Figure 2. In both semantic space and surface space, good positive examples
should have close semantic distance but distant surface distance from the
anchor sentence, while good negative examples should have distant semantic
distance but close surface distance from the anchor sentence.

In this paper, we argue that it’s the simple training data construction rules
adopted in contrastive sentence learning that introduce many biases into
sentence embedding and have huge distribution gap with training data used in
supervised learning. How to fill this distribution gap is key to a high
quality sentence embedding. We call this distribution gap between the
contrastive and supervised learning ”training data distribution bias” in this
paper.

Figure 3. The overall architecture of DebCSE

##  3\. Problem Formulation

In this section, we specify the research problem of unsupervised contrastive
sentence embedding learning in the debiasing perspective. As discussed above,
the training data distribution gap between contrastive and supervised learning
hinders the success of unsupervised contrastive sentence embedding learning.

To tackle this problem, we propose to fill the distribution gap as a debiasing
task by ”mimicking” the training data distribution of supervised learning in
an unsupervised way, which allows our proposed approach to possess both the
low-cost advantage of contrastive learning and avoid various biases in
training data it may introduce. Specifically, we take both the surface and
semantic similarity of positive and negative pairs into account when we
automatically construct training data of contrastive learning. Our basic
assumption is as follows: Given the input sentence $x_{i}$, an informative
positive example $x_{i}^{+}$ should have close semantic distance in semantic
space and distant surface distance in surface space with $x_{i}$
simultaneously while a good negative example $x_{i}^{-}$ should have close
surface distance but distant semantic distance with $x_{i}$ at the same time.
Figure 2 shows our proposed assumption.

Following this basic assumption of constructing training data, we can produce
candidates pool $C_{x_{i}}=\\{x_{j},j<n\\}$ for sentence $x_{i}$ in
unsupervised way and sample positive or negative pairs from $C_{x_{i}}$ by
inverse propensity weighting probability which can be computed according to
the basic assumption.

Inverse propensity weighting is a widely used debiasing technology in
recommendation systems (Ai et al., 2018) (Wang et al., 2016) (Joachims et al.,
2017), which is unbiased for any probabilistic assignment mechanism and
leverages the inverse of the propensity score (observed item clicked
probability in recommendation system) to align the biased item distribution
over the click space with that over the exposure space. Similarly, we use
inverse propensity weighting to align the biased training data distribution
over the contrastive learning with that over the supervised learning. By
applying this alignment, we can effectively mitigate the disparities in data
distribution between the two approaches. However, in this paper we leverage
inverse propensity weighting score as sampling probability to sample positive
or negative pairs from candidates pool $C_{x_{i}}=\\{x_{j},j<n\\}$, which is
different from the common usage in recommendation systems.

##  4\. Our Proposed Method

In this section, we present our DebCSE to effectively alleviate the training
data distribution bias by negative and positive sampling. The overall
architecture of DebCSE is illustrated in Figure 3, consisting of three main
components: negative sampling by inverse propensity weighting (Section 4.1),
positive sampling by inverse propensity weighting (Section 4.2), and
alternative normalization contrastive loss (Section 4.3).

###  4.1. Negative Sampling by Inverse Propensity Weighting Probability

In this section, we describe how to sample negative pairs by inverse
propensity weighting probability, which consists of two phases: First, we
filter out negative candidates to form the negative candidates pool based on
the input sentence; Then, we sample high-quality negatives according to the
sampling probability.

#### Negative Candidates Pool

Given an input sentence $x_{i}$, we firstly form the negative candidates pool
$C_{x_{i}}^{NCP}=\\{x_{j},j<n\\}$ by filtering out sentences $x_{j}$ with
semantic similarity $Sim(x_{i},x_{j})$ within specific range. Specifically,
letting $h_{i}$ denotes sentence embedding of input sentence $x_{i}$, we
firstly generate sentence embeddings for sentences in dataset using the
trained encoder such as unsupervised SimCSE model. Then, for any sentence
$x_{i}$ in dataset, we leverage $cos(h_{i},h_{j})$ as semantic similarity
metric to find out those $x_{j}$ which have semantic similarity with $x_{i}$
within specific range. We filter out those sentences with not too high or not
too low semantic similarity with $x_{i}$ as candidate negative pairs. For
those $x_{j}$ with too low semantic similarity with $x_{i}$, they are too easy
negative examples just like random negatives containing little knowledge for
model to learn. For those $x_{j}$ with too high semantic similarity with
$x_{i}$, they have much higher probability of being the “false negatives”
introducing noise into the model. Therefore, we ignore these examples and pay
more attention on much informative examples.

#### Negative Sampling

Given the input sentence $x_{i}$ and its corresponding negative candidates
pool $C_{x_{i}}^{NCP}$, we opt to select few high quality negatives among them
because the candidate number is still too large . One straightforward method
is to rank them according to semantic similarity with $x_{i}$ and select top
ranked sentences as negative examples for $x_{i}$. However, higher semantic
similarity means higher ”false negatives” probability.

To resolve this problem, we leverage inverse propensity weighting probability
as sampling probability by taking both sentence surface similarity and
semantic similarity into account. Specifically, edit distance (Ristad and
Yianilos, 1998) $edit(x_{i},x_{j})$ and embedding cosine function
$cos(h_{i},h_{j})$ are used as similarity metrics to measure the surface
dissimilarity and semantic similarity of $x_{j}$ with $x_{i}$, respectively.
To make the two similarity score comparable, we use $Softmax(\cdot)$ function
to normalize each score as follows:

(1)
$S_{sur}(x_{i},x_{j})=1-\frac{e^{edit(x_{i},x_{j})}}{\sum_{k=1}^{n}e^{edit(x_{i},x_{k})}}$

and

(2)
$S_{sem}(x_{i},x_{j})=\frac{e^{cos(x_{i},x_{j})}}{\sum_{k=1}^{n}e^{cos(x_{i},x_{k})}}$

where $n$ is the number of negative examples in the negative candidates pool
$C_{x_{i}}^{NCP}$, while $S_{sur}(x_{i},x_{j})$ and $S_{sem}(x_{i},x_{j})$ are
surface similarity and semantic similarity scores, respectively.

In order to find out those $x_{j}$ with higher surface similarity and lower
semantic similarity with $x_{i}$ simultaneously, we take both the surface and
semantic similarity into consideration and propose to sample the negative
examples according to the following inverse propensity weighting probability:

(3)
$\tilde{P}_{neg}(x_{j}|x_{i})=(1-\lambda_{n})S_{sur}(x_{i},x_{j})+\lambda_{n}(1-S_{sem}(x_{i},x_{j}))$

where $\lambda_{n}$ is the balanced hyper-parameter for adjusting the
importance of semantic similarity.

To make the model pay more attention to more informative and better quality
negative examples among the candidates during the sampling procedure,
$Softmax(\cdot)$ function is used to sharpen the sampling probability as
follows:

(4)
$P_{neg}(x_{j}|x_{i})=\frac{e^{\tilde{P}_{neg}(x_{j}|x_{i})}}{\sum_{k=1}^{n}e^{\tilde{P}_{neg}(x_{k}|x_{i})}}$

Then, we sample from the pool of $n$ negative candidates, each with a
probability of ${P}(x_{j}|x_{i})$, according to the rules that higher
probability to be selected as negative of $x_{i}$ for those examples with
higher sampling probability. As a result, we obtain a set
$C_{x_{i}}^{neg}=\\{x_{1},x_{2}....,x_{m}\\}$ consisting of $m$ sampled
negative examples for input sentence $x_{i}$. Here we let $m=2$ in our
experiments.

###  4.2. Positive Sampling by Inverse Propensity Weighting Probability

In this section, we describe how to sample positive pairs by inverse
propensity weighting probability, which also consists of two phases: First, we
generate positive candidates to form the positive candidates pool based on the
input sentence; Then, high-quality positives are sampled according to the
sampling probability.

#### Positive Candidates Pool

Given the input sentence $x_{i}$, we try to generate augmented positive
candidate samples exhibiting certain kinds of variation in an unsupervised
way. We can leverage strong text generators such as ChatGPT to automatically
produce positive candidates through in context learning prompt. However, we
intent to show that a weaker text generator also works. To this end, we
utilize back-translation model111Opus-mt https://huggingface.co/Helsinki-
NLP/opus-mt-en-zh 222Opus-mt https://huggingface.co/Helsinki-NLP/opus-mt-zh-en
or summary model as the text generator. To produce as many different
candidates as possible, we inject one or two high frequency words as noise or
mask one or two words on random position in the sentence $x_{i}$ before
sending the input sentence into text generator. It should be noticed that if
ChatGPT is used as a tool for generating positive examples, our proposed model
should achieve much better performance. We will leave this kind of adopting
large language model to produce training data of contrastive learning as our
future work.

#### Positive Sampling

After obtaining corresponding positive candidates pool
$C_{x_{i}}^{PCP}=\\{x_{j},j<n\\}$ of the input sentence $x_{i}$ , we also use
inverse propensity weighting probability as sampling probability to select
informative positive examples in an unsupervised manner, just like we do in
negative sampling stage. However, we intent to regard those $x_{j}$ with lower
surface similarity and higher semantic similarity with $x_{i}$ simultaneously
as high quality positive examples. Therefore, we propose to sample the
positive examples according to the following inverse propensity weighting
probability:

(5)
$\tilde{P}_{pos}(x_{j}|x_{i})=(1-\lambda_{p})(1-S_{sur}(x_{i},x_{j}))+\lambda_{p}S_{sem}(x_{i},x_{j})$

where $\lambda_{p}$ is the balanced hyper-parameter.

We also leverage $Softmax(\cdot)$ to let model focus on more informative
positive examples as follows:

(6)
${P_{pos}}(x_{j}|x_{i})=\frac{e^{\tilde{P}_{pos}(x_{j}|x_{i})}}{\sum_{k=1}^{n}e^{\tilde{P}_{pos}(x_{k}|x_{i})}}$

We can sample positive examples according to the above probability to obtain
the set $C_{x_{i}}^{pos}=\\{x_{1},x_{2}....,x_{m}\\}$ consisting of $m$
sampled positive examples for input sentence $x_{i}$. And we also sample $m=2$
positive examples for each input sentence in our experiments.

Table 2. Sentence representation performance on the STS tasks. We employ our
method to BERT and RoBERTa in both base and large versions and report
Spearman’s correlation.

| Models | STS12 | STS13 | STS14 | STS15 | STS16 | STS-B | SICK-R | Avg.
---|---|---|---|---|---|---|---|---|---
BERTbase | BERT (first-last avg.) | 39.7 | 59.38 | 49.67 | 66.03 | 66.19 | 53.87 | 62.06 | 56.7
BERT-flow | 58.4 | 67.1 | 60.85 | 75.16 | 71.22 | 68.66 | 64.47 | 66.55
BERT-whitening | 57.83 | 57.83 | 60.9 | 75.08 | 71.31 | 68.24 | 63.73 | 66.28
IS-BERT | 56.77 | 69.24 | 61.21 | 75.23 | 70.16 | 69.21 | 64.25 | 66.58
ConSERT | 64.64 | 78.49 | 69.07 | 79.72 | 75.95 | 73.97 | 67.31 | 72.74
SimCSE | 68.4 | 82.41 | 74.38 | 80.91 | 78.56 | 76.85 | 72.23 | 76.25
ESimCSE | 73.4 | 83.27 | 77.25 | 82.66 | 78.81 | 80.17 | 72.3 | 78.27
SNCSE | 70.67 | 84.79 | 76.99 | 83.69 | 80.51 | 81.35 | 74.77 | 78.97
DiffCSE | 72.28 | 84.43 | 76.47 | 83.9 | 80.54 | 80.59 | 71.23 | 78.49
ArcCSE | 72.08 | 84.27 | 76.25 | 82.32 | 79.54 | 79.92 | 72.39 | 78.11
DebCSE | 76.15 | 84.67 | 78.91 | 85.41 | 80.55 | 82.99 | 73.6 | 80.33
RoBERTabase | RoBERTa (first-last avg.) | 40.88 | 58.74 | 49.07 | 65.63 | 61.48 | 58.55 | 61.63 | 56.57
RoBERTa-whitening | 46.99 | 63.24 | 57.23 | 71.36 | 68.99 | 61.36 | 62.91 | 61.73
SimCSE | 70.16 | 75.19 | 65.52 | 75.08 | 71.31 | 68.24 | 63.73 | 76.57
DiffCSE | 70.05 | 83.43 | 75.49 | 82.81 | 82.12 | 82.38 | 71.19 | 78.21
ESimCSE | 69.9 | 82.5 | 74.68 | 83.19 | 80.3 | 80.99 | 70.54 | 77.44
SNCSE | 70.62 | 84.42 | 77.24 | 84.85 | 81.49 | 83.07 | 72.92 | 79.23
DebCSE | 74.29 | 85.54 | 79.46 | 85.68 | 81.2 | 83.96 | 74.04 | 80.60
BERTlarge | ConSERT | 70.69 | 82.96 | 74.13 | 82.78 | 76.66 | 77.53 | 70.37 | 76.45
unsup-SimCSE | 70.88 | 84.16 | 76.43 | 84.5 | 79.76 | 79.26 | 73.88 | 78.41
ArcCSE | 73.17 | 86.19 | 77.9 | 84.97 | 79.43 | 80.45 | 73.5 | 79.37
ESimCSE | 73.21 | 85.37 | 77.73 | 84.3 | 78.92 | 80.73 | 74.89 | 79.31
SNCSE | 71.94 | 86.66 | 78.84 | 85.74 | 80.72 | 82.29 | 75.11 | 80.19
DebCSE | 76.82 | 86.36 | 79.81 | 85.8 | 80.83 | 83.45 | 74.67 | 81.11
RoBERTalarge | unsup-SimCSE | 70.88 | 84.16 | 76.43 | 84.5 | 79.76 | 79.26 | 73.88 | 78.90
ESimCSE | 73.20 | 84.93 | 76.88 | 84.86 | 81.21 | 82.79 | 72.27 | 79.45
SNCSE | 73.71 | 86.73 | 80.35 | 86.8 | 83.06 | 84.31 | 77.43 | 81.77
DebCSE | 77.68 | 87.17 | 80.53 | 85.90 | 83.57 | 85.36 | 73.89 | 82.01

###  4.3. Alternative Normalization Contrastive Loss

Recently, a few methods in image contrastive learning such as BYOL (Grill et
al., 2020) and SimSiam (Chen and He, 2021) have proved that batch-wise or
feature-wise normalization helps produce high-quality representations.
However, we find direct normalization on sentence embedding is harmful to good
text representation. Therefore, in this paper we propose an alternative
normalization contrastive loss, which is a kind of indirect normalization
approach. More precisely, we compute an normalized sentence embedding from one
sentence in the positive pair and predict this normalized representation from
another sentence in positive pair.

Given an input sentence $x_{i}$, we let $x_{i}^{+}$ and $x_{i}^{-}$ denote the
mined positive and negative example, respectively. The parameter shared dual-
tower encoder with different dropout is adopted to produce the sentence
embedding, just as SimCSE does. However, $x_{i}$ and $x_{i}^{+}$ are used as
positive example to be feed into the dual-tower encoder as input.

Obtaining two sentence embedding $\mathbf{h}_{x_{i}}$ and
$\mathbf{h}_{x_{i}^{+}}$ produced by encoder from two positive examples , we
compute their normalized representation $\mathbf{z}_{x_{i}}$ and
$\mathbf{z}_{x_{i}^{+}}$ leveraging batch normalization (Ioffe and Szegedy,
2015). Then we setup an alternative prediction problem with the following loss
function:

(7)
$\mathcal{L}(h_{x_{i}},h_{x_{i}^{+}})={L}(z_{x_{i}},h_{x_{i}^{+}})+{L}(z_{x_{i}^{+}},h_{x_{i}})$

This loss function contains two terms that setup the alternative prediction
problem of predicting the normalized representation $\mathbf{z}_{x_{i}}$ from
the positive example’s sentence embedding $\mathbf{h}_{x_{i}^{+}}$, and
$\mathbf{z}_{x_{i}^{+}}$ from $\mathbf{h}_{x_{i}}$. Each term represents the
Debias-infoNCE loss as follows:

(8)
${L}(z_{x_{i}},h_{x_{i}^{+}})=-log\frac{e^{{cos(\mathbf{z}_{x_{i}},\mathbf{h}_{x_{i}^{+}})}/\tau}}{\sum_{k=1}^{m}{e}^{cos(\mathbf{z}_{x_{i}},\mathbf{z}_{x_{i}^{-}})/\tau}+\sum_{j=1}^{N-m}e^{cos(\mathbf{z}_{x_{i}},\mathbf{z}_{j})/\tau}}$

and

(9)
${L}(z_{x_{i}^{+}},h_{x_{i}})=-log\frac{e^{{cos(\mathbf{z}_{x_{i}^{+}},\mathbf{h}_{x_{i}})}/\tau}}{\sum_{k=1}^{m}{e}^{cos(\mathbf{z}_{x_{i}^{+}},\mathbf{z}_{x_{i}^{-}})/\tau}+\sum_{j=1}^{N-m}e^{cos(\mathbf{z}_{x_{i}^{+}},\mathbf{z}_{j})/\tau}}$

where $\mathbf{\tau}$ is temperature parameter and we set $\mathbf{\tau=0.05}$
as default configurations of SimCSE does. ${N}$ is the size of mini-batch and
$\mathbf{m}$ is the number of the mined hard negative examples.

Compared with infoNCE used by SimCSE, Debias-infoNCE loss replaces the same
input sentence with the mined positive example and introduces hard negative
examples into random negatives. This reduces the training data distribution
gap between the contrastive and supervised learning and helps produce high-
quality sentence representation.

##  5\. Experiments

###  5.1. Setups

#### Evaluation Tasks

Consistent with previous works (Gao et al., 2021), this work uses the SentEval
toolkit (Conneau and Kiela, 2018) and semantic textual similarity (STS)
datasets for evaluating unsupervised sentence embedding models. The STS
dataset includes the following tasks: STS tasks, 2012 - 2016 (Agirre et al.,
2012, 2013, 2014, 2015, 2016), STS benchmark (STSb) (Cer et al., 2017) and
SICK Relatedness (SICK-R) (Marelli et al., 2014). STS dataset is one of the
most widely used benchmark for the evaluation of unsupervised sentence
embedding and aims to measure the semantic similarity of sentence pairs and
tests the semantic representation ability of models. Each sample in the STS
dataset contains a pair of sentences and a corresponding similarity score,
which ranges from 0 to 5 and is used to measure the similarity between
sentence pair.

#### Baselines

To verify the effectiveness of DebCSE, we compare our model with strong
unsupervised baselines, including SimCSE (Gao et al., 2021), ArcCSE (Zhang et
al., 2022), DiffCSE (Chuang et al., 2022), SNCSE (Wang et al., 2022b), ESimCSE
(Wu et al., 2022), BERT-flow (Li et al., 2020), ConSERT (Yan et al., 2021),
IS-BERT (Zhang et al., 2020), and Bert-whitening (Su et al., 2021). Some
baseline methods are evaluated on both the base and large parameter scales,
while others are only evaluated on the base parameter scale..

#### Implementation Details

We use Wikipedia-1M data as the training dataset, which is randomly sampled
1,000,000 sentences from Wikipedia. During training, we train our models for 1
epoch with temperature $\tau$ = 0.05 using an Adam optimizer. The batch size
is 64 and the learning rate is 2.5e-5 and 5e-6 respectively, for base and
large parameter scale models. For the four backbone models, the dropout rate
is set to 0.1. Those models start from the pre-trained checkpoints of BERT-
uncased (Devlin et al., 2018) and RoBERTa-cased (Liu et al., 2019) on Wiki-1M.

When constructing negative candicates pool, we use a pre-trained unsupervised
SimCSE model with similarity score range between 0.25 and 0.75 as a semantic
sentence filter to find out those sentences which have not too low and not too
high semantic similarity with the anchor sentence.

Following the evaluation process of SimCSE (Gao et al., 2021) for a fair
comparison, we evaluate the model every 125 training steps on the development
set of STS-B and SICK-R to keep the best checkpoint for the final evaluation
on test sets. We implement our framework with Huggingface’s transformers (Wolf
et al., 2020).

###  5.2. Main Results

Table 2 shows different approaches’ performances on seven semantic textual
similarity (STS) test sets. It can be seen that DebCSE outperforms other SOTA
methods substantially and achieves the best performance in different parameter
scale settings. Specifically, DebCSE achieves on average 4.1 points absolute
improvements in terms of Spearman’s correlation on BERTbase compared with
SimCSE. Meanwhile, DebCSE obtains the best performance with the highest
average Spearman’s correlation coefficient on all backbone encoders, which are
respectively 80.33% for BERT-base, 81.11% for BERT-large, 80.60% for RoBERTa-
base and 82.01% for RoBERTa-large.

These experimental results demonstrate that DebCSE has better generalization
ability over SOTA unsupervised sentence embedding methods and prove that
reducing the ”training data distribution bias” is indeed the key problem for
sentence embedding learning.

(a) Similarity distribution of positive pairs

(b) Similarity distribution of negative pairs

Figure 4. Effectiveness of Debias(Semantic)

##  6\. Analysis

In this section, we firstly perform an extensive ablation studies to DebCSE
with BERTbase on the development set of STS-B. Then, we further analyze the
effect of DebCSE on reducing training data distribution bias. We also explore
the distribution of sentence embedding based on Uniformity and Alignment, and
the effect of hyper-parameters.

###  6.1. Ablation Studies

Table 3. Ablation study of different components Method | STS Avg.
---|---
DebCSE | 80.33
w/o Alternative Norm | 79.43
w/o Negative-Debias | 78.96
w/o Postive-Debias | 78.82

To verify each component’s contribution to the DebCSE’s final result, we
conduct ablation study by removing each of the components on seven STS tasks
and report the average value of the Spearman’s correlation metric. As shown in
Table 3, removing any component would lead to the sharp performance
degradation of DebCSE. It indicates that all the three key components are
important in our proposed framework.

###  6.2. Analysis of Training Data Distribution Bias

Table 4. Effectiveness of Debias(Surface) | Supervised | Contrastive | DebCSE
---|---|---|---
Positive | 0.38 | 1.0 | 0.56
Negative | 0.28 | 0.09 | 0.21

In this section we try to explore the effect on training data distribution
bias after introducing our proposed DebCSE.

We use the mined positive and negative pairs as described in Section 4 as the
training data of DebCSE and compare both the training data surface similarity
distribution and semantic similarity distribution of DebCSE with that of
commonly used contrastive and supervised learning.

We compute the lexical overlap between the positive pairs or negative pairs as
surface similarity of training data and show the results in Table 4. It can be
seen that the surface similarity of both positive and negative pairs of DebCSE
are between the corresponding values of contrastive and supervised learning.
This demonstrates that the training data of DebCSE is much harder and
informative than that of contrastive sentence learning method while DebCSE
indeed alleviates the surface training data distribution bias problem.

We also use a pre-trained SimCSE as sentence encoder and cosine function as
similarity metric to measure semantic similarity of positive and negative
pairs. Figure 4 shows the comparison results of three approaches. We can see
that distribution of training data of DebCSE is near the corresponding
distribution of supervised learning, compared with the training data
distribution of contrastive learning, which demonstrates DebCSE reduces the
semantic training data distribution gap between contrastive and supervised
learning bias.

###  6.3. Uniformity and Alignment

The quality of learned representations can also be measured by Uniformity and
Alignment. Alignment is computed based on the expected distance between
representations of positive pairs. Lower alignment indicates close
representations between positive pairs. Besides, Uniformity measures how well
representations are uniformly scattered across the representation space.

Table 5. Uniformity and Alignment | unsup-SimCSE | sup-SimCSE | DebCSE
---|---|---|---
Uniformity | -2.26 | -1.13 | -1.61
Alignment | 0.38 | 0.16 | 0.24

We follow prior studies (Wang and Isola, 2020; Gao et al., 2021) to measure
Uniformity on the whole STS-B test set, and Alignment on sentence pairs in
STS-B test set with correlation scores higher than 4. As shown in Table 5,
DebCSE shows better Alignment and higher Uniformity than the unsupervised
SimCSE model based on Bert-Base. We conjecture that DebCSE obtains higher
uniformity and better Alignment because of the sampled hard negative and
positive pairs, which makes DebCSE behave more like supervised contrastive
models. Compared to the supervised SimCSE model, DebCSE achieves lower
Uniformity but larger Alignment and we think that DebCSE obtains lower
uniformity by still adopting random in-batch negatives.

###  6.4. Hyper-parameters Analysis

Table 6. Hyper-parameters Analysis Result

${\lambda_{n}}$ ${\lambda_{p}}$ | 1 | 0.8 | 0.6 | 0.4 | 0.2 | 0
---|---|---|---|---|---|---
1 | 79.46 | 79.93 | 80.14 | 79.64 | 80.11 | 79.24
0.8 | 79.85 | 80.33 | 79.96 | 79.59 | 79.55 | 79.37
0.6 | 80.12 | 80.25 | 80.23 | 80.13 | 80.17 | 79.95
0.4 | 79.91 | 79.91 | 80.24 | 80.07 | 80.24 | 79.32
0.2 | 79.33 | 79.70 | 80.12 | 79.66 | 80.06 | 79.20
0 | 79.27 | 79.49 | 79.62 | 79.35 | 79.53 | 79.18

In this section, we conduct extensive experiments to explore the effects of
weighting hyper-parameter of semantic similarity $\lambda_{p}$ and
$\lambda_{n}$ on model’s performance.

Here $\lambda_{p}$ denotes weighting parameter of semantic similarity when we
sample positive examples while $\lambda_{n}$ means weighting parameter of
semantic similarity when we sample negative examples. Setting the parameter a
larger value means the semantic similarity is more important than surface
similarity during the sampling procedure. We adopt various values for
$\lambda_{p}$ and $\lambda_{n}$ and Table 6 shows the experimental results.

From the experimental results, we mainly have the following observations:

(1) When $\lambda_{p}=0.8$ and $\lambda_{n}=0.8$, we have the best performance
model. This demonstrates the semantic similarity is more important than
surface similarity during the training data sampling procedure, no matter
positive or negative example sampling.

(2) When we only use semantic similarity and ignore the effect of surface
similarity($\lambda_{p}=1$ or $\lambda_{n}=1$), model performance is sub-
optimal. This indicates surface similarity is beneficial to model performance.

All in all, we can draw the conclusion that surface similarity and semantic
similarity both contribute to the final performance gain while semantic
similarity is more important during the training data sampling procedure.

##  7\. Related Work

Both supervised and unsupervised learning have been widely studied in the
field of sentence embedding learning. Early supervised sentence embedding
learning studies such as (Reimers and Gurevych, 2019) mainly focus on
supervised learning methods based on Natural Language Inference. Unsupervised
approaches such as (Mikolov et al., 2013) try to use pooling word2vec
embeddings to derive representations without labeled datasets. A plethora of
studies, such as (Reimers and Gurevych, 2019; Li et al., 2020; Yan et al.,
2021) have established that native sentence embedding generated directly from
pre-trained language models lack the desired level of quality. And it’s
embeddings is often highly similar and concentrated, which are unable to
accurately reflect the semantic similarity and relatedness between sentences.
According to the (Li et al., 2020), word frequency bias is an important cause
of the poor performance of sentence embedding and lead to the collapse and
concentration of sentence vectors due to the uneven distribution of word
frequencies in the representation space. To address this issue, it proposes a
method that aims to improve the performance of the BERT model by reducing word
frequency bias through the use of a flow model. While (Su et al., 2021)
introduced whitening method to resolved the problem.

In recent years, contrastive learning methods such as SimCSE(Gao et al., 2021)
and ConSERT (Yan et al., 2021) have achieved great success in unsupervised
sentence representation learning. Contrastive learning is based on the
principle of maximizing the mutual information between different views and has
demonstrated to be highly effective in learning robust sentence
representations in an unsupervised setting. For example, SimCSE use dropout as
data augment function to make positive examples to obtain high-quality
sentence representations. ConSERT (Yan et al., 2021) explores various
effective data augmentation strategies, and ESimCSE (Wu et al., 2021) attempts
to attenuate the length bias by randomly repeating words. (Zhou et al., 2022)
try to reduce the sampling biased cause by improper negatives pairs. Besides,
ArcCSE (Zhang et al., 2022) design triplet loss to enhance the pairwise
discriminative ability of the model. DiffCSE (Chuang et al., 2022) learns
sentence embeddings that are sensitive to the difference between the original
sentences and an edited sentences, which improves the quality of
representation. While SNCSE (Wang et al., 2022b) proposes contrastive learning
for unsupervised sentence embedding with soft negative samples.

Different from all the current SOTA methods, we regard the contrastive
sentence learning as a debiasing procedure, which tries to fill the training
data distribution gap between the contrastive learning and supervised
learning. It’s not hard to see that many of current SOTA approaches are just a
special case of our proposed DebCSE framework, which try to reduce either the
positive example bias or negative example bias introduced by the simple
training data construction rules in contrastive sentence learning.

##  8\. Conclusion and Future Work

In this paper, we proposed the DebCSE, a debiased contrastive learning
framework for unsupervised sentence representation learning. We think the
simple rules used to produce training data in contrastive learning bring
various biases and propose to sample positive and negative examples by inverse
propensity weighting, which reduces the training data distribution gap between
the contrastive learning and supervised learning. We demonstrate the
effectiveness of our framework on seven STS tasks and experimental results
have shown that our approach outperforms several competitive baselines.

Additionally, we believe that using large language models such as ChatGPT to
create positive and negative examples for contrastive learning tasks will
greatly diminish the various biases present in the current contrastive
learning training data. In the future, we plan to adopt this strategy to
enhance the representation quality of producing sentence embeddings by
contrastive learning.

## References

  * (1)
  * Agirre et al. (2015) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Inigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, et al. 2015\.  Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability. In _Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015)_. 252–263.
  * Agirre et al. (2014) Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M Cer, Mona T Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014\.  SemEval-2014 Task 10: Multilingual Semantic Textual Similarity.. In _SemEval@ COLING_. 81–91.
  * Agirre et al. (2016) Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez Agirre, Rada Mihalcea, German Rigau Claramunt, and Janyce Wiebe. 2016.  Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In _SemEval-2016. 10th International Workshop on Semantic Evaluation; 2016 Jun 16-17; San Diego, CA. Stroudsburg (PA): ACL; 2016\. p. 497-511._ ACL (Association for Computational Linguistics).
  * Agirre et al. (2012) Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012\.  SemEval-2012 Task 6: A Pilot on Semantic Textual Similarity. In _*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_. Association for Computational Linguistics, Montréal, Canada, 385–393.  https://aclanthology.org/S12-1051
  * Agirre et al. (2013) Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013.  *SEM 2013 shared task: Semantic Textual Similarity. In _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity_. Association for Computational Linguistics, Atlanta, Georgia, USA, 32–43.  https://aclanthology.org/S13-1004
  * Ai et al. (2018) Qingyao Ai, Keping Bi, Cheng Luo, Jiafeng Guo, and W Bruce Croft. 2018.  Unbiased learning to rank with unbiased propensity estimation. In _The 41st international ACM SIGIR conference on research & development in information retrieval_. 385–394.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017.  Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation.  _arXiv preprint arXiv:1708.00055_ (2017).
  * Chen and He (2021) Xinlei Chen and Kaiming He. 2021.  Exploring simple siamese representation learning. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 15750–15758.
  * Chuang et al. (2020) Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. 2020\.  Debiased contrastive learning.  _Advances in neural information processing systems_ 33 (2020), 8765–8775.
  * Chuang et al. (2022) Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Li, Wen-tau Yih, Yoon Kim, and James Glass. 2022\.  DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings. In _Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
  * Conneau and Kiela (2018) Alexis Conneau and Douwe Kiela. 2018.  Senteval: An evaluation toolkit for universal sentence representations.  _arXiv preprint arXiv:1803.05449_ (2018).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Gao et al. (2018) Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tieyan Liu. 2018.  Representation Degeneration Problem in Training Natural Language Generation Models. In _International Conference on Learning Representations_.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  Simcse: Simple contrastive learning of sentence embeddings.  _arXiv preprint arXiv:2104.08821_ (2021).
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. 2020\.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ 33 (2020), 21271–21284.
  * Ioffe and Szegedy (2015) Sergey Ioffe and Christian Szegedy. 2015.  Batch normalization: Accelerating deep network training by reducing internal covariate shift. In _International conference on machine learning_. pmlr, 448–456.
  * Jiang et al. (2022) Ting Jiang, Shaohan Huang, Zihan Zhang, Deqing Wang, Fuzhen Zhuang, Furu Wei, Haizhen Huang, Liangjie Zhang, and Qi Zhang. 2022.  Promptbert: Improving bert sentence embeddings with prompts.  _arXiv preprint arXiv:2201.04337_ (2022).
  * Joachims et al. (2017) Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017\.  Unbiased learning-to-rank with biased feedback. In _Proceedings of the tenth ACM international conference on web search and data mining_. 781–789.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of NAACL-HLT_. 4171–4186.
  * Li et al. (2020) Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020\.  On the Sentence Embeddings from Pre-trained Language Models. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. 9119–9130.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014.  A SICK cure for the evaluation of compositional distributional semantic models. In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_. 216–223.
  * Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.  Distributed representations of words and phrases and their compositionality.  _Advances in neural information processing systems_ 26 (2013).
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014.  Glove: Global vectors for word representation. In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_. 1532–1543.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 3982–3992.
  * Ristad and Yianilos (1998) Eric Sven Ristad and Peter N Yianilos. 1998.  Learning string-edit distance.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 20, 5 (1998), 522–532.
  * Su et al. (2021) Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021\.  Whitening sentence representations for better semantics and faster retrieval.  _arXiv preprint arXiv:2103.15316_ (2021).
  * Wang et al. (2022b) Hao Wang, Yangguang Li, Zhen Huang, Yong Dou, Lingpeng Kong, and Jing Shao. 2022b.  SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samples.  _arXiv preprint arXiv:2201.05979_ (2022).
  * Wang and Isola (2020) Tongzhou Wang and Phillip Isola. 2020.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In _International Conference on Machine Learning_. PMLR, 9929–9939.
  * Wang et al. (2022a) Wei Wang, Liangzhu Ge, Jingqiao Zhang, and Cheng Yang. 2022a.  Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negatives. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2159–2165.
  * Wang et al. (2016) Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016.  Learning to rank with selection bias in personal search. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_. 115–124.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020\.  Transformers: State-of-the-Art Natural Language Processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. Association for Computational Linguistics, Online, 38–45.  https://www.aclweb.org/anthology/2020.emnlp-demos.6
  * Wu et al. (2021) Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2021\.  ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding.  _arXiv preprint arXiv:2109.04380_ (2021).
  * Wu et al. (2022) Xing Wu, Chaochen Gao, Liangjun Zang, Jizhong Han, Zhongyuan Wang, and Songlin Hu. 2022\.  ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding. In _Proceedings of the 29th International Conference on Computational Linguistics_. International Committee on Computational Linguistics, Gyeongju, Republic of Korea, 3898–3907.  https://aclanthology.org/2022.coling-1.342
  * Yan et al. (2021) Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. 2021\.  ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Online, 5065–5075.  https://doi.org/10.18653/v1/2021.acl-long.393
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019\.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ 32 (2019).
  * Zhang et al. (2020) Yan Zhang, Ruidan He, Zuozhu Liu, Kwan Hui Lim, and Lidong Bing. 2020.  An unsupervised sentence embedding method by mutual information maximization.  _arXiv preprint arXiv:2009.12061_ (2020).
  * Zhang et al. (2022) Yuhao Zhang, Hongji Zhu, Yongliang Wang, Nan Xu, Xiaobo Li, and Binqiang Zhao. 2022\.  A Contrastive Framework for Learning Sentence Representations from Pairwise and Triple-wise Perspective in Angular Space. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. 4892–4903.
  * Zhou et al. (2022) Kun Zhou, Beichen Zhang, Wayne Xin Zhao, and Ji-Rong Wen. 2022\.  Debiased Contrastive Learning of Unsupervised Sentence Representations.  _arXiv preprint arXiv:2205.00656_ (2022).