#  EnterpriseEM:
Fine-tuned Embeddings for Enterprise Semantic Search

Kamalkumar Rathinasamy
<EMAIL_ADDRESS>Jayarama Nettar
<EMAIL_ADDRESS>Amit Kumar
<EMAIL_ADDRESS>Vishal Manchanda
<EMAIL_ADDRESS>Arun Vijayakumar
<EMAIL_ADDRESS>Ayush Kataria
<EMAIL_ADDRESS>Venkateshprasanna Manjunath
<EMAIL_ADDRESS>Chidambaram GS
<EMAIL_ADDRESS>Jaskirat Singh Sodhi
<EMAIL_ADDRESS>Shoeb Shaikh
<EMAIL_ADDRESS>Wasim Akhtar Khan
<EMAIL_ADDRESS>Prashant Singh
<EMAIL_ADDRESS>Tanishq Dattatray Ige
<EMAIL_ADDRESS>Vipin Tiwari
<EMAIL_ADDRESS>Rajab Ali Mondal
<EMAIL_ADDRESS>Harshini K
<EMAIL_ADDRESS>S Reka
<EMAIL_ADDRESS>Chetana Amancharla
<EMAIL_ADDRESS>Faiz ur Rahman
<EMAIL_ADDRESS>Harikrishnan P A
<EMAIL_ADDRESS>Indraneel Saha
<EMAIL_ADDRESS>Bhavya Tiwary
<EMAIL_ADDRESS>Navin Shankar Patel
<EMAIL_ADDRESS>Pradeep T S
<EMAIL_ADDRESS>Balaji A J
<EMAIL_ADDRESS>Priyapravas
<EMAIL_ADDRESS>Mohammed Rafee Tarafdar
<EMAIL_ADDRESS>
Infosys Limited

###### Abstract

Enterprises grapple with the significant challenge of managing proprietary
unstructured data, hindering efficient information retrieval. This has led to
the emergence of AI-driven information retrieval solutions, designed to
adeptly extract relevant insights to address employee inquiries. These
solutions often leverage pre-trained embedding models and generative models as
foundational components. While pre-trained embeddings may exhibit proximity or
disparity based on their original training objectives, they might not fully
align with the unique characteristics of enterprise-specific data, leading to
suboptimal alignment with the retrieval goals of enterprise environments. In
this paper, we propose a methodology to fine-tune pre-trained embedding models
specifically for enterprise environments. By adapting the embeddings to better
suit the retrieval tasks prevalent in enterprises, we aim to enhance the
performance of information retrieval solutions. We discuss the process of
fine-tuning, its effect on retrieval accuracy, and the potential benefits for
enterprise information management. Our findings demonstrate the efficacy of
fine-tuned embedding models in improving the precision and relevance of search
results in enterprise settings.

##  1 Introduction

In the context of enterprises accumulating proprietary unstructured data, AI-
driven information retrieval solutions have emerged as vital tools for
extracting relevant answers to employee queries. Traditional methods for
developing such solutions often involve choosing between Retrieval Augmented
Generation (RAG) or fine-tuned Large Language Models (LLMs). However, fine-
tuned LLMs, comprising only generative models, lack a guarantee of factual
accuracy, while RAG, comprising an embedding model and a generative model,
assures factual precision (Lewis at al., 2020 [1]). Despite their superior
performance in general, RAG based solutions often rely on pre-trained models,
potentially leading to suboptimal alignment with enterprise-specific data.

Addressing this challenge entails exploring two potential avenues: Firstly,
recent studies such as RAFT (Zhang et al., 2024 [2]) explore the integration
of fine-tuned generative models within a RAG pipeline to enhance accuracy,
albeit requiring substantial domain-specific data to fine-tune the generative
models. Alternatively, leveraging domain-specific embedding models within a
RAG pipeline to enhance accuracy remains an underexplored area.

Earlier efforts, such as BioBERT (Lee et al., 2019 [3]), SciBERT (Beltagy et
al., 2019 [4]), and LEGAL-BERT (Chalkidis et al., 2020 [5]) have effectively
demonstrated the efficacy of domain-specific embeddings in information
retrieval tasks. These endeavors primarily investigated two methodologies: (a)
extending the pre-training of BERT and (b) pre-training BERT from scratch,
both employing domain-specific corpora. Despite yielding commendable results,
these methodologies necessitated substantial domain-specific corpora, with
figures as staggering as 21.3B words for BioBERT, 3.17B tokens for SciBERT,
and 11.5GB of text data for LEGAL-BERT, thereby posing significant challenges,
particularly in low-resource domains like enterprises. Hence, for such
domains, an alternative methodology is to fine-tune an embedding
model111Models such as Sentence-T5 (Ni at al., 2022 [6]) which produce single
vector embeddings might be more suitable over BERT models for retrieval tasks.
with enterprise corpus to arrive at an EnterpriseEM (Enterprise Embedding
Model), such as the InfosysEM (Infosys Embedding Model), tailored for all
enterprise (Infosys Limited) information retrieval tasks. Leveraging an
EnterpriseEM in lieu of pre-trained embedding model within a RAG pipeline
bolsters the accuracy of semantic search within enterprise contexts.

##  2 Datasets

Our dataset comprised a diverse range of internal Infosys Ltd. data, including
technical course contents, internal knowledge base articles, standard
operating procedures for technical tasks, a repository of internal technical
queries with corresponding resolutions, sales data, and employee blogs. Text
data was extracted from various source document formats such as PDF, MS Word,
Excel, PowerPoint, and web pages. However, data from audio, video, and image
files were not included in this version of the dataset.

###  2.1 Extraction

Langchain’s document
loaders222https://python.langchain.com/v0.1/docs/modules/data_connection/document_loaders/,
including UnstructuredFileLoaders from the community package, provided a
convenient abstraction atop renowned parsing libraries, offering comprehensive
support for various file formats such as PDFs, MS Word documents, PowerPoint
presentations, Excel sheets, HTMLs, markdowns, emails, Evernote files, ODT
files, and text files, thereby facilitating efficient data extraction. In
addition to this, specialized tools such as
PDFMiner333https://pdfminersix.readthedocs.io/en/latest/,
PyMuPDF444https://pymupdf.readthedocs.io/en/latest/, python-
pptx555https://python-pptx.readthedocs.io/en/latest/, and
BeautifulSoup666https://www.crummy.com/software/BeautifulSoup/ were utilized
for extracting text from PDFs, PowerPoint presentations and HTML files. Non-
textual data, such as images and graphics in these files were disregarded
during the extraction process.

###  2.2 Preprocessing

The collected data underwent a multi-step preprocessing pipeline to ensure its
suitability for synthetic question generation.

#### Masking

The Presidio Analyzer777https://microsoft.github.io/presidio/analyzer/ and
Anonymizer888https://microsoft.github.io/presidio/anonymizer/ engines were
employed to detect and mask personally identifiable information (PII) within
text data. Configured with the spaCy model
en_core_web_lg999https://spacy.io/models/en#en_core_web_lg for English
language text analysis, the Analyzer Engine identifies potential PII
instances, while the Anonymizer Engine masks the identified PII with
asterisks. This meticulous approach ensured thorough detection and
anonymization of any PII present in the input text, thereby safeguarding
individual privacy and ensuring compliance with data protection regulations.

#### Cleaning

The data cleaning stage addressed various undesirable elements, including XML
tags, HTML scripts, and non-ASCII characters. Python libraries,
lxml101010https://pypi.org/project/lxml/ and clean-
text111111https://pypi.org/project/clean-text/, were employed to effectively
remove these noise elements. However, certain noise types, such as tables,
lists, and non-English sentences, required manual intervention.

#### Chunking

The chunking process aimed to segment the cleaned data into contextually
relevant units suitable for synthetic question generation. This process
involved three key steps:

  * •

Denormalization of Structured Data: A subset of our data which was structured,
such as tabular or hierarchical data, was denormalized or flattened into plain
text, represented as key-value pairs, with the help of a custom python script.
The data was then split into chunks using Langchain
RecursiveTextSplitter121212https://python.langchain.com/docs/modules/data_connection/document_transformers/recursive_text_splitter/,
and the important key-value pairs were duplicated in each chunk to preserve
the contextual information.

  * •

Paragraph Separation: For unstructured data consisting of paragraphs,
contextually independent paragraphs were identified and separated from the
continuous text stream.

  * •

Sentence Chunking: The separated paragraphs were further segmented into
smaller chunks using the Langchain RecursiveTextSplitter with appropriate
delimiters. The HuggingFace tokenizer BertTokenizerFast was employed, with a
maximum chunk size of 512 tokens and an overlap of 40-70 tokens to preserve
contextual information across chunks.

This diverse data collection amounted to approximately 17 million tokens
distributed across 65,200 data chunks. The histogram of chunk sizes in our
dataset as shown in Figure 1 indicates that most chunks consisted of 300-500
tokens, with many smaller chunks containing less than 100 tokens.

Figure 1: Histogram of the chunk sizes showing the number of chunks with
different token counts.

###  2.3 Synthetic Question generation

The final stage involved feeding the cleaned and chunked dataset into a pre-
trained large language model (LLM),
Mixtral-8x7B-Instruct-v0.1131313https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1,
to generate a diverse range of questions (approximately 300,000). These
questions encompassed various types, including factual, clarification,
interpretation, consequence-related, extractive, subjective, and reasoning-
based, achieved using engineered prompts. The synthetic question generation
process was carried out in multiple iterations, where the questions generated
from each iteration were curated through rigorous review, and substandard
questions were discarded to ensure the overall quality of the generated
questions.

The following is an example prompt used for generating different types of
questions:

⬇

INSTRUCTIONS: Generate a variety of questions spanning different types,
including factual, clarification, interpretation, scenario-based, consequence-
related, extractive, subjective, reasoning-based, etc., which can be answered
using the information from the provided CONTEXT only without any prior
knowledge as demonstrated in the EXAMPLE.

CONTEXT: {text}

EXAMPLE:

Example Context: Finacle help banks engage better with their customers,
employees, and partners. We do this by helping banks design and deliver truly
personalized products and services. Built on a unique engagement hub, our
suite helps banks onboard, sell, service, and converse better with customers.
In fact, banks running on Finacle have realized an average 19% improvement in
their NPS scores.

Example Questions:

1. Factual: What is the primary focus of Finacle in helping banks?

2. Clarification: Can you elaborate on how Finacle helps banks engage with their customers, employees, and partners?

3. Scenario based: Suppose a bank wants to improve its customer engagement. How can Finacle help achieve this goal?

4. Consequence related: What benefits have banks experienced after implementing Finacle, particularly in terms of their NPS scores?

5. Extractive: What percentage improvement have banks seen in their NPS scores when using Finacle?

6. Reasoning based: What might be some reasons behind the effectiveness of Finacle in enhancing customer engagement for banks?

#### Alignment@Scale (Custom Entity Annotation)

Annotating domain-specific entities in the dataset, synthesizing questions
around these annotated entities, and subsequently training the model with
these questions facilitates enhanced ranking of answers by prioritizing
relevance to specific entities within the inquiry. The domain-specific nature
of the annotations contributes to the model’s acclimatization with Infosys
terminology and subtleties, thereby refining its capacity to grasp contextual
nuances within the domain-specific questions, ultimately enhancing the overall
performance of the question-answering system.

In pursuit of improving question generation effectiveness and refining the
query dataset, initial attempts were made to identify Infosys-specific
entities using state-of-the-art libraries like
spaCy141414https://spacy.io/api/entityrecognizer and
NLTK151515https://www.nltk.org/, as well as pre-trained entity recognition
models. However, these methods exhibited limitations in accurately recognizing
Infosys-specific entities. To address this, a novel approach was implemented,
combining automated pre-annotation with manual refinement.

A curated dictionary of around 500 organization-wide entities with their types
was developed, and a script was created to recognize these entities within the
dataset. The entire dataset was then transformed into multiple batches of 50
chunks each. Subsequently, every batch was independently annotated by three
annotators trained in manual entity recognition and labelling techniques with
a specialized course. The course, created in collaboration with Infosys
Education, Training, and Assessment (ETA) department, provided in-depth
explanations of entity recognition concepts alongside numerous illustrative
examples, equipping annotators with the skills necessary to produce high-
quality annotations. Following an automated review process, where only
entities mutually agreed by all three annotators in each batch, were
incorporated into the final annotated dataset. This process facilitated the
creation of a comprehensive high-quality annotated dataset optimized for
entity-specific synthetic question generation.

The Table 1 illustrates how the relevancy of the generated questions to the
domain improves when the prompt is engineered around the custom entities.

Table 1: Comparison of synthetic question generation prompts and the corresponding results, with and without custom entity annotations. |  Synthetic questions generated using a prompt without custom entity annotations |  Synthetic questions generated using a prompt with custom entity annotations
---|---|---
Prompt: |  INSTRUCTIONS: Generate 3 questions spanning different types, including factual, clarification, scenario-based, reasoning-based, etc., which can be answered using the information from the provided CONTEXT only without any prior knowledge.

CONTEXT: Infosys and Tennis Australia have joined hands to redefine the tennis experience by leveraging cloud, 3D virtual experiences, AI and mobility. A recent press release further highlights how Infosys is applying technology to take the game to those virtually who are unable to attend the matches physically. Learn more about the Virtual Hub - Powered by Infosys Meridian - which engages partners of the Australian Open and their guests in premium digital hospitality across the vast expanse of Melbourne Park. |  INSTRUCTIONS: Generate 3 questions spanning different types, including factual, clarification, scenario-based, reasoning-based, etc., which can be answered using the information from the provided CONTEXT only without any prior knowledge.
Please generate the questions around the entity ‘Infosys Meridian’ of type
Platform.
CONTEXT: Infosys and Tennis Australia have joined hands to redefine the tennis
experience by leveraging cloud, 3D virtual experiences, AI and mobility. A
recent press release further highlights how Infosys is applying technology to
take the game to those virtually who are unable to attend the matches
physically. Learn more about the Virtual Hub - Powered by Infosys Meridian -
which engages partners of the Australian Open and their guests in premium
digital hospitality across the vast expanse of Melbourne Park.
Generated Questions: |  1\. Factual: What companies have collaborated to enhance the tennis experience by utilizing cloud, 3D virtual experiences, AI, and mobility? 2\. Clarification: What is the purpose of the Virtual Hub - Powered by Infosys Meridian? 3\. Reasoning-based: How does Infosys’ partnership with Tennis Australia demonstrate the application of technology to overcome real-world limitations in sports events? |  1\. Factual: What is the name of the platform that Infosys is using to engage partners and their guests digitally at the Australian Open? 2\. Reasoning-based: How does the Infosys Meridian platform contribute to redefining the tennis experience at the Australian Open? 3\. Scenario-based: Imagine you are a guest of a partner of the Australian Open who cannot attend the event physically. How might the Infosys Meridian platform help you engage with the tournament?
|  |

###  2.4 Benchmark Evaluation Dataset

For benchmark evaluation, a dataset comprising 2500 diverse question-chunk
pairs derived from actual end-user queries was prepared. This dataset was
meticulously curated to ensure the highest quality, closely mirroring real-
world end-user inquiries. This dataset was used to assess and compare the
accuracy of the models.

##  3 Training

###  3.1 Dataset Preparation

The dataset was split into train and validation datasets at a 95:5 ratio for
each type of Infosys data.

###  3.2 Foundation Model Selection

For selecting the foundation models, we considered models that are currently
used as foundation models for existing Semantic Search solutions at Infosys.
Additionally, we included state-of-the-art models based on HuggingFace MTEB
Leaderboard161616https://huggingface.co/spaces/mteb/leaderboard, with a
preference for open-source models over closed models. Other factors considered
during this process included embedding model size, embedding dimensions,
context length, and performance on benchmark evaluation datasets. After the
selection process, the shortlisted foundation models were evaluated using our
benchmark evaluation dataset using BeIR171717https://github.com/beir-
cellar/beir library. Based on the evaluation results, we selected three
models: e5-large-v2181818https://huggingface.co/intfloat/e5-large-v2, A state-
of-the-art general purpose embedding model from E5 family (Wang at al., 2024
[7]). for retrieval, mxbai-rerank-
large-v1191919https://huggingface.co/mixedbread-ai/mxbai-rerank-large-v1 for
reranking, and colbertv2.0202020https://huggingface.co/colbert-ir/colbertv2.0,
The latest version of ColBERT (Khattab at al., 2020 [8]), a contextual late
interaction model. for both retrieval and reranking tasks, which consist of
335M, 435M and 110M parameters respectively.

###  3.3 Training Process

The selected models – e5-large-v2, mxbai-rerank-large-v1, and colbert2.0 –
were fine-tuned for retrieval and reranking tasks, optimizing their
performance for their respective objectives.

#### Bi-encoder model

Fine-tuning of the e5-large-v2 foundation model for retrieval tasks was done
using the Sentence Transformer212121https://sbert.net/ library.

Experiments encompassed various training data formats, including [query,
positive text], [query, positive text, negative text], and [query, positive
text, multiple negative texts] where the negative pairs were identified by
hard negative mining technique222222The hard negative mining technique
introduced by Wang at al., 2022 [9] in their work on Generative Pseudo
Labeling for Unsupervised Domain Adaptation of Dense Retrieval, was adopted
for hard-mining negative pairs in the training data. using msmarco-distilbert-
base-v3 and msmarco-MiniLM-L-6-v3 models. The [query, positive text] format,
which omitted the negative pairs, demonstrated optimal performance.

Multiple experiments with different combinations of hyperparameters were
conducted, with the MultipleNegativeRankingLoss function proving most
effective. Evaluation utilizing the InformationRetrievalEvaluator class
yielded superior results. Optimal hyperparameters included epoch = 3,
warmup_steps = 400, weight_decay = 0.01, scheduler = WarmupLinear, and
learning_rate = 1e-5.

Fine-tuning was carried out on a single NVIDIA A100 80GB GPU, spanning
approximately 8 hours.

#### Cross-encoder model

For reranking tasks, fine-tuning of the mxbai-rerank-large-v1 model was
conducted using the Sentence Transformer library.

Experiments explored various input data formats, including [query, positive
text], [query, positive text, negative text], and [query, positive text,
multiple negative texts] where the negative pairs were identified by hard
negative mining technique using msmarco-distilbert-
base-v3232323https://huggingface.co/sentence-transformers/msmarco-distilbert-
base-v3 and msmarco-MiniLM-L-6-v3242424https://huggingface.co/sentence-
transformers/msmarco-MiniLM-L-6-v3 models. The [query, positive text, negative
text] format exhibited superior accuracy during evaluation.

Optimal hyperparameters, including weight_decay = 0.01, scheduler =
WarmupLinear, epoch = 3, train_batch_size = 16, warmup_steps = 400, and
learning_rate = 1e-5, were determined utilizing CECorrelationEvaluator.

Fine-tuning was performed on a single NVIDIA A100 80GB GPU, lasting
approximately 12 hours.

#### Colbert model

Fine-tuning of the colbertv2.0 foundation model was accomplished via the
RAGatouille252525https://github.com/bclavie/RAGatouille library. This model
can serve as a standalone retrieval model, leveraging a ColBERT index without
requiring further reranking. Alternatively, it can also be used for reranking
the results obtained from other retrieval models.

Experiments explored various data formats, with the [query, positive text,
negative text] format outperforming others in terms of result accuracy.
Negative pairs were generated using the RAGatouille library with the help of
bge-small-en-v1.5262626https://huggingface.co/BAAI/bge-small-en-v1.5 model.

Optimal results were achieved utilizing default RAGatouille hyperparameters,
including train_batch_size = 32, learning_rate = 5e-6, and warmup_steps = 10%
of the total steps (9500 in our case).

Fine-tuning was executed on a single NVIDIA A100 80GB GPU, for about 6 hours.

##  4 Evaluation and Results

A thorough analysis was conducted, exploring different combinations of pre-
trained and fine-tuned models across diverse architectures. The evaluation was
performed using the BeIR library on our benchmark evaluation dataset, with
assessment metrics, including Normalized Discounted Cumulative Gain (NDCG),
Mean Average Precision (MAP), Precision, and Recall, computed at k_values of
@1 and @3. The aggregated scores of all these metrics provided a comprehensive
comparison of the overall model accuracy. Table 2 summarizes the key findings
and performance metrics obtained from this extensive evaluation.

Table 2: Summary of Model Configuration and Evaluation Results. Exp # |  Retrieval Model |  Reranking Model | Score
---|---|---|---
1 |  Pre-trained e5-large-v2 |  <none> | 68.7%
2 |  Pre-trained e5-large-v2 |  Pre-trained mxbai-rerank-large-v1 | 77.0%
3 |  Pre-trained e5-large-v2 |  Pre-trained colbertv2.0 | 76.9%
4 |  Pre-trained colbertv2.0 |  <none> | 80.6%
5 |  Fine-tuned e5-large-v2 |  <none> | 77.2%
6 |  Fine-tuned e5-large-v2 |  Pre-trained mxbai-rerank-large-v1 | 80.1%
7 |  Fine-tuned e5-large-v2 |  Pre-trained colbertv2.0 | 80.4%
8 |  Fine-tuned e5-large-v2 |  Fine-tuned mxbai-rerank-large-v1 | 81.2%
9 |  Fine-tuned e5-large-v2 |  Fine-tuned colbertv2.0 | 83.0%
10 |  Fine-tuned colbertv2.0 |  <none> | 83.8%
|  |  |

These findings highlight the versatility of ColBERT models in various stages
of the information retrieval pipeline and the efficacy of fine-tuning in
enhancing overall information retrieval accuracy. Experiments without re-
rankers using only the fine-tuned retrieval models exhibited comparable or
even superior performance to their pre-trained counterparts, even when
accompanied by pre-trained re-rankers. Overall, these evaluation results
reveal that fine-tuned bi-encoder retrieval models, particularly when paired
with fine-tuned re-rankers, consistently outperform equivalent pre-trained
models by a significant margin, making them best suited for enterprise
information retrieval tasks.

Besides the improvements in accuracy, an EnterpriseEM eliminates the need for
hosting and serving multiple embedding models and enables one embedding (such
as InfosysEM Retriever & InfosysEM Re-ranker) model to power all semantic
search applications within the enterprise.

##  5 Conclusion and Future Work

In conclusion, this paper has proposed a methodology aimed at enhancing the
performance of information retrieval solutions within enterprise environments
by fine-tuning pre-trained embedding models. By addressing the challenge of
efficiently managing proprietary unstructured data, our approach offers
promising prospects for improving the precision and relevance of search
results. Particularly when coupled with our techniques for synthetic question
generation based on annotated custom entities, our methodology demonstrates
its potential to adapt embeddings to better suit prevalent information
retrieval tasks in enterprises, thereby facilitating more efficient knowledge
discovery and decision-making processes.

Moving forward, future research in the field of fine-tuned embedding models
for information retrieval tasks could explore emerging techniques and
strategies such as semantic
chunking272727https://python.langchain.com/docs/modules/data_connection/document_transformers/semantic-
chunker/, which aims to improve the semantic search accuracy by considering
the semantic similarity of sentences within chunks. Additionally, exploring
techniques for multimedia data extraction presents an opportunity to maximize
the utilization of available enterprise data, enriching the training corpus
and potentially improving model performance in capturing diverse types of
information. Continuous integration and adaptation of state-of-the-art models
and methodologies will always remain crucial, ensuring embedding models evolve
to meet the evolving needs of enterprise information retrieval tasks. By
pursuing these avenues for future exploration, we aim to further enhance the
effectiveness and applicability of fine-tuned embedding models in enterprise
information retrieval tasks.

## References

  * [1] Patrick S.. Lewis et al.  “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks”  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020  URL: https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html
  * [2] Tianjun Zhang et al.  “RAFT: Adapting Language Model to Domain Specific RAG”  In _CoRR_ abs/2403.10131, 2024  DOI: 10.48550/ARXIV.2403.10131
  * [3] Jinhyuk Lee et al.  “BioBERT: a pre-trained biomedical language representation model for biomedical text mining”  In _CoRR_ abs/1901.08746, 2019  DOI: 10.48550/arxiv.1901.08746
  * [4] Iz Beltagy, Kyle Lo and Arman Cohan  “SciBERT: A Pretrained Language Model for Scientific Text”  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ Association for Computational Linguistics, 2019, pp. 3613–3618  DOI: 10.18653/V1/D19-1371
  * [5] Ilias Chalkidis et al.  “LEGAL-BERT: The Muppets straight out of Law School”  In _CoRR_ abs/2010.02559, 2020  DOI: 10.48550/arxiv.2010.02559
  * [6] Jianmo Ni et al.  “Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models”  In _Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022_ Association for Computational Linguistics, 2022, pp. 1864–1874  DOI: 10.18653/V1/2022.FINDINGS-ACL.146
  * [7] Liang Wang et al.  “Text Embeddings by Weakly-Supervised Contrastive Pre-training”  In _CoRR_ abs/2212.03533, 2022  DOI: 10.48550/ARXIV.2212.03533
  * [8] Omar Khattab and Matei Zaharia  “ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT”  In _CoRR_ abs/2004.12832, 2020  DOI: 10.48550/arxiv.2004.12832
  * [9] Kexin Wang, Nandan Thakur, Nils Reimers and Iryna Gurevych  “GPL: Generative Pseudo Labeling for Unsupervised Domain Adaptation of Dense Retrieval”  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ Association for Computational Linguistics, 2022, pp. 2345–2360  DOI: 10.18653/V1/2022.NAACL-MAIN.168