langcache-embed-v3 / README.md

Aditeya Baral

Add new SentenceTransformer model

a87594a verified 2 months ago

16.7 kB

metadata

language:
  - en
license: apache-2.0
tags:
  - biencoder
  - sentence-transformers
  - text-classification
  - sentence-pair-classification
  - semantic-similarity
  - semantic-search
  - retrieval
  - reranking
  - generated_from_trainer
  - dataset_size:1056095
  - loss:SoftmaxLoss
base_model: Alibaba-NLP/gte-modernbert-base
widget:
  - source_sentence: >-
      In 2015 Adolf Hitler appeared in the kickstarter short movie `` Kung Fury
      `` as Taccone ( A.K.A .
    sentences:
      - >-
        In 2015 , Adolf Hitler appeared in the Kickstarter - short film `` Kung
        Fury `` as Taccone ( A.K.A .
      - >-
        In 1795 , the only white residents were Dr. John Laidley and two
        brothers with the surname Ainslie .
      - >-
        The 125th University Match was played in March 2014 at the Rye Golf Club
        , Oxford , East Sussex won the game 8.5 - 6.5 .
  - source_sentence: >-
      From 1973 to 1974 , Aubrey toured with the Cambridge Theatre Company as
      Diggory in `` She Stoops to Conquer `` and again as Aguecheek .
    sentences:
      - >-
        Oxide can be reduced to metallic samarium at higher temperatures by
        heating with a reducing agent such as hydrogen or carbon monoxide .
      - >-
        From 1973 to 1974 Aguecheek toured with the Cambridge Theatre Company as
        Diggory in `` You Stoops to Conquer `` and again as Aubrey .
      - >-
        The medals were presented by Barry Maister , IOC member , New Zealand
        and Sarah Webb Gosling , Vice President of World Sailing .
  - source_sentence: >-
      There is no official wall on the border , although there are sections of
      fence near populated areas and continuous border crossings .
    sentences:
      - >-
        The 2014 -- 15 Boston Bruins season was the 91st season for the National
        Hockey League franchise that was established on November 1 , 1924 .
      - He was trained by the Inghams and owned by John Hawkes .
      - >-
        There is no continuous wall on the border , although there are fence
        sections near populated areas and official border crossings .
  - source_sentence: >-
      Capital . `` The French established similar hill stations in Indochina ,
      such as Dalat built in 1921 .
    sentences:
      - >-
        Lubuk China is a small town in Alor Gajah District , Melaka , Malaysia .
        It is situated near the border with Negeri Sembilan .
      - >-
        The French established similar hill stations in Indochina , such as
        Dalat , built in 1921 .
      - >-
        John Potts ( or Pott ) was a doctor and colonial governor of Virginia in
        the Jamestown settlement at Virginia Colony in the early 17th century .
  - source_sentence: >-
      The band pursued `` signals `` in January 2012 in three weeks , and drums
      were recorded in a day and a half .
    sentences:
      - >-
        It was repaired at the beginning of the 20th century and is listed as
        closed in our records .
      - >-
        The band tracked `` Signals `` in three weeks in January 2012 . Drums
        were recorded in a day and a half .
      - >-
        Contributors include actor Anton LaVey , Satanist Christopher Lee ,
        serial killer expert Clive Barker , author Karen Greenlee , and
        necrophile Robert Ressler .
datasets:
  - aditeyabaral-redis/langcache-sentencepairs
pipeline_tag: sentence-similarity
library_name: sentence-transformers
metrics:
  - cosine_accuracy
  - cosine_accuracy_threshold
  - cosine_f1
  - cosine_f1_threshold
  - cosine_precision
  - cosine_recall
  - cosine_ap
  - cosine_mcc
model-index:
  - name: Redis fine-tuned BiEncoder model for semantic caching on LangCache
    results:
      - task:
          type: binary-classification
          name: Binary Classification
        dataset:
          name: val
          type: val
        metrics:
          - type: cosine_accuracy
            value: 0.7629982153480072
            name: Cosine Accuracy
          - type: cosine_accuracy_threshold
            value: 0.8640064001083374
            name: Cosine Accuracy Threshold
          - type: cosine_f1
            value: 0.6907391673746814
            name: Cosine F1
          - type: cosine_f1_threshold
            value: 0.8261547684669495
            name: Cosine F1 Threshold
          - type: cosine_precision
            value: 0.6290946608202218
            name: Cosine Precision
          - type: cosine_recall
            value: 0.7657770800627943
            name: Cosine Recall
          - type: cosine_ap
            value: 0.7350929175906749
            name: Cosine Ap
          - type: cosine_mcc
            value: 0.47714361581572273
            name: Cosine Mcc
      - task:
          type: binary-classification
          name: Binary Classification
        dataset:
          name: test
          type: test
        metrics:
          - type: cosine_accuracy
            value: 0.7035036519888425
            name: Cosine Accuracy
          - type: cosine_accuracy_threshold
            value: 0.8520700931549072
            name: Cosine Accuracy Threshold
          - type: cosine_f1
            value: 0.7118460123901542
            name: Cosine F1
          - type: cosine_f1_threshold
            value: 0.8109649419784546
            name: Cosine F1 Threshold
          - type: cosine_precision
            value: 0.5979034259235814
            name: Cosine Precision
          - type: cosine_recall
            value: 0.8794413407821229
            name: Cosine Recall
          - type: cosine_ap
            value: 0.6473553527394227
            name: Cosine Ap
          - type: cosine_mcc
            value: 0.4408784752892243
            name: Cosine Mcc

Redis fine-tuned BiEncoder model for semantic caching on LangCache

This is a sentence-transformers model finetuned from Alibaba-NLP/gte-modernbert-base on the LangCache Sentence Pairs (all) dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for sentence pair similarity.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: Alibaba-NLP/gte-modernbert-base
Maximum Sequence Length: 8192 tokens
Output Dimensionality: 768 dimensions
Similarity Function: Cosine Similarity
Training Dataset:
- LangCache Sentence Pairs (all)
Language: en
License: apache-2.0

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("aditeyabaral-redis/langcache-embed-v3")
# Run inference
sentences = [
    'The band pursued `` signals `` in January 2012 in three weeks , and drums were recorded in a day and a half .',
    'The band tracked `` Signals `` in three weeks in January 2012 . Drums were recorded in a day and a half .',
    'Contributors include actor Anton LaVey , Satanist Christopher Lee , serial killer expert Clive Barker , author Karen Greenlee , and necrophile Robert Ressler .',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.9599, 0.4944],
#         [0.9599, 1.0000, 0.5097],
#         [0.4944, 0.5097, 1.0000]])

Evaluation

Metrics

Binary Classification

Datasets: val and test
Evaluated with BinaryClassificationEvaluator

Metric	val	test
cosine_accuracy	0.763	0.7035
cosine_accuracy_threshold	0.864	0.8521
cosine_f1	0.6907	0.7118
cosine_f1_threshold	0.8262	0.811
cosine_precision	0.6291	0.5979
cosine_recall	0.7658	0.8794
cosine_ap	0.7351	0.6474
cosine_mcc	0.4771	0.4409

Training Details

Training Dataset

LangCache Sentence Pairs (all)

Dataset: LangCache Sentence Pairs (all)
Size: 62,021 training samples
Columns: sentence1, sentence2, and label
Approximate statistics based on the first 1000 samples:
sentence1 sentence2 label
type string string int
details
min: 8 tokens
mean: 27.46 tokens
max: 53 tokens

min: 9 tokens
mean: 27.36 tokens
max: 52 tokens

0: ~50.30%
1: ~49.70%

	sentence1	sentence2	label
type	string	string	int
details	min: 8 tokens mean: 27.46 tokens max: 53 tokens	min: 9 tokens mean: 27.36 tokens max: 52 tokens	0: ~50.30% 1: ~49.70%

Samples:

sentence1	sentence2	label
`The newer Punts are still very much in existence today and race in the same fleets as the older boats .`	`The newer punts are still very much in existence today and run in the same fleets as the older boats .`	`1`
`Turner Valley , was at the Turner Valley Bar N Ranch Airport , southwest of the Turner Valley Bar N Ranch , Alberta , Canada .`	`Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada .`	`0`
`After losing his second election , he resigned as opposition leader and was replaced by Geoff Pearsall .`	`Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall .`	`1`

Loss: SoftmaxLoss

Evaluation Dataset

LangCache Sentence Pairs (all)

Dataset: LangCache Sentence Pairs (all)
Size: 62,021 evaluation samples
Columns: sentence1, sentence2, and label
Approximate statistics based on the first 1000 samples:
sentence1 sentence2 label
type string string int
details
min: 8 tokens
mean: 27.46 tokens
max: 53 tokens

min: 9 tokens
mean: 27.36 tokens
max: 52 tokens

0: ~50.30%
1: ~49.70%

	sentence1	sentence2	label
type	string	string	int
details	min: 8 tokens mean: 27.46 tokens max: 53 tokens	min: 9 tokens mean: 27.36 tokens max: 52 tokens	0: ~50.30% 1: ~49.70%

Samples:

sentence1	sentence2	label
`The newer Punts are still very much in existence today and race in the same fleets as the older boats .`	`The newer punts are still very much in existence today and run in the same fleets as the older boats .`	`1`
`Turner Valley , was at the Turner Valley Bar N Ranch Airport , southwest of the Turner Valley Bar N Ranch , Alberta , Canada .`	`Turner Valley Bar N Ranch Airport , , was located at Turner Valley Bar N Ranch , southwest of Turner Valley , Alberta , Canada .`	`0`
`After losing his second election , he resigned as opposition leader and was replaced by Geoff Pearsall .`	`Max Bingham resigned as opposition leader after losing his second election , and was replaced by Geoff Pearsall .`	`1`

Loss: SoftmaxLoss

Training Logs

Epoch	Step	val_cosine_ap	test_cosine_ap
-1	-1	0.7351	0.6474

Framework Versions

Python: 3.12.3
Sentence Transformers: 5.1.0
Transformers: 4.55.0
PyTorch: 2.8.0+cu128
Accelerate: 1.10.0
Datasets: 4.0.0
Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers and SoftmaxLoss

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}