File size: 6,708 Bytes
da02d00 767f673 4a26b36 6ea84e2 813718f 4a26b36 767f673 4a26b36 6ea84e2 813718f 4a26b36 813718f 767f673 4a26b36 813718f da02d00 f287dd4 d9e70e8 f287dd4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 |
---
library_name: transformers
tags: []
---
## Model Details
### Model Description
Dataset: [GreenNode/GreenNode-Table-Markdown-Retrieval](https://huggingface.co/datasets/GreenNode/GreenNode-Table-Markdown-Retrieval-VN)
| Model Name | MAP@5 ↑ | MRR@5 ↑ | NDCG@5 ↑ | Recall@5 ↑ | Mean ↑ |
|----------------------------------------|---------|---------|----------|------------|--------|
| **Multilingual Embedding models** | | | | | |
| me5_small | 33.75 | 33.75 | 35.68 | 41.49 | 36.17 |
| me5_large | 38.16 | 38.16 | 40.27 | 46.62 | 40.80 |
| M3-Embedding | 36.52 | 36.52 | 38.60 | 44.84 | 39.12 |
| OpenAI-embedding-v3 | 30.61 | 30.61 | 32.57 | 38.46 | 33.06 |
| **Vietnamese Embedding models (Prior Work)** | | | | | |
| halong-embedding | 32.15 | 32.15 | 34.13 | 40.09 | 34.63 |
| sup-SimCSE-VietNamese-phobert_base | 10.90 | 10.90 | 12.03 | 15.41 | 12.31 |
| vietnamese-bi-encoder | 13.61 | 13.61 | 14.63 | 17.68 | 14.89 |
| **GreenNode-Embedding** | | | | | |
| M3-GN-VN | 41.85 | 41.85 | 44.15 | 57.05 | 46.23|
| M3-GN-VN-Mixed | 42.08 | 42.08 | 44.33 | 51.06 | 44.89 |
| **Ours – Multi-vector embedding** | | | | | |
| Vintern-Embedding-1B | 57.01 | 57.01 | 59.17 | 65.65 | 59.71 |
Dataset: [GreenNode/zalo-ai-legal-text-retrieval-vn](https://huggingface.co/datasets/GreenNode/zalo-ai-legal-text-retrieval-vn)
| Model Name | MAP@5 ↑ | MRR@5 ↑ | NDCG@5 ↑ | Recall@5 ↑ | Mean ↑ |
|----------------------------------------|---------|---------|----------|------------|--------|
| **Multilingual Embedding models** | | | | | |
| me5_small | 54.68 | 54.37 | 58.32 | 69.16 | 59.13 |
| me5_large | 60.14 | 59.62 | 64.17 | 76.02 | 64.99 |
| M3-Embedding | 69.34 | 68.96 | 73.70 | 86.68 | 74.67 |
| OpenAI-embedding-v3 | 38.68 | 38.80 | 41.53 | 49.94 | 41.74 |
| **Vietnamese Embedding models (Prior Work)** | | | | | |
| halong-embedding | 52.57 | 52.28 | 56.64 | 68.72 | 57.55 |
| sup-SimCSE-VietNamese-phobert_base | 25.15 | 25.07 | 27.81 | 35.79 | 28.46 |
| vietnamese-bi-encoder | 54.88 | 54.47 | 59.10 | 79.51 | 61.99 |
| **GreenNode-Embedding** | | | | | |
| M3-GN-VN | 65.03 | 64.80 | 69.19 | 81.66 | 70.17 |
| M3-GN-VN-Mixed | 69.75 | 69.28 | 74.01 | 86.74 | 74.95 |
| **Ours – Multi-vector embedding** | | | | | |
| Vintern-Embedding-1B | 68.90 | 69.06 | 72.32 | 82.29 | 73.14 |
Dataset: [ViDoRe Benchmark](https://huggingface.co/collections/vidore/vidore-benchmark-667173f98e70a1c0fa4db00d)
| Model | Model_Size | Average_Score | ArxivQA | DocVQA | InfoVQA | Artificial Intelligence | Energy | Government | Healthcare Industry | TAT-DQA |
|-----------------------------------------------|------------|---------------|---------|--------|---------|-------------------------|--------|------------|----------------------|---------|
| royokong/e5-v | 8.3B | 62.88 | 48.3 | 34.7 | 69.2 | 78.9 | 78.1 | 82.2 | 82.3 | 29.3 |
| TIGER-Lab/VLM2Vec-Full | 4.2B | 51.16 | 42.8 | 26.7 | 66.7 | 53.5 | 63.5 | 64 | 70.7 | 21.4 |
| nvidia/llama-nemoretriever-colembed-3b-v1 | 4.4B | 90.42 | 88.4 | 66.2 | 94.9 | 99.6 | 96.6 | 97.8 | 99.3 | 80.6 |
| nvidia/llama-nemoretriever-colembed-1b-v1 | 2.4B | 89.8 | 87.6 | 64.5 | 93.6 | 100 | 96.6 | 96.7 | 99.6 | 79.8 |
| jinaai/jina-embeddings-v4 | 3.8B | 89.38 | 88.5 | 60.1 | 93.8 | 99.3 | 97.3 | 96.6 | 99.1 | 80.3 |
| nomic-ai/colnomic-embed-multimodal-3b | 3B | 89.25 | 88.1 | 61.3 | 92.8 | 96.3 | 97.4 | 96.6 | 98.3 | 83.2 |
| nomic-ai/colnomic-embed-multimodal-7b | 7B | 89.00 | 88.3 | 60.1 | 92.2 | 98.8 | 96.3 | 95.9 | 99.3 | 81.1 |
| vidore/colqwen2.5-v0.2 | 3B | 89.58 | 88.9 | 63.6 | 92.5 | 99.6 | 96.1 | 95.8 | 98 | 82.1 |
| vidore/colqwen2-v1.0 | 2.2B | 89.18 | 88 | 61.5 | 92.5 | 99 | 95.9 | 95.5 | 98.8 | 82.2 |
| ibm-granite/granite-vision-3.3-2b-embedding | 3B | 85.98 | 84.2 | 54.6 | 89.7 | 98.9 | 96.3 | 97.3 | 98.9 | 67.9 |
| vidore/colpali-v1.3 | 3B | 85.44 | 83.3 | 58.4 | 85.5 | 97.4 | 94.6 | 96.1 | 97.4 | 70.8 |
| vidore/colpali-v1.2 | 3B | 83.16 | 77.8 | 56.6 | 82.2 | 97.5 | 93.8 | 94.4 | 94.9 | 68.1 |
| ColVintern-1B | 0.9B | 78.8 | 71.6 | 48.3 | 84.6 | 92.9 | 88.7 | 89.4 | 95.2 | 59.6 |
| Vintern-Embedding-1B | 0.9B | 82.85 | 75.37 | 51.79 | 86.2 | 97.52 | 93.19 | 93.97 | 97.09 | 67.72 |
|