Update README.md

README.md

@@ -1,3 +1,160 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+datasets:
+- lmms-lab/LLaVA-One-Vision-1.5-Insturct-26M
+- lmms-lab/LLaVA-One-Vision-1.5-Mid-Training-85M
+base_model:
+- Qwen/Qwen3-8B-Base
+- DeepGlint-AI/rice-vit-large-patch14-560
+---
+# LLaVA-OneVision-1.5: Fully Open-Source State-of-the-Art VLM Model
+
+# ✨ Key Features
+
+**LLaVA-OneVision-1.5** introduces a novel family of **fully open-source** Large Multimodal Models (LMMs) that achieves **state-of-the-art performance**  with substantially **lower cost** through training on **native resolution** images.
+
+1. **Superior Performance**
+A family of fully open-source large multimodal models demonstrating **superior performance** across multiple multimodal benchmarks, **outperforming Qwen2.5-VL** in most evaluation tasks.
+
+2. **High-Quality Data at Scale**
+Meticulously curated **pre-training and SFT data** with rigorous filtering and quality control, achieving **superior data efficiency** with only **5B tokens** (1.2% of Qwen2.5-VL's training data).
+- Concept-balanced, highly diverse, high-quality caption data
+- Comprehensive instruction fine-tuning data covering a wide range of tasks
+
+3. **Ultra-Efficient Training Framework**
+Complete end-to-end training framework designed for maximum efficiency:
+- **$16K total budget** for full model training
+- **45% HFU efficiency** on A100 GPUs ($0.6 per GPU/Hour)
+- Built on **MegatronLM** with support for **MoE**, **FP8**, and **long sequence parallelization**
+- Optimized codebase for cost-effective scaling
+
+4. **Fully Open Framework** for community access and reproducibility:
+- ✅ High-quality pre-training & SFT data
+- ✅ Complete training framework & code
+- ✅ Training recipes & configurations
+- ✅ Base & instruct model checkpoints
+- ✅ Comprehensive training logs & metrics
+
+## Dataset
+| Description | Link |
+|-------------|------|
+| Pretrain data for LLaVA-OneVision-1.5 | [🤗 Download (Uploading!)](https://huggingface.co/datasets/lmms-lab/LLaVA-One-Vision-1.5-Mid-Training-85M) |
+| SFT data for LLaVA-OneVision-1.5 | [🤗 Download (Uploading!)](https://huggingface.co/datasets/lmms-lab/LLaVA-One-Vision-1.5-Insturct-26M) |
+
+## Evaluation Results
+All evaluations were conducted using lmms_eval.
+
+|                                  | **LLaVA-OV-1.5-8B** | **Qwen2.5 VL 7B** | **LLaVA-OV-1.5-4B** | **Qwen2.5 VL 3B** |
+|:----------------------------------|:---------------:|:-------------:|:---------------:|:-------------:|
+| MMMU (Validation)                 |    **55.44**    |     51.33     |    **51.44**    |     46.44     |
+| MMMU-Pro (Standard)               |    **37.40**    |     36.30     |    **33.24**    |     31.10     |
+| MMMU-Pro (Vision)                 |      25.15      |   **32.83**   |    **23.53**    |     21.27     |
+| MMBench (English; Test)           |    **84.14**    |     83.40     |    **82.29**    |     77.97     |
+| MMBench (Chinese; Test)           |      81.00      |   **81.61**   |    **76.73**    |     74.55     |
+| MME-RealWorld (English)           |    **62.31**    |     57.33     |    **57.16**    |     51.60     |
+| MME-RealWorld (Chinese)           |    **56.11**    |     51.50     |      21.38      |   **45.38**   |
+| AI2D (With Mask)                  |    **84.16**    |     82.58     |    **84.62**    |     78.56     |
+| AI2D (Without Mask)               |    **94.11**    |     93.36     |    **92.84**    |     90.74     |
+| CV-Bench                          |    **80.82**    |     79.95     |    **74.00**    |     71.53     |
+| VL-RewardBench                    |      45.90      |   **49.65**   |    **45.90**    |     42.06     |
+| V*                                |    **78.01**    |     76.96     |      66.49      |   **69.63**   |
+| PixmoCount                        |      62.19      |   **63.33**   |    **59.17**    |     50.85     |
+| CountBench                        |    **88.19**    |     86.35     |    **77.80**    |     72.51     |
+| ChartQA                           |    **86.48**    |     84.08     |    **85.11**    |     83.36     |
+| CharXiv (Direct Questions)        |    **74.10**    |     69.80     |    **70.70**    |     58.20     |
+| DocVQA (Test)                     |    **95.00**    |     94.93     |    **93.48**    |     92.67     |
+| InfoVQA (Test)                    |      78.42      |   **81.67**   |    **75.27**    |     75.63     |
+| WeMath                            |    **33.62**    |     33.33     |    **28.00**    |     18.38     |
+| MathVista (Mini)                  |    **69.57**    |     68.60     |    **67.36**    |     60.23     |
+| MathVision                        |    **25.56**    |     22.37     |    **22.76**    |     21.25     |
+| MMStar                            |    **67.72**    |     62.54     |    **64.22**    |     55.86     |
+| SEED-Bench (Image)                |      77.32      |   **77.53**   |    **76.74**    |     74.81     |
+| ScienceQA                         |    **94.98**    |     88.75     |    **92.05**    |     83.33     |
+| SEED-Bench 2-Plus                 |      69.21      |   **70.93**   |    **68.42**    |     68.64     |
+| OCRBench                          |      82.90      |   **84.20**   |      77.80      |   **79.20**   |
+| RealWorldQA                       |      68.10      |   **68.50**   |    **64.05**    |     60.00     |
+
+### Using 🤗  Transformers to Chat
+
+Here we show a code snippet to show you how to use the chat model with `transformers` and `qwen_vl_utils`:
+
+```python
+from transformers import AutoTokenizer, AutoProcessor, AutoModelForCausalLM
+from qwen_vl_utils import process_vision_info
+model_path = "Deep-VLM/LLaVA-One-Vision-1.5-4B-instruct"
+
+# default: Load the model on the available device(s)
+model = AutoModelForCausalLM.from_pretrained(
+    model_path, torch_dtype="auto", device_map="auto", trust_remote_code=True
+)
+
+# default processer
+processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
+
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {
+                "type": "image",
+                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
+            },
+            {"type": "text", "text": "Describe this image."},
+        ],
+    }
+]
+
+# Preparation for inference
+text = processor.apply_chat_template(
+    messages, tokenize=False, add_generation_prompt=True
+)
+image_inputs, video_inputs = process_vision_info(messages)
+inputs = processor(
+    text=[text],
+    images=image_inputs,
+    videos=video_inputs,
+    padding=True,
+    return_tensors="pt",
+)
+inputs = inputs.to("cuda")
+
+# Inference: Generation of the output
+generated_ids = model.generate(**inputs, max_new_tokens=1024)
+generated_ids_trimmed = [
+    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
+]
+output_text = processor.batch_decode(
+    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
+)
+print(output_text)
+```
+
+## Citation
+
+If you find *LLaVA-OneVision-1.5* useful in your research, please consider to cite the following related papers:
+
+```
+@inproceedings{LLaVA-OneVision-1.5,
+  title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},
+  author={},
+  booktitle={arxiv},  
+  year={2025}
+ }
+
+@inproceedings{xie2025region,
+  title={Region-based Cluster Discrimination for Visual Representation Learning},
+  author={Xie, Yin and Yang, Kaicheng and An, Xiang and Wu, Kun and Zhao, Yongle and Deng, Weimo and Ran, Zimin and Wang, Yumeng and Feng, Ziyong and Miles, Roy and Elezi, Ismail and Deng, Jiankang},
+  booktitle={ICCV},
+  year={2025}
+}
+
+@article{lillava,
+  title={LLaVA-OneVision: Easy Visual Task Transfer},
+  author={Li, Bo and Zhang, Yuanhan and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Hao and Zhang, Kaichen and Zhang, Peiyuan and Li, Yanwei and Liu, Ziwei and Li, Chunyuan},
+  journal={Transactions on Machine Learning Research}
+  year={2024}
+}
+```
+
+
+