File size: 3,182 Bytes
87ce6da b88a047 87ce6da b88a047 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 |
---
base_model: Qwen/Qwen2.5-14B-Instruct-1M
---
This is a quantization of the [Qwen2.5-14B-Instruct-1M](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct-1M).
Qwen2.5-14B-Instruct-1M, developed by Alibaba Cloud, is a standout model in the world of large language models due to its exceptional capability to handle ultra-long contexts, supporting up to 1 million tokens. This feature makes it significantly more effective for long-context tasks compared to previous versions, while still maintaining strong performance on shorter tasks. With an architecture incorporating advanced techniques like RoPE, SwiGLU, and RMSNorm, Qwen2.5-1M offers a balanced blend of sophistication and efficiency. It is designed as a causal language model and demonstrates considerable prowess in generating coherent and contextually aware text, marking a substantial advancement in handling complex language generation tasks.
## Evaluations
This model provides an accuracy recovery of 100.09%.
| __English__ | __[Qwen2.5-14B-Instruct-1M](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct-1M)__ | __[Qwen2.5-14B-Instruct-1M-FP8-Dynamic (this)](https://huggingface.co/cortecs/Qwen2.5-14B-Instruct-1M-FP8-Dynamic)__ |
|:--------------|-------------------------------------------------------------------------------------:|-----------------------------------------------------------------------------------------------------------------------:|
| Avg. | 74.79 | 74.86 |
| ARC | 70 | 70.3 |
| Hellaswag | 74.6 | 74.5 |
| MMLU | 79.77 | 79.77 |
We did not check for data contamination.
Evaluation was done using [Eval. Harness](https://github.com/EleutherAI/lm-evaluation-harness) with `limit=1000`.
## Usage
Install **vLLM** and
run the [server](https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#openai-compatible-server):
```
python -m vllm.entrypoints.openai.api_server --model cortecs/Qwen2.5-14B-Instruct-1M-FP8-Dynamic --max-model-len 42000 --gpu-memory-utilization 0.95
```
Access the model:
```
curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d ' {
"model": "cortecs/Qwen2.5-14B-Instruct-1M-FP8-Dynamic",
"prompt": "San Francisco is a"
} '
```
⚡ This model is optimized to handle heavy workloads providing a total throughput of ️**4497 tokens per second** using one NVIDIA L40S ⚡ |