|
|
--- |
|
|
license: apache-2.0 |
|
|
base_model: mistralai/Mistral-7B-v0.1 |
|
|
language: |
|
|
- en |
|
|
tags: |
|
|
- mistral |
|
|
- onnxruntime |
|
|
- onnx |
|
|
- llm |
|
|
--- |
|
|
|
|
|
# Mistral-7b for ONNX Runtime |
|
|
|
|
|
## Introduction |
|
|
|
|
|
This repository hosts the optimized versions of **Mistral-7B-v0.1** to accelerate inference with ONNX Runtime CUDA execution provider. |
|
|
|
|
|
See the [usage instructions](#usage-example) for how to inference this model with the ONNX files hosted in this repository. |
|
|
|
|
|
## Model Description |
|
|
|
|
|
- **Developed by:** MistralAI |
|
|
- **Model type:** Pretrained generative text model |
|
|
- **License:** Apache 2.0 License |
|
|
- **Model Description:** This is a conversion of the [Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) for [ONNX Runtime](https://github.com/microsoft/onnxruntime) inference with CUDA execution provider. |
|
|
|
|
|
|
|
|
## Performance Comparison |
|
|
|
|
|
#### Latency for token generation |
|
|
|
|
|
Below is average latency of generating a token using a prompt of varying size using NVIDIA A100-SXM4-80GB GPU, taken from the [ORT benchmarking script for Mistral](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/models/llama/README.md#benchmark-mistral) |
|
|
|
|
|
| Prompt Length | Batch Size | PyTorch 2.1 torch.compile | ONNX Runtime CUDA | |
|
|
|-------------|------------|----------------|-------------------| |
|
|
| 32 | 1 | 32.58ms | 12.08ms | |
|
|
| 256 | 1 | 54.54ms | 23.20ms | |
|
|
| 1024 | 1 | 100.6ms | 77.49ms | |
|
|
| 2048 | 1 | 236.8ms | 144.99ms | |
|
|
| 32 | 4 | 63.71ms | 15.32ms | |
|
|
| 256 | 4 | 86.74ms | 75.94ms | |
|
|
| 1024 | 4 | 380.2ms | 273.9ms | |
|
|
| 2048 | 4 | N/A | 554.5ms | |
|
|
|
|
|
## Usage Example |
|
|
|
|
|
Following the [benchmarking instructions](https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/python/tools/transformers/models/llama/README.md#mistral). Example steps: |
|
|
|
|
|
1. Clone onnxruntime repository. |
|
|
```shell |
|
|
git clone https://github.com/microsoft/onnxruntime |
|
|
cd onnxruntime |
|
|
``` |
|
|
|
|
|
2. Install required dependencies |
|
|
```shell |
|
|
python3 -m pip install -r onnxruntime/python/tools/transformers/models/llama/requirements-cuda.txt |
|
|
``` |
|
|
|
|
|
5. Inference using manual model API, or use Hugging Face's ORTModelForCausalLM |
|
|
```python |
|
|
from optimum.onnxruntime import ORTModelForCausalLM |
|
|
from onnxruntime import InferenceSession |
|
|
from transformers import AutoConfig, AutoTokenizer |
|
|
|
|
|
sess = InferenceSession("Mistral-7B-v0.1.onnx", providers = ["CUDAExecutionProvider"]) |
|
|
config = AutoConfig.from_pretrained("mistralai/Mistral-7B-v0.1") |
|
|
|
|
|
model = ORTModelForCausalLM(sess, config, use_cache = True, use_io_binding = True) |
|
|
|
|
|
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1") |
|
|
|
|
|
inputs = tokenizer("Instruct: What is a fermi paradox?\nOutput:", return_tensors="pt") |
|
|
|
|
|
outputs = model.generate(**inputs) |
|
|
|
|
|
print(tokenizer.decode(outputs[0], skip_special_tokens=True)) |
|
|
``` |
|
|
|
