|
|
--- |
|
|
tags: |
|
|
- deepsparse |
|
|
--- |
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from deepsparse import TextGeneration |
|
|
|
|
|
prompt = "How to get in a good university?" |
|
|
formatted_prompt = f"<s> [|User|]\n{prompt}</s>[|Assistant|]\n" |
|
|
|
|
|
model = TextGeneration(model="deployment") |
|
|
|
|
|
print(model(formatted_prompt, max_new_tokens=500).generations[0].text) |
|
|
""" |
|
|
Hi there! I'm an AI assistant and I can help you with any questions you have. Here's how you can get into a good university: |
|
|
1. Choose a university that you are interested in. |
|
|
2. Read the university's website and learn more about their programs and courses. |
|
|
3. Reach out to the university's alumni or staff members and ask for advice or guidance. |
|
|
4. Attend a campus tour or visit day to get a better understanding of the university's culture and atmosphere. |
|
|
5. Finally, don't wait to apply for the university of your choice. |
|
|
I hope this helps you get into a good university! Let me know if you have any other questions or need any more information. |
|
|
""" |
|
|
``` |
|
|
## One-shot and Export |
|
|
|
|
|
```bash |
|
|
git clone https://github.com/neuralmagic/sparseml |
|
|
pip install -e "sparseml[transformers]" |
|
|
python sparseml/src/sparseml/transformers/sparsification/obcq/obcq.py TinyLlama/TinyLlama-1.1B-Chat-v0.4 open_platypus --recipe recipe.yaml --save True |
|
|
python sparseml/src/sparseml/transformers/sparsification/obcq/export.py --task text-generation --model_path obcq_deployment |
|
|
cp deployment/model.onnx deployment/model-orig.onnx |
|
|
python onnx_kv_inject.py --input-file deployment/model-orig.onnx --output-file deployment/model.onnx |
|
|
``` |
