File size: 7,433 Bytes

---
library_name: transformers
license: bsd-3-clause
base_model:
- Qwen/Qwen2.5-0.5B-Instruct-GPTQ-Int4 
tags:
- Qwen
- Qwen2.5-0.5B-Instruct
- Qwen2.5-0.5B-Instruct-GPTQ-Int4 
- GPTQ
- Int4
---

# Qwen2.5-0.5B-Instruct-GPTQ-Int4 

This version of Qwen2.5-0.5B-Instruct-GPTQ-Int4 has been converted to run on the Axera NPU using **w4a16** quantization.

This model has been optimized with the following LoRA: 

Compatible with Pulsar2 version: 4.2(Not released yet)

## Convert tools links:

For those who are interested in model conversion, you can try to export axmodel through the original repo : https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GPTQ-Int4

[Pulsar2 Link, How to Convert LLM from Huggingface to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/appendix/build_llm.html) 

[AXera NPU LLM Runtime](https://github.com/AXERA-TECH/ax-llm) 

## Support Platform

- AX650
  - AX650N DEMO Board
  - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html)
  - [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html)
- AX630C
  - *developing*
 
|Chips|w8a16|w4a16|
|--|--|--|
|AX650| 28 tokens/sec|44 tokens/sec|

## How to use

Download all files from this repository to the device

```
root@ax650:/mnt/qtang/llm-test/qwen2.5-0.5b# tree -L 1
.
├── qwen2.5-0.5b-gptq-int4-ax650
├── qwen2.5_tokenizer
├── qwen2.5_tokenizer.py
├── main_axcl_aarch64
├── main_axcl_x86
├── main_prefill
├── post_config.json
├── run_qwen2.5_0.5b_gptq_int4_ax650.sh
├── run_qwen2.5_0.5b_gptq_int4_axcl_aarch64.sh
└── run_qwen2.5_0.5b_gptq_int4_axcl_x86.sh
```

#### Start the Tokenizer service

```
root@ax650:/mnt/qtang/llm-test/qwen2.5-0.5b# python3 qwen2.5_tokenizer.py --port 12345
None None 151645 <|im_end|>
<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
hello world<|im_end|>
<|im_start|>assistant

[151644, 8948, 198, 2610, 525, 1207, 16948, 11, 3465, 553, 54364, 14817, 13, 1446, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 14990, 1879, 151645, 198, 151644, 77091, 198]
http://localhost:12345
```

#### Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board

Open another terminal and run `run_qwen2.5_0.5b_gptq_int4_ax650.sh`

```
root@ax650:/mnt/qtang/llm-test/qwen2.5-0.5b# ./run_qwen2.5_0.5b_gptq_int4_ax650.sh
[I][                            Init][ 125]: LLM init start
bos_id: -1, eos_id: 151645
  3% | ██                                |   1 /  27 [0.00s<0.08s, 333.33 count/s] tokenizer init ok
[I][                            Init][  26]: LLaMaEmbedSelector use mmap
100% | ████████████████████████████████ |  27 /  27 [1.34s<1.34s, 20.10 count/s] init post axmodel ok,remain_cmm(3427 MB)
[I][                                       Init][ 241]: max_token_len : 1024
[I][                            Init][ 246]: kv_cache_size : 128, kv_cache_num: 1024
[I][                            Init][ 254]: prefill_token_num : 128
[I][                     load_config][ 281]: load config:
{
    "enable_repetition_penalty": false,
    "enable_temperature": true,
    "enable_top_k_sampling": true,
    "enable_top_p_sampling": false,
    "penalty_window": 20,
    "repetition_penalty": 1.2,
    "temperature": 0.9,
    "top_k": 10,
    "top_p": 0.8
}

[I][                            Init][ 268]: LLM init ok
Type "q" to exit, Ctrl+c to stop current running
>> who are you
[I][                             Run][ 466]: ttft: 134.66 ms
I am Qwen, a Qwen AI created by Alibaba Cloud. I am here to assist you with various topics and provide help to the best of my ability. I am here to help  
with any questions you have about science, technology, or any other topic you might have for help or guidance. I am always happy to help you!
[N][                             Run][ 605]: hit eos,avg 42.11 token/s

>> 1+1=?
[I][                             Run][ 466]: ttft: 135.07 ms
1+1=2
[N][                             Run][ 605]: hit eos,avg 43.04 token/s
```

#### Inference with M.2 Accelerator card

[What is M.2 Accelerator card?](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html), Show this DEMO based on Raspberry PI 5.

```
(base) axera@raspberrypi:~/samples/qwen2.5-0.5b $ ./run_qwen2.5_0.5b_gptq_int4_axcl_aarch64.sh
build time: Feb 13 2025 15:44:57
[I][                            Init][ 111]: LLM init start
bos_id: -1, eos_id: 151645
100% | ████████████████████████████████ |  27 /  27 [11.64s<11.64s, 2.32 count/s] init post axmodel okmain_cmm(6788 MB)
[I][                            Init][ 226]: max_token_len : 1024
[I][                            Init][ 231]: kv_cache_size : 128, kv_cache_num: 1024
[I][                     load_config][ 282]: load config:
{
    "enable_repetition_penalty": false,
    "enable_temperature": true,
    "enable_top_k_sampling": true,
    "enable_top_p_sampling": false,
    "penalty_window": 20,
    "repetition_penalty": 1.2,
    "temperature": 0.9,
    "top_k": 10,
    "top_p": 0.8
}

[I][                            Init][ 288]: LLM init ok
Type "q" to exit, Ctrl+c to stop current running
>> who are you
I am Qwen, a Qwen-like language model created by Alibaba Cloud. I am designed to assist users in answering questions, generating text,
and participating in conversations. I am here to help you with your questions and to engage in meaningful exchanges with you.
If you have any questions, you can ask me, and if you want, you can even write to me!
[N][                             Run][ 610]: hit eos,avg 25.88 token/s

>> 1+1=?
1+1=2
[N][                             Run][ 610]: hit eos,avg 29.73 token/s
>> q

(base) axera@raspberrypi:~/samples/qwen2.5-0.5b $ axcl-smi
+------------------------------------------------------------------------------------------------+
| AXCL-SMI  V2.26.0_20250205130139                                Driver  V2.26.0_20250205130139 |
+-----------------------------------------+--------------+---------------------------------------+
| Card  Name                     Firmware | Bus-Id       |                          Memory-Usage |
| Fan   Temp                Pwr:Usage/Cap | CPU      NPU |                             CMM-Usage |
|=========================================+==============+=======================================|
|    0  AX650N                    V2.26.0 | 0000:01:00.0 |                170 MiB /      945 MiB |
|   --   43C                      -- / -- | 2%        0% |                392 MiB /     7040 MiB |
+-----------------------------------------+--------------+---------------------------------------+

+------------------------------------------------------------------------------------------------+
| Processes:                                                                                     |
| Card      PID  Process Name                                                   NPU Memory Usage |
|================================================================================================|
|    0   474440  /home/axera/samples/qwen2.5-0.5b-gptq-int4/main_axcl_aarch64         370172 KiB |
+------------------------------------------------------------------------------------------------+
(base) axera@raspberrypi:~ $
```