--- library_name: transformers license: bsd-3-clause base_model: - Qwen/Qwen2.5-0.5B-Instruct-GPTQ-Int4 tags: - Qwen - Qwen2.5-0.5B-Instruct - Qwen2.5-0.5B-Instruct-GPTQ-Int4 - GPTQ - Int4 --- # Qwen2.5-0.5B-Instruct-GPTQ-Int4 This version of Qwen2.5-0.5B-Instruct-GPTQ-Int4 has been converted to run on the Axera NPU using **w4a16** quantization. This model has been optimized with the following LoRA: Compatible with Pulsar2 version: 4.2(Not released yet) ## Convert tools links: For those who are interested in model conversion, you can try to export axmodel through the original repo : https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GPTQ-Int4 [Pulsar2 Link, How to Convert LLM from Huggingface to axmodel](https://pulsar2-docs.readthedocs.io/en/latest/appendix/build_llm.html) [AXera NPU LLM Runtime](https://github.com/AXERA-TECH/ax-llm) ## Support Platform - AX650 - AX650N DEMO Board - [M4N-Dock(爱芯派Pro)](https://wiki.sipeed.com/hardware/zh/maixIV/m4ndock/m4ndock.html) - [M.2 Accelerator card](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html) - AX630C - *developing* |Chips|w8a16|w4a16| |--|--|--| |AX650| 28 tokens/sec|44 tokens/sec| ## How to use Download all files from this repository to the device ``` root@ax650:/mnt/qtang/llm-test/qwen2.5-0.5b# tree -L 1 . ├── qwen2.5-0.5b-gptq-int4-ax650 ├── qwen2.5_tokenizer ├── qwen2.5_tokenizer.py ├── main_axcl_aarch64 ├── main_axcl_x86 ├── main_prefill ├── post_config.json ├── run_qwen2.5_0.5b_gptq_int4_ax650.sh ├── run_qwen2.5_0.5b_gptq_int4_axcl_aarch64.sh └── run_qwen2.5_0.5b_gptq_int4_axcl_x86.sh ``` #### Start the Tokenizer service ``` root@ax650:/mnt/qtang/llm-test/qwen2.5-0.5b# python3 qwen2.5_tokenizer.py --port 12345 None None 151645 <|im_end|> <|im_start|>system You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|> <|im_start|>user hello world<|im_end|> <|im_start|>assistant [151644, 8948, 198, 2610, 525, 1207, 16948, 11, 3465, 553, 54364, 14817, 13, 1446, 525, 264, 10950, 17847, 13, 151645, 198, 151644, 872, 198, 14990, 1879, 151645, 198, 151644, 77091, 198] http://localhost:12345 ``` #### Inference with AX650 Host, such as M4N-Dock(爱芯派Pro) or AX650N DEMO Board Open another terminal and run `run_qwen2.5_0.5b_gptq_int4_ax650.sh` ``` root@ax650:/mnt/qtang/llm-test/qwen2.5-0.5b# ./run_qwen2.5_0.5b_gptq_int4_ax650.sh [I][ Init][ 125]: LLM init start bos_id: -1, eos_id: 151645 3% | ██ | 1 / 27 [0.00s<0.08s, 333.33 count/s] tokenizer init ok [I][ Init][ 26]: LLaMaEmbedSelector use mmap 100% | ████████████████████████████████ | 27 / 27 [1.34s<1.34s, 20.10 count/s] init post axmodel ok,remain_cmm(3427 MB) [I][ Init][ 241]: max_token_len : 1024 [I][ Init][ 246]: kv_cache_size : 128, kv_cache_num: 1024 [I][ Init][ 254]: prefill_token_num : 128 [I][ load_config][ 281]: load config: { "enable_repetition_penalty": false, "enable_temperature": true, "enable_top_k_sampling": true, "enable_top_p_sampling": false, "penalty_window": 20, "repetition_penalty": 1.2, "temperature": 0.9, "top_k": 10, "top_p": 0.8 } [I][ Init][ 268]: LLM init ok Type "q" to exit, Ctrl+c to stop current running >> who are you [I][ Run][ 466]: ttft: 134.66 ms I am Qwen, a Qwen AI created by Alibaba Cloud. I am here to assist you with various topics and provide help to the best of my ability. I am here to help with any questions you have about science, technology, or any other topic you might have for help or guidance. I am always happy to help you! [N][ Run][ 605]: hit eos,avg 42.11 token/s >> 1+1=? [I][ Run][ 466]: ttft: 135.07 ms 1+1=2 [N][ Run][ 605]: hit eos,avg 43.04 token/s ``` #### Inference with M.2 Accelerator card [What is M.2 Accelerator card?](https://axcl-docs.readthedocs.io/zh-cn/latest/doc_guide_hardware.html), Show this DEMO based on Raspberry PI 5. ``` (base) axera@raspberrypi:~/samples/qwen2.5-0.5b $ ./run_qwen2.5_0.5b_gptq_int4_axcl_aarch64.sh build time: Feb 13 2025 15:44:57 [I][ Init][ 111]: LLM init start bos_id: -1, eos_id: 151645 100% | ████████████████████████████████ | 27 / 27 [11.64s<11.64s, 2.32 count/s] init post axmodel okmain_cmm(6788 MB) [I][ Init][ 226]: max_token_len : 1024 [I][ Init][ 231]: kv_cache_size : 128, kv_cache_num: 1024 [I][ load_config][ 282]: load config: { "enable_repetition_penalty": false, "enable_temperature": true, "enable_top_k_sampling": true, "enable_top_p_sampling": false, "penalty_window": 20, "repetition_penalty": 1.2, "temperature": 0.9, "top_k": 10, "top_p": 0.8 } [I][ Init][ 288]: LLM init ok Type "q" to exit, Ctrl+c to stop current running >> who are you I am Qwen, a Qwen-like language model created by Alibaba Cloud. I am designed to assist users in answering questions, generating text, and participating in conversations. I am here to help you with your questions and to engage in meaningful exchanges with you. If you have any questions, you can ask me, and if you want, you can even write to me! [N][ Run][ 610]: hit eos,avg 25.88 token/s >> 1+1=? 1+1=2 [N][ Run][ 610]: hit eos,avg 29.73 token/s >> q (base) axera@raspberrypi:~/samples/qwen2.5-0.5b $ axcl-smi +------------------------------------------------------------------------------------------------+ | AXCL-SMI V2.26.0_20250205130139 Driver V2.26.0_20250205130139 | +-----------------------------------------+--------------+---------------------------------------+ | Card Name Firmware | Bus-Id | Memory-Usage | | Fan Temp Pwr:Usage/Cap | CPU NPU | CMM-Usage | |=========================================+==============+=======================================| | 0 AX650N V2.26.0 | 0000:01:00.0 | 170 MiB / 945 MiB | | -- 43C -- / -- | 2% 0% | 392 MiB / 7040 MiB | +-----------------------------------------+--------------+---------------------------------------+ +------------------------------------------------------------------------------------------------+ | Processes: | | Card PID Process Name NPU Memory Usage | |================================================================================================| | 0 474440 /home/axera/samples/qwen2.5-0.5b-gptq-int4/main_axcl_aarch64 370172 KiB | +------------------------------------------------------------------------------------------------+ (base) axera@raspberrypi:~ $ ```