Hugging Face's logo

RedHatAI
/
Llama-4-Scout-17B-16E-Instruct-quantized.w4a16

Image-Text-to-Text
Safetensors
PyTorch
llama4
facebook
meta
llama
neuralmagic
redhat
llmcompressor
quantized
W4A16
INT4
conversational
compressed-tensors
Model card Files
xet
Community
4
ekurtic commited on
Commit
cab2453
·
verified ·
1 Parent(s): 5fa5124

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +180 -0
README.md ADDED
@@ -0,0 +1,180 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: vllm
3
+ language:
4
+ - ar
5
+ - de
6
+ - en
7
+ - es
8
+ - fr
9
+ - hi
10
+ - id
11
+ - it
12
+ - pt
13
+ - th
14
+ - tl
15
+ - vi
16
+ base_model:
17
+ - meta-llama/Llama-4-Scout-17B-16E-Instruct
18
+ pipeline_tag: image-text-to-text
19
+ tags:
20
+ - facebook
21
+ - meta
22
+ - pytorch
23
+ - llama
24
+ - llama4
25
+ - neuralmagic
26
+ - redhat
27
+ - llmcompressor
28
+ - quantized
29
+ - W4A16
30
+ - INT4
31
+ license: other
32
+ license_name: llama4
33
+ ---
34
+
35
+ # Llama-4-Scout-17B-16E-Instruct-quantized.w4a16
36
+
37
+ ## Model Overview
38
+ - **Model Architecture:** Llama4ForConditionalGeneration
39
+ - **Input:** Text / Image
40
+ - **Output:** Text
41
+ - **Model Optimizations:**
42
+ - **Activation quantization:** None
43
+ - **Weight quantization:** INT4
44
+ - **Release Date:** 04/25/2025
45
+ - **Version:** 1.0
46
+ - **Model Developers:** Red Hat (Neural Magic)
47
+
48
+
49
+ ### Model Optimizations
50
+
51
+ This model was obtained by quantizing weights of [Llama-4-Scout-17B-16E-Instruct](https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct) to INT4 data type.
52
+ This optimization reduces the number of bits used to represent weights from 16 to 4, reducing GPU memory requirements by approximately 75%.
53
+ Weight quantization also reduces disk size requirements by approximately 75%. The [llm-compressor](https://github.com/vllm-project/llm-compressor) library is used for quantization.
54
+
55
+
56
+ ## Deployment
57
+
58
+ This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below.
59
+
60
+ ```python
61
+ from vllm import LLM, SamplingParams
62
+ from transformers import AutoTokenizer
63
+
64
+ model_id = "RedHatAI/Llama-4-Scout-17B-16E-Instruct-quantized.w4a16"
65
+ number_gpus = 4
66
+
67
+ sampling_params = SamplingParams(temperature=0.7, top_p=0.8, max_tokens=256)
68
+
69
+ tokenizer = AutoTokenizer.from_pretrained(model_id)
70
+
71
+ prompt = "Give me a short introduction to large language model."
72
+
73
+ llm = LLM(model=model_id, tensor_parallel_size=number_gpus)
74
+
75
+ outputs = llm.generate(prompt, sampling_params)
76
+
77
+ generated_text = outputs[0].outputs[0].text
78
+ print(generated_text)
79
+ ```
80
+
81
+ vLLM also supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
82
+
83
+
84
+ ## Evaluation
85
+
86
+ The model was evaluated on the OpenLLM leaderboard tasks (v1 and v2), long context RULER, multimodal MMMU, and multimodal ChartQA.
87
+ All evaluations are obtained through [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness).
88
+
89
+ <details>
90
+ <summary>Evaluation details</summary>
91
+
92
+ **OpenLLM v1**
93
+ ```
94
+ lm_eval \
95
+ --model vllm \
96
+ --model_args pretrained="RedHatAI/Llama-4-Scout-17B-16E-Instruct-quantized.w4a16",dtype=auto,add_bos_token=True,max_model_len=4096,tensor_parallel_size=8,gpu_memory_utilization=0.7,enable_chunked_prefill=True,trust_remote_code=True \
97
+ --tasks openllm \
98
+ --batch_size auto
99
+ ```
100
+
101
+ **OpenLLM v2**
102
+ ```
103
+ lm_eval \
104
+ --model vllm \
105
+ --model_args pretrained="RedHatAI/Llama-4-Scout-17B-16E-Instruct-quantized.w4a16",dtype=auto,add_bos_token=False,max_model_len=16384,tensor_parallel_size=8,gpu_memory_utilization=0.5,enable_chunked_prefill=True,trust_remote_code=True \
106
+ --tasks leaderboard \
107
+ --apply_chat_template \
108
+ --fewshot_as_multiturn \
109
+ --batch_size auto
110
+ ```
111
+
112
+ **Long Context RULER**
113
+ ```
114
+ lm_eval \
115
+ --model vllm \
116
+ --model_args pretrained="RedHatAI/Llama-4-Scout-17B-16E-Instruct-quantized.w4a16",dtype=auto,add_bos_token=False,max_model_len=524288,tensor_parallel_size=8,gpu_memory_utilization=0.9,enable_chunked_prefill=True,trust_remote_code=True \
117
+ --tasks ruler \
118
+ --metadata='{"max_seq_lengths":[131072]}' \
119
+ --batch_size auto
120
+ ```
121
+
122
+ **Multimodal MMMU**
123
+ ```
124
+ lm_eval \
125
+ --model vllm-vlm \
126
+ --model_args pretrained="RedHatAI/Llama-4-Scout-17B-16E-Instruct-quantized.w4a16",dtype=auto,add_bos_token=False,max_model_len=1000000,tensor_parallel_size=8,gpu_memory_utilization=0.9,enable_chunked_prefill=True,trust_remote_code=True,max_images=10 \
127
+ --tasks mmmu_val \
128
+ --apply_chat_template \
129
+ --batch_size auto
130
+ ```
131
+
132
+ **Multimodal ChartQA**
133
+ ```
134
+ export VLLM_MM_INPUT_CACHE_GIB=8
135
+ lm_eval \
136
+ --model vllm-vlm \
137
+ --model_args pretrained="RedHatAI/Llama-4-Scout-17B-16E-Instruct-quantized.w4a16",dtype=auto,add_bos_token=False,max_model_len=1000000,tensor_parallel_size=8,gpu_memory_utilization=0.9,enable_chunked_prefill=True,trust_remote_code=True,max_images=10 \
138
+ --tasks chartqa \
139
+ --apply_chat_template \
140
+ --batch_size auto
141
+ ```
142
+
143
+ </details>
144
+
145
+ ### Accuracy
146
+
147
+ | | Recovery (%) | meta-llama/Llama-4-Scout-17B-16E-Instruct | RedHatAI/Llama-4-Scout-17B-16E-Instruct-quantized.w4a16<br>(this model) |
148
+ | ---------------------------------------------- | :-----------: | :---------------------------------------: | :-----------------------------------------------------------------: |
149
+ | ARC-Challenge<br>25-shot | ? | 69.37 | ? |
150
+ | GSM8k<br>5-shot | ? | 90.45 | ? |
151
+ | HellaSwag<br>10-shot | ? | 85.23 | ? |
152
+ | MMLU<br>5-shot | ? | 80.54 | ? |
153
+ | TruthfulQA<br>0-shot | ? | 61.41 | ? |
154
+ | WinoGrande<br>5-shot | ? | 77.90 | ? |
155
+ | **OpenLLM v1<br>Average Score** | **?** | **77.48** | **?** |
156
+ | IFEval<br>0-shot<br>avg of inst and prompt acc | ? | 86.90 | ? |
157
+ | Big Bench Hard<br>3-shot | ? | 65.13 | ? |
158
+ | Math Lvl 5<br>4-shot | ? | 57.78 | ? |
159
+ | GPQA<br>0-shot | ? | 31.88 | ? |
160
+ | MuSR<br>0-shot | ? | 42.20 | ? |
161
+ | MMLU-Pro<br>5-shot | ? | 55.70 | ? |
162
+ | **OpenLLM v2<br>Average Score** | **?** | **56.60** | **?** |
163
+ | RULER<br>seqlen = 131072<br>niah_multikey_1 | ? | 88.20 | ? |
164
+ | RULER<br>seqlen = 131072<br>niah_multikey_2 | ? | 83.60 | ? |
165
+ | RULER<br>seqlen = 131072<br>niah_multikey_3 | ? | 78.80 | ? |
166
+ | RULER<br>seqlen = 131072<br>niah_multiquery | ? | 95.40 | ? |
167
+ | RULER<br>seqlen = 131072<br>niah_multivalue | ? | 73.75 | ? |
168
+ | RULER<br>seqlen = 131072<br>niah_single_1 | ? | 100.00 | ? |
169
+ | RULER<br>seqlen = 131072<br>niah_single_2 | ? | 99.80 | ? |
170
+ | RULER<br>seqlen = 131072<br>niah_single_3 | ? | 99.80 | ? |
171
+ | RULER<br>seqlen = 131072<br>ruler_cwe | ? | 39.42 | ? |
172
+ | RULER<br>seqlen = 131072<br>ruler_fwe | ? | 92.93 | ? |
173
+ | RULER<br>seqlen = 131072<br>ruler_qa_hotpot | ? | 48.20 | ? |
174
+ | RULER<br>seqlen = 131072<br>ruler_qa_squad | ? | 53.57 | ? |
175
+ | RULER<br>seqlen = 131072<br>ruler_qa_vt | ? | 92.28 | ? |
176
+ | **RULER<br>seqlen = 131072<br>Average Score** | **?** | **80.44** | **?** |
177
+ | MMMU<br>0-shot | ? | 53.44 | ? |
178
+ | ChartQA<br>0-shot<br>exact_match | ? | 65.88 | ? |
179
+ | ChartQA<br>0-shot<br>relaxed_accuracy | ? | 88.92 | ? |
180
+ | **Multimodal Average Score** | **?** | **69.41** | **?** |