hugging-quants

/

Meta-Llama-3.1-8B-Instruct-AWQ-INT4

like 89

Follow

Hugging Quants 290

I was installing autoawq but I am seeing an error "no module named torch" even if torch is installed. so i used github using "pip install git+https://github.com/casper-hansen/AutoAWQ.git --no-deps". this was success but when I try to run the example script I am coming across another error.

from awq.utils.module import try_import
ImportError: cannot import name 'try_import' from 'awq.utils.module' (C:\Users\rajes\Documents\GitHub\quantization_code\quant-env\Lib\site-packages\awq\utils\module.py)

I changed version from GEMM to GEMV since windows doesn't support flash attention yet and I see this error.
RuntimeError: Error(s) in loading state_dict for WQLinear_GEMV:
size mismatch for qweight: copying a param with shape torch.Size([14336, 512]) from checkpoint, the shape in current model is torch.Size([4096, 1792]).

Can someone help me how to run the example script atleast to run some basic testing on AWQ quantized models?