3 bits or 4 bits GPTQ in this project?
#13
by
wyklq
- opened
quantize_config.json contains
"bits": 3,
But the model is in 4 bits in the name "gptq_model-4bit".
This caused automatic loading failed with the example code in the model card.
It looks like the bits shall be 4.
Sorry! Yes this is the 4bit model, quantize_config.json was wrong. I have corrected it now.
There is a separate 3bit version as well, linked in the README.
TheBloke
changed discussion status to
closed