TheBloke

/

h2ogpt-oasst1-512-30B-GPTQ

like 5

I've not had a chance to do the readme yet.

GPTQ params are:
wbits = 4
groupsize = None
model_type = llama

I get a size mismatch error if I use wbits=4

I get a size mismatch error if I use wbits=4

Show me the errors. Are you using pre_layer as well?

I run this this command python server.py --model TheBloke_h2ogpt-oasst1-512-30B-GPTQ --model_type llama --wbits 4 --chat

and I get this error (only showing the last two lines):

        size mismatch for model.layers.59.mlp.up_proj.qzeros: copying a param with shape torch.Size([1, 2240]) from checkpoint, the shape in current model is torch.Size([52, 2240]).
        size mismatch for model.layers.59.mlp.up_proj.scales: copying a param with shape torch.Size([1, 17920]) from checkpoint, the shape in current model is torch.Size([52, 17920]

Can you try without --chat and let me know if that makes any difference?

I run this this command python server.py --model TheBloke_h2ogpt-oasst1-512-30B-GPTQ --model_type llama --wbits 4 --chat

and I get this error (only showing the last two lines):

        size mismatch for model.layers.59.mlp.up_proj.qzeros: copying a param with shape torch.Size([1, 2240]) from checkpoint, the shape in current model is torch.Size([52, 2240]).
        size mismatch for model.layers.59.mlp.up_proj.scales: copying a param with shape torch.Size([1, 17920]) from checkpoint, the shape in current model is torch.Size([52, 17920]

You need to specifity --groupsize 512

You need to specifity --groupsize 512

The 512 in the model size is not the group size. The group size is None.

I now know what's causing this. It appears to be a text-gen-ui bug related to groupsize. It's forcing the model to use a groupsize of 128, ignoring the --groupsize -1 default. For the fix, please see: https://huggingface.co./TheBloke/OpenAssistant-SFT-7-Llama-30B-GPTQ/discussions/3#6463434bab15db2fa5661b31