Size mismatch error when loading Mistral-Small
#1
by
Srirakshith
- opened
I'm trying to load the 4 bit quantised model with an L4 GPU, and it keeps throwing up the following error.
ValueError: Trying to set a tensor of shape torch.Size([83886080, 1]) in "weight" (which has shape torch.Size([5120, 32768])), this looks incorrect.
Is there a setting that I need to change to overcome this?