Alex Daminger

Yes, exactly. When converting from float16 to float32 for fine-tuning (as I thought), we need to fill 13 bits of the mantissa and 3 bits of the exponent with zeros, rather than simply filling the last 16 bits.

Ok I get your point now.

I don't understand much about this, but maybe the model in F32 is just redundant. Maybe the other half of most weights are filled with zeros. It was scaled this way to fine-tune it or to make it impossible for people with few resources to run it😁

32 and 16 is the memory that each weight takes, not a number of weights. You can look into float point 32 and 16 in computer science to better grasp what does it mean.