It is needed to use bnb 4bit?
Hi!
I'm just wondering if it's needed to use a bnb 4 bit model as the base for the fine tune instead of a full precision model, or this is just to speed up the process but the quality of the adapter generated would be better if we used the full precision as the base?
Then to merge the adapter, is it merged with the bnb 4 bit version or into the full precision model?
I'm a bit confused by this. Thanks!!
Hi!
I'm just wondering if it's needed to use a bnb 4 bit model as the base for the fine tune instead of a full precision model, or this is just to speed up the process but the quality of the adapter generated would be better if we used the full precision as the base?
Then to merge the adapter, is it merged with the bnb 4 bit version or into the full precision model?
I'm a bit confused by this. Thanks!!
Hey sorry for the extremely late reply.
You use the 4bit version for 4x faster downloading, training and 4x less vram use in general. IF you use the original 16bit model you will need at least 40GB of VRAM