Meraj-Mini.Q8_0.gguf
#2
by
Majoude
- opened
Getting this while initiating the model from pretrained:
llm_load_vocab: missing pre-tokenizer type, using: 'default'
llm_load_vocab:
llm_load_vocab: ************************************
llm_load_vocab: GENERATION QUALITY WILL BE DEGRADED!
llm_load_vocab: CONSIDER REGENERATING THE MODEL
llm_load_vocab: ************************************
It took so much time (5+ minutes) to generate an answer on Colab and generated none-sense.
this is a large model, you can't run the Q8 on a Colab that has 2 cores. It requires either more compute or you have to lower the quant size, like Q3 or maybe even Q2 just for testings.
MaziyarPanahi
changed discussion status to
closed