Meraj-Mini.Q8_0.gguf

#2
by Majoude - opened

Getting this while initiating the model from pretrained:
llm_load_vocab: missing pre-tokenizer type, using: 'default'
llm_load_vocab:
llm_load_vocab: ************************************
llm_load_vocab: GENERATION QUALITY WILL BE DEGRADED!
llm_load_vocab: CONSIDER REGENERATING THE MODEL
llm_load_vocab: ************************************

It took so much time (5+ minutes) to generate an answer on Colab and generated none-sense.

this is a large model, you can't run the Q8 on a Colab that has 2 cores. It requires either more compute or you have to lower the quant size, like Q3 or maybe even Q2 just for testings.

MaziyarPanahi changed discussion status to closed

Sign up or log in to comment