Weird completions
Is anyone also having weird completions with the model?
For example, I gave a simple few-shot example of a sentiment classification task and it is generating completions such as:
"786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786786"
Also, accuracy on Winogrande is close to random (53%).
I encountered similar issues: https://huggingface.co./google/gemma-7b/discussions/33
Can you send us the particular prompts you're using (I imagine this is with the pretrained model)?
I was accidentally using add_special_tokens=False, which does not properly prepend a <bos> token (token_id=2) to the sequence:
tokenizer("I like", return_tensors="pt", add_special_tokens=False)
>>> {'input_ids': tensor([[235285, 1154]]), 'attention_mask': tensor([[1, 1]])}
tokenizer("I like", return_tensors="pt", add_special_tokens=True)
{'input_ids': tensor([[ 2, 235285, 1154]]), 'attention_mask': tensor([[1, 1, 1]])}
Just to be clear, using add_special_tokens=True solved the problem.