Update generate to not ramble/repeat
#1
by
ctranslate2-4you
- opened
Please update the output to specify the extra eos token. For example, here is a code snippet before:
output = model.generate(**inputs, max_new_tokens=512, do_sample=False)
And here is the corrected snippet:
output = model.generate(**inputs, max_new_tokens=512, do_sample=False, eos_token_id=processor.tokenizer.convert_tokens_to_ids("<|eot_id|>"))
Otherwise, THANKS for the model, works great!
The rambling/repeat issue is not related to the EOS/eot token. It's a problem of the model tuning itself.
It sometimes properly ends the stream with the eot token, sometimes it repeats.
The repeating issues are in a wide variety. If you ask it to generate a list of N things of the image you've a nice change to get 1-2 things repeated all over without eot