Tokenizer needs to be fixed for BOS handling
#18
by
dzhulgakov
- opened
Same issues as https://huggingface.co./nltpt/Llama-3.2-1B-Instruct/discussions/8 - raw encode() doesn't prepend BOS
Same issues as https://huggingface.co./nltpt/Llama-3.2-1B-Instruct/discussions/8 - raw encode() doesn't prepend BOS