quantflex commited on
Commit
fc56aa1
1 Parent(s): 58f1df1

Disable bos token

Browse files

Hello 🙂,

Lets disable bos as according to line 8 in config.json:

https://huggingface.co./Infinigence/Megrez-3B-Instruct/blob/main/config.json#L8

Thank you

Files changed (1) hide show
  1. tokenizer_config.json +1 -0
tokenizer_config.json CHANGED
@@ -1,4 +1,5 @@
1
  {
 
2
  "added_tokens_decoder": {
3
  "120000": {
4
  "content": "<|eos|>",
 
1
  {
2
+ "add_bos_token": false,
3
  "added_tokens_decoder": {
4
  "120000": {
5
  "content": "<|eos|>",