adamo1139/Yi-34B-200K-Un-Instruct-1906

Basic Model Info

1 epoch on adamo1139/uninstruct-v1-experimental-chatml. I used GaLore.
Purpose of this model is to make the model un-learn to use chatml-specific code words such as: <|im_start|>, <|im_end|>, user, assistant. This is a base model meant for further finetuning. I think much of OpenAI slop is still left in there, so it's probably best combined with preference optimization method like DPO, ORPO or SPO for best results.