Question about the chat_template
Hi, I found there is a slight difference on the chat template definition among training samples, tokenizer config and suggeted prompt.
In the README, the chat template is suggested as:
<s>[INST] SYSTEM MESSAGE
USER MESSAGE[/INST] ASSISTANT MESSAGE</s>[INST] USER MESSAGE[/INST]
And in the tokenizer_config, the official template is defined as (which means the system message is placed in the last message):
{%- if loop.last and system_message is defined %}
{{- \"[INST]\" + system_message + \"\
\
\" + message[\"content\"] + \"[/INST]\" }}
{%- else %}
{{- \"[INST]\" + message[\"content\"] + \"[/INST]\" }}
In addition, I notice that all training samples do not have system message. Does this affect the performance of the model?
It's not that the datasets don't have a system prompt - it's the same dataset, but with the system message prepended to the first user message.
The chat template included in tokenizer_config.json is simply the unmodified official template, which, as you said, prepends the system message to the last user message. I'd suggest prompting the model the way it was finetuned, which is with the system message prepended to the first user message rather than the last.