Issues with Incomplete Translation Using webbigdata/C3TR-Adapter_gguf with llama.cpp
I am currently using the webbigdata/C3TR-Adapter_gguf model loaded via llama.cpp for translation tasks. However, I am encountering an issue where the translation output is often incomplete, with sentences being cut off mid-way.
For instance, when translating the Japanese sentence:
"識別子は任意（あとで地図表示対象のデータをロケーションログの環境設定で設定する際に、この識別子を指定するのでユーザ自身がわかる文字列を記述）"
the translation output is:
"The identifier is optional (you can specify it when setting map display data in the..."
The expected behavior would be a complete and coherent translation. Here is the configuration I am using:
llm = Llama(
model_path="C3TR-Adapter.f16.Q4_k_m.gguf"
)
prompt = f"""You are a highly skilled professional {from_lang}-{to_lang} translator.
Translate the given text accurately, taking into account the context and specific instructions provided.
Steps may include hints enclosed in square brackets [] with the key and value separated by a colon:.
Only when the subject is specified in the {from_lang} sentence, the subject will be added when translating into {to_lang}.
If no additional instructions or context are provided, use your expertise to consider what the most appropriate context is
and provide a natural translation that aligns with that context. When translating, strive to faithfully reflect the meaning
and tone of the original text, pay attention to cultural nuances and differences in language usage, and ensure that the translation
is grammatically correct and easy to read. After completing the translation, review it once more to check for errors or unnatural expressions.
For technical terms and proper nouns, either leave them in the original language or use appropriate translations as necessary.
Take a deep breath, calm down, and start translating.
### Instruction:
Translate {from_lang} to {to_lang}.
When translating, please use the following hints:
[writing_style: web-fiction]
Input:
{text}
Output:
"""
output = llm(prompt)
translated_text = output['choices'][0]['text']
I would appreciate any guidance or suggestions on how to resolve this issue and achieve complete translations. Thank you!
Hi.
With small models like the C3TR, it is important to make sure you follow the template.
・The location and number of line breaks in the prompt
・The special tokens <start_of_turn> are set in two places and <end_of_turn> is set in one place
・Please also make sure that the command line option -n is set to -2
The basic structure is as follows:
./llama-cli -m ./C3TR-Adapter.f16.Q4_k_m.gguf -e --temp 0 --repeat-penalty 1.0 -n -2 -p "You are a highly skilled professional Japanese-English and English-Japanese translator. Translate the given text accurately, taking into account the context and specific instructions provided. Steps may include hints enclosed in square brackets [] with the key and value separated by a colon:. Only when the subject is specified in the Japanese sentence, the subject will be added when translating into English. If no additional instructions or context are provided, use your expertise to consider what the most appropriate context is and provide a natural translation that aligns with that context. When translating, strive to faithfully reflect the meaning and tone of the original text, pay attention to cultural nuances and differences in language usage, and ensure that the translation is grammatically correct and easy to read. After completing the translation, review it once more to check for errors or unnatural expressions. For technical terms and proper nouns, either leave them in the original language or use appropriate translations as necessary. Take a deep breath, calm down, and start translating.
<start_of_turn>### Instruction:
<<Translate Japanese to English. or Translate English to Japanese.>>
When translating, please use the following hints:
[writing_style: <<casual or formal or technical or journalistic or web-fiction or business or nsfw or educational-casual or academic-presentation or slang or sns-casual>>]
### Input:
<<YOUR_TEXT>
<end_of_turn>
<start_of_turn>### Response:
"
What you can change is
<<Translate Japanese to English. or Translate English to Japanese.>><<casual or formal or technical or journalistic or web-fiction or business or nsfw or educational-casual or academic-presentation or slang or sns-casual>><YOUR_TEXT>
your case.
./llama-cli -n -2 -e --temp 0 --repeat-penalty 1.0 -m c3tr-v3/v3_gguf/C3TR-Adapter.f16.Q4_k_m.gguf -p "You are a highly skilled professional Japanese-English and English-Japanese translator. Translate the given text accurately, taking into account the context and specific instructions provided. Steps may include hints enclosed in square brackets [] with the key and value separated by a colon:. Only when the subject is specified in the Japanese sentence, the subject will be added when translating into English. If no additional instructions or context are provided, use your expertise to consider what the most appropriate context is and provide a natural translation that aligns with that context. When translating, strive to faithfully reflect the meaning and tone of the original text, pay attention to cultural nuances and differences in language usage, and ensure that the translation is grammatically correct and easy to read. After completing the translation, review it once more to check for errors or unnatural expressions. For technical terms and proper nouns, either leave them in the original language or use appropriate translations as necessary. Take a deep breath, calm down, and start translating.\n\n<start_of_turn>#### Instruction:\nTranslate Japanese to English.\nWhen translating, please use the following hints:\n[writing_style: technical]\n\n### Input:\n識別子は任意（あとで地図表示対象のデータ をロケーションログの環境設定で設定する際に、この識別子を指定するのでユーザ自身がわかる文字列を記述）\n<end_of_turn>\n<start_of_turn>### Response:\n"
Hi dahara1,
Thank you for your prompt response and suggestions.
After a thorough review, I discovered that the issue was due to not configuring max_tokens properly. Specifically, the lack of proper max_tokens settings was causing the model to truncate the translation output, resulting in incomplete sentences.
I have now adjusted the configuration to correctly set max_tokens, and the translation results have significantly improved. I appreciate your helpful advice. If I encounter any more issues or have further questions, I will reach out.