Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The tokenizer does not have a {% if add_generation_prompt %} #1312

Open
Galaxy-Husky opened this issue Nov 21, 2024 · 3 comments
Open

The tokenizer does not have a {% if add_generation_prompt %} #1312

Galaxy-Husky opened this issue Nov 21, 2024 · 3 comments

Comments

@Galaxy-Husky
Copy link

Hi,

After I upgraded unsloth from 2024.11.5 to 2024.11.7, it raised the error that the tokenizer does not have a {% if add_generation_prompt %}.
The model is shenzhi-wang/Llama3.1-8B-Chinese-Chat. It has the following chat template:

{% set loop_messages = messages %}
{% for message in loop_messages %}
    {% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>
    
    '+ message['content'] | trim + '<|eot_id|>' %}
    {% if loop.index0 == 0 %}
        {% set content = bos_token + content %}
    {% endif %}
    {{ content }}
{% endfor %}
{{ '<|start_header_id|>assistant<|end_header_id|>

' }}

Could you check and fix it?

@danielhanchen
Copy link
Contributor

@Galaxy-Husky Ye so the chat template for that model looks incorrect - it should not have {{ '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }} by default, but rather {% if add_generation_prompt %}{ '<|start_header_id|>assistant<|end_header_id|>\n\n'`

@Galaxy-Husky
Copy link
Author

@Galaxy-Husky Ye so the chat template for that model looks incorrect - it should not have {{ '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }} by default, but rather {% if add_generation_prompt %}{ '<|start_header_id|>assistant<|end_header_id|>\n\n'`

Yes, I agree.
But 2024.11.5 will help me add {% if add_generation_prompt %} to fix the template while 2024.11.7 will not and I can't use the model.
Is this the expected behavior?

@Galaxy-Husky
Copy link
Author

@danielhanchen Hi! Would you consider continuing to support fixing such templates?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants