Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

modellist: automatically replace known chat templates with our versions #3327

Merged
merged 8 commits into from
Dec 19, 2024

Conversation

cebtenzzre
Copy link
Member

@cebtenzzre cebtenzzre commented Dec 19, 2024

This is a temporary fix for issues such as #3309, #3282, and #3263 until we can improve Jinja2Cpp itself.

The workaround for users loading a GGUF they downloaded from HuggingFace and then getting an ugly chat template that is in many cases incompatible with Jinja2Cpp, is to make a list of known templates and automatically replace them when found.

This is currently done silently and transparently, aside from a log message that looks like this:

[Warning] (Thu Dec 19 14:25:09 2024): automatically substituting chat template for "SummLlama3.2-3B-Q4_0.gguf"

We can make this list less necessary by improving Jinja2Cpp (see the comments in jinja_replacements.cpp for next steps), but for now this is better than what the current release does with these models.

This allows us to be compatible with e.g. any finetune of Llama 3.2 3B Instruct that does not alter the chat template, without having to touch models3.json.


models3.json has been touched in this PR for consistency with the list of substitutions, which now covers all official GPT4All models, even the ones that were working before. This makes them prettier and easier to edit.

@cebtenzzre cebtenzzre marked this pull request as ready for review December 19, 2024 19:33
@cebtenzzre cebtenzzre requested a review from manyoso December 19, 2024 19:33
Signed-off-by: Jared Van Bortel <[email protected]>
Signed-off-by: Jared Van Bortel <[email protected]>
@manyoso manyoso merged commit 6bbeac2 into main Dec 19, 2024
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment