Beyond simple inference

[ElCondor1969]

Hi Jarrod.
Do you have a date in mind to implement inference via the chat template paradigm, with the possibility to specify the various roles of individual messages, as was done in Infero?
Bye and thanks again for your great work.

Sergio

[jarroddavis68]

Hi, thanks! This feature is next on my list, and I’m excited to dive into it within the next week or two. Stay tuned!

[ElCondor1969]

Thanks a lot Jarrod!
I am eager to test the new feature.

[jarroddavis68]

I sent you a msg on discord.

[ElCondor1969]

Hi Jarrod.
I saw your message on Discord.
I will run the test as soon as I can and let you know the result soon after.
See you soon.

[ElCondor1969]

Hi Jarrod.
I did some tests, using the different models that were indicated in the test project and everything worked correctly.
In one test, using the model 'gemma-2-2b-it-abliterated-Q8_0', I simulated an exchange of responses between the user and the model:

jiAddMessage(jiROLE_USER, 'A quale città sto pensando ora?');
jiAddMessage(jiROLE_ASSISTANT, 'Tu stai pensando alla città di Ancona');
jiAddMessage(jiROLE_USER, 'E a quale città sto pensando adesso?');
jiAddMessage(jiROLE_ASSISTANT, 'Adesso invece stai pensando alla città di Roma');
jiAddMessage(jiROLE_USER, 'Indicami un tragitto per arrivare alla seconda città partendo dalla prima');

and the result was this:

So I would say that everything works correctly.
Thanks!

[jarroddavis68]

Great! Thanks, will have an update to this project soon. Stay tuned!

[ElCondor1969]

Hi Jarrod.
Thanks!
While we're at it, I would like to give a tip on how to handle the model's chat template.
Often, inside the GGUF files, you can find the template that the model uses:

(the key to see is also "general.chat_template")

So, when loading a model with Lumina, I would suffer to implement the following points:

1. If the template is specified by the user, then it is used.
2. If the template is not specified by the user, then it will use the one specified in the GGUF file.
3. If the template is not specified by the user and does not exist inside the GGUF file, then it will use a default one or raise an exception.

What do you think?
Bye and thanks again!

[jarroddavis68]

https://github.com/ggerganov/llama.cpp/wiki/Templates-supported-by-llama_chat_apply_template

This is the way I do it in Lumina, but llama.cpp does not get those templates from the GGUF file, they are hard coded in llama.cpp and based on the model, it will look it up and if not found just default to chatML (OpenAI format). That template can have logic it and as stated in the link, llama.cpp does not currently support this due to its complexity.

Anyway, all ideas are welcome, and I will take everything into consideration.

Since it's not released yet and is still WIP, can I ask you to communicate with me on discord. I've already sent you an updated build. I will be setting up a repo for it soon. The new build I sent you has a pretty much complete API now as well as docs.

[ElCondor1969]

OK Jarrod, I got it.
I'll test it and I let know you the results.
See you soon and thank you!