Llama-3.2 11B Vision Support

[yukiarimo]

I would like to run Llama-3.2 11B Vision in KoboldCPP. Ollama recently added the support, so I guess it is possible to pull it here :)

Also, there is support needed for GGUF conversion!

[LostRuins]

Ollama does it in golang, their implementation is not compatible.

KoboldCpp's upstream is llama.cpp, so you'd have to wait until ggerganov#9643 is implemented (looking very unlikely since nobody is interested in implementing it)

Alternatively, you can use MiniCPM v2, which already has support here and works well enough.

[yukiarimo]

Oh, I see. That’s sad! By the way, in KoboldCPP, you can send plain raw text to the model. Do you know how to do that using Ollama without using any template? And is a model card mandatory?

[LostRuins]

I don't think it's possible in Ollama (most backends nowadays only allow chat completions). That said, I have not extensively tried their software and would naturally be biased against it.

In KoboldCpp, no model card is necessary and you are able to use whatever prompt format or instruct format you like. In Ollama, I believe they restrict it based on whatever you have downloaded.