LM Studio Granite GGUF model loading error

[dukeinps]

Thank you for the GGUF models on huggingface. They download and have the correct parameters to run but but sadly they fail to load in LM Studio. Getting this error:
"llama.cpp error: 'check_tensor_dims: tensor 'output.weight' not found'" In researching this on the llama.cpp github discussions, there is no clear answer. The message infers that something is missing or incomplete.

[dprosper]

similar issue attempting to use Ollama, the error is sliglty different then above.

error loading model vocabulary: unknown pre-tokenizer type: 'refact

in contrast one from instructlab granite-7b-lab-GGUF works fine.

[mayank31398]

Hi, currently you need to use llama.cpp from main branch I think.
I am unsure if ollama is supported currently.

[akiemiq4g]

I am running on Ollama pre-release version 0.1.39 and that is how I got around @dprosper the refact error but the chat response has not been properly parsed. Therefore, it looks like this but I bet it probably looks better with the proprietary app IBM developed to use their model:

I feel like I'll end up just downloading the IBM app just to get my feet wet with it but not a fan of this initial road of installing it. But if you intend on integrating it into an app and will be doing parsing downstream of your api calls then this should work for you. I just had to download this version: https://github.com/ollama/ollama/releases/tag/v0.1.39 and install it by dragging it into my Applications folder on Mac overwriting my previous 0.1.38 version. I will probably revert back as I'd rather run on the stable version as I'm not developing apps that require any pre-release functionality as of yet and not sure if it has caused any side-effects to the other models I've been using via Ollama server.

[mayank31398]

I see. I have tried LM Studio btw.
Its working fine for me.