Followed instructions to swap LLM model, ggetting "Error calling the chat endpoint: Connection error.One Conversation Loop Completed"

[TheChief0117]

I followed the instructions to swap the LLM model from what it was currently using to a local version of llama-2-7b.Q5_0.gguf; what am I doing wrong?

My conf.yaml as follows:

Ollama & OpenAI Compatible inference backend

ollama:

BASE_URL: "http://localhost:11434"

BASE_URL: "http://localhost:11434/v1"
LLM_API_KEY: "somethingelse"
ORGANIZATION_ID: "org_eternity"
PROJECT_ID: "project_glass"

LLM name

MODEL: "llama3.1:latest"

system prompt is at the very end of this file

VERBOSE: false

Claude API Configuration

claude:
BASE_URL: "https://api.anthropic.com"
LLM_API_KEY: "YOUR API KEY HERE"
MODEL: "claude-3-haiku-20240307"
VERBOSE: False

llamacpp:
MODEL_PATH: "C:\Users\Server\Desktop\MadeInChinaNeurosama\Open-LLM-VTuber\llm\llama-2-7b.Q5_0.gguf"
VERBOSE: True

[t41372]

Have you resolved the problem? Sorry for the communication delay.

The Error calling the chat endpoint error means your LLM_PROVIDER option in conf.yaml is probably set to ollama, claude, or anything other than llamacpp. If you would like to use llama cpp to run the gguf files directly, please set so in the conf.yaml file and remember to check out the related section in README for instruction.