Ollama local LLM step-by-step guide?

[robscurity]

Hi, I have ollama installed on my mac and it´s working. I would like to use my local ollama LLM:s with fabric and I tried this:

ollama list (to start ollama)
fabric --listmodels (here I can see my local ollama LLM:s but their names seems to be shortened)
pbpaste | fabric -m "codelama:7b-code" -p extract_wisdom (I get Error: Connection error. Connection error.)

My ~/.config/fabric/.env file contains
OPENAI_API_KEY=NotShownHere
OPENAI_BASE_URL=http://localhost:114341

Any suggestions?

[xssdoctor]

Show me the results of your —listmodels

[robscurity]

(fabric)% fabric --listmodels
GPT Models:
gpt-3.5-turbo
gpt-3.5-turbo-0125
gpt-3.5-turbo-0301
gpt-3.5-turbo-0613
gpt-3.5-turbo-1106
gpt-3.5-turbo-16k
gpt-3.5-turbo-16k-0613
gpt-3.5-turbo-instruct
gpt-3.5-turbo-instruct-0914

Local Models:
codellama:7b-cod
llama2
llama2-uncensored:7b-ch
llava:7b
mistral:7b-instruct-q4_K_M
mixtr

Claude Models:
claude-3-opus-20240229

[robscurity]

If I check with ollama, the names are longer:
% ollama list
NAME ID SIZE MODIFIED
codellama:7b-code fc84f39375bc 3.8 GB 3 weeks ago
llama2:latest 78e26419b446 3.8 GB 7 minutes ago
llama2-uncensored:7b-chat 44040b922233 3.8 GB 3 weeks ago
llava:7b 8dd30f6b0cb1 4.7 GB 4 weeks ago
mistral:7b-instruct-q4_K_M 1a85656b534f 4.4 GB 3 weeks ago
mixtral:latest 7708c059a8bb 26 GB 2 weeks ago

[xssdoctor]

Try just codelama or llama2

[xssdoctor]

I just pushed a fix to that problem with the --listmodels that it shows weird info after the :. Try a git pull and tell me what you get. But with the local models the names should be something like llama2 or llama2-uncensored. these are also the names that go into the ollama api

[TerrenceMiao]

This is my workaround approach:

$ ollama cp mixtral:instruct mixtral
copied 'mixtral:instruct' to 'mixtral'

$ ollama list
NAME                   	ID          	SIZE  	MODIFIED
mixtral:instruct       	7708c059a8bb	26 GB 	2 weeks ago  	
mixtral:latest         	7708c059a8bb	26 GB 	5 seconds ago	

$ echo "An idea that coding is like speaking with rules." | fabric --pattern write_essay --model mixtral

Fabric can automatically detect if Ollama runs on default port 11434. No need to add OPENAI_BASE_URL in configuration file ~/.config/fabric/.env.

[robscurity]

Ok thanks. I will try this. /Rob11 mars 2024 kl. 23:20 skrev xssdoctor ***@***.***>: I just pushed a fix to that. i dont know what the problem is. But with the local models the namesshould be something like llama2 or llama2-uncensored. these are also the names that go into the ollama api —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>