Improving Goose performance with local models

[ahau-square]

Improving Goose Performance with Local Models

We're working on measuring and improving Goose performance with local models (focused on those available through Ollama) to increase accessibility and enhance the fully open source experience.

Current Focus Areas

1. Open models with native tool calling capabilities

   • Models like llama3.1-3.3, mistral 7B, qwen2+

2. Open models without native tool calling

   • Models like deepseek-r1, phi4, gemma

Our Approach

We're benchmarking open models against closed models (from Anthropic/OpenAI) on simple tasks to establish current performance baselines. Our initial findings suggest tool calling is a significant limitation for many open models today, even those with the technical capability.

It seems like tool calling is a key inhibitor for open models so far - even if they have the capability, they might not perform well. Recent research supports this:

• https://sierra.ai/blog/benchmarking-ai-agents
• https://arxiv.org/pdf/2408.04682

Consumer Hardware Used for Benchmarking

We're running our benchmarks on:

• NVIDIA GeForce RTX 4080 (16GB VRAM)
• MacBook Pro M1 (16GB unified memory)

We acknowledge that many Goose users may not have comparable hardware, but we believe providing these benchmarks will still be valuable as a reference point for consumer hardware performance.

Anticipated Improvements

• Working on updating Goose to allow users to set custom context lengths for Ollama models (current default is only 2048 tokens)
  • Related PR: config: allow setting context length through env var ollama/ollama#8938
  • This addresses silent prompt truncation issues that degrade performance (our system prompt alone is ~1k tokens)

Experimental Work

• For models without native tool calling, we're developing a "shim" that uses an interpretive model to parse responses into appropriate tool calls. This approach might prove useful for future open models as well.
• We're exploring promising models like Functionary (https://github.com/MeetKai/functionary) which show good tool calling results, but compatibility issues with Ollama exist that we are investigating solutions for
  • Functionary uses XML for tool formatting vs. Ollama's required JSON
  • Running Functionary reliably requires VLLM, which isn't feasible for many users without GPUs

How You Can Help

• Share your experiences using Goose with any open source models
• Recommend open source models you've found that excel at function calling
• Share prompts that make open source models work better with Goose
• Share Goose performance results on different hardware configurations
• Contribute to benchmarking efforts - we will share more on this soon! (a way for anyone to add and run their own eval tasks via the Goose cli)

[alicehau]

PR in progress for tool shim: #1448

[michpohl]

Hey! Thanks so much for doing this!

Just wanted to share some of my experiences so far, running on a M1 Macbook:

1. MacOS (especially Sonoma) seems to block all tool calling attempts by default. I didn't get any prompts about what Goose is attempting to use, so for now I gave it "Full Disk Access", "Accessibility" and "Developer Tools". These seemed to be most likely to help.
2. With these enabled I'm starting to see models with tool calling at least try to do work (tested Qwen2.5:7b and @michaelneale's deepseek model today). Both attempt to do tool calling, but fail.
3. It appears to me they dream up what tools they have available. I've tried two tasks: List files in this directory and Create a file called testfile, both failed.

I'm not very experienced with LLMs, so I'm just poking around. If there are any specific things I could provide, I'm happy to try and collect some proper data!

[michaelneale]

yes depending on config macos may pop up at various points (usually when launching, but can be easy to dismiss).

I think tool calling is still weak with local models - the most promising so far is #1448 - which when ollama can support larger context input, should be able to get that in and help it work with a wider variety of models. So - it is a rapidly evolving space, but still some work to go before all local on reasonable hardware (affordable) is common.