Autocomplete Strategy - October 25

[hassoncs]

Ghost Autocomplete: Strategic Direction

Current State

Ghost Autocomplete exists today as a working but limited system:

• What Works: Users can get code completions using any AI model they configure
• The Limitation: We only let users customize the model, not how we prompt it
• The Problem: Different models need different prompting strategies to work well
  • Local models (Ollama) need special "fill-in-middle" format
  • Specialized models (Mercury Coder) need rich context and diff-based output
  • Chat models (GPT-4, Claude) need XML-style guidance

Result: Users get suboptimal completions because we use a one-size-fits-all approach.

---

The Challenge

We have lots of things working in a massive development branch, but it's too large to review and merge safely. We need to decide our path forward and break it into manageable pieces. #2723

---

Two Paths

Path A: MVP - Single Model Excellence

"Make one thing work exceptionally well"

Focus: Optimize exclusively for Mercury Coder

What We Build:

1. Mercury-optimized prompting with rich context
2. Advanced snippet collection system
3. Benchmarking to prove quality

Pros:

• Fastest path to "best-in-class" autocomplete
• Clear, focused scope
• Easier to market ("optimized for Mercury")
• Lower maintenance burden

Cons:

• Locks users into one model
• Doesn't leverage our "any model" philosophy
• Harder to pivot if Mercury doesn't gain traction
• Misses opportunity with local model users

---

Path B: Configurable Platform

"Enable users to choose their own path"

Focus: Build a flexible system where users can mix models with strategies

What We Build:

1. Profile system (couple any model with any strategy)
2. Multiple strategies (Mercury, FIM for local models, HoleFiller for chat models)
3. UI for users to experiment and switch
4. Benchmarking to compare approaches

Pros:

• Aligns with Kilo Code's "your choice" philosophy
• Serves all user segments (cloud, local, specialized)
• Future-proof architecture
• Competitive differentiation

Cons:

• Longer development time
• More complex to maintain
• Harder to market ("works with everything")
• Risk of spreading too thin

---

Recommendation

Path B: Configurable Platform - but with a phased approach:

Phase 1: Mercury MVP

• Extract Mercury strategy from mega branch
• Get it working exceptionally well
• Ship to users

Phase 2: Add Flexibility

• Add FIM strategy for local models
• Add profile system for switching
• Let users experiment

Phase 3: Polish & Scale

• Add remaining strategies
• Use benchmarking system to optimize more

Why This Works:

• Gets something excellent in users' hands quickly (Mercury)
• Doesn't lock us into single-model approach
• Validates architecture with real usage
• Maintains strategic flexibility