Prompting

Overview

tldr for this section: {CODE}, {SELECTED_TEXT} and {CONTEXT} are keywords in the messages that are replaced by code and context returned from the memory backend, and selected text from the user's editor. Note that the file_store backend always returns an empty context so if you are using the file_store backend, do not include the {CONTEXT} keyword. The file_store backend injects <CURSOR> in the code at the user's current cursor location.

There are currently three available prompt formats:

• Completion
• FIM
• Chat

See the Configuration page for how to enable them.

Completion and FIM involve no additional prompts from the user. Chat requires the user to provide some messages like:

{
  "messages": [
    {
      "role":"system",
      "content":"You are a programming completion tool. Replace <CURSOR> with the correct code."
    },
    {
      "role": "user",
      "content": "{CODE}"
    }
  ]
}

There are three steps involved before these messages are passed to the model:

1. The Memory Backend gets the code and context for the prompt
2. The {CODE} and {CONTEXT} keys are replaced with the code and context from the Memory Backend
3. The Transformer Backend Formats the Messages

1. The Memory Backend Gets the Code and Context

When requested, the memory backend returns the following struct:

struct Prompt {
  code: String,
  context: String
}

If the user is working on the following file with their cursor located right after the n of return on line 3.

def fib(n):
    if n <= 1:
        return
    else:
        return fib(n-1) + fib(n-2)

The file_store backend context is always empty, but the value of code will be one of three different formats depending on whether the user has specified Completion, FIM, or Chat.

Value of code for Completion

def fib(n):
    if n <= 1:
        return

Value of code for FIM

<fim_prefix>def fib(n):
    if n <= 1:
        return<fim_suffix>
    else:
        return fib(n-1) + fib(n-2)<fim_middle>

Note that the <fim_prefix>, <fim_suffix> and <fim_middle> are configurable.

Value of code for Chat

def fib(n):
    if n <= 1:
        return<CURSOR>
    else:
        return fib(n-1) + fib(n-2)

2. The Keys Are Replaced

After the Prompt struct is returned from the memory backend, the prompt keys are replaced in the messages.

There are three available prompt keys:

• {CODE} - the code prompt for the model. See the section above for what this looks like
• {CONTEXT} - the context around the code. This is blank if using the file_store backend but populated by a vector search if using the vector_store backend.
• {SELECTED_TEXT} - the text that was selected when the user performed the action request. This is only populated when performing custom actions.

Given the following messages:

{
  "messages": [
    {
      "role":"system",
      "content":"You are a programming completion tool. Replace <CURSOR> with the correct code."
    },
    {
      "role": "user",
      "content": "{CODE}"
    }
  ]
}

and assuming the same code as in the section above, the messages would be transformed to:

{
 "messages": [
   {
     "role":"system",
     "content":"You are a programming completion tool. Replace <CURSOR> with the correct code."
   },
   {
     "role": "user",
     "content": "def fib(n):\n    if n <= 1:\n        return<CURSOR>\n    else:\n        return fib(n-1) + fib(n-2)"
   }
 ]
}

3. The Transformer Backend Formats the Messages

After replacing the prompt keys, the messages are passed to the backend for final formatting. Typically, api providers like OpenAI handle this automatically and messages are just passed as a key in the JSON body. Some backends like llama.cpp require some manual formatting for which we use built in function like llama.cpp's llama_chat_apply_template or MiniJinja.

Recommended Prompts

I would love for this section to grow into some best practices and recommendations of good prompts but we are not there yet. Feel free to contribute!