Where do we go next?

[JoelKronander]

Thanks @2mawi2 and @EdwardTheLegend for great PRs and additions to the code. Just released as 1.2.0.

@kafischer and I initially had the idea for copilot when we were frustrated it's so hard to remember terminal commands.

Where do we go from here? What are you all thinking we could improve further?
Where does copilot work well and where does it not currently work well?

[2mawi2]

I believe if we want to maintain this project for longer we have to come up with something better than this: copilot-cli. Copilot-cli has multiple versions (possibly trained) for git, jq and a general version. It would be nice if that would work fully automatically and the user does not have to decide. We could for instance classify the command of the user in advance and then decide for a specific context for the prompt that fits to the users request.

[JoelKronander]

Interesting, have you gotten access yet?
How does it compare to terminal-copilot in terms of quality of the commands it proposes?

I agree that one should just have a classifier that routes you to the right tool, and the users would not have to pick. Don't seem that hard even? like is it git related vs not etc.? Could just probably train a huggingface classifier for that real quick with AutoTrain e.g.

[JoelKronander]

Also @2mawi2 not sure if you saw this? Quite interesting breakdown of how GitHub copilot constructs prompts: https://thakkarparth007.github.io/copilot-explorer/posts/copilot-internals.html

[2mawi2]

It sound like Copilot requires quite a big engineering effort to get it working properly. But I am not quite sure if we have to deal with such a complexity here. I believe some small fine-tuning can be enough. I mean it already works surprisingly well already.

[JoelKronander]

yeah agree. fine tuning is def intresting, how would we collect the right data for that?

[2mawi2]

Ah sry. I didn't mean fine tuning training. I meant fine tuning by finding the correct prompt with k-shot examples. I believe fine-tuning a model directly would be even better, but you would need 100reds of training samples.

[2mawi2]

One other future idea: There appears the be a huge need for correcting bash commands (see https://github.com/nvbn/thefuck. Where the user tries a specific command and then is stuck with an error. We could include an option to correct a bash command and include as context the last error and its wrong command. The output of GPT would then be the corrected command.

[JoelKronander]

haha, the user interface is interesting for that tool.

Yeah that would be intresting to try out, something like
copilot --correct-last or copilot -cl
or similar?

Also feels related to what you did re including history as well? Could make the prompt even more narrow for that use-case though might work better. There is also the "edit" endpoint at OpenAI that might work better for that use-case actually?

[kafischer]

Correct last would be a great addition

[2mawi2]

In theory it might be possible to include the output of the last prompts as well at then let GPT learn from it. This partially works already with the current history implementation. But since terminal-copilot does not have access to the result of its last suggested prompts, it cannot learn from them. However there might be a bunch of data security questions here. Because sending the output of bash command to OpenAI can contain a lot of sensitive information. GPT could have access to many kubernetes cluster in no time. And I am pretty sure they log they prompts in plaintext into the db.

[kafischer]

Similar - imagine there's a bunch of error output in your console.

copilot --fix-errors

and it suggests some commands to run

[2mawi2]

I am not quite sure how to attain access to the last console output. But there must be some way I assume, since thefuck does it as well.